Consensus is God

I believe humans cause global warming. I am not part of a consensus that states, “Humans cause global warming.” If this sounds crazy, it’s because it is. It’s also seems to be what Dana Nuccitelli of Skeptical Science fame claimed in a comment at Collin Maessen’s blog.

To those who don’t know, Skeptical Science announced the Cook et al paper with a post which described their results (Cook et al are from Skeptical Science):

The 97% Consensus Results

Based on our abstract ratings, we found that just over 4,000 papers expressed a position on the cause of global warming, 97.1% of which endorsed human-caused global warming. In the self-ratings, nearly 1,400 papers were rated as taking a position, 97.2% of which endorsed human-caused global warming.

This has caused some controversy as to what the “consensus” measures. In reference to that, Nuccitelli explained:

We used multiple definitions of ‘consensus’, one being simply ‘humans are causing global warming’ and another being ‘humans are the main cause of global warming’.

We should take a moment to admire the audacity of this. Skeptical Science has claimed they found “a consensus.” They’ve talked about “the consensus.” Not once did they say they found “some consensuses.” Not once did they discuss “the consensuses.” The existence of multiple consensuses is not present in their paper. It is not present in their discussion of their paper’s results. As far as I know, it isn’t present in any of the media coverage either.

It seems Skeptical Science has gone to great lengths to hide the fact they used multiple definitions. As an example, here is me discussing the lack of definition for the “consensus” in the paper at their site:

2) There is no clear, much less explicit, definition of “consensus” in the Cook et al paper. None was provided on Skeptical Science either. Obama’s tweet about this paper, which Skeptical Science has used for publicity, describes the consensus found by this paper as something it could not possibly be. It’s cheeky to criticize people for using a different criterion for the “consensus” when you never bothered to provide one of your own.

Multiple authors of the paper responded, but not a one answered this simple question. Not a one corrected me by pointing out more than one definition was used. However, one of the participants in the project said this:

Ergo though John Cook may have lacked an explicit definition of endorsement [of the consensus], he and the raters had an implicit definition which is in the paper. What is more, that implicit definition is, or is very close to the tacit definition actually used by raters in rating abstracts.

None of the authors of the paper contradicted him, and before I could try, the moderators falsely claimed I was breaking their site’s rules, and I got banned. This shows Skeptical Science was determined not to give an answer to the simple question, “What is the consensus?”

Now we know why. Think about how different things would have been if Nuccitlli had said this six months ago. What would newspapers have said if people knew the paper used more than one definition for what the “consenus” is? They’d never have accepted claims about “the consensus.” They would have asked about the other consensuses. It was only by intentionally hiding the use of multiple definitions, apparently even from participants in the project, that Skeptical Science’s PR campaign could work.

But still, that doesn’t address the key issue here. How can I say I believe humans cause global warming yet not be part of the consensus measured by agreement with the statement, “Humans are causing global warming”? How can Roy Spencer? Spencer may not believe “humans are the main cause of global warming,” but he does believe they cause some global warming.

So how can Nuccitelli dispute this? I don’t know. There may be a clue in the last paragraph of his comment:

The point you (and he, and others) continue to miss is that the < 3% includes papers that minimize the human influence (implicitly or explicitly). That's why Spencer is in that category. I don't care where he believes he should go based on a misunderstanding of our paper and categories. He is in the < 3%.

Nuccitelli claims Spencer is in the 3% that rejects the consensus. Spencer agrees humans are causing warming, so this “consensus” cannot be the first Nuccitelli listed. We know the 97% is not based on agreeing humans cause most global warming, so this “consensus” cannot be the second Nuccitelli listed.

Maybe it’s not that Dana Nuccitelli is claiming that despite the fact I believe humans cause global warming, I am not part of a consensus that states, “Humans cause global warming.” Maybe the truth is their paper used a third, unknown definition for the “consensus” along with the two Nuccitelli has already said were used.

But wait. Even if Roy Spencer didn’t fit some third definition, he’d still fit the first. Since the authors of the paper, Skeptical Science and everyone else refer to “the consensus” when there’s really three (or more?), it’d be fair for Spencer to do the same. He’d just be referring to a different consensus when he says “the consensus.” He’d have his consensus, they’d have their consensus, Barack Obama would have his own consensus, and everybody would insist their consensus is “the consensus.”

Because consensus is God, and Cook et al have created a new holy text!



  1. From the private Skeptical Science forum (made public by an admin crew up)
    Topic – The Consensus Project – headline – Defining the Consensus

    we see, they used ari’s porno approach to defining the consensus AGW with NO quantification:

    2012-01-24 14:46:03 Defining the scientific consensus
    John Cook

    One of the problems with drawing up a methodology is pinpointing what the scientific consensus is. Dana talks about rejection of the consensus being “no human impact”. But that’s lowering the bar very low for endorsing the consensus. I would suggest therefore that the consensus be defined as one of the following:
    •IPCC: most of global warming over the last 50 years is likely caused by humans
    •Doran 2009: humans have caused significant warming
    •UNFCCC: “dangerous anthropogenic interference with the climate system”

    Okay, I don’t really think we should go with the third option but I include it to illustrate a point. The consensus position is that humans have had a significant impact on climate – and therefore what humans do matters. If humans only cause 10% of global warming, then while it may be discernable, it doesn’t really matter. So anthropogenic global warming is about whether our contribution to global warming is significant and therefore an important issue.

    The problem is “significant” is fuzzy so I’m actually leaning back towards where I first started – “most of global warming is human caused”

    So I suggest a rejection of AGW is not “humans have no impact on climate” but instead, “the human influence on global warming is insignificant, minor or less than half”. An implicit rejection is to say “natural factors dominate global warming”.

    That raises the problem – almost all endorsements of AGW don’t specify the quantity. So I suggest that when a paper refers to anthropogenic global warming, the null hypothesis is “most of global warming is human caused”. That means when a paper refers to anthropogenic global warming as a given fact, they are endorsing the null hypothesis that most of it is anthropogenic.

    UPDATE: or upon reflection, is the null hypothesis idea turning it into a self-fulfilling prophecy – all these papers endorse the consensus because we say they do? I don’t know, I just want to start coding and getting the rating going but these issues need to be locked down tight before we can start. Very frustrating!

    2012-01-24 16:02:51
    Dana Nuccitelli

    Maybe we need to trade a bit of simplicity for accuracy. What about this? Wrong discussion thread, but relevant here:
    •Explicitly endorses and quantifies AGW as 50+% cause of the observed warming (or consistent with the IPCC, or something similar)
    •Explicitly endorses but does not quantify AGW
    •Implicitly endorses AGW (by definition does not quantify)
    •Explicitly minimizes AGW (less than 50%, less than IPCC, less than consensus, etc.) = rejection

    That way the consensus is better (most warming is anthropogenic), but we’re not necessarily assuming that the papers are all endorsing 50+% AGW.

    One issue here is that you’ll get more ‘rejections’ than in your paper with Jim, because it’s a different consensus definition.

    2012-01-24 16:19:43 Inconsistency with Jim
    John Cook

    I had that lurching thought this morning on the way to campus – all this haggling over the definition of rejection means it’s possible we come to a different sample of rejection papers. So that’s another wrinkle that we’re just going to have to put out of our minds for now.

    I’m not a big fan of complicating the endorsement categories unnecessarily but on the other hand, we can always collapse it down to a simpler final result and get the best of both worlds. However, the distinction you’ve made – “Explicit endorse with quantification” vs “Explicit endorse without quantification” – well, are there actually many endorsements with quantification, other than those handful of papers you recently blogged about that directly address the issue? I don’t recall seeing many when we trolled for explicit endorsements – do you? If not, it would be an unnecessary complication?

    However, I have also been keen for a “Provides evidence for AGW” category that goes above and beyond “explicit endorsement” and provides evidence for AGW (eg – all your recent papers) which is very close to the “Endorse with quantification” category (except a paper can endorse with quantification but not provide evidence – I want to tag the ones that do present evidence).

    And don’t forget Implicitly Minimizes AGW (eg – “solar is the dominant cause, nudge nudge, wink wink”). Doesn’t explicitly reject AGW but we know what they’re getting at.

    2012-01-24 16:29:47
    Dana Nuccitelli

    There aren’t a lot that specifically quantify AGW, but there are a good number that essentially endorse the IPCC report, which is equivalent to endorsing AGW as greater than 50%.

    I just really want to avoid the assumption that if a paper says ‘humans are causing global warming’, for example, that they’re saying humans are causing the majority of the warming. Maybe they are, maybe they aren’t. If we assume they are, we’ll get busted. So if your consensus is ‘humans are causing most of the observed warming’, then you need the papers that don’t somehow quantify the effect in a different category, because you then have to add an assumption in order to get an endorsement for your consensus definition.

    But like I said, you don’t have to be totally literal and only accept if they quantify it. For example, endorsing the IPCC is a consensus endorsement.

    2012-01-24 16:47:16
    Dana Nuccitelli

    As another option, we could have two different consensuses (consensi?). The first being ‘humans are causing most of the observed warming’, the second just being ‘humans are causing global warming’.

    The first is a stronger point, but relatively few papers will qualify as endorsements (but it should still be a clear consensus). The second is a weaker point, but you’ll have a much larger consensus.

    We don’t even necessarily have to decide now (and we could even discuss both in the paper), because the categories I listed will work either way, with some slight tweaking:
    1.Explicitly endorses and quantifies AGW as 50+% cause of the observed warming (or consistent with the IPCC, or something similar)
    2.Explicitly endorses but does not quantify AGW
    3.Implicitly endorses AGW (by definition does not quantify)
    5.Implicitly minimizes AGW (i.e. says the sun is playing a big role)
    6.Explicitly minimizes AGW (less than 50%, less than IPCC, less than consensus, etc.)
    7.Explicitly says there’s no anthropogenic effect on climate/temperature

    For the first consensus position, only #1 qualifies as an endorsement, while #6 and 7 are rejections.

    For the second consensus position, #1 through 3 are endorsements while only #7 is a rejection.

    Personally I kind of like this idea. By keeping the second consensus position, we can stay consistent with Oreskes and Jim. But by adding the first consensus position, we can make a much stronger point.

    2012-01-24 18:00:13
    Ari Jokimäki

    Well, I think the consensus is what most studies show, not what some organization or single paper has said. There are two separate issues here:

    1. Endorsement/rejection of the theory of AGW.

    2. What concensus says.

    These are different issues. Don’t mix them with each other. Papers endorse or reject (or are neutral) the theory of AGW. If majority of papers endorse it, then it’s consensus position. I think what the opening post is actually trying to define is what the theory of AGW says. You can’t define consensus that way.

    2012-01-25 03:28:57
    Dana Nuccitelli

    Ari’s #1 is my ‘consensus’ #2, and his #2 is basically my #1. We’re basically saying the same thing – the question is whether you’re checking on the support for the AGW theory in general, or for the ‘consensus’ that most of the warming is anthropogenic. Strictly speaking, what Oreskes and Jim did was the former. I don’t see why we shouldn’t do both, to add onto what they did.

    2012-01-25 05:09:44
    Ari Jokimäki

    This is largely nit-picking but we can’t check for the consensus that most of the warming is anthropogenic. We can check if the consensus is that most of the warming is anthropogenic. The point I’m trying to make here is that we can’t define the consensus but we can only measure it. So, from my point of view, consensus is not something we take as an input to this study but it is something that comes as output of this study.

    2012-01-25 05:43:49
    Dana Nuccitelli

    Super nitpicky Ari! Our hypothesis is that that’s the consensus, and we’re testing if our hypothesis is correct. You need the hypothesis beforehand in order to set up the categories – i.e., the ‘majority of warming is anthropogenic’ category is based on the hypothesized consensus.

    2012-01-25 14:46:21 Let’s get technical
    John Cook

    What we’re doing here is measuring the number of papers that endorse/reject the theory of AGW. Forget semantics about consensus, we’re just adding up numbers. Talk of consensus comes at the end when we peruse the results.
    So what I want is clarification – exactly what is “the theory of AGW” and practically how do we measure it? There are several approaches and all are problematic:
    •Humans are causing most of global warming. How we measure it, given most papers don’t quantify, is to assume that when a paper endorses “AGW”, then without any evidence to say otherwise, we assume they mean “most of global warming is human caused”. This assumption might generate a lot of criticism.
    •Humans are causing global warming. Presumably this means any amount of warming equates to AGW and a rejection is to say “humans are causing no warming”. This sets the bar so low, this will also generate criticism. 90% of deniers would probably say “some warming” but not much.
    •Humans are causing significant warming. This isn’t clearly defined but seems closest to Ari’s approach. By leaving himself some wiggle room, categorizing a rejection becomes more subjective. But given there’s no standard of quantification, perhaps subjective and wiggly is the only option left to us?

    2012-01-25 15:02:16
    Tom Curtis

    John, IMO, the “theory” of AGW is that:

    A) The GMST has risen in the 20th century, and will continue to rise in the 21st;

    B) The primary cause (>50%) of this rise is the emission of GHG by human activity; and

    C) Absent changes in policy, technology and/or economic activity to reduce GHG emissions, the consequent temperature rise in the 21st will reach levels where they have serious adverse impacts on human well being and the health of eco-systems.

    There is certainly a consensus among climate scientists about the first two, and a majority beleive the third, but I am not sure it is a consensus.

    I do not believe it is possible to rate many scientific papers for agreement with all three based on the abstract alone, or even on the whole paper because most papers do not discuss all three. The solution, IMO, would be to first rate the issues adressed by each paper, and then rate agreement or disagreement with the theory with regard to the issues adressed. You could then report both the detailed data, and overall agreement with AGW with the overall figure based on the minimum level of agreement in the paper for any issue discussed. So, a paper that is category (1) (using Dana’s categories of 24th Jan, 4:47 pm) for issue (A), but category rejects the possibility of significant harm (issue C) would be classed as rejecting AGW. However, if it does not mention issue C, it would be rated as supporting AGW.

    2012-01-27 03:37:53
    Dana Nuccitelli

    Tom’s is an interesting approach, but a multiple step process would add time and complexity. There are virtually no papers which dispute the planet has warmed, so then it would boil down to “will the warming continue?” and “is the warming mostly anthropogenic”?. If the answer to the second question is yes, then the answer to the first is also yes, so it still boils down to answering the second. The only exception would be a paper that says the warming will continue, but doesn’t specify that it’s anthropogenic, in which case I’d hesitate to call that an AGW endorsement.

    I haven’t seen a reason why we shouldn’t follow the approach I suggested here, using these categories:
    1.Explicitly endorses and quantifies AGW as 50+% cause of the observed warming (or consistent with the IPCC, or something similar)
    2.Explicitly endorses but does not quantify AGW
    3.Implicitly endorses AGW (by definition does not quantify)
    5.Implicitly minimizes AGW (i.e. says the sun is playing a big role)
    6.Explicitly minimizes AGW (less than 50%, less than IPCC, less than consensus, etc.)
    7.Explicitly says there’s no anthropogenic effect on climate/temperature

    For ‘humans are causing most of the warming’, #1 qualifies as an endorsement, while #5 through 7 are rejections.

    For ‘humans are causing warming’, #1 through 3 are endorsements, while only #7 is a rejection.

    With those categories, we can just go ahead and rate the papers and see how the percentages turn out for each possible consensus position. It gives us more flexibility than if we choose one or the other definition of AGW beforehand.

    2012-02-08 21:12:54 Trying to kick start this conversation again
    John Cook

    Been busy in other areas but would like to sew this up so I can start preparing Phase 2b (the email project caused me to split Phase 2 into two parts). Dana, I could probably get on board with your system although I do have reservations about it being possibly unnecessarily complicated. But let’s assume for now that we do go with your “have your cake and eat it” approach. I have two concerns with your categories:
    First, would like to see symmetry so that means tweaking the rejections to:
    6. Explicitly minimizes AGW but does not quantify
    7. Explicitly minimizes AGW as less than 50%

    I also worry that category 5 is a big tent, may capture a lot of papers, really cause a big inconsistency with Jim’s paper and it’s a little fuzzy. Is a paper that says “natural cycles may be bigger than thought” implicitly reject?

    Also, am really happy to see so many SkSers help collect emails but would be great if a few more weighed in on this discussion. We’re laying the foundation of TCP here, establishing the methodology for what will probably be a month’s worth of crowd sourcing. We really want to get this part right.

    2012-02-08 22:06:51
    Ari Jokimäki

    What is this “Jim’s paper” and if it’s relevant, should we read it (sorry if I have missed the information on this)?

    I think simple pro-AGW/neutral/against AGW would be best classification scheme because it only has two middlegrounds (what is the proper English term for this?) between different categories. Each middleground introduces its own problems. Dana’s categories have 6 places where we need to scratch our heads whether a paper belongs to this category or that category.

    However, introducing fourth category of unrelated papers would increase the amount of information we can squeeze out of this project. So, I suggest following categories:

    1. Endorses the theory of AGW.

    2a. Is neutral on AGW but relates to the issue.

    2b. Is unrelated to the issue.

    3. Is against the theory of AGW.

    2012-02-09 13:11:32 Jim’s paper
    John Cook

    Jim’s paper is “Phase 1” of TCP (see Jim’s paper was submitted to Science but got rejected, it’s now under consideration at Eos. Currently we’re discussing the methodology of Phase 2.

    Re your idea of “Unrelated papers”, I agree we capture that but that information goes in the “Categories” categorisation, not the “Endorsement” categorisation. For example, a suggested set of categorisations could be:

    Categories Endorsements
    Explicit Endorsement of AGW
    Implicit Endorsement of AGW
    Implicit Rejection of AGW
    Explicit Rejection of AGW

    The “Oreskes method” would be to put any NOT RELATED papers under methods. We have the flexibility of collapsing all NOT RELATED papers into the Methods category if we want to produce an Oreskes style result.

    I am keen to record Explicit and Implicit endorsements and then for the sake of symmetry would need to do the same for rejections.

    Dana, am still leaning towards the simpler endorsement categorisation. Ari has a point that with too many categories, we’ll be scratching our heads and we are talking multiple rating of over 12,000 papers. I just don’t see the quantification category as adding that much to the analysis. The simpler it is, the less likely things will go wrong and I do envisage a possibility of making this a publicly interactive thing in some way or another in the future so a public friendly method is good too.

    2012-02-09 16:45:53 objectivity is key
    Dana Nuccitelli

    There’s a trade-off between simplicity and objectivity. More categories may leave you scratching your head for a few seconds in some cases, but it also removes a lot of the subjectivity. For example, what exactly is this “AGW” that the papers are endorsing? Is it any anthropogenic influence at all, or is it that the majority of warming is anthropogenic? If you just leave it as endorsing or rejecting AGW, you lose that distinction.

    I’m also confident the first criticism of the paper will be “I’m a denier and based on your definition, I endorse AGW”, assuming our definition is any anthropogenic warming. If deniers fall into our ‘endorse’ category, that substantially weakens our result.

    My other concern is that when I was helping to categorize papers for Jim’s project, I felt too often like I was making a subjective decision, and that my own ratings weren’t even consistent. With the categories I list above, you remove a lot of that subjectivity. Not altogether – a paper that says “natural cycles are understimated” or something like that will still require a subjective call (yes, probably category 5), but the less subjectivity, the better. We want these results to be easily reproducible.

    John – I don’t think symmetry is all that important, but I don’t have any problem with your suggested tweaks either. Seems fine to me.

    2012-02-09 17:23:19
    Ari Jokimäki

    So, if Jim’s paper is relevant to our work, shouldn’t we be able to review it as soon as possible, i.e. now?

    Dana, I disagree with you on the amount of subjectivity. Both in your and in my schemes we have to define “AGW” somwhow. That introduces same amount of subjectivity to both schemes. In my scheme there’s subjectivity relating to decisions if paper is pro-AGW or neutral and if it is against AGW or neutral. I’m assuming here that there’s no middleground between pro-AGW and against AGW directly, so one doesn’t need to decide if paper is pro-AGW or against AGW (although it can be argued that even such papers can exist, presenting strong evidence for both sides). Let us quantify this and assume that these major middlegrounds between categories introduce one subjectivity unit per each middleground. That would make my scheme to have subjectivity of 2.

    In your scheme you would have these same middlegrounds, although they are worded a little differently, but equivalent middlegrounds in your scheme would be 1-3 to 4 and 5-7 to 4, so that would make your scheme subjectivity 2. However, you also have middlegrounds inside 1-3 and 5-7. These are not major middlegrounds so let’s give them subjectivity value of 0.5. Such middlegrounds are 1 to 2, 2 to 3, 5 to 6, and 6 to 7. That is 4 x 0.5 = 2, which makes the total subjectivity of your scheme 4.

    So, as you can see, this is irrefutable proof that my scheme has twice less subjectivity than your scheme. 😉

    2012-02-10 15:06:58 Let’s look at a few examples
    John Cook

    Here is an example of an explicit endorsement of AGW:

    The signing of the UN Framework Convention on Climate Change in Rio de Janeiro in June 1992 by 160 nations has firmly identified global climate change due to human pollution as a pressing global environmental concern.

    However, it doesn’t specify “most of warming”, just that human pollution is causing climate change. This is the nature of the vast bulk of the AGW endorsements and many of the rejections. They just say “we are or are not causing climate change”, they don’t talk amounts. So Dana, when you argue that we need to keep track of the “most of GW is man-made” papers, I’m pretty sure that will be a very small amount. So I question the value in measuring that. I’m not dead set against the idea. I’m just a quantification skeptic, not a denier 🙂

    Also, measuring papers that say “humans are causing NO global climate change” will also be vanishingly small – maybe a couple out of our 12,000 papers. Measuring that statistic is next to useless and I question even more the value of measuring that. I’ve been able to only identify one paper that says NO warming from humans:

    Computations based on the adiabatic theory of greenhouse effect show that increasing CO2 concentration in the atmosphere results in cooling rather than warming of the Earth’s atmosphere.

    So to the “no human influence at all” category, I am a denier.

    Let’s look at some examples of rejections, considering Dana’s categories (with my amendments):

    5. Implicitly minimizes AGW (i.e. says the sun is playing a big role)
    6. Explicitly minimizes AGW but does not quantify
    7. Explicitly minimizes AGW as less than 50%

    So here are examples of what I consider rejection papers, with commentary:

    These rate as 5, don’t explicitly mention AGW but implicitly minimise AGW.

    It is found that at least 60% of the global warming observed since 1970 has been induced by the combined effect of the above natural climate oscillations.
    Nicola Scafetta

    The mid-point of that range, a change in solar irradiance of 0.4%, is sufficient to explain all or most of the recovery from the Little Ice Age of the 17th Century and most of the half-degree global warming observed during the last 100 years.
    Sallie Baliunas

    It is established here that these global temperature patterns are currently in the process of undergoing a sunspot-related change from the post-1940 relatively less turbulent variability back into relatively more turbulent variability. This apparently imminent state of more turbulent variability is expected to stop and at least slightly reverse the global warming trend, which has been going on since about 1965.

    This is a 7, “relatively minor” is surely less than 50%

    In view of the relatively minor effects that anthropogenic carbon dioxide has had, and will presumably continue to have, on the climate scenario it must be wondered whether the considerable effort expended on countermeasures is always justifiable from an economic point of view.

    Category 7, again certainly less than 50%

    The writers show that the present-day global warming is not due to the increase in the volume of greenhouse gases, but rather to the increased solar activity.

    Category 6, minimises without quantification

    However, the recent studies of some of the Himalayan glaciers indicate that the rate of recession of most of the glaciers in general is on decline. These observations are in contradiction to the widely popularized concept of anthropogenically induced global warming.

    Category 6, rejection without quantification:

    The stabilization of the global temperature in the last decades at a constant increase in the amount of greenhouse gases in the atmosphere contradicts the concept, according to which an increase in the global temperature in the last decades is only explained by the anthropogenic impact.

    Category 6, no specifics, just casts doubt and minimizes AGW:

    The bottom line is simple-despite a public perspective to the contrary, the global temperature record provides little support for the catastrophic view of the greenhouse effect.
    Robert Balling

    I don’t think Dana’s method removes subjectivity. Category 3 and 5 still retain the same subjectivity problem as before. So you could say we’ve added complexity without removing the underlying problem. On the plus side, you could also say it’s added an extra level of information about the quantified endorsements/rejections which could be interesting (I do love extra data). But don’t be mistaken into thinking adding the quantification categories solves the problem of subjectivity.

    2012-02-10 17:51:53
    Dana Nuccitelli

    I still think it reduces subjectivity, but more importantly it retains a lot of important information. Maybe it turns out that very few papers explicitly endorse AGW > 50%. That’s fine, then don’t use that as the consensus definition. But if it turns out that a lot papers do explicitly endorse AGW>50%, that’s critical information. I’m just saying don’t limit yourself before you’ve even started the project. If you leave your options open and collect as much info as possible, who knows what you’ll find. In this case I think oversimplifying would be a big mistake.

    2012-02-10 18:05:05
    Ari Jokimäki

    I don’t see how your scheme could reduce subjectivity. Compared to three level scheme the subjectivity in your scheme is at best the same. However, I think in practice your sublevels in each side of neutral will increase subjectivity.

    2012-02-10 20:59:36 Dana’s exploiting my weak spot
    John Cook

    He knows if there’s one thing I love, it’s more data. So I must admit, the argument of retaining more info is compelling 🙂
    Ari, what I think will likely happen is for the final result, we collapse all the endorsement categories into one category and all rejections likewise. So subjectivity between endorsement categories isn’t that much of an issue. And the delineation between “explicit with quantification” and “explicit without quantification” is pretty clear. It’s more the grey area between endorsement and neutral or rejection and neutral that is the trickier subjective area. But that will be an issue no matter which scheme we go for.

    So allow me to recap where we’re at with this conversation. We’ve ruled out “humans cause no global warming” as the definition for rejection. Ari suggests a non-quantified “humans cause global warming” definition. Dana suggests we also use the non-quantified “humans cause global warming” but include a subset “quantified >50% human warming”. So the only difference between the two is whether to include the extra subset. Have I got right?

    2012-02-11 03:52:02
    Dana Nuccitelli

    Yes John, basically right, plus the other quantified subsets. Ari’s ‘humans cause global warming’ is my #2, > 50% is my #1, and then there’s the implicit, neutral, and the equivalent categories for rejecting AGW.

    The reason my method is less subjective is that there are more options. Fewer options mean you have to force papers into categories where they don’t really fit, which leads to a subjective choice. For example, a Scafetta paper which accepts AGW but says it’s small and solar influence is large, where does that go? In my categories, there’s no question it goes into #6. If you only have endorse/neutral/reject, frankly you could make an argument for any one of the three. It accepts AGW, but says it’s small. Technically it’s an “endorse”, but if you go with your gut, it’s “reject”, and since it could go either way, you might even be tempted to call it “neutral”.

    You can bypass that subjectivity through your definition of ‘AGW’, i.e. if you say ‘AGW’ means any anthropogenic warming. But then you’ve got Scafetta in the ‘endorse’ category, and that’s a problem. With my method, he’s in the correct category. My method gives you a place to put those sorts of papers without having to make a judgment call.

    2012-02-11 06:59:05
    John Cook

    Well, no because we’ve ruled out “AGW means any anthropogenic warming” as a definition so under Ari’s “porno” method (aka ‘I know rejection or endorsement when I see it’), Scafetta would never be considered “implicitly endorses anthropogenic warming”, it would be “implicitly rejects”.

    This is where I think the confusion in this conversation is. To me now, there is no difference between your scheme and Ari’s and they should get the exact same result as far as # of endorsements/rejections go. The “implicitly endorses AGW” and “implicitly rejects AGW” categories will capture all those problematic papers, regardless of whether you are categorising some papers as quantified. The quantified papers are the hard centre but the soft outside is still the same.

    So Dana, is this your categories now?
    1.Explicitly endorses and quantifies AGW as 50+% cause of the observed warming (or consistent with the IPCC, or something similar)
    2.Explicitly endorses but does not quantify AGW
    3.Implicitly endorses AGW (by definition does not quantify)
    5.Implicitly minimizes/rejects AGW (i.e. says the sun is playing a big role)
    6.Explicitly minimizes/rejects AGW but does not quantify AGW
    7.Explicitly minimizes/rejects AGW as less than 50%

    2012-02-11 07:22:35
    Ari Jokimäki

    Dana, you need to realize that my category 1 is your categories 1-3 and my 3 is your 5-7. Your categories do not introduce more objectivity. What they do is they introduce more work for classifiers. Is the extra information gained worth the effort? I have no idea.

    The extra information for papers quantifying the thing to more than 50 % doesn’t seem to be needed in my opinion because it really is just a handful of papers that do the quantification (basically only the attribution studies do that and there’s not many of them – I have a list in my blog of those). So I would definitely get rid of the 50 % categories at the least to simplify things.

    What these extra categories do, even if they won’t introduce more subjectivity, they introduce more complicated setting for the classifiers.

    Regarding my “porno” method, I don’t think there’s other way as the diversity of the papers is so huge that you cannot control it by setting specific classification rules which would cover each situation. Complicated rules would just be confusing and they would, well, complicate things, wouldn’t they?

    2012-02-11 07:25:16
    Dana Nuccitelli

    Yeah, I like those categories.

    So using Ari’s method, how are we defining ‘AGW’? If it’s “most” (and you do need to be specific and clear about this definition), then again you run into that problem of assuming that papers are agreeing that >50% warming is anthropogenic, and then a ton of subjectivity is introduced into the process (i.e. is this paper suggesting most warming is anthropogenic or not?). If a paper just says “humans are causing global warming”, is that an endorsement? In my case it clearly falls in category #2, no question. In Ari’s, well, that depends how you define ‘AGW’. You can assume that it’s implicitly endorsing AGW>50%, but that’s a major assumption.

    If that’s how we’re defining AGW (“most”), then if you collapse my categories I agree the end result should be the about the same. But my categories are much better because in #2 and #3, we’re not making that assumption that papers are saying >50%.

    This was a problem with Oreskes’ paper. She defined AGW as >50%, but a ton of the papers she categorized as explicit/implicit didn’t quantify AGW, and so she was criticized for making that assumption. I’m saying we shouldn’t repeat that mistake, we should remove the >50% assumption by using category #2. Then in the paper we can say x% endorsed the AGW theory, and y% went further and explicitly endorsed it as >50%.

    It may not sound as good as “75% explicitly or implicitly endorsed,” or whatever it was that Oreskes said, but it will be more accurate and more thorough. You have to be specific about what the papers are endorsing or you’ll get reamed with criticisms, and rightly so, IMO.

    2012-02-11 07:34:58
    Dana Nuccitelli

    In short, in case I’m not being clear, my problem is that if you define AGW as “>50% observed warming over the past ~century is anthropogenic”, then either:

    •you’re forced to make major assumptions in claiming that a large percentage of papers are endorsing that definition (for which you’ll be criticized, and rightly so), or

    •you can break out those which explicitly endorse that definition and those which simply endorse “AGW”, in which case you’re not making any such assumptions (which IMO also makes it less subjective, though perhaps you can replace much of my ‘subjectivity’ concerns with ‘assumption’ concerns).

    If you go back up to John’s update on the first post above, he asked:

    “…all these papers endorse the consensus because we say they do?”

    That’s my concern too. Replace “we say they do” with “we assume they do” – same thing.

    Ultimately it boils down to how you define ‘AGW’. In my method, you don’t have to define it as >50%, but you do get the >50% category as well.

    2012-02-13 10:22:18 What is really annoying about Dana is this…
    John Cook

    Keep reading and I’ll explain what’s really annoying about Dana at the end of this post…

    Okay, so we’ve ruled out a definition of AGW being “any amount of human influence” or “more than 50% human influence”. We’re basically going with Ari’s porno approach (I probably should stop calling it that 🙂 which is AGW = “humans are causing global warming”. Eg – no specific quantification which is the only way we can do it considering the breadth of papers we’re surveying.

    That means this now boils down to one issue. Is it worth the trouble of subdividing “endorsing AGW” and “rejecting AGW” into separate categories. They make it more complicated for the reviewers so is it worth it?

    Firstly, it’s definitely worth subdividing into “explicit” and “implicit”, I suggest we must do this. Firstly, because Naomi did that and we’re replicating her research. Secondly, because Naomi’s effort at subdividing into implicit and explicit was a hand-wavy “well, I’m guestimating that 25% is explicit and 50% is implicit”, which frankly doesn’t really cut it. And critiques of her work (by Benny Peiser) rebutted her work by actually going through the 928 papers and specifying the # of explicit and implicit endorsements. So we both improve on Naomi’s research by specifically rating each paper and also head off possible criticisms by having hard, quantified data.

    The next question is do we subdivide explicit into “quantified explicit” and “unquantified explicit”. Dana argues that this captures more information. Ari argues it complicates things with little benefit as the # of quantified papers will be small. I’m sympathetic to both arguments. However, to me, what had me falling one way or the other was this. As I was looking through the papers in Phase 1, I noticed a number of papers that were kick-arse papers that not only endorsed AGW but also quantified and found evidence for AGW. These were papers I would’ve liked to flag for future reference. So on this point alone, I would recommend adding “quantified >50% explicit endorsement of AGW” as a category, in order to flag those great papers. There are other potential benefits too. It will be interesting to see how the “hard centre” of endorsements looks – does it grow or shrink over time, does it grow proportionally compared to the total # of endorsements and how does it compare to the # of rejections. These are all interesting questions and we will only be able to answer them with the extra questions.

    So basically, I’ve come to the conclusion that we rate papers according to this:

    AGW = humans are causing global warming
    1.Explicitly endorses and quantifies AGW as 50+% cause of the observed warming (or consistent with the IPCC, or something similar)
    2.Explicitly endorses but does not quantify AGW
    3.Implicitly endorses AGW (by definition does not quantify)
    5.Implicitly minimizes/rejects AGW (i.e. says the sun is playing a big role)
    6.Explicitly minimizes/rejects AGW but does not quantify AGW
    7.Explicitly minimizes/rejects AGW as less than 50%

    So what is annoying about Dana is when we have a long, convoluted discussion and at the end, I end up just agreeing with what Dana was saying at the very start of the conversation. This is not the first time this has happened to me. At my house, we (I) have a rule, WIAR (Wendy Is Always Right). When she volunteers advice (eg – “should take an umbrella today”) and I ignore it, I always end up regreting it. However, I’m not saying I’m endorsing a DIAR (Dana is always right) policy just yet though, wouldn’t want him to get too cocky 🙂

    2012-02-13 12:02:57
    Dana Nuccitelli

    I told you John, DAWAAR (Dana and Wendy are always right).

    I think you guys are too pessimistic about the >50% explicit papers too. I saw quite a few papers that effectively endorsed AGW as defined by the IPCC, which is >50% and thus fall within category #1. But the only way to find out is to do the survey.

    I also think you’re overestimating the amount of effort the 7 categories will add. I don’t think it will lead to much, if any more head-scratching. The categories are pretty clear, all you have to do is look at whether or not a paper endorses and quantifies AGW. Pretty straightforward.

    2012-02-13 17:54:53
    Ari Jokimäki

    A paper that just agrees with IPCC doesn’t quantify the issue, it just accepts the mainstream position. I don’t understand what additional information we would gain by counting all the papers that agree with IPCC vs. papers that “explicitly endorse” some other way – I don’t see a distinction between these. I thought this extra category would have been the papers that actually quantify the thing themselves, and not just accept the quantifications of others.

    2012-02-13 20:43:55
    John Cook

    Note the category is “explicitly endorsing AGW with quantification”. So merely agreeing with the IPCC is implicit endorsement. They have to say the words explicitly to be categorized as explicit.
    But it’s not necessarily papers that do the quantification research themselves. This is about whether they endorse AGW, not whether they prove or quantify AGW with their own research.

    2012-02-13 21:04:00
    Ari Jokimäki

    Yes, I understood that, but I fail to see the value in the category, because it seems to be basically the same as explicit endorsement. Take two papers; another one says “we agree with mainstream concensus of climate science” and another one says “we agree with IPCC conclusions”. Both say basically the same thing but in this classifying scheme you place them in different categories. What’s the extra value in that?

    2012-02-14 03:06:40
    Dana Nuccitelli

    We have to take a step back now and define exactly what qualifies as “quantifying AGW.”

    The IPCC has quantified AGW as >50%, and that’s accepted as the consensus position. So my feeling is that a paper which explicitly agrees with either the IPCC or consensus position goes into category #1. That’s why I defined category #1 as:

    Explicitly endorses and quantifies AGW as 50+% cause of the observed warming (or consistent with the IPCC, or something similar)

    So Ari, I don’t think those papers would go in different categories. But it’s something we need to discuss. My feeling is that something which explicitly endorses something which has quantified AGW (IPCC or consensus, for example) itself counts as explicitly quantifying AGW as well. Do you guys agree?

    On the other hand, a paper that just says “anthropogenic warming” without referring to the IPCC or consenus or any quantification belongs in a different category (#2).

    2012-02-14 08:20:07
    Ari Jokimäki

    Ok, that explains that dilemma, but now I don’t quite understand which papers would then go to category 2 as it seems to me that all explicit endorsements would actually be in category 1 (endorses AGW = endorses with consensus position = endorses IPCC). You say that paper that says anthropogenic warming goes to category 2, but there are some papers that suggest a little anthropogenic warming but say that other factors are bigger and that AGW doesn’t actually matter. In your scheme these would be classified as explicitly endorses AGW?

    To answer your question, I don’t think I agree. There probably are some papers that endorse IPCC’s position but have next to nothing to do with climate science. They are just adopting the mainstream position on the issue while studying some climate related issue in another branch of science. And these would go to highest category?

    2012-02-14 09:45:37
    Dana Nuccitelli

    What you’re describing Ari is Category 5/6 – Implicitly/Explicitly minimizes/rejects AGW but does not quantify AGW. I guess category 2 should be ‘endorses and does not minimize AGW’.

    “There probably are some papers that endorse IPCC’s position but have next to nothing to do with climate science. They are just adopting the mainstream position on the issue while studying some climate related issue in another branch of science. And these would go to highest category?”

    Yes. What we’re testing here is what percentage of papers endorse ‘AGW’ (with a few possible definitions of ‘AGW’, one being with a quantification, one without).

    I guess I wasn’t clear about that. Category #1 isn’t just papers that quantify AGW, it’s papers that endorse AGW>50%. It doesn’t matter if they have little to do with climate science themselves, because we’re just testing the strength of the consensus.

    Now if we can also break these out by jounal as you’ve suggested, such that we seperate papers in climate and non-climate related journals, that would be interesting too. But the main purpose is just to determine what percentage of peer-reviewed papers endorse AGW.

    2012-02-14 17:54:17
    Ari Jokimäki

    Sorry for being a pain here, but I still don’t understand what papers are supposed to go to category 2. How can a paper say “anthropogenic warming” without going to categories 1 or 5/6? If a paper says that mankind is the main driver of climate, then that paper endorses consensus -> category 1. If a paper says mankind influences climate but there are some other main drivers -> category 5 or 6.

    2012-02-14 20:27:16 Category 2 papers
    John Cook

    I’ve seen many papers that would go in category 2. These are papers that refer to “anthropogenic global warming” or “humans are causing global warming” as a given fact, without quantification. I think there’ll be more category 2 than category 1 (unless we decide endorsement of IPCC means cat 1)

    2012-02-14 22:54:53
    Ari Jokimäki

    But with the description Dana has given here, every paper that agrees with mainstream consensus are category 1 papers. I understand this so that if a paper refers to AGW as given fact, then they endorse consensus -> endorse IPCC -> endorse >50 % quantification. I have currently no idea what papers are category 2 papers. This is what Dana said above:

    “The IPCC has quantified AGW as >50%, and that’s accepted as the consensus position. So my feeling is that a paper which explicitly agrees with either the IPCC or consensus position goes into category #1.”

    Paper that takes AGW as given fact explicitly agrees with consensus position, don’t you think? Hence it should go to category 1 according to Dana’s description.

    2012-02-14 23:19:55
    John Cook

    I’m not sold on Dana’s idea that any paper that endorses the IPCC automatically explicitly quantifies AGW as >50%. The whole point of “explicit endorsement” is they need to explicitly say it. Endorsing the IPCC is implicit endorsement of AGW, not explicit endorsement of AGW and not explicit quantification of AGW.
    Papers that take AGW as a given fact are an explicit endorsement of AGw but not an explicit quantification of AGW. If there’s no explicit mention of how much humans are contributing, there’s no explicit quantification. Explicit means you have to say the words. I dunno, it seems pretty straightforward to me 🙂

    2012-02-14 23:46:26
    Ari Jokimäki

    Yes, that’s what I think too, and I thought it was straightforward until Dana started messing with my head here. But we do need to get Dana and everyone else on the same page with us before we continue.

    2012-02-15 13:50:01 I’m working on Dana in the other thread
    John Cook

    He’s proving a tough cookie though 🙂 He and I think so alike, it’s very rare we disagree, I expect we’ll resolve this soon.

    2012-02-15 15:59:40
    Dana Nuccitelli

    Ari, categories 5 and 6 are only for papers that minimize AGW.

    I’m still not convinced on the IPCC issue but it’s not a big sticking point for me. I guess if we put that sort of endorsement in category 2, it will make that category stronger anyway.

    2012-02-15 16:44:58
    Ari Jokimäki

    Well, Dana, all you need to do is to clarify what you mean here. I know what categories 5 and 6 are for. Papers suggesting less than 50 % for AGW go there. My real dilemma is with category 2. If papers with >50% for AGW go to category 1, category 1 is meaningless. So, how can a paper support explicitly the consensus position and still end up in category 2 in your scheme?

    2012-02-16 03:13:05
    Dana Nuccitelli

    They’re not endorsing “the consensus position” – we have to demonstrate what the consensus is. They’re endorsing the AGW theory in general without being specific about the human contribution to AGW.

    This is the assumption problem I was talking about earlier. We can’t assume that just because a paper says “anthropogenic global warming” that they agree the human contribution is >50%, but they have explcitly endorsed that humans are contributing. Thus they go in category #2.

    The way I see the final paper is that we’ll conclude ‘There’s an x% consensus supporting the AGW theory, and y% explicitly put the human contribution at >50%’.

    2012-02-16 04:09:02
    Ari Jokimäki

    You can’t assume that either for the papers that say “IPCC is right”, because that’s just other way of saying “anthropogenic global warming” especially to some researchers outside the field. You can assume that only for the papers that say IPCC is right in their quantification that human contribution is >50%. Otherwise you’re just quessing what they actually mean by their IPCC endorsement. To me this division feels arbitrary. Two similar papers end up in different categories just because other one mentioned IPCC and other one didn’t.

    Also, if we can’t assume >50% then how can we assume that they belong to category 2? It seems to me that they should go to neutral, if we feel that there is a possibility that they might suggest less than 50% for human contribution.

    2012-02-16 05:51:23
    Dana Nuccitelli

    That’s fine – I think we’ve agreed not to put IPCC endorsements in category #1. I’m okay with that.

    Category 2 is “Explicitly endorses but does not quantify (or minimize) AGW.” Thus it doesn’t require an assumption of >50%.

  2. may have included an email address and an IP or 2, please delete them.
    Comment was for your information ((please don’t publish in full form)

  3. I removed the e-mail and IP addresses. Otherwise, I’m posting it in its entirety because while I’ve read all those e-mails (I believe I was even the first person to write about Ari’s P0rno Approach), there’s no ready way to refer to them. Having them in one comment will be far more convenient than having to point to a text document inside a zip file.

    Update: I’ve also gone ahead and added quote tags for legibility.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s