Author: Brandon Shollenberger

When Fact Checkers Aren’t

I am sometimes troubled by the growing popularity of “fact checkers.” I think fact checking is great. The problem is the fact checkers becoming more and more popular don’t seem to be checking facts. For instance, Washington Post’s Glenn Kessler recently published an article “fact checking” statements by former president George W. Bush, including this one:

The man, Saddam Hussein, would have a lot of revenue as a result of high prices of oil. And even though there wasn’t, you know, a – we found a dirty bomb, for example – he had the capacity to make chemical, biological and nuclear weapons. And so there’s – you know, it’s all very hypothetical. But yeah, I could argue that we’re much safer without Saddam. And I would argue that the people of Iraq have a better shot at living in a peaceful – a peaceful state.

The article begins on a troubling note:

The Fact Checker was puzzled by Bush’s reference to finding a dirty bomb in Iraq. It certainly sounded like he said that such a weapon was found in Iraq. But after listening to the tape a few times, we concluded that the former president, in an offhand manner, was giving that as an example of something that was not found after the United States invaded Iraq.

I find it puzzling anyone would think Bush’s statement was meant to indicate the United States found a dirty bomb. Taking out a few superfluous words in the sentence gives us the obvious meaning, “And even though there wasn’t a dirty bomb, for example, he had the capacity to make chemical, biological and nuclear weapons.”

The meaning is clear. I don’t know why someone would have to listen to a tape multiple times to realize the words “you know, a – we found” were little more than verbal ticks. It’s obviously nothing more than someone saying, “Uh, you know” to fill some space while they’re trying to think of how to express their idea.

It gets worse though. The next paragraph of this “fact check” says:

Who knew that the United States invaded Iraq because of a potential dirty-bomb threat? That’s certainly not what the administration suggested before the war.

This is incredibly misleading. The transcript shows Bush was asked a specific question:

You talk about the current situation in Iraq and the growth of ISIS. And you look at that country today, there are atrocities. There’s violence. There’s chaos. Can you argue today that that country is a safer place and a better place than it was when Saddam Hussein was in power?

This question clearly asks if Iraq is a safer and better place due to the invasion. That has nothing to do with the stated reasons for the invasion. Those reasons could have been wrong, false or even intentionally dishonest. That still wouldn’t tell us whether or not Iraq is a safer and better place.

Bush answered the question he was asked. Kessler’s “fact check” calls Bush’s answer misleading because if he had given it in response to an entirely different question, it’d be misleading. That’s ridiculous. You can’t strip a statement of all context in order to pretend it was given in response to an entirely different question then call it misleading like Kessler does when he says:

the former president misleadingly twists the initial rationale behind the invasion

The former president did not discuss “the initial rationale behind the invasion” in this interview. A “fact checker” simply made that up. I have no idea how that is supposed to be “fact checking.”


Arbitrary Aggregations

Should the IPCC be allowed to change values based entirely upon arbitrary, and undisclosed, assumptions? I don’t think so. My recent post discussing undisclosed changes to the latest IPCC report discussed how several changes were made to values which had the note, “Results aggregated by Tol (2013).” A question I raised at the time is, how can the IPCC change values yet claim to have gotten them from a source which lists the original values?

Where Did the IPCC Get its Numbers From?

I was recently reminded of an issue I discovered several months ago. It seems worth revisiting. The Summary for Policymakers of the latest IPCC report (Working Group II) says:

Global economic impacts from climate change are difficult to estimate. Economic impact estimates completed over the past 20 years vary in their coverage of subsets of economic sectors and depend on a large number of assumptions, many of which are disputable, and many estimates do not account for catastrophic changes, tipping points, and many other factors. With these recognized limitations, the incomplete estimates of global annual economic losses for additional temperature increases of ~2°C are between 0.2 and 2.0% of income (±1 standard deviation around the mean) (medium evidence, medium agreement).

The reference given for this text is Chapter 10, Section 9. When we look at that chapter, we find the same text in its Executive Summary. Like in the SPM, it cites 10.9. However, nothing in the text of 10.9 gives those numbers. They seem to have been pulled out of thin air.

Another troubling issue is the text in Chapter 10’s Executive Summary was changed to match that in the SPM. In the “Final Draft” of Chapter 10, the text originally said:

Globally aggregated economic impacts of global warming are a small fraction of income up until 3°C [10.9.2, medium evidence, high agreement]. A global mean average temperature rise of 2.5C may lead to global aggregated economic losses between 0.2 and 2.0% of income (medium evidence, medium agreement) and losses increase with greater warming. Little is known about aggregate economics impacts above 3°C. Impact estimates are incomplete and depend on a large number of assumptions, many of which are disputable.

General differences aside, the thing which stands out is the previous version of the text said a rise of 2.5C may cause 0.2 – 2.0% economic losses. The new version says “economic losses for additional temperature increases of ~2°C are between 0.2 and 2.0%.” They’ve shifted the amount of warming necessary for these damages up by half a degree, and they’ve done so without any explanation.

A clue might be present in a previous version of the SPM. It said:

Global mean temperature increase of 2.5C above preindustrial levels may lead to global aggregate economic losses between 0.2 and 2.0%

Notice it says a rise above preindustrial levels. We could perhaps assume the 2.5C figure is in relation to preindustrial levels while the 2C figure is in relation to modern levels even though there has been a greater amount of warming than .5C. This assumption requires assuming the “Final Draft” of Chapter 10’s text was wrong to refer to a “rise of 2.5C” as ~20% of that rise had already happened.

Even with that assumption though, nothing much is explained. Even if we know exactly what value the IPCC intends, we have no explanation for where the values came from. It seems they were pulled out of thin air. However, if they were somehow based upon Section 10.9, then they would be affected by the changes I highlighted in my recent post. It’s difficult to see how they could get the exact same results for these two sets of data:


So where did the IPCC get its numbers from? Did it just make them up? Did somebody, somewhere, perform some secret calculations we’re not allowed to see?

It’s impossible to read Section 10.9 and think the numbers came from it. Does that mean the IPCC lied about where it got those numbers from?

Knowingly Promoting False Conclusions(?)

Richard Tol has a blog post responding to an article in the Guardian discussing one of the undisclosed changes to IPCC report I mentioned in my last post. There are a variety of things worth commenting on in it, but one stands out to me more than the rest. The Guardian article focused on the fact this sentence had been removed from the IPCC report:

Climate change may be beneficial for moderate climate change but turn negative for greater warming.

Tol’s post explains:

Here is the story. The old data (the blue circles) roughly fit a parabola: first up, then down, and ever faster down.

The new data do not fit a parabola: The initial impacts are positive, but the progression to negative impacts is linear rather than quadratic.

If you fit a parabola to this data, you will find that the mildly negative estimate at 5.5K dominates the positive estimate at 1.0K and the sharply negative estimate at 3.2K. The parabola become essentially a straight line through the origin and the right-most observation.

I think the appropriate conclusion from this is to fit a bi-linear relationship to the data, rather than stick with a parabolic one. This was not yet in the peer-reviewed literature when the window for AR5 closed, so we decided to just show the data.

To better understand what Tol is saying, we can look at the old and new curves he refers to. Here’s the image he tweeted of them:

As we can see, the old curve says some amounts of global warming will have (net) benefits. The new curve says no amount of global warming will have (net) benefits. That’s a significant change.

Tol’s post explains that change is not due to the multitude of data errors in his earlier work as claimed by some. We can verify that by looking at the effect of the corrections:


The corrections clearly are not the cause of the change in his conclusions. That means, as he says, the change is due entirely to the newer, more up-to-date, data. That would be the data he showed as diamonds in this figure he added to the IPCC report:


But think about that. The figure he added has the new data, meaning the curve for it would show no benefits. Tol says that means a different regression should be used to generate the curve, but that hadn’t been done at the time. The only regression which had been published would show no benefits. Given that, why would Tol try to make the IPCC say:

Climate change may be beneficial for moderate climate change but turn negative for greater warming.

When he knew there was no published work to support that conclusion for the data he was showing? Doesn’t that mean he knowingly added a conclusion to the IPCC report which couldn’t be supported by any published literature? And doesn’t that mean the IPCC lets authors add conclusions to its report not supported by the data shown in the report, much less any published literature?

Or am I missing something?

Side note, Tol’s claim to have fit a parabola to the data is false. He actually used a linear + parabolic fit. That’s why the new curve was so linear – he used a model with a linear component.

Undisclosed Changes in the IPCC AR5 Report

I’ve previously documented changes in the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) which were made absent any external review. This happened despite the IPCC principles stating:

Three principles governing the review should be borne in mind. First, the best possible scientific and technical advice should be included so that the IPCC Reports represent the latest scientific, technical and socio-economic findings and are as comprehensive as possible. Secondly, a wide circulation process, ensuring representation of independent experts (i.e. experts not involved in the preparation of that particular chapter) from developing and developed countries and countries with economies in transition should aim to involve as many experts as possible in the IPCC process. Thirdly, the review process should be objective, open and transparent.

Working Group/TFI Co-chairs should arrange a comprehensive review of reports in each review phase, seeking to ensure complete coverage of all content.

It is difficult to see how a review process can be “open and transparent” if significant changes are made absent any review. It is difficult to see how a review process can “ensure complete coverage of all content” when entire sections are (re)written absent any review.

I’m not going to dwell on that today though. The changes I discussed before were between various drafts of the IPCC report. Today, I’m going to discuss changes made between the “Final Draft” and “final version” of the IPCC WGII report. The Final Draft was released to the public on March 31st with this word of caution:

The Final Draft Report has to be read in conjunction with the document entitled “Climate Change 2014: Impacts, Adaptation, and Vulnerability. Working Group II Contribution to the IPCC 5th Assessment Report — Changes to the Underlying Scientific/Technical Assessment” to ensure consistency with the approved Summary for Policymakers (IPCC-XXXVIII/DOC.3) presented to the Panel at its 38th Session. This document lists the changes necessary to ensure consistency between the full Report and the Summary for Policymakers, which was approved line-by-line by Working Group II and accepted by the Panel at the above-mentioned Sessions. A listing of substantive edits additionally indicates corrections of errors for the Final Draft Report.

But has been widely treated as being the “IPCC Report.” The actual final version was only published two days ago, on October 15th. It has changes as indicated would be made in that disclaimer. The list of “substantive edits” is available here. An errata listing several minor changes is available here. A list of changes necessary for consistency with the Summary for Policymakers (SPM) is available here. That these documents are made publicly available suggests a genuine interest in transparency. That suggestions is completely undermined, however, by the multitude of changes the documents don’t mention.