Undisclosed Changes in the IPCC AR5 Report

I’ve previously documented changes in the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) which were made absent any external review. This happened despite the IPCC principles stating:

Three principles governing the review should be borne in mind. First, the best possible scientific and technical advice should be included so that the IPCC Reports represent the latest scientific, technical and socio-economic findings and are as comprehensive as possible. Secondly, a wide circulation process, ensuring representation of independent experts (i.e. experts not involved in the preparation of that particular chapter) from developing and developed countries and countries with economies in transition should aim to involve as many experts as possible in the IPCC process. Thirdly, the review process should be objective, open and transparent.

Working Group/TFI Co-chairs should arrange a comprehensive review of reports in each review phase, seeking to ensure complete coverage of all content.

It is difficult to see how a review process can be “open and transparent” if significant changes are made absent any review. It is difficult to see how a review process can “ensure complete coverage of all content” when entire sections are (re)written absent any review.

I’m not going to dwell on that today though. The changes I discussed before were between various drafts of the IPCC report. Today, I’m going to discuss changes made between the “Final Draft” and “final version” of the IPCC WGII report. The Final Draft was released to the public on March 31st with this word of caution:

The Final Draft Report has to be read in conjunction with the document entitled “Climate Change 2014: Impacts, Adaptation, and Vulnerability. Working Group II Contribution to the IPCC 5th Assessment Report — Changes to the Underlying Scientific/Technical Assessment” to ensure consistency with the approved Summary for Policymakers (IPCC-XXXVIII/DOC.3) presented to the Panel at its 38th Session. This document lists the changes necessary to ensure consistency between the full Report and the Summary for Policymakers, which was approved line-by-line by Working Group II and accepted by the Panel at the above-mentioned Sessions. A listing of substantive edits additionally indicates corrections of errors for the Final Draft Report.

But has been widely treated as being the “IPCC Report.” The actual final version was only published two days ago, on October 15th. It has changes as indicated would be made in that disclaimer. The list of “substantive edits” is available here. An errata listing several minor changes is available here. A list of changes necessary for consistency with the Summary for Policymakers (SPM) is available here. That these documents are made publicly available suggests a genuine interest in transparency. That suggestions is completely undermined, however, by the multitude of changes the documents don’t mention.

Consider, for instance, Figure 10-1. Here is the previous version (left) and the new version (right):

10-17-Fig10-1_comparison

That it was redrawn to look better is fine. The problem is the data represented in the two figures are not the same. The diamond (representing an estimate published in the last five years) between 2 and 2.5 degrees has shifted almost to 3 degrees. Two points previously at 2.5 degrees have shifted left to 2.2 degrees. A new diamond was added at 3 degrees. A circle at three degrees and about 5%, which previously stood out, has vanished. The diamond at the far right side has shifted even farther right, going from just under five to just to under 5.5 while also dropping quite a bit. The diamond at the bottom has fallen as well, being lower by nearly one full point.

Not a single one of these changes was disclosed. We can verify them, however, by examining the tables provided for both figures. I’ve previously displayed the table for the first figure:

AR5-10-B-1

Here is the new version of the table (found in the IPCC Supplementary Material):

10-17-Figure10-Table

Every difference I highlighted in the figures can be confirmed in these tables. More differences can be found as well. For instance, we can see the estimate from Nordhaus 1994a was changed from -4.8 (-30.0 to 0.0) to -1.9 (median), -3.6 (mean) [-21 to 0.0]. This change is neither disclosed nor explained. Also unexplained is why only both the median and mean values are shown yet only the median value is displayed in the graph.

A partial explanation is available if we examine Nordhaus 1994a. It turns out the original value was taken from the wrong figure in the paper. The earlier value was the mean “likelihood of a high-consequence event” instead of the “impact on global output.” The change fixes an obvious error. What isn’t obvious, however, is why the table previously listed only the mean value while the new one lists both the mean and median. Also unexplained is why now only the median value is plotted while previously the mean value plotted.

Another difference can be found in the entry for Hope 2006. This entry has some history to it as the creator of the table previously published its estimate as +0.9 instead of -0.9. That error wasn’t present in either of these versions of the table though. Instead, a different error was published. The range given for the paper was originally listed as -0.2 to 2.7, a range which excludes the -0.9 estimate listed. This error was one of the few listed in any of the IPCC documents. Its list of substantive edits says:

In cell intersection Hope (2006a) row and Impact (% GDP) column, change ‘-0.2 to 2.7’ to ‘-2.7 to 0.2’

I’d have no problem with this change as it is clearly an error and the error is disclosed, save for one thing. The range given in the final version is not -2.7 to 0.2 as specified. Instead, it’s somehow changed to -2.7 to 0.0. This is despite the fact both entries have a footnote which says, “Results aggregated by Tol (2013).”

How can there be two different ranges given when both are supposedly taken from the same paper? I don’t know. No explanation is given. When we look at the latest version of that paper, a version published to correct and update the table now used in the IPCC report, we see:

10-17-Tol-corrected_table

The range for Hope 2006 is listed as -2.7 to 0.2. The new version of the IPCC table does not match the published values of the paper it cites. It’s interesting to examine the Nordhaus 1994a entry for this table as well. Unlike the new IPCC table, it only lists the mean value, -3.6, which was not plotted. The median value the IPCC report plots, -1.9, is not listed.

Another issue is the two Mendelsohn entries previously had the footnote saying their “Results [were] aggregated by (Tol 2013).” That footnote has inexplicably vanished. It is difficult to explain this change, along with the fact their degree value changed from 2.5 to 2.2. We can try to figure out what happened by examining the papers as we did with Nordhaus 1994a, but that’s complicated by fact the reference section shows:

10-17-References

Only one Mendelsohn 2000 paper is listed. Why are there two entries but only one paper listed? It’s hard to say. Examining the paper the IPCC report cites doesn’t help. It doesn’t provide either estimate listed. It doesn’t even examine a scenario with 2.5 or 2.2 degrees of warming. It examines a different scenario:

Country-specific results indicate that the 2°C global-mean warming projected for 2060 will result in net market benefits for most OECD countries and net market damages for most non-OECD countries.

All hope is not lost, however. There is a different, uncited, Mendelsohn 2000 paper titled “Comparing impacts across climate models.” This paper says it examines a scenario with a climate sensitivity of 2.5C and a predicted change of 2.21C:

Third, we specify a global temperature sensitivity of 2.5C. The model predicts a global-average temperature of 2.21C by 2100.

It seems plausible the original version mixed up these two values. This is especially true since the paper says:

The market impacts predicted in this analysis do not exceed 0.1% of global GDP and are likely to be smaller.

This does not provide the two values listed in the IPCC table, but it shows the paper’s results are in line with what that table shows. Also, the paper does use two different methodologies which could perhaps explain how two different answers were obtained.

It does not, however, explain why the two estimates would be cited as two different papers. Nor does it explain why one estimate would be described as generated via “Enumeration” while the other is generated via a “Statistical” method. Nor does it explain why one is listed as using “Agriculture, forestry, sea level rise, energy demand, water resources” data while the other is listed as using only “Agriculture, forestry energy demand data.” And of course, it doesn’t explain why values from it would be listed while it isn’t cited, yet values from another paper which was cited aren’t listed.

But perhaps the most troubling aspect is we still don’t have an explanation for where the numbers listed came from. The table published in the IPCC report inexplicably dropped the note the results were aggregated by Tol 2013. That’s true even though the same note has accompanied this table time after time in the past. For instance, it is noted in the same table of Tol 2012. Interestingly, that table has an entry for only one Mendelsohn 2000 paper, and its description matches the paper I suggested the values came from. Also interesting is the fact Tol 2012 doesn’t actually cite any Mendelsohn 2000 paper.

In any event, the issue of where the numbers come from goes back to the Hope 2006 entry whose given range inexplicably changed. You see, Richard Tol had never published the calculations which went into his “aggregations,” even after being asked to repeatedly. That means nobody knew how he generated the “estimates” from those papers. They weren’t taken directly from the papers he cited, but rather, were calculated in some undisclosed manner. Tol’s reaction to people trying to check his work was to say things like:

Finally, Professor Abraham repeats Mr Ward’s lament about lack of transparency. In fact, all data are in the public domain. It does take a bit of multiplication, addition and division to go from disaggregate data to aggregate data, but that should not be beyond someone with a bachelor’s degree in geology.

Which is obviously not how a scientist should behave. It’s also strange to hear him insist the process is so easy when he produces inconsistent results with it. You see, Richard Tol is the person responsible for this section of the IPCC report, the one heavily relying upon his own work. And you see, the IPCC reacted to pressure about Tol’s refusal to say how he got his results by forcing him to upload the calculations. There are a number of issues with those calculations, including the fact several of them don’t produce the results Tol mocked people for being unable to replicate, but the most important aspect at the moment is they don’t include any calculations for the Mendelsohn papers. Because the note saying their results were aggregated by Tol 2013 was dropped, Tol managed to avoid having to upload the calculations for them.

Does it matter? I don’t know. What I know is the lack of explanation for any of this is quite troubling. Similarly, the extreme estimate at the bottom of the figure changed because the Maddison and Rehdanz entry in the tables changed. Originally it was -11.5%, but that inexplicably changed to -12.4%. Again, both of these results were attributed to the aggregation done in Tol 2013, the aggregation Richard Tol refused to document. Now that we have documentation, it doesn’t match his previous answers.

Whatever. I’ll move onto another example. The table shows two diamonds which changed locations belong to the Roson and van der Mensbrugghe 2012 paper. The previously listed results were -1.8 for 2.3C and -4.6 for 4.9C. The paper says:

According to our preliminary estimates, at the global level, the most serious consequence from climate change will be changes to labor productivity that would induce 84% of the global damage in 2050 (-1.8% of global GDP) and 76% in 2100 (-4.6% of global GDP).

It would appear the IPCC report changed 1.8 to 2.1 because 1.8 is 84% of 2.1. Similarly, it would appear 4.6 was changed to 6.1 because 4.6 is 76% of 6.1. This is wrong. The values in parentheses are the total damage, not the damage caused solely by changes to labor productivity. This can be easily confirmed by examining another paper by the same authors. It says:

At the global level when all of the climate change impacts are included, global output is some 1.8 percent less than in the standard baseline in the year 2050, i.e. the rise in mean temperature of 2.3º C induces an overall economic decline of 1.8 percent

Pushing the simulation out to 2100 leads to an overall loss in 2100 of 4.6 percent, with the labor productivity impact still dominating the overall losses.

It appears the undisclosed changes to the percent values for this paper were made to fix a problem which didn’t actually exist. I’m not sure why the temperature values were changed though. The original temperature values had been 2.3 and 4.9. These were changed to 2.9 and 5.4. This is strange as a figure in that paper shows those values are not correct:

10-17-Roson

In fact, the paper explicitly states the one value:

The global average temperature is found to change in the 21st century by 4.87 °C, reduced to 4.79°C when climate impact feedbacks are taken into account

While the other paper I mentioned lists the other value in its Table 2 (as 2.34). That the values were changed to be something completely different when its clear what values the authors of the paper used is inexplicable.

Also inexplicable is why the figure shows five diamonds when there are only four new entries. The text of the report makes this perfectly clear. The legend for the figure says diamonds are for estimates published after the last IPCC report (AR4). The text says:

Since AR4, four new estimates of the global aggregate impact on human welfare of moderate climate change were published (Maddison and Rehdanz, 2011; Bosello et al., 2012; Roson and van der Mensbrugghe, 2012), including two estimates for warming greater than 3°C. Estimates agree on the size of the impact (small relative to economic growth), and 17 of the 20 impact estimates shown in Figure 10-1 are negative. Losses accelerate with greater warming, and estimates diverge. The new estimates have slightly widened the uncertainty about the economic impacts of climate.

If there are “four new estimates,” there should be four diamonds. Instead, there are five. By eyeballing the figure and looking at the table, I’d say the Nordhaus 2008 estimate (which inexplicably changed from -2.5 to -2.6) is being listed as “published after IPCC AR4,” but I have no idea why that would be the case. What I also don’t know is why the text for this paragraph has changed. Previously, it said:

Since AR4, four new estimates of the global aggregate impact on human welfare of moderate climate change were published (Bosello et al., 2012; Maddison and Rehdanz, 2011; Roson and van der Mensbrugghe, 2012), including two estimates for warming greater than 3°C. Estimates agree on the size of the impact (small relative to economic growth) but disagree on the sign (Figure 10-1). Climate change may be beneficial for moderate climate change but turn negative for greater warming. Impacts worsen for larger warming, and estimates diverge. The new estimates have slightly widened the uncertainty about the economic impacts of climate.

Notice how the old version implied the “four new estimates” disagree on the sign of the impact. The new version improves this by making it clear it is discussing all 20 estimates, not just the four new ones. Additionally, it is good the new version removes the claim climate change “may be beneficial for moderate climate change but turn negative for greater warming” as there was only one data point which suggested there might be any sort of (net) meaningful benefits from global warming. These change are improvements. That just does not excuse the IPCC’s failure to disclose them.


I don’t claim this discussion has been exhaustive. There are likely other examples of undisclosed changes to be found. There is certainly more to be said about this one section. For now though, I’m content with what I’ve shown here. The Final Draft of the IPCC WGII Report has been out for nearly seven months. It has been used by the public and media as “the IPCC Report.” And yet, we now see substantial changes were made to it before the official final version was published, changes made without any disclosure, explanation or transparency. To add to the strangeness, we see such changes were made to a section, figure and table which were never reviewed in the first place as they were added to the IPCC report only after the last round of reviews.

One could easily believe the apparent need for inexplicable and undisclosed changes wouldn’t be necessary if the IPCC would bother to have “a comprehensive review of [its] reports” where “experts not involved in the preparation of” material actually reviewed what went into the report.

But that might require a review process which is “objective, open and transparent.”

Advertisements

11 comments

  1. Richard S.J. Tol, if you’d like to contribute something to the discussion, feel free to. Otherwise, people are just going to see you refusing to deal with your mistakes.

    And if they look at your comments elsewhere, they’ll see you insulting anyone who points out those mistakes.

  2. Richard S.J. Tol, three things. Just yesterday you called Bob Ward a liar for saying multiple changes were made to address his concerns. You claim only one change was made because of him, that a 4.9 was changed to 4.8. I’ve shown many “errors” were “corrected,” including a number his correspondence highlighted. It’s difficult to reconcile this.

    Two, I’ve demonstrated some of the changes made were did not result in “errors” being “corrected.” They resulted in mistakes being added.

    Three, the IPCC report falsely cites your Tol 2013 for values given by your posted aggregation calculations. Those calculations give different results than any of your previous calculations, including those published in Tol 2013. The reality is the IPCC now cites calculations which were never published in any literature but lies about it. Additionally, it now falsely claims to take values from a Mendelsohn paper without any aggregation by you even though you’ve consistently you aggregated the results yourself. This is especially odd as the IPCC cites a Mendelsohn paper it couldn’t possibly have gotten its values from.

    Oh, and by the way, making changes secretly without review or disclosure is wrong. Given you are the person responsible for the work this was done regarding, and a person responsible for the IPCC chapter it was done in, it seems you ought to step up and address the problems.

    Or, you know, you could just try to cover up whatever role you had in introducing false citations into the IPCC report.

  3. Brandon

    Thanks for your efforts. It’s a shame we have have to spend valuable time unravelling what others have ravelled up with the specific intent of concealing their actions. That always puts us on the back foot because we have to reverse engineer their intricate manipulations of the data and text before we can reveal that sleight of hand. We know when we are on the right track when the perpetrators come up with startling well-prepared answers that seem reasonable at first until we then unravel those talk.

    A perfect example of this phenomenon at work is in the following article I wrote. It highlights exactly the same sort of graph manipulation in the SPM presented by Thomas Stocker of WG1. I spent hours unravelling what they had done before posting. But the really telling part is when Tim Osborn of CRU enters the fray in the comments. It’s quite clear they have thought of every eventuality if they should be found out.

    Reading those comments, it seems as though they had said to themselves, “what if someone notices that, what shall we say?” and then crafted a carefully spun excuse. It was as if Tim Osborn was producing a fig leaf at every stage, that had been left for him in advance by the wayside. All the excuses were thought out in advance, all of them totally inadequate when seen in the context of the media coverage that misinterpreted the original spin at the press conference.

    I (Scute/Andrew Cooper) took Tim Osborn to task on all his comments.

    I have linked my article at today’s WUWT repost of your article (1st comment).

    Scute

  4. Scute, I remember that example being discussed. If I may though, your comment reminded me of another oddity related to this that I want to comment on before I forget. The WGII SPM says:

    With these recognized limitations, the incomplete estimates of global annual economic losses for additional temperature increases of ~2°C are between 0.2 and 2.0% of income (±1 standard deviation around the mean) (medium evidence, medium agreement).

    While citing Chapter 10. Chapter 10’s Executive Summary section has the exact same text, citing section 10.9. Section 10.9 does not have any text to support those values. There is absolutely nothing in 10.9 which gives a total, quantified estimate. That means the IPCC SPM cites text in Chapter 10 which has absolutely no given basis.

    Additionally, the changes I highlighted in this post are all in Section 10.9. They all deal with the attempt to quantify the potential economic damage of global warming. It stands to reason the values given should be tied to the data I’m discussing. If that’s true, the changes I highlight could well change the results given in the SPM.

    But we can’t tell because the SPM uses values not present anywhere in the report, and the changes to the data were never disclosed, much less explained.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s