University of Queensland Doubles Down on Hiding Data

I was just in bed failing to fall asleep when I got an e-mail alerting me to a public statement by the University of Queensland acting Pro-Vice-Chancellor (Research and International) Professor Alastair McEwan. The statement is regarding the publicity my recent posts about the “97% consensus” paper by John Cook and colleagues. It’s also highly misleading, if not completely wrong.

The idea behind the statement is the University of Queensland is not trying to hide data. According to McEwan, all data “that are of any scientific value were published.” He says only “information that might be used to identify the individual research participants was withheld.” It’s difficult to reconcile these claims with reality.

I’ll begin with an indisputable issue. We’ve established data for 521 papers was not included in any data files. There’s no way data for these 521 papers would be problematic if the exact same data could be made available for the other 11,944 papers. And surely, if the data for the other 11,944 papers is of “scientific value,” the data for these 521 papers is as well.

Now we’ll move onto a slightly more complicated issue, timestamps. There were over 20,000 ratings done for this project. How could knowing when those ratings were done possibly “be used to identify the individual research participants”? Does anyone really believe it’d be possible to look at a time and date then conclude one particular person, out of the entire world, must have done the rating at that point? Of course not.

Or are we to believe the time of ratings is of “no scientific value”? That can’t be true either. I’ve recently shown there are patterns in some of the ratings determined (partially) by when the ratings were done. Professor Richard Tol has shown the same thing over the entire data set. If data is dependent in part on when it was collected, it is certainly of “scientific value” to take note of the time that data was collected.

This brings us to a third type of data which hasn’t been released – rater IDs. The only reason rater IDs could identify anyone is the ID numbers of participants are the same as their ID numbers at Skeptical Science. That problem could be resolved by simply anonymizing the ID numbers (by assigning new, random values to them). That would make it impossible to identify the participants.

A secondary issue is rater ID data could allow people to associate specific ratings with specific raters. This is inconsequential. If we already know Mr. X was a participant in this study, what harm is caused by knowing which ratings he contributed? The only way someone could be harmed by being associated with specific ratings is if there were issues with those ratings. That would be of “scientific value.” If individual participants were biased or flawed in some other way, that’d be good to know. It’d be so good to know, several participants of the study suggested they should look at it:

I supect that we have all felt that our rating criteria have drifted with time and it would be useful to review those rating where there is disagreement. Some ratings discrepancies may also be fat-finger entry mistakes and those, of course, need to be fixed. It may be helpful to have some stats., for example, the percentage of certain ratings that we have done compared to the group overall. This may reveal some systematic bias of certain individuals, well, maybe not bias but failure to grasp the criteria correctly.

Following Andy, I would like to see the percent I rated in each category relative to the whole sample, i.e. 50% of mine are neutral vs 55% of everyone’s

If the authors of the paper thought data was of “scientific value” when they were doing the study, how can anyone now claim it isn’t of “scientific value? That’s just silly.

But lets move beyond all that. Suppose it truly is important to keep the identity of raters private. Why then did I just load this image at Skeptical Science:

tcp_raters2

That shows the identity of 11 raters, and it’s been viewable on Skeptical Science for a couple years now (archived for posterity here). So too has this one (archived here):

tcp_raters3

This one also identifies nearly a dozen individual participants. It’s true we only found out about these images because of a hack, but that hack happened nearly two years ago. Surely the authors of the paper shouldn’t leave confidential information in a publicly accessible location for two years, even if people have already seen it.

Beyond that, we have to ask, why was this data ever available in the first place? None of the identities of the participants were keep secret from one another. Heck, people not involved in the project could post in the same forum this data was posted in! How can anyone claim it was confidential? Did everyone involved in the project, and everyone with access to that forum, all sign a confidentiality agreement? If not, the data was never kept confidential.

I’d love to know if there were such confidentiality agreements. That’s why I specifically asked the University of Queensland for them. I wanted to know what data was confidential so I could keep that in mind when considering what data I should or should not release.

John Cook refused to tell me. Later, when the University of Queensland sent me a threatening letter, they invited me to respond. I did, asking for information about these confidentiality issues. They ignored me. They were apparently willing to threaten me with a lawsuit to try to get me to shut up, but they weren’t willing to answer a simple question.


I get Alastair McEwan is in a difficult spot. I don’t sympathize with him though. It’s his own fault. This all began when I asked to be told what data should be kept confidential and why. I’ve asked that question multiple times, and everyone has refused to answer it.

McEwan could have resolved this entire situation by answering that one simple question. He chose not to. He chose not to address this simple issue in a direct and reasonable manner. Instead, he chose to make false and misleading statements to the media.

Advertisements

46 comments

  1. One new item (to me anyway).

    McEwan claims that the Cook study was given ethical approval by UQ. If true, this would go towards confirming their ownership of the study / data etc

  2. SkS files: Ari Hits 3000-

    Ari “the other day I practiced my guitar playing and rated papers at the same time. 🙂 “

  3. Amazing. The UQ is claiming that the raters were to be protected due to the ethics approval? So are they admitting that this “research” was actually a study about the raters skills to classify scientific papers according to a set of parameters and not about the actual content of the papers at all? Otherwise, why would they need ethics approval for rating scientific papers? This is going to get really entertaining, time to dig up the ethics approval? If raters are to be understood as research participants then we will be seeing a lot of anon-science in the future without any possibility of falsification.

  4. Barry, this Ari fellow is a true “debunking-warrior” of the AGW, just take a look at his blog: http://agwobserver.wordpress.com. It is exactly raters like him why this kind of “research” needs to stand the test of falsification which the UQ is now is trying to stop. By doing this they are destroying science in order to save “climate science”.

  5. McEwan claims that the Cook study was given ethical approval by UQ.

    If so, the ethics approval for the 97% study should be available and it should predate the study itself. Someone Australian might FOI it. Also, they could FOIA other discussions about the approval, including information about the plan and so on. I suspect someone will.

  6. Ari Jokimaki was one of the defenders that another Alistair put in place by demonstrating Mann’s upside usage.

  7. they are destroying science in order to save “climate science”.

    Not really a trade worth making, is it.

  8. The following is a statement from UQ acting Pro-Vice-Chancellor (Research and International) Professor Alastair McEwan.

    [regarding the John Cook et al (2013) paper ‘Consensus’ published in ERL]

    “Only information that might be used to identify the individual research participants was withheld.

    This was in accordance with University ethical approval specifying that the identity of participants should remain confidential.”

    – – – – – – – – –

    Alastair McEwan,

    Then perhaps you should talk to the ‘Concensus’ paper authors John Cook et al about their violation of the University ethics approval for their paper, because they identified some “individual research participants” (the raters) in their paper. Here is a quote from the paper identifying some raters by name.

    From the Cook et al (2013) paper published in ERL,

    “Acknowledgements

    Thanks to James Powell for his invaluable contribution to this analysis, Stephan Lewandowsky for his comments and to those who assisted with collecting email addresses and rating abstracts: Ari Jokimäki, Riccardo Reitano, Rob Honeycutt, Wendy Cook, Phil Scadden, Glenn Tamblyn, Anne-Marie Blackburn, John Hartz, Steve Brown, George Morrison, Alexander C Coulter, Martin B Stolpe (to name just those who are not listed as (co-)author to this paper).”

    John

  9. AndyL, lucia, I didn’t even notice that when I first read the press release. I got distracted by all the other problems. It’s an interesting point. I’d love to know what John Cook submitted for an ethics approval regarding this project.

  10. Brandon,

    There is a saying from my time in the service for when someone screws up big time. Alastair McEwan and the UQ have not just managed to step on their crank, they are doing an Irish Jig on it.

  11. Time for a paper reviewing the peer review process as it relates to CAGW.
    If , as it might appear to the layman, there has been a significant bias then the University should be conducting a serious examination of its own motivation. Massively induced peer pressure springs to mind and “I was only following orders ” is scant excuse in these enlightened days. If you do not reveal your participation in the fraud right now , expect zero mercy when the entire corrupt regime is exposed.
    Time is on my side,don’t ya know?
    17 years and 6 months ,,,,,,,,,,,get it yet?

  12. or whether it was a Queensland sponsored paper at all.. that they are just attaching themselves to it (whilst SKS did all the work)
    It would be nice if they clarified this.

  13. Ari Hits 3000 – article, SkS hacked forum

    John Cook:
    Damn, I only find out now that I could’ve been rating all this time on the iPad? So much lost opportunity! I just did a half hour exercise on the cross trainer and knocked off 30 ratings while I exercised.

  14. If there was a real confidentiality agreement, that figure should never have been made, not without anonymizing the evaluators. (E.g., 1, … 24.)

    The fact that the figure exists either points to gross negligence and incompetence on Cook’s part or it is a recognition that there was no assumption of privacy from the beginning on the part of the evaluators.

  15. I’ve done human subjects studies, and it is a rule that the subjects aren’t identified to each other, let alone to third parties not involved in the project.

    (Is there a count of the number of participants of the secret forum at the time of the hack?)

  16. This was (as far as can be established from the lies…er, information available) not a study of human subjects. It was a study of research publications. The references to secrecy, confidentiality, and so on, do not apply and are obviously a red herring to divert attention from the fact that release of the data would immediately reveal that the paper is worthless.

  17. Carrick, I agree. I really don’t see how the University of Queensland could expect to justify its arguments of confidentiality.

    As for the secret forum, there were a large number of people who had access to it, but I’m not sure if they could all access the TCP subforum. It’s possible most people who were active in the forum couldn’t see the discussion. I don’t think so though. This was said in a topic in General Chat:

    Just letting you all know you can now rate papers for TCP:

    http://www.skepticalscience.com/tcp.php?Action=Rate_Papers

    The goal is to get through the rating within 2 months. The reason for the timeframe is I hope to have this submitted to a journal and accepted before the IPCC AR5 deadline (which I think is August 2012). But if our first journal doesn’t accept our paper, then we have to resubmit. So the sooner we get through the paper rating, the more likely our chances. Getting in the IPCC AR5 is a long shot but we should give ourselves every chance.

    Any questions about the rating of papers, post in this thread:

    http://www.skepticalscience.com/thread.php?t=4566&r=0

    That seems to indicate everyone who had access to the forum could participate if they wanted. If so, you’re talking about dozens of people who didn’t participate in the study, but had access to “confidential” data from it.

  18. Carrick, if I’ve understood the “User_LevelId” values (in the users.csv released after Skeptical Science was hacked) correctly, there were over 200 people with access to the SkS forum while this study was going on. I don’t know how many were active, but that’s a large number of people with potential access to “confidential” material.

    Also, this isn’t just a matter of those images. From my reading of the discussions in the forums, raters could see each others’ progress via the TCP web interface.

  19. I’m really confused.
    The research had nothing to do with how the individuals rated. Hell, they are part of the research team, not the subject of the research. They should be named as co-researchers. The subject of the research is the scientist’s papers. The research should all be made public.
    What’s to hide?
    cn

  20. lucia, there are only six self-ratings that weren’t released by Cook et al. I’ve barely mentioned them. I doubt most people who saw my posts even realized I had any undisclosed self-ratings.

  21. Chuck Nolan:

    The research had nothing to do with how the individuals rated. Hell, they are part of the research team, not the subject of the research. They should be named as co-researchers. The subject of the research is the scientist’s papers. The research should all be made public.

    Well that is what most of us are leaning to, of course.

    Brandon says that as many as 200 people had access to this supposedly confidential information. So it doesn’t look like confidentiality was even a “concern”, until people started examining the paper and finding problems with it.

    Then all of a sudden there was an outcrop of worry over “protecting the confidentiality of the subjects”.

  22. I meant to say “the confidentiality of the evaluators”. They obviously aren’t the subjects. (It doesn’t help to insert further linguistic confusion into this mix.)

  23. The contortions going on in climate science and in many universities associated with it, is astounding. I guess they’re hoping to ride it out and wait for all the interest to die down. Trouble is, their shenanigans make it all the more interesting. They must realize that more people are looking their way and beginning to wonder. Keep doing what you are doing, Brandon, the UQ have painted themselves into a corner.

  24. Do those names constitute the complete set of raters? If not, what portion are they responsible for?

  25. “McEwan claims that the Cook study was given ethical approval by UQ.”

    IF it is truly an ethics approved UQ work, then it is subject to a full FOIA request. Something they should be well aware of.

    Why they would undertaking this poking of the hornets nest with a stick – making what we all understand to be silly and meaningless threats, and thus exposing themselves to further investigation and ridicule, is simply amazing.

  26. lucia, good luck with that. I suspect the press release has little connection to reality.

    Carrick, I agree, but I should point out I’m not positive of that argument. I haven’t proven the 200+ figure or that the 200+ could all see the data. I think I can make a strong case for it, but I haven’t taken the time to confirm it with certainty.

    MikeN, those images show 11 of 24 raters. An earlier one showed eight of 24 (archived here). I don’t care to work out the relative contributions at the time the various images were made, but I can easily tell you what the relative contributions were for the top 8 and top 11 when the project was completed. Once all ratings were completed, the eight most active raters contributed 79% of all ratings. The eleven most active raters contributed 94%.

    In other words, those images don’t show data for all raters, but they show an overwhelming majority of the data.

  27. As so many have notices, If there is a ethics approval, it can not affect the paper they published since there were no research participants. The raters were researchers or at the least research assistants. All this is a pathetic smoke screen. However, there could be an ethics approval that relates to another paper that they could be writing for example about the raters themselves. We will never know unless someone uses FOI to call their cards. In any case, sad episode for science, business as usual for climate science from down under.

  28. Before going the FOI route, why not simply as for a copy of the approval? Brandon, as this was mentioned by McEwan in a press release about you, I think you are entitled to request it.

  29. UQ has a form with research malfeasance, admissions corruption, etc., may it is time for University of Queensland acting Pro-Vice-Chancellor (Research and International) Professor Alastair McEwan to consider his future. Is this the level of incompetence that UQ accepts?

  30. Over at WUWT a valid point was raised concerning the anonymity of the self-rating scientist (that is scientist who wrote the papers being rated by the Cook “research” project). I agree that there are valid reasons for concealing the anonymity of such self-rating scientists since they could be understood as research participants, but that does not change the fact that for the most part the UQ is just pulling up a smoke screen. Cook’s other raters (many of them being AGW-activists) have to be considered as researchers or research assistants and their claims of anonymity are therefore ridiculous.

  31. AndyL, I made simple a request for some basic documentation when I first got the threatening letter. The ethics approval would have been covered by it. I was ignored.

    Pethefin, that issue is largely a red herring. I’ve pretty much not talked about self-rating data because it’s largely inconsequential. I have no names, e-mail addresses or other information regarding the identities of the self-raters.

    The only new material I have access to regarding them is six ratings withheld from the original release because the unique pattern in them meant the rated paper could be identified. Six data points is hardly meaningful. That’s why I haven’t brought them up in my posts.

    Incidentally, I suggested I should withhold those data points in my first e-mail to John Cook, to which the university referenced. There’s no way they could be a central issue.

  32. Brandon,

    I agree. The reason I brought this up was the fact that argumentation becomes easier ones you understand the logic of your opponent, no matter how ridiculous it is. Since you have already promised to withhold that particular part of the data, their behaviour is even more ridiculous.

  33. Pethefin, agreed, except I didn’t promise to withhold that data. I specifically suggested it was data that might need to be withheld while requesting I be told what data should be withheld and why. I intended to wait for an explanation of what data should be withheld before making any promises. I almost certainly would have agreed to withhold that data if anyone had requested I do so. I just couldn’t agree to withhold it since nobody ever requested I withhold it.

    I never intended to release that data, but because John Cook and University of Queensland have adamantly refused to request I not release it, I kind of feel like I should. Why should I keep data confidential if nobody wants it kept confidential? It’s ironic. Their approach to trying to get me to not release data (avoiding all discussion and relying entirely upon legal threats) could justify me releasing more data than I would if they had said nothing.

  34. I didn’t realize that this was only partway thru the study, or that it didn’t match the list in the paper. Has the paper identified all of the raters? I count 12 in the acknowledgments, plus 9 coauthors, plus James Powell + Stephan Lewandowsky.

  35. I have to wonder (don’t we all ) about the raters progress chart. (tcp_raters3.gif)

    Based on the totals in that diagram, Mr Nuccitelli rated around 7% of the rated papers (at the time that last screenshot was taken).
    The oddity which is yet to be explained is how most raters showed regular updates of stochastic progress but Mr Nuccitelli is the only person capable of the superhuman feat of apparently rating 900 papers in a single day. A programmatic batch update, perhaps?
    Or perhaps he was so diligent that he rated an equal number of papers every day, thus forming the straight line on that chart from genuine data points that the charting software didn’t display. Yes, that’s probably it. He’s just particularly methodical about keeping to schedule. Including every weekend after the 25 February. What a machine.

    The other question about the meaning of the chart is the same question MikeN asked. If there’s 24 raters in total, why are only 11 shown in the progress chart?

    There’s probably a perfectly rational explanation for all of this. Since the chart leaked 2 years ago these questions may have been answered before but I don’t know where.

  36. Brandon,

    This is very late but hopefully useful to you. On May 20 over at WUWT, you asked how I knew Baerbel Winkler was not a rater. Well, I have been sparring with her for months on Coursera climate courses, where she is SkS’ official attack dog. She admitted that her only role was researching author’s email addresses and that she has no scientific background – she’s in IT.

  37. Brandon – did you ever get this link? If Jan Christoffersen’s claim is true then someone was fibbing.

  38. I never got Brandon’s response to my comment of June 6. During the U. of Melbourne’s Coursera climate course, which ended in June 2014, I finally got her to admit on a discussion forum that her background was in IT and that she had contributed to Cook et al (2013) by researching email addresses of authors. At the time, I commented to her that it was curious that she was credited with authorship for that task but Ari, a big-time rater, was not.

    I have no existing links to the discussions forums relating to that course.

  39. Sorry I couldn’t give you some solid back-up but my memory of her statement is pretty firm. Maybe Baerbel lied as she’s very good at obfuscation as SkS’s attack dog on Coursera discussion forums.

    Otherwise, I am not familiar with the Sekret website and what it contains.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s