My Super Secret Confidential Data

As most of you know, a couple months ago I came across some super secret data for the infamous Cook et al “consensus” paper. I haven’t released that data because the University of Queensland threatened to sue me if I did then threatened to sue me if I showed anyone their letter threatening to sue me:

5-15-copyright

I, of course, thumbed my nose at them and published their letter, challenging them to sue me. I even said:

Tell me what material I possess could cause harm if disseminated. Tell me what agreements or contractual obligations would be impinged upon if that material were released to the public.

If you are unable or unwilling to meet such a simple challenge, I’ll release the data and you can bite me. I mean, sue me.

They didn’t sue me. In fact, they didn’t respond at all. I was tempted to release the data then and there. I didn’t because I wanted to try to find out what the basis for their threats were first. I wanted to know where this idea the data was confidential was coming from. It took two months, but I’m happy to say I have. It was six data points.

I kid you not. Initially, John Cook responded to me on this topic in e-mail by saying:

Brandon, I am waiting for direction from the University of Queensland legal department re the implications of stolen data being released that violates participants’ and raters’ privacy. It’s already public knowledge that the release of raterids can be linked to public individuals (in fact, Richard Tol has posted about how to do so using content from the stolen SkS forum). You alone are responsible, legally, for any actions you takes with stolen information.

His later e-mail contained a subtle change I failed to catch:

I don’t have the rights to waive the confidentiality of participants of scientific research. You alone are responsible, legally and ethically, for any actions you take with stolen information that violates the confidentiality of participants.

Note the difference in what he talks about being violated. In the earlier e-mail, he referred to “participants’ and raters’ privacy.” In the later e-mail, he referred only to “the confidentiality of participants.” I didn’t think anything of that until today, when I read John Cook’s application for an ethics approval (obtained by Simon Turnill via FOI) and saw:

7-11-participants

The question asks who the “participants” of the study will be. Cook defined them as the scientists who would be contacted to rate their own work. That made me realize what had happened. John Cook didn’t get any ethics approval for the ratings he and the others from Skeptical Science did.

John Cook stopped referring to the raters privacy because the raters didn’t sign confidentiality agreements. There were no legal obligations regarding them. Cook just used the fact there were confidentiality agreements with the scientists he contacted to create the illusion there was confidentiality for the people he worked with.

This point is emphasized by the fact Cook’s application was received on May 25th, months after he and his pals started rating the abstracts. In case that’s not enough, he specifically refers to those ratings having been done in his application:

7-11-summary

Since there was never any confidentiality issue with the data involving Cook and his pals, you might wonder, what data was confidential? To answer that, I’ll quote John Cook’s description of a data file:

there were six self-rated papers with a unique combination of year and abstract endorsement level. So to maintain privacy, I removed those six papers from the sample of 2,142 papers.

The version of the file he released had 2,136 entries. The version I found has 2,142 entries. Those six extra entries are all the data I found that was covered by any confidentiality agreement.


Now that Cook’s trick has been figured out, it’s worth pointing out I found ~40 data files. I found over 20,000 datestamps. I found over 20,000 rater ID#s listed. I found hundreds of comments left by raters. I found 24 rater usernames. None of that was covered by any confidentiality agreements.

But John Cook and the University of Queensland said I had to keep all that hidden because six data points were.

Advertisements

33 comments

  1. There’s a related point I didn’t bring up in this post. Simon Turnill also requested copies of:

    all agreements entered into by the University relating to the intellectual property in the data comprising the Paper

    The University of Queensland provided none. As Turnill said, “Conclusion, therefore, is that that data is not subject to any IP restrictions by the university.” Which makes the letter it sent me all the more strange. It seems like they just made everything they said in the letter up purely to try to intimidate me.

    That’s pretty poor behavior for a university.

  2. It does seem to make sense that the only party that could require privacy are the scientists who responded to the survey.

    I don’t know if there’s a name for this tendency of ‘trying it on’ i.e. where a person tries to leave others to pick up incorrect inferences, but I would say the SkS crowd, if not masters of it, seem to instinctively to rely on on that tendency when dealing with people.

    So e.g. if you say you have some previously unreleased details of theirs on subject A they will instinctively claim you are prevented from using it because they have legal protection of privacy without telling you that protection is only for subject B.

    The thing is, one might excuse that tendency in the rough and tumble of personal interplay and disputes, but the worst thing about this slippery mindset is that it seems to extend to their whole ethos in conveying information. So you get the quote fabrications, omissions, hype and use of third party misunderstandings used as a tool to shape every bit of information they reveal, even in their so called educational material.

    I.e. their every word is not to be trusted and needs to be checked for validity. I now feel this is such a blank statement of observable fact about them (as you yourself have documented here) that my previous annoyance when seeing supposedly smart people refer to them as purveyors of information has changed, via utter bemusement, to a realisation that for the non-partisans who do this it can only be from laziness, ignorance, or lack of real care of the subject, that anyone can use them as a source of information. I find this quite a fascinating realisation.

    Do you know if Simon Turnill is writing up the details of this FOI on his blog? I don’t see it mentioned there.

  3. Richard Tol, one of the things I wanted to do after receiving this update was look and see what John Cook said about the ethics approval for his paper. I didn’t because I wanted to get the post up before I fell asleep. I’ll probably review what I can this evening.

    tlitb1, it is remarkable how often John Cook and others at Skeptical Science resort to that approach. The weirdest part to me is I’m not sure any of it is intentional. I’ve called Cook out on this before, and he defended what he said in a way that made it seemed like he genuinely believed it. It’s like they trick their own minds with this or something.

    As for Simon Turnill’s FOI request, I’m not sure if he’s intending to discuss it himself or not. I know he had been planning on resolving one last point of dispute with the University of Queensland before passing the results on. He might have intended to write a post at that point, or he might have just e-mailed the results to the people he knew would be interested.

  4. I don’t think you are bound even for the six. You made no promise to anyone. It’s Cook who violated the privacy by posting on a publicly accessible board. But if you have any qualms, I think you should just post all but those 6.

    You could wait until after you are no longer busy with your move.

  5. lucia, I know I don’t have any obligations in regard to any of the data. I don’t really care about those six data points though. I wouldn’t have hesitated to withhold them if I had been asked to.

    As for waiting, the problem with that is who knows how long I’d wind up waiting for? I’ve already had the data for two months. If I wait until things have settled down from moving, that could wind up being six months. That seems silly.

  6. Tom Curtis made reference to the confidentiality of the self rating scientists in a comment at SKS back in February.

    “I suspect it is fortunate for a number of AGW “skeptics” who self rated that their self ratings are confidential (unless they choose to release them), for I suspect quite a few of them will have rated them as rejecting the concensus, and are now publicly declaring that they ratings must be interpreted such that they are part of the 97%.  As I have not seen the data, that is, of course, just a guess.”

    http://www.skepticalscience.com/stringer-cnn-misrepresent-consensus.html#102351

  7. This is even a bigger, bloodier mess than I had imagined.

    First there is no question that John Cook needed to have ethics approval before conducting the initial rankings by “team members”. The fact that his participants were all “team members” (from the IRB perspective, that just means they are volunteers taken from a smaller pool) does not inoculate you from this. The only thing it does is reduce the paper work in making public identifying information about you.

    Secondly, having performed the rankings prior to seeking any approval has significant negative consequences for this data under UQ rules:

    http://www.compliance.iastate.edu/irb/policies/university.html

    Retroactive Approval

    It is very important that careful consideration be given to the possibility of an eventual desire to publish, present the material, or use any collected data in future studies, etc. Retroactive approval will not be given for studies conducted without IRB approval. For example, if a class project was conducted without IRB approval and resulted in unexpected but important findings or data, those findings or data may not be presented at a national meeting or used in a future project or research study.

    Note the request specifically says “The ratings by team members will be compared to results from the authors of the papers rating their own papers.. “.

    The IRB approved the ethics application, which means they approved the comparison of under UQ rules unpublishable data with the results of the authors of the papers.

    It seems to me the most appropriate action would have been to turn down the ethics application.

    It is possible to have “strict” rules then waive them, especially if these are just policy statements rather than legal requirements.

    It is a legal requirement though to have ethics approval before publishing the team rankings. I suppose UQ, if pressed, could argue that this IRQ approval also (implicitly) retrospectively approved the original rankings from “team members.” I think the better argument is that the approval was made in error.

    Nonetheless, there does seem to be a substantial level of bungling at all levels on this project combined with ethically compromised decision making.

  8. DGH, yeah, but I don’t think anyone has really doubted the self-ratings were done under the assumption of confidentiality. It seemed an obvious thing to me, and really, it’s not like the withheld self-ratings contain any useful information.

  9. Carrick, I suspect the University of Quernsland will take a different approach. I imagine they’d offer two defenses. First, they’d say no ethics approval was needed because the raters were all project team members, thus not requiring ethical oversight. Second, they’d say they had no involvement with the collection of the abstract ratings so they have no obligations relating to those. They merely allowed John Cook to use data he already had under the presumption he obtained it ethically.

    I don’t know how successful either argument would be. I suspect it would depend upon who is asking the questions. The attitude which allowed the Climategate whitewashes would certainly allow those arguments.

  10. Brandon:

    First, they’d say no ethics approval was needed because the raters were all project team members, thus not requiring ethical oversight

    I am certain they will not offer this as a defense, because it is not a legitimate argument, and they certainly know better.

    It is easy to for me to show this with US regulations (which I’m more familiar with), but the general concept holds internationally.

    From Code of Federal Regulations, 45 CFR 46.102(f):

    http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html#46.102

    (f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

    (1) Data through intervention or interaction with the individual, or
    (2) Identifiable private information.

    Since an interaction between Cook and his team members was present they qualify as human subjects. Note there is no exemption for being a “team member” from this definition. As we discussed previously, Cook should not have allowed identifiable private information to be shared with no-“team members”. That’s an ethics issue too.

    The only exception from IRB review in the US is for surveys where there is neither intervention nor interaction nor public sharing of private information.

    Anyway it’s not a survey, they are collecting psychometric measurements using the “team members” as subjects. That’s straight-up human subjects research.

    The most similar example I can think of is we did a helicopter quieting study once, where the subjects had dials that they could set from 1 to 4 for how loud the helicopter noise was. Of course this required human subjects approval. Because there was the potential for harm to the subject from helicopter noise exposure, that had to be addressed too. (We also paid our subjects, we recruited them, and there was travel involved, so those were additional issues that were considered.)

    Both studies involving using the results of psychometric measurements and both require IRB approval.

    Now we could go back and examine Australian law, but as far as I can tell, they are actually a lot tighter than they are in the US. It’s unlikely we’ll find it to be the case that a straight-up psychometrics-based study could be be exempted. As far as I know, there are no exemptions at all, when there are human participants in a research study. All studies, as I understand it, must go through institutional level ethics review.

    Do you know if they released a blank copy of the human subjects consent form?

    They should have, it’s normally considered to be part of the dossier for IRB applications.

  11. Carrick, I agree that argument is wrong. I just don’t agree it being wrong means it won’t be offered. I can think of plenty of times during the global warming debate where people and organizations have knowingly offered untrue arguments. Even if they weren’t willing to do it in a formal setting like some sort of hearing (which I wouldn’t assume), I can certainly see them doing it in informal settings like press releases. And I can certainly see someone like John Cook saying it.

    Do you know if they released a blank copy of the human subjects consent form?

    There was a Participation Consent Form released, but since this project was done online, it isn’t actually what was filled out by the participants. It describes the web page participants were shown.

    But of course, that’s just for the scientists contacted for self-ratings. As far as I know, no sort of consent form was signed by any of the 24 raters. They certainly didn’t sign one in relation to this ethics approval or anything involving the University of Queensland.

    Since there may be any number of questions about what was in the ethics approval application, I’ve gone ahead and uploaded a copy. You can find it here.

    By the way guys, I’m planning on uploading the leaked Skeptical Science forum to my web server so anyone can view the material. I hope to have it up later today.

  12. Carrick, it appears Cook and group carried out their project without ethics approval. That has nothing to do with Cook’s group members being human subjects. The study was not an examination of subjects’ opinions of global warming papers and Cook’s volunteers are not the subjects of the study.

  13. Shub, no that isn’t correct. In fact, in the US, surveys (studies of opinions) are the only exempted activity that involves human participants (i.e., doesn’t need IRB review). In Australia, even that activity requires IRB approval.

    If you have a research project involving rankings, regardless of its 1 to 7 for AGW or 1 to 4 for loudness of sound from a helicopter (the example I gave, that I was involved, in involving just ranked data), you will need IRB approval. Both involve interaction between researcher and the participants, and as soon as you have interaction, that’s a human research study that requires IRB approval.

  14. About a month ago, I wrote to the University of Queensland lawyer seeking copyright permission to use various images produced by UQ employee John Cook, the copyright of which, under University of Queensland policy, would appear to be held by the University. In particular, I requested copyright permission for the image of Cook in a Nazi uniform and of several skeptics in Spartan outfits. For greater certainty, I included copies of the images in my request. Thus far, I have not received permission – or even had my request acknowledged.

  15. Shub, the people who do the rankings are participants. It would help you if you knew what you were talking about here.

  16. Carrick, there are no human subjects in Cook’s project. There are humans involved but no subjects.

    Your entire description of what Cook et al studied is wrong.

    -Best

  17. shub, good grief.

    Humans ranked abstracts. That’s subjective data.

    Therefore there were human participants.

    Stop being an idiot.

  18. Carrick
    No, I am not hurt from the previous thread. I think a bit about outstanding issues in the climate debate and then I take a position. I had been thinking about temperature adjustments for a good while and I ran into Stockwell’s little paper. It helped crystallize things and I wrote down what I thought. This was more than a year back – in February 2013. I look back and see it holds up well.

    The concept of pairwise homogenization is circular. You can say whatever you want – it does not matter. No amount of mathematics will rescue a method from a flaw in its inferential chain.

    Read Zeke Hausfather’s paper, his Realclimate post and his Climate Etc post. He’s recycled the same thing including the very text. His position has not changed.

    So, the whole thing is just the usual suspects jogging along their well-trodden tracks. What an absolute shame. And Zeke, as someone whose thinking on this matter has not evolved or moved one bit, has gotten more preachy. Again, a sham. Mind you, this is not a personal opinion about him, only about his stance on this question.

    You write: “Since an interaction between Cook and his team members was present they qualify as human subjects”

    Who are human subjects according to you? Nuccitelli, Jokimakki, Skuce, Rob Painting etc? Are they whom you are referring to?

  19. “…”I don’t think anyone has really doubted the self-ratings were done under the assumption of confidentiality…”

    No but I don’t recall anyone discussing their confidentiality in the context of John Cook’s claims.

    For example, Rud Istvan posted an open letter to UQs admin at WUWT suggesting they had an ethical dilemma related to the SKS team raters’ ids. He was not referring to the self raters. And I do not recall anyone on that thread or related threads on the topic even considering that Cook might be referring to the self rating scientists. There were many, many comments on the topic and I could be wrong but that wasn’t the focus of the conversations. As you pointed out here It becomes clear from the ethics approval application that is exactly whom Cook was referring to.

    Tom Curtis’s comment provided an earlier clue about this. Indeed the post in which Cook released the self ratings includes, “In accordance with the confidentiality conditions stipulated beforehand, we had to anonymise the self-rating data in order to protect the privacy of the scientists who filled out our survey.”

    http://www.skepticalscience.com/Consensus-Project-self-rating-data-now-available.html

  20. Hey, sorry to spam your blog this morning but the ethics approval application is interesting and very funny.

    Q5. “[The Skeptical Science website] will be used because of its facility to encrypt private details and integrate entered data with the existing database of papers. Participants will be presented a secure, encrypted web form with their authored papers and a drop down allowing them to select the level of endorsement of their papers.”

    Here we have John cook using terms like secure and encrypted. That might be the funniest thing I’ve read since we learned that Cook stored his SQL transaction log in a public directory and that he allowed cookies to be manipulated to bypass his security.

    As to the manner in which Cook protected the ID’s of the self raters it seems to me that Cook’s answers are inconsistent with one another and with what actually occurred.

    Q2. “Data will be de-individualized so that individual ratings will not be published. User identities will not be stored on the same server as the ratings data.”

    Q5, “Data is saved on an online database that contains only a unique identifying number for each author (e.g. – no personal details such as name or email are kept in the online database)”

    Q6. “Ratings data is stored on the Skeptical Science website which is password protected. User identities will be stored in a separate password protected database on a different server, completely fire walled from the ratings database.”

    How can “no personal details such as name or email are kept in the online database” be consistent with “User identities will be stored in a separate password protected database on a different server”?

    And didn’t you find the ratings data stored on the super sekret website not the SKS website?

  21. DGH, three comments on two posts in a couple isn’t really spam. I’ve had people post four comments on one post in under 20 minutes. That’s when it becomes a problem.

    Anyway, you’re right nobody really discussed the confidentiality of the self-raters in regard to John Cook’s claims. We had little reason to. Only one of the files I found had any data related to the self-raters, and even then, it only had six data points that were relevant. The other 30+ files with tens of thousands of new data points is what everyone was interested in. I’m guessing John Cook realized that and took advantage of the conflation.

    As for what you quoted from the ethics approval, it’s important to remember all of that is in relation to the self-rating data. I found data on a totally different website, but that data was the Skeptical Science team’s ratings. It’s funny to hear Cook talk about security, but none of his answers seem wrong to me. As for the second server he refers to, what he says could easily be true if the data were stored on something like a home server.

  22. Shub, just ask yourself this question… how did they obtain the rankings of abstracts?

    If it involved subjective judgement by humans, those were “human participants” (the term used in Australia). In the US, we use the somewhat unfortunate moniker “human subjects” but it refers to “research where humans are used in any capacity” not just e.g., research of the effects of a drug on an individual.

    So even in the US “human subject research” includes even surveys, though it happens under certain circumstances that the survey is exempted from IRB approval.

    We aren’t studying the individuals who ranked data in the Cook study, but we are using their opinions as part of a further data analysis. That means they are participants and warrant IRB protection

    Since you are unwilling to produce any evidence to support your argument, I’m going to assume you are just trolling and ignore you until you use an argument besides “is not”.

  23. Regarding homogenization and “circularity”… you’re using the wrong word. It’s an “ill-posed problem” if you can’t measure the trend and the homogenization correction separately. And you can determine whether this is true by hand-waving. You have to use mathematical analysis to determine that.

    I mentioned on that thread how one can make an objective determination. If you want to satisfy for yourself whether Stockwell is correct, just create a dummy problem and see if you can separate the trend from the homogenization corrections.

    Of course this has already been done.

    As I pointed out on that other thread, Stockwell might have somewhat of a point for a single station, but generally people are using multiple stations to look for station moves, instrument changes, etc. What people find who do Monte Carlo studies find is that generally the methods work.

    I think it works because multiple stations are used in the algorithms and because there isn’t a lot of overlap between a step-function and a trend. When you make a step-wise correction to a series it is true this affects the trend, but what you are looking for are jumps in the record when compared to near by stations.

    If you want to counter this argument you do need something besides hand-waving to be taken seriously. For example, present Monte Carlo simulation of the homogenization process which introduces a net bias in the trend estimate. (By “net bias” I mean bias when you average over stations. It’s okay for the homogenization correction to produce random error, as large as the error isn’t too large and has a zero mean.)

  24. Carrick, I wrote: “… there are no human subjects in Cook’s project.”

    You write: “We aren’t studying the individuals who ranked data in the Cook study …”

    Furthermore, you write: “If it involved subjective judgement(sic) by humans, those were “human participants” …”

    By this definition, there would be no research without an institutional ethics committee because no research can be conducted without humans.

  25. Shub:

    Furthermore, you write: “If it involved subjective judgement(sic) by humans, those were “human participants” …”

    By this definition, there would be no research without an institutional ethics committee because no research can be conducted without humans.

    To start with “judgement” and “judgment” are both correct spellings of the word. Look it up.

    Secondly, that definition very much does not imply that “there would be no research without an institutional ethics committee”.

    If it’s subjective, that means it’s internal to the person. If it’s objective, it exists outside of them.

    Subjective data definitionally involves a human reseaech participant. In a few cases, research involving subjective data are exempted from IRB oversight. In the US, one of those is the opinions of properly anonymized individuals where there no interaction with or intervention by the researcher.

    Objective data, unless it involves measurements of a human subject, not not involve a human research participant and therefore does not require IRB supervision.

  26. “Subjective data definitionally involves a human reseaech participant.”

    The data generated by Cook et al was not the subjective opinions of volunteers, i.e., it was not a survey of the opinions of Nuccitell, Jokkimakki, Skuce etc. It was not a survey of individuals. It is not billed as a survey of individuals.

    The authors have claimed several times if anyone else picked up the criteria list and applied to abstract text their results would be equally valid.

  27. I think you need a course in measurement theory. We are too far apart in training for continuing this to be useful.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s