A Strange Omission by Cook et al

This post is to document an oddity I never noticed before. It’s not particularly important, but if I were more cynical, I might believe it showed misconduct on the part of Cook et al.

People who’ve followed my discussion of Cook et al will remember the data files for the paper included data for only 11,944 abstracts even though 12,465 abstracts were rated. They’ll also remember I’ve said some of the data for the missing 521 abstracts is inexplicably available via the online search engine provided at Skeptical Science. Somehow, I missed an interesting aspect to this. That search engine’s page says:

Search through the title and abstract of the 12,464 papers analysed in Quantifying the Consensus on Anthropogenic Global Warming in the Scientific Literature.

Notice the number. It says 12,464 papers, not 12,465. That suggests one paper was not included in their searchable database. That is strange. Why would data from only one paper be excluded from their searchable database? What was that one paper? We have no way to know. It’s never been listed anywhere. At least, not officially. I think I’ve found it in an old, unofficial data file.

About a year ago, before the Cook et al paper was released, John Cook hosted a survey where readers could rate abstracts, ten at a time. His description of the survey was deceptive. That’s not important though. What is important is the code for the survey was wonky. It was possible for a glitch to happen which would cause the server to dump the entire table of abstracts. The resulting page would show 12,465 abstracts.

Finding one extra abstract in 12,465 could be difficult. I think I’ve done it though. The survey protocol said raters would not be allowed to see anything but the title and abstract of each paper. However, one entry said this after the title:

Article from: E&MJ – Engineering & Mining Journal | February 1, 1999 | Doerell, Peter E. | Copyright

The title of that paper was Anti-global warming measures would hit mining hard. I was able to find a link to it online (behind a paywall), but I couldn’t find it with the Skeptical Science search engine. I also couldn’t find the corresponding ID number in any data file (meaning it is, at least, one of the 521 excluded abstracts).

I get the impression somehow that entry got mangled, and for some unspecified reason it got filtered out. That’s problematic. It suggests Cook et al screwed up in yet another way, this time in a way which led to the arbitrary exclusion of a piece of data.

A cynical person might view it as worse than that though. According to Cook et al, only 78 of ~12,000 papers “Reject AGW.” What then, are the odds the only paper excluded from the searchable database would have an abstract that begins:

For more than five years the world has been bombarded with gloomy prophecies that “within the next hundred years” there will be a “climatic disaster.” The culprit is said to be the increasing C[O.sub.2] emissions caused by the burning of fossil fuels.

However, not a single word of these allegations is true and every detail of this scare-mongering is distorted. It begins with the emotional, fear-provoking term “greenhouse effect.” A greenhouse is an enclosed space in which the temperature can be regulated as desired. However, our atmosphere is an open space without a roof or side walls. It has free access to icy polar winds and hot winds from the desert, and man has no influence on it at all.

I don’t know. What I do know is this shows how stupid it is Cook et al were allowed to filter out data in ways that are completely unverifiable because they hide the excluded data.


On a side note, I’d like to point out University of Queensland acting Pro-Vice-Chancellor (Research and International) Professor Alastair McEwan insists:

All data relating to the “Quantifying the Consensus on Anthropogenic Global Warming in the Scientific Literature” paper that are of any scientific value were published on the website Skepticalscience.com in 2013.

Advertisements

37 comments

  1. Cook & gang are many things. Serious scientists and careful academics, as well as high quality internet/software engineers do not appear to be among those many things.

  2. Brandon, a cynical person might wonder why you describe the introductory paragraph of an article as an abstract (the two are not the same). They might especially wonder in light of the fact that it is not possible to rate the abstract of an article without an abstract. They might further wonder why you did not point out that the Engineering and Mining Journal is a trade journal, not a scientific journal and that hence the article in question was not peer reviewed. Given that Cook et al excluded all papers from their search that either had no abstract, or were not peer reviewed (as stated in the paper), they might wonder why you are making a song and dance about this particular paper while not revealing those relevant facts.

  3. Tom Curtis, that cynical person would have to be a fool. I said that paper was one of the 521 paper filtered from the data set. Anyone familiar with this topic would know that means it was filtered out for either not being peer reviewed, not having an abstract or not being climate-related. That means they’d know it was not one of the 11,944 abstracts included in the data files for the paper.

    However, anyone familiar with this topic would also know abstracts filtered from the data files were included in the Skeptical Science searchable database. After having read this post, they’d know 520 extra papers were listed in the searchable database, instead of the 521 papers missing from the data files. That means they would know there are three different lists of papers. One has 11,944, another has 12,464, and another has 12,465. As any informed reader would also know, the amount of data available for papers on each list is different.

    The only information you claim I didn’t provide that was actually missing to any informed reader* is what I listed as an abstract is really “the introductory paragraph of an article.” Leaving aside the fact there are two paragraphs, not one, your group listed it as an abstract. I don’t think it’s appropriate to fault me for calling something an abstract when you guys said it was an abstract.

    Here’s an idea Tom Curtis. Instead of making comments which consist of nothing but snide, backhanded remarks that fail to address anything anyone has ever said, how about you try having a real discussion with me. I’ve raised plenty of issues about this paper. Why don’t you pick one and discuss it?

    To get the ball rolling, I’ll suggest a topic. You claimed I stole data from a web server. I’ve explained how I came by that data. How would you reconcile my explanation with your description? What did I do that qualifies as stealing?

    If you don’t like that topic, you’re welcome to choose a different one.

    *I’ve provided that information in multiple posts prior to this one. I can’t be expected to revisit every issue in every post.

  4. lucia, here. Technically, he didn’t say I stole the data; he said the data is stolen. I don’t think the difference is relevant though. I was the only person involved, so if the data was stolen, I had to have stolen it (unless you believe I lied about doing it on my own).

  5. Brandon,
    A bit more finesse and you could play a dullard like Tom Curtis like a banjo. Try a lighter touch.

  6. hunter, I don’t feel like playing games in conversations. If Tom Curtis wants to participate in anything resembling a discussion, he can. If not, he can choose pettiness and hostility over actual discourse. It’s his call.

    I have other things I’d rather spend my efforts on. For example, I just got a web rating system set up that lets people register an account, log in and rate abstracts. That much more interesting to me, and there’s still a ton of work to be done on it.

    For example, I have to figure out why my code isn’t letting people log out.

  7. Brandon, the second sentence of your post reads:

    “It’s not particularly important, but if I were more cynical, I might believe it showed misconduct on the part of Cook et al.”

    That is an odd sentence. Of logical necessity, you are not more cynical than you are. Therefore the antecedent is false and consequent irrelevant. The conditional conveys exactly zero information. Indeed, tautologically, it conveys zero information. But it sends a message, not by statement but by providing a suggestion as to how people should react. It is a signal marking the “evidence” you provide as indicating “misconduct” by Cook et al, but indirectly. It shows what you wanted people to understand, but lacked the integrity to say directly.

    Given that, your concern over whether I choose ” choose pettiness and hostility over actual discourse” is entirely hypocritical, in that your blog post has chosen pettiness and hostility over discourse with respect to Cook et al.

    However, in the interests of discussion:

    1) Cook et al never call the introduction to “Anti-“global warming” measures would hit mining hard” (Doerell, 1999) an “abstract”. In the paper they say:

    “The ISI search generated 12 465 papers. Eliminating papers that were not peer-reviewed (186), not climate-related (288) or without an abstract (47) reduced the analysis to 11 944 papers written by 29 083 authors and published in 1980 journals.”

    They only refer to the abstracts of the 11944 papers actually rated. In the publicly released list of the papers rated, they refer to the tcp_articles. The only public release that can be construed as referring to Doerell (1999) is the searchable database, which does not include it.

    2) You claim that Doerell (1999) was unusual in not being included in the searchable database, ie, that it was the one (and only) paper not included from the 12,465 found by the search. In fact, the searchable database includes only 12,280 “abstracts” as can be determined by doing a search for a space, comma or the letter “a”. It can also be checked against Carrick’s data scrape. The supposed uniqueness of Doerell (1999) on which your snide comments are premised is an illusion based on careless research. It is clear from the number of “abstracts” included that the 12,464 on which you place so much emphasis is merely a misprint. It is also clear that the number is misleading, though unintentionally so, and not on a matter of substance.

    Expanding on those last two points, the searchable database contains 283 papers that are “not climate related”,39 that are “not peer reviewed”, and 7 with “no abstract”. leaving an additional 7 “abstracts” that are not accounted for. According to the paper there were 186 “not peer reviewed” (147 more), 288 “not climate related” (5 more), and 47 with “no abstract) (40 more). From among the 12,280 “abstracts”, 3933 were rated as endorsing (ie, 37 more than reported in the paper), and 8269 were rated as neutral (ie, 261 more than in the paper). The number disendorsing is the same in both the searchable database and the paper. The ratio of endorsing to neutral papers was 0.476 among the 12,280 searchable “abstracts”, but 0.487 in the paper. The slightly lower ratio is unsurprising given that “not climate related” papers, and papers with “no abstract” will be disproportionately rated neutral.

    From these facts it can be seen that excluding the additional papers weakened the Cook et al result. That is, its effect was in the reverse direction to that expected from bias, and from your snide comments above. That may or may not also apply to the additional papers pre-screened. The greater proportional removal of papers prior to inclusion in the 12280 for “not peer reviewed” and “no abstract” categories suggests an initial automatic pre-screening based on comparing journals to members of the ISI list of peer reviewed journals, and checking the abstract field for contents. That in turn suggests the 12280 papers from the searchable database were the actual papers assessed by raters. Hence the reason I do not consider the “12464” misleading in any signficant way. Parenthetically, it also indicates that “abstract” in the description of the searchable database is just shorthand for “abstract and/or title”, and cannot be interpreted as a claim that any particular paper.

    3) You claim that:

    “I said that paper was one of the 521 paper filtered from the data set. Anyone familiar with this topic would know that means it was filtered out for either not being peer reviewed, not having an abstract or not being climate-related.”

    However, you were very careful to suggest a fourth reason (misconduct) for exclusion, and made not attempt to mention the two clear criteria under which it was appropriately excluded. Further, you fudged the issue on one of those criteria by describing an introduction as an “abstract”. Given that, your pleading of what “everybody knows” is pure misdirection.

  8. Brandon, you ask:

    “You claimed I stole data from a web server. I’ve explained how I came by that data. How would you reconcile my explanation with your description? What did I do that qualifies as stealing?”

    A few years ago, my sister came around to visit me. We talked downstairs for a while, and she put her lap top down. We then went upstairs to the lounge for some coffee, leaving the front door open. A stranger entered the house, took the computer and drove of. They stole the computer.

    Had the case come to court, and their lawyer stood up and pleaded not guilty on the grounds that the door was left wide open, he would have been laughed out of court. The lack of security had no bearing on whether or not the computer was illegally and immorally acquired (stolen). The only relevant facts was did the owner of the computer give permission for the stranger to acquire the computer? She did not, so taking the computer was theft.

    You plead that you did not steal the data because the security protecting the data was lax, indeed virtually non-existent. That is irrelevant in the same way as a similar plea by the computer thief would have been irrelevant. The only questions are:
    1) Did the owner of the data (John Cook) give you specific permission to obtain the data? or
    2) Did the owner of the data give you implicit permission to obtain the data by posting a public link to the data?

    He did neither. Ergo, you took data he owned without his permission. You stole it. The case could not be more clear cut.

    I will allow that you may not have thought it through clearly, and may genuinely think you have not stolen the data. In that case, you need to immediately correct your error by deleting the data from your devices or accessible memory, removing any discussion of the data from sites you control, informing Cook of anybody you may have passed the data on to, and emailing those people requesting that they also delete the data etc.

    Of course, I strongly suspect that you will not do that. (I think there is a snowballs chance in hell of your doing that.) I think you do not really care about the ethics of your act, and will simply use casuistry to cloak the fact that your only true concern is the efficacy of the act in your apparent vendetta against Cook.

  9. Tom Curtis, you seem to be willfully misinterpreting my second sentence. You call it odd because the logic of it is supposedly shaky (it isn’t, as one can discuss hypotheticals), then you discuss its rhetorical effect. I don’t know why shaky logic in a sentence written with a clear rhetorical aim would make you think the sentence is odd.

    More confusingly, you act as though the sentence’s rhetorical effect applies to everything in the post when any high school student could see it doesn’t. The sentence was an obvious precursor to this sentence near the end of the piece:

    A cynical person might view it as worse than that though.

    Which discussed one specific point. Any high school student would recognize the technique. They’d realize I referred to a point in my introductory paragraph to inform people it would be discussed later on, and later on, I discussed it. It’s the same thing they are taught to do when they learn how to write a thesis statement. I have no idea how anyone could fail to miss the obvious structure of the post.

    In any event, you offer three points in response to me. I’ll respond to each. Your 1) says Cook et al never referred to what I provided as an abstract. You justify this by quoting the paper. You can only do this by pretending that’s the only time the material was listed. It wasn’t. The material was listed in a database in an a column for abstracts. The Skeptical Science page I discussed referred to “the title and abstract of the 12,464 papers.” The page I got the material from specifically said, “Below are listed the titles and abstracts.” You pretend only one quote matters by ignoring things like these.

    I have to thank you for your 2) though. You just informed everybody data for 185 papers was excluded from your searchable database, 184 more than the one I said was excluded. You also informed them the description of the searchable database was misleading. It’s good to know my “careless research” has led to you pointing out the data I said was missing from the database is only .5% of the data that is actually missing from that database. I’m sure it’ll mean to people like Alastair McEwan coming from a member of the project than it would coming from me.

    As for the numbers you presented, I think it’s kind of funny you focus on the 336 papers included in the searchable database, not the 521 excluded from the data files. You make the point none of the 336 papers you refer to were rated as rejecting the “consensus.” We know at least one paper which rejects it was excluded even from those 336 papers. People could reasonably wonder if any of the other 184 rejected it.

    As for your idea “the 12280 papers from the searchable database were the actual papers assessed by raters,” I think that’s great. You say that is the reason you “do not consider the ‘12464’ misleading in any signficant way,” but if it is true, that means you guys used a pre-screening technique which has never been disclosed before. You’re saying the value on that page isn’t meaningfully misleading because you guys did something you never told anyone about!

    For your 3), you claim I “made not[sic] attempt to mention the two clear criteria under which it was appropriately excluded.” I find this argument amusing. Your group made it impossible to know just why the paper was excluded from your data files by not disclosing that information. You then made it impossible to know why the paper was excluded from your online database by not disclosing the data’s existence. It’s amusing to think I’m the bad guy because I didn’t discuss the reasons you guys hid from everybody.

    You guys didn’t release data, and I’m bad for not discussing what that data would have shown.

  10. As for your claim of stealing Tom Curtis, I have to say your analogy is terrible. It involves depriving people of their property. There’s no reason for that. I didn’t deprive anyone of anything. If we fixed that aspect of your analogy, your argument is greatly weakened:

    A few years ago, my sister came around to visit me. We talked downstairs for a while, and she put some files down. We then went upstairs to the lounge for some coffee, leaving the front door open. A stranger entered the house, took pictures of the files drove of.

    However, your argument crumbles if we fix another faulty aspect of your analogy. A home is a private residence. A public facing web server is not. Your analogy would be more appropriate if it took place in a public business:

    A few years ago, my sister came around to visit me at my store. We talked at the counter for a while, and she put some files down on it. We then went upstairs to the lounge for some coffee, leaving the front door to the store open. A stranger entered the store, took pictures of the files drove of.

    There isn’t a court in the world which would find that person’s behavior to be criminal. That shows the entirety of your argument rests upon misrepresenting the situation. When we move away from the analogy:

    You plead that you did not steal the data because the security protecting the data was lax, indeed virtually non-existent. That is irrelevant in the same way as a similar plea by the computer thief would have been irrelevant. The only questions are:
    1) Did the owner of the data (John Cook) give you specific permission to obtain the data? or
    2) Did the owner of the data give you implicit permission to obtain the data by posting a public link to the data?

    He did neither. Ergo, you took data he owned without his permission. You stole it. The case could not be more clear cut.

    I think this argument is interesting. I’m not going to discuss what I see as its flaws right now. Before doing that, I’d like to ask for clarification. When a user visits a page, they are given a copy of that page. There is effectively no difference between copying a page and simply looking at it.

    As such, it is my understanding you claim merely visiting a page where the answer to your two questions is no qualifies as stealing. Is that correct?

    In the meantime, if John Cook deletes all of his copies of the stolen Heartland documents, deletes all discussion of them from his website and publicly apologizes for having ever disseminated them, I’ll do as you suggest. When Cook shows he holds to the standards your promote, I’ll hold to them regarding his data. Until then, I’m going to assume he either doesn’t believe as you believe, or he’s a hypocrite. Either way, I don’t feel obliged to hold to your standard.

  11. Brandon:

    1) I point out the logic of your second sentence to focus on the fact that you clearly suggest wrong doing by Cook without having the integrity to come out and say he actually did something wrong. You are dog whistling in a contemptible way.That applies as much to your final paragraph, which leads of with a mere statement of possibility (“a cynical person might”) and expands on that with a rhetorical question. Consequently, I felt no need to expand on that fact with regard to the final paragraph when I could show the fact with your second sentence.

    Further, your dog whistling is not restricted to the second sentence and the last paragraph. You have a series of rhetorical questions in the third paragraph whose obvious intent is to lead up to the later dog whistling.

    Absent the dog whistling, all the information you have provided and have available to you is that some of the screened papers were not listed in the searchable data base, and that you have found one of those papers which obviously should have been filtered out of the data on two of the three grounds for filtering out papers (cleaning the data in Tol’s terms) specified on the paper. You have twisted those simple facts into an indictment of Cook by pure rhetoric.

    2)

    “I think it’s kind of funny you focus on the 336 papers included in the searchable database, not the 521 excluded from the data files”

    On the contrary. I focused on the data I had available, making the further point that “That may or may not also apply to the additional papers pre-screened”, where “that” refers to the claim that:

    “… it can be seen that excluding the additional papers weakened the Cook et al result”

    Further, there were only 185 papers, not 521, excluded from publicly released data files. You are clearly overstating the number of papers I purportedly ignored, even leaving aside that I did not ignore them. Rather, you ignored my discussion of them.

    3)

    “that means you guys used a pre-screening technique which has never been disclosed before.

    The paper states the following about the pre-screening technique:

    “Eliminating papers that were not peer-reviewed (186), not climate-related (288) or without an abstract (47) reduced the analysis to 11 944 papers written by 29 083 authors and published in 1980 journals.”

    There is, therefore, no pretense that the pre-screening technique was fully described. It was simply not described as a concession to the space limits in scientific papers. In that situation, if you are interested in the technique used, you merely contact the lead author for clarification. The issue has been considered of so little importance that apparently nobody has , and certainly you haven’t, even though it has been known for over a year that the searchable data base contains 185 papers less than those produced by the search.

    Further, the papers are not hidden. Anybody interested and with a limited level of programming ability could repeat the search used, and filter for papers containing empty abstract fields, etc to find the relevant articles. You could also compare search results to the papers listed in the searchable database. Just because the data is not given to you in a convenient package does not mean the data was hidden.

    4)

    “Your group made it impossible to know just why the paper was excluded from your data files by not disclosing that information.”

    First, “my group” made it perfectly easy for you to know why the paper was excluded. They did so by stating explicitly the only reasons for excluding data, ie, that it did not have an abstract, that it was not peer reviewed, or that it was not climate related. Two simple checks show that two of the three reasons for exclusion apply. When two of the three stated reasons apply, fishing for another unstated reason is mere mud slinging.

    Second, Cook et al did not even make impossible to know that the paper was excluded. It has always been possible for you to repeat their search and compare the results (excluding any articles published after the final date of their search). Such an exercise would have turned up all, or nearly all (given that databases change their contents) of the 185 papers you are interested in.

    Your laziness is not the same as Cook’s culpability. Neither is your insistence that they hand over, uncompensated, the fruits of their labours; or as is now shown, your willingness to take the fruit of their labour without their permission.

    “You guys didn’t release data, and I’m bad for not discussing what that data would have shown.”

    And that is a clear distortion of my argument. My claim was that the reasons for excluding papers was clearly stated in the paper, and the excluded paper that you turned up would have been excluded under two of the stated criteria, and clearly so. Not revealing that is what makes your original post deceptive.

  12. Regarding theft:

    1) The words “theft”, “steal”, etc are commonly used of obtaining data without consent even when the original copies of the data are left intact. This can be verified easily by googling “data” & “steal”.

    2) Two make your shop analogy accurate you would need to not only enter the store, but go into the back section through the door marked “staff only”. There was no public invitation to the particular page(s) from which you obtained the data. Therefore it was not a public space.

    3) John Cook did not, and would not in so far as I know him, attempt to obtain information from the Heritage Foundation by any means other than following public links to their website, or otherwise accessing information the Heritage Foundation has publicly invited people to access. If you want an analogy to the Heritage Foundation incident, you are in the position Peter Gleick, As Cook has not even hosted the stolen documents, he is at worst in the position of somebody directing people to a fence. So, if you want to feel morally justified on no more basis than tu quoque, by all means continue to keep a copy of the sks forum hack. But don’t pretend that it justifies you yourself obtaining and distributing data without the owners consent.

  13. In the interest of time Tom Curtis, I’m going to let a few points drop. Please don’t assume that means I accept your unanswered claims.

    Further, there were only 185 papers, not 521, excluded from publicly released data files. You are clearly overstating the number of papers I purportedly ignored, even leaving aside that I did not ignore them.

    You claim I am “clearly overstating the number of papers [you] puportedly ignored.” Leaving aside the fact I didn’t claim you ignored those papers, you are wrong on an obvious, factual point. Data for only 11,944 papers was released in the data files (such as here).

    You claim I am “clearly overstating” a value, based solely upon you fabricating a claim about the data files released for this paper. Anyone who spent any time with those files would know your claim is false, as would anyone who bothered to verify what I’ve said in this post. In other words, you’ve just denied the discrepancy between the data files for this paper and the searchable database by simply making things up.

    There is, therefore, no pretense that the pre-screening technique was fully described. It was simply not described as a concession to the space limits in scientific papers.

    You can only argue this by cherry-picking quotes. Prior to the paragraph you quote, the paper said:

    In March 2012, we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for ‘global warming’ or ‘global climate change’. Article type was restricted to ‘article’, excluding books, discussions, proceedings papers and other document types. The search was updated in May 2012 with papers added to the Web of Science up to that date.

    We classified each abstract according to the type of research (category) and degree of endorsement.

    There was no mention of any pre-screening here. There’s a description of how the data was retrieved, and a description of what was done with it. By not listing anything in between the two steps, it is clearly implied nothing was done between the two steps. This implication is evident in the fact not a single non-participant has ever known some sort of pre-screening was used.

    Which isn’t to say I believe such pre-screening happened. I’ve seen no evidence for your notion, and I’ve seen evidence against it. I’d wager there was none.

    Further, the papers are not hidden. Anybody interested and with a limited level of programming ability could repeat the search used, and filter for papers containing empty abstract fields, etc to find the relevant articles. You could also compare search results to the papers listed in the searchable database. Just because the data is not given to you in a convenient package does not mean the data was hidden.

    This is another claim you’ve made up. While people can filter a search to cover the same period as that used by Cook et al, the data retrieved will be different than that used by Cook et al. This is because the delay between a paper being published and being added to the Web of Science is variable. Some papers within months of being published, others a year after publication.

    First, “my group” made it perfectly easy for you to know why the paper was excluded.

    No, it didn’t. The fact I know some potential reasons a paper may have been filtered out in no way means I know why it was filtered out. For all I know, there was some mythical pre-screening which filtered out papers whose abstracts were exactly 1,943 characters.

    Heck, someone could have gone through and removed 185 papers simply because they would have been rated as rejecting the consensus. It’s impossible to tell Without knowing which papers were removed (and preferably, why).

    Your laziness is not the same as Cook’s culpability. Neither is your insistence that they hand over, uncompensated, the fruits of their labours; or as is now shown, your willingness to take the fruit of their labour without their permission.

    Claiming I’m demanding people “hand over, uncompensated, the fruits of their labours” is ridiculous. The only data I have ever called to be released is that which was said to have been released. There is nothing unreasonable about that. Your portrayal is grossly misleading.

  14. On the issue of theft, I’ll note you didn’t answer my simple request for clarification. As it stands, I cannot tell what your standard is. Rather than making irrelevant remarks (the fact “theft” and “steal” can be used somewhat interchangeably does not change the fact your metaphor was inappropriate), would you please clarify where you draw the line for “theft”?

    Specifically, at what step in the process of accessing this data did my actions become unacceptable to you?

  15. 1) If you go the Consensus Project searchable database and do a search, you will find results like the following (being the :

    “Results 1 to 25 out of 12280:

    A 20-year Record Of Alpine Grasshopper Abundance, With Interpretations For Climate Change

    Authors: White, Eg; Sedcole, Jr (1991)
    Journal: New Zealand Journal Of Ecology
    Category: Impacts
    Endorsement Level: 4. No Position”

    The first line gives the number of results displayed on the page, along with the total number recovered. The rest of the quoted portion shows the first result displayed. The data then, gives the title, authors, year of publication, journal, category (as rated for the paper) and endorsement level (as rated for the paper). The database can be searched by year, category, and endorsement level to obtain subtotals if desired. You can also obtain the listed data for all 12280 papers if you desire. Further, this is the database to which you are linked to by IOP when you look for supporting data for Cook et al.

    If follows that it is a matter of record that Cook et al have released the data for 12280 papers in a publicly searchable database, and have done so for more than a year. As 12465 papers were found in the initial plus supplemental searchs, that leaves just 185 papers for which no data has been released, beyond the summary data of numbers of papers excluded due to not being peer reviewed, having an abstract, or being climate related.

    2) If indeed no non-participant has figured out there was a prescreening, how dull you all must be. At least, how dull must be those of you who have spent so much time trying to find any flaw imaginable with Cook et al, right down to carrying out mathematically impossible analyses (Tol). I suspect, however, that many of you have figured it out, in simply considered it inconsequential.

    For your benefit, however, it is clear from the statement on results that in addition to research categories 4 (not climate related) and 5 (not peer reviewed), there was an unspecified category of (no abstract), and that members of these three categories were excluded from the analysis. It is then perfectly clear that such a screening took place, but the method of the screening was not specified in the paper. Given the sheer number of papers involved, that alone might lead you to speculate that an algorithmic pre-screening was used to cut down the load of papers needing rating by hand. Even without that the discovery of the missing 185 papers from the searchable database should instantly suggest that possibility, resulting in an inquiry to John Cook if you thought it important. Simple arithmetic then shows that the paper reports 192 more papers in the three excluded categories than found in the database, showing that the 185 missing papers were excluded on those grounds, and that the 7 paper discrepancy already mentioned in a previous comment were also all excluded on those grounds, which leads also to an inference of pre-screening.

    Of course, if you were so lax as to not twig how to find the number of papers in that database when you first came across it, and missed the public discussion of the number in the database by Carrick at Lucia’s and me at SkS – well perhaps you wouldn’t twig.

    3)

    “This is another claim you’ve made up. While people can filter a search to cover the same period as that used by Cook et al, the data retrieved will be different than that used by Cook et al. This is because the delay between a paper being published and being added to the Web of Science is variable. Some papers within months of being published, others a year after publication.

    I believe I have already mentioned that databases change overtime so that the search results will not be an exact match. In the case of the Web of Science change over time for a given time period will be small so that you will net nearly all of the missing papers. You will certainly net enough to show whether or not papers are frequently excluded for other than the stated grounds.

    4)

    “The fact I know some potential reasons a paper may have been filtered out in no way means I know why it was filtered out. “

    The fact is you have prima facie evidence as to the reason for exclusion and no grounds for doubting that reason. Your only grounds for doubting Cook et als claim is that you want to doubt their claim. Absent actual grounds to doubt that reason, you are indulging in simple muck raking. All your pleading has shown is that you have checked one case and found it would have been excluded on those grounds – therefore tending to confirm Cook et al’s claim – and that you are too lazy to take further measures to check those claims, even though such measures are reasonably easy to take. Instead of taking those measures, however, you simply resort to muck raking.

  16. Tom Curtis, before continuing any further, I must take a stand on a moderation issue. As a rule, people on this site are not allowed to fail to address disputes regarding their factual claims. If a person challenges a factual claim you make, you must address it. As such, I must insist you address the fact this claim of yours is false:

    Further, there were only 185 papers, not 521, excluded from publicly released data files. You are clearly overstating the number of papers

    I provided you this link to one of the data files in question. That data file has entries for 11,944 papers, 521 fewer than the total 12,465 papers collected by your group.

    The file I linked to above is a data file officially hosted by the journal which published the paper. It is indisputable data for 521 papers was excluded from it. It is also indisputable data for the same papers was excluded from this listing of the papers at Skeptical Science, as well as this listing of ratings performed by your group.

    You claimed I was “clearly overstating the number of papers” excluded from these data files. That your claim is false is indisputable. As such, I must insist you either acknowledge your statement was false or provide evidence those data files contain data for more than 11,944 papers.

    No further conversation can happen until you address this moderation issue.

  17. Brando,:
    1) Your original link was broken.
    2) It is irrelevant if one publicly accessible data base is restricted to only those files included in the final ratings for the paper if another publicly accessible database released by Cook et al includes the larger quantity. That the released the former in no way indicates that they did not release the later. In this case, the larger database is linked to from exactly the same page as that which contains the link to the file you refer to. Further, the larger database was the first released in chronological order.
    3) I directly addressed the factual issue by directly linking to the larger database, explaining its structure and linking to the IOP page from which it was linked. That you choose to ignore these facts shows conclusively that there is no moderation issue and that you are only pretending to raise one as a spurious means to avoid admitting your rather glaring error.

  18. Tom Curtis, your response is not responsive. You specifically said:

    Further, there were only 185 papers, not 521, excluded from publicly released data files. You are clearly overstating the number of papers

    Not only did this comment specifically refer to the “publicly released data files,” it was made in response to me discussing those data files. I described the number of papers excluded from those files with complete accuracy. You made a bold, and completely baseless, claim that I was “clearly overstating the number of papers” excluded from those files. Your response does nothing to address that issue.

    Moreover, and this is not an encouragement to discuss a separate issue prior to resolution of the moderation issue, your response is deceptive. While the database you refer to is obviously not a “publicly released data file,” a more important point is it does not contain as much information as the data files. Those data files include categories of information not available via the searchable database. That means even if one could accept calling a searchable database, which is temporary and non-local to users, a data file, it still would not address the fact data for many papers was excluded from what was made available.

    At this point, it will be abundantly obvious to any reader you accused me of “clearly overstating” a value by misrepresenting the available data, and you are now avoiding addressing your mistake. You can call me a liar all you wish. Nobody will be fooled.

  19. Brandon, one last time:

    There are three data files released listing the papers from Cook et al. There is the file to which you link, and hosted by IOP that lists the 11,944 actually rated in Cook et al. There is the file hosted at SkS and labelled tcp_articles, and there is the file from the searchable database. A paper is not missing from the files (plural) unless it is missing from all three. Consequently, 521 papers from the search are not found in the IOP file. That does not mean they are “missing”, a tendentious description that presumes the file was intended to include them. More than 521 papers are absent from the tcp_articles files, which appears to be corrupted, and in any event does not even include the full 11,944 papers rated in Cook et al., But only 185 are absent from the searchable database, and hence only 185 are absent from the datafiles collectively.

    I will concede that a misunderstanding arose between us if you intended by data files just the IOP file and the tcp_articles file. In that case, however, your original claim that I was ignoring the 521 papers is bizarre given that I explicitly discusses the majority of those papers (the 336 included in the searchable data base) and went on to discuss the remaining 185 in more general terms.

    Given, however, that I have very clearly been including the searchable data base among the data files which I have discussed, that you have not noticed that and persisted in your “moderation” shows it is clearly intended to shut down a discussion in which you have been shown to be ignorant, and vindictive. You have shown, in other words, all the personal qualities I have come to expect from you.

  20. Tom Curtis, it’s no surprise that would be your last comment. Your fabricated claims are getting more and more outlandish:

    More than 521 papers are absent from the tcp_articles files, which appears to be corrupted, and in any event does not even include the full 11,944 papers rated in Cook et al.

    This file has 11,944 papers listed in it. You’ve simply made up a problem with the file with no basis or apparent reason.

    Given, however, that I have very clearly been including the searchable data base among the data files which I have discussed

    A searchable database, by definition, is not a data file. I’ve pointed this simple fact out. You ignored it.

    I will concede that a misunderstanding arose between us if you intended by data files just the IOP file and the tcp_articles file. In that case, however, your original claim that I was ignoring the 521 papers is bizarre given that I explicitly discusses the majority of those papers (the 336 included in the searchable data base) and went on to discuss the remaining 185 in more general terms.

    The only “misunderstanding” here is you used the word “data file” in a way it cannot possibly be used. As for what you claim was “bizarre,” that’s yet another fabrication. I never claimed you ignored the 521 papers. I said “you focus on the 336 papers.” Focusing on one thing does not require ignoring anything else.

    Put simply, you’ve fabricated an argument and attributed it to me. You based this argument upon your use a phrase with a clear meaning (data files) in a way clearly contrary to the phrase’s meaning. Finally, you randomly made things up about the data file even though anyone can see you’re wrong about it.

    As though all that’s not bad enough, you then criticize me as a person based upon a nearly impossible interpretation:

    I have very clearly been including the searchable data base among the data files which I have discussed, that you have not noticed that and persisted in your “moderation” shows it is clearly intended to shut down a discussion in which you have been shown to be ignorant, and vindictive. You have shown, in other words, all the personal qualities I have come to expect from you.

    You say I “have not noticed that” you have “been including the searchable data base among the data files which I have discussed.” The opposite is true:

    While the database you refer to is obviously not a “publicly released data file”… That means even if one could accept calling a searchable database, which is temporary and non-local to users a data file…, it still would not address the fact data for many papers was excluded from what was made available.

    Not only did I refer to you calling the database a data file, I highlighted the inappropriateness of it.

    Beyond which, your own words contradict your claim, as you say “there is the file from the searchable database.” That shows you don’t think the searchable database database is a file in and of itself. Given there is no data file output by the database, that proves your interpretation has no basis.

    You can insult me as much as you want. You’ve said multiple, obviously untrue things (such as a data file being incomplete when it wasn’t) and insulted people for not figuring out an undisclosed methodology was used by Cook et al, when there is absolutely no evidence that methodology was actually used:

    If indeed no non-participant has figured out there was a prescreening, how dull you all must be.

    I don’t think anyone but the most close-minded people will agree with your insults, especially not if I take the time to go find a half dozen examples from past exchanges where you admitted to having made things up about me. You have a history of making things up to insult me. All you’re doing here is continuing it.

  21. “I don’t think anyone but the most close-minded people will agree with your insults, especially not if I take the time to go find a half dozen examples from past exchanges where you admitted to having made things up about me. “

    That is a straight forward lie. I have never made up anything about you, and consequently have never admitted making things up. I on one occasion made a mistake about the frequency of your posting on WUWT due to your posting on a high proportion of the WUWT comment threads I looked at, When challenged, I checked the facts and admitted the error, However, making a mistaken inference is not the same as making things up. That you should misrepresent it as such, however, shows that I am not mistaken about your integrity, or rather your lack of the same.

    As to whether fair minded people agree with me about you, they need only consider whether you were correct about the number of papers missing from the searchable database; and determine what evidence you had for your slurs against the authors of Cook et al. You were mistaken, ignorant, in point of fact – and your slurs were not justified by any evidence, being based solely on your vindictive nature.

  22. Tom Curtis:

    That is a straight forward lie…. However, making a mistaken inference is not the same as making things up.

    I’ll leave aside the fact I never said that was one of the examples I had in mind, thus you’re accusing me of lying without even being able to be sure of what I’m talking about.

    I want to focus on the semantic issue you raise between “making things up” and “making a mistaken inference.” The two are obviously not one and the same, but there is plenty of overlap between them. Most of the time somebody makes something up, they do so based upon “a mistaken inference.” That’s sort of like how John Cook used a fabricated quote (which you acknowledged). I’ve always assumed it was unintentional. I even assumed it was unintentional when Skeptical Science used a fabricated quote on multiple occasions. (Though I’m not sure the continued use of that fabricated quote can be considered unintentional as I know for a fact you guys discussed the issue in your forum.)

    Those are examples of mistakes where people made things up without meaning to. You claiming I am “a regular commentator at WUWT” is similar. It was a factual claim you believed to be true, but wasn’t. That you looked at some evidence before jumping to a wrong conclusion doesn’t mean your conclusion wasn’t made up. Similarly, I assume your fabrication of a quote in that comment was unintentional. That doesn’t change the fact you made the quote up.

    As to whether fair minded people agree with me about you, they need only consider whether you were correct about the number of papers missing from the searchable database

    Here’s a thought you might want to consider (head’s up hunter). I stated one paper was missing from the searchable database. You, a Skeptical Science representative, responded by going on and on about how there wasn’t just one paper missing, but 185. Had I simply written a post saying 185 papers were missing from that database, no Skeptical Science representative would have ever posted here acknowledging that fact.

    In other words, if I hadn’t done the incredibly stupid thing you say I did, I wouldn’t have confirmation from a Skeptical Science representative that Alastair McEwan;s press release is wrong. It might be good for people to consider that when judging our performances here.

  23. Sorry to jump in, but Brandon is obviously right about a searchable database from a website site is not being the same thing as a data file that was released by Cook’s group. Because the contents of the database could change over time, then this simply isn’t a substitute for a dump of the contents of the data file at the time of the submission of the final (accepted) manuscript.

    Tom Curtis–isn’t that kind of obvious?

    That said… how do we know the database wasn’t modified to include XXX additional papers after the submission of the final manuscript?

    On a side note, I am hoping that Tom Curtis would address the question of whether he signed a human subjects consent statement written by Cook and approved by University of Queensland when he initially agreed to participate in the survey. In fact, just a summary of what he was assured would help in un-muddy the waters.

  24. Carrick:

    1) A searchable database has an underlying data file that supports it. Therefore the distinction you and Shollenberger are trying to raise it is trivial and irrelevant to the discussion. At best Shollenberger could claim that I used non-standard vocabulary to make an obvious point that devestated his argument. He would be wrong, but that is the best he could claim.

    2) Anybody interested could have done a data scrape of the searchable database, thereby reconstructing the underlying data file. They could have done it a year ago when it was first available, and done it again now to compare the two. Therefore there is no substantive difference between the two even if you are right about nomenclature (which you are not).

    3) We can detect any change of contents by said data scrapes. Absent such data scrapes we have a good idea that there has been no change because we can compare the numbers from your data scrape from two days after publication (which I confirmed independently at the time) with those we obtain now. Unless you wish to suggest Cook removed files, substituting new ones with the same classification and endorsement rating for no apparent purpose, we thereby confirm no alteration. If you do want to affirm Cook has made such a substitution, you demonstrate a large degree of paranoia, and that intelligent conversation will be impossible with you.

    4) I have never been the subject of research by Cook or the UQ, and so no human subjects consent forms were required. Frankly the line of argument in relation to Cook et al (which I believe originates from Tol) is completely ungrounded in reality. I had thought better than you than that you would take it up. However, while Cook et al was not research into individual raters, any attempt to use the data Shollenberger stole to assess individual raters would be. I hope Tol and he have got their consent forms all signed and lined up.

  25. Tom Curtis, this is pushing my moderation limits to an extreme, but since you’re such a staunch critic of me, I’ll allow you to continue. I think it’s hilarious you say:

    1) A searchable database has an underlying data file that supports it. Therefore the distinction you and Shollenberger are trying to raise it is trivial and irrelevant to the discussion. At best Shollenberger could claim that I used non-standard vocabulary to make an obvious point that devestated his argument. He would be wrong, but that is the best he could claim.

    If one accepts the underlying data used by the searchable database is a data file, your argument is false on its face. The searchable database does not present all the underlying information. That means it does not present all the information in what you now say is the data file. As such, the searchable database cannot possibly be considered a data file. At most, you could argue it presents us windows through which we can look at parts of a data file.

    4) I have never been the subject of research by Cook or the UQ, and so no human subjects consent forms were required.

    This would seem to confirm there were no confidentiality agreements regarding rater identity. If so, there was never any obligation to withhold that data as that data was not confidential.

  26. Brandon:

    “The searchable database does not present all the underlying information. That means it does not present all the information in what you now say is the data file. As such, the searchable database cannot possibly be considered a data file.”

    The searchable database does present all the information in the underlying data file. It merely displays them 25 papers at a time (with the correct search terms). It can also display subsets of the data whose union is the complete data from the underlying file. Therefore it does present all the information in the underlying data file.

    “This would seem to confirm there were no confidentiality agreements regarding rater identity. If so, there was never any obligation to withhold that data as that data was not confidential.”

    On the contrary. From memory assurances of confidentiality were made when the raters were recruited. Such assurances create an ethical requirement of confidentiality regardless of whether or not any formal paper work was signed. Further, even if it was open to Cook to release the data, that in no way grants you the right to release the data.

  27. Tom Curtis, at some point, it would be nice if you’d stop making things up at some point:

    The searchable database does present all the information in the underlying data file. It merely displays them 25 papers at a time (with the correct search terms). It can also display subsets of the data whose union is the complete data from the underlying file. Therefore it does present all the information in the underlying data file.

    You don’t have access to the underlying database so you couldn’t possibly know this is true. You have no basis for this claim.

    Moreover, everyone should already know your claim is false. At a minimum, paper IDs are not accessible via that search engine, but they were included in the database. We know this because they were used as an index of the table, something we’ve seen in the output for the actual data files.

    On the contrary. From memory assurances of confidentiality were made when the raters were recruited. Such assurances create an ethical requirement of confidentiality regardless of whether or not any formal paper work was signed.

    Even if John Cook gave assurances of confidentiality, that would not be enough to make the data confidential. Confidentiality requires all people given access to the data agree to keep that data confidential. If nobody signed any sort of agreements limiting what could be disclosed, the data was never confidential, regardless of what Cook said he would or would not release.

    And that’s ignoring the fact simply telling people you’ll keep their identity secret doesn’t automatically mean you’re allowed to keep their identity secret. Professional work has rules about how confidentiality works. It requires more than just saying, “I won’t tell anyone.” Subjects of research have a lot of rights regarding confidentiality, but the researchers themselves don’t.

    On a moderation note, you are refusing to address the fact you’ve made multiple false claims on this page. For example, you still have not corrected your claims regarding this file (such as it not having 11,944 entries). I’m giving you a lot of leeway when you at least respond on the general topic of a dispute, but if you won’t even attempt to respond to factual disputes, moderation isn’t optional.

  28. Tom:

    1) A searchable database has an underlying data file that supports it. Therefore the distinction you and Shollenberger are trying to raise it is trivial and irrelevant to the discussion. At best Shollenberger could claim that I used non-standard vocabulary to make an obvious point that devestated his argument. He would be wrong, but that is the best he could claim.

    Of course it’s not trivial. Unless you have a “frozen” version of the file, e.g., stored on the web-server for the journal, in principle the file could be modified, increased in length, or even deleted.

    I wonder for example if the difference in the length of the database and the file is just that….we also don’t know which is supposed to be canonical here for the Cook article: the database available from the webserver or the

    ) Anybody interested could have done a data scrape of the searchable database, thereby reconstructing the underlying data file.

    And they could do it in the future… if only they had a time machine!

    But again, the data file that is included with the paper is supposed to be the official version. We can’t know which is the official version, unless the paper or the SI addresses it. The paper doesn’t even discuss the webserver does it?

    Unless you wish to suggest Cook removed files, substituting new ones with the same classification and endorsement rating for no apparent purpose, we thereby confirm no alteration

    No. Unless he states that this is the official version of the database used for his paper, we can make no such assumption.

    I have never been the subject of research by Cook or the UQ, and so no human subjects consent forms were required. Frankly the line of argument in relation to Cook et al (which I believe originates from Tol) is completely ungrounded in reality. I had thought better than you than that you would take it up.

    What!? Of course we should be able to ask research ethics questions, and we shouldn’t have to feel guilty when we do so. But thank you in any case for clearly answering the question, even if you have to grief me for asking it.

  29. Brandon, you asked on Lucia’s thread about what constitutes a human subject. Here’s the best answer I can give.

    A person becomes a human subject when they serve as the instrument of measure or when they are the object of measure.

    In the former of these, the evaluation tool is a psychometric tool that the subject uses to measure some other object (e.g., abstracts or excerpts in Cook’s TCP or from web comments in Lew’s Fury). In the latter of these cases, the psychometric tool is not “driven” by the subject, but by a researcher, e.g., applying DSM-IV diagnostics to subjects e.g., suffering from a common trauma.

    It is very common in social sciences to refer to the psychometric tool as the “instrument”, but I object to that word usage. Think for example “thermometer”. That’s the thing that is used to measure. Thermal expansion or junction of dissimilar materials is how the measurement is done.

    Anyway, it’s possible that TCP, even though it involves human subjects is exempted from IRB. But I think it’s unlikely that it would not need IRB approval in either the US or Australia.

    I am pretty sure that Cook would not be in the US because there was interaction between Cook and the human subjects. Lew’s Fury wouldn’t be exempted because there was interaction between Lew and his fellow authors and the people who wrote the comments being evaluated.

    The comments authors are not irrelevant, because Fury wasn’t just a study on discourse, it focused on denialism and the reactions of the authors to the supposed stimulus of Hoax.

    For Australia, I think neither study is rightfully exempted without review, because under Australian code, it is a requirement that all studies involving human participants undergo IRB review. (I dug this out in connection with Lew’s Fury.)

    Remember that Lew seemed to think his Fury was exempt, but he was quickly corrected by the IRB secretary. My ethics complaint about his research wasn’t that there wasn’t a IRB approval, but that the IRB approval had nothing to do with the research he was doing. (Moreover, Lew didn’t ask for permission until he had started it, and IRB approval cannot retrospectively be given.)

    So if I’m right (which I’m pretty sure I am, but I’ve been sure and wrong before), TCP is a seriously compromised piece of literature. Technically, it should be withdrawn and all public copies destroyed. That won’t happen of course, but that is standard practice for badly compromised studies.

  30. Carrick, that’s interesting if true. I’m still struggling to see how it’d work to have raters be human subjects though. I just don’t see where you’d draw a line between them and one of the researchers. Unless researchers can be human subjects and researchers at the same time? I don’t know.

    I do know at least one University of Queensland representative has said an ethics approval was granted for this work though (and that was the basis for not releasing data). If true, the approval could provide some useful information. They refused to provide it to me when I asked for it, but an FOI request was submitted for it a little while back. Hopefully we’ll have it (or know it doesn’t exist) within a couple weeks. I wish we’d know sooner, but it’s my understanding the university has 30 days to respond.

    In any event, the paper is screwed up no matter what. If there was an ethics approval which does what they claim it does, John Cook failed to meet it by failing to protect raters’ confidentiality (such as by making a graph to share raters’ identities and publishing it where a bunch of people could see). If there wasn’t an ethics approval, the University of Queensland lied to people in order to hide data they had no legal basis for hiding.

  31. Brandon, the definition includes either of the two categories I described above, either being measured, or used for measuring. It actually makes no difference whether one of the subjects is also a researcher, you can easily be both. There are, as I mentioned, examples where you are exempted from direct IRB oversight (in the US you’re allowed to self-exempt if your research meets certain conditions, but you take on any responsibility for harm in that case).

    Imagine instead of a web-based interface, you were having subjects grading from 1 to 7 (Likert scale) how satisfactory a paint finish is by sitting in a room comparing samples…there’s not really so much difference between these two activities.

    The biggest difference, the politicization of climate science, actually increases the risk of harm to the subjects, were their anonymity broken.

    I had this discussion relating to Lewandowsky’s Fury that you might find interesting.

  32. Since Tom Curtis brings up again the question of whether human subjects approval is necessary, this link from The University of Queensland is apropos.

    The section on “Who is a ‘participant'”, in particular the first line, removes any question:

    A ‘human participant’ in research is any person who, for example:

    • Provides research data, e.g. completes surveys
    • Participates in interviews, discussions, a focus group, or observations
    • Undergoes psychological, physiological or medical assessment that is not part of their clinical treatment
    • Tests software or a device, or equipment
    • Grants access to personal data or records, photographs, etc
    • Provides human tissue, e.g. blood, urine, saliva, hair, etc
    • Is identified in a record, e.g. employment record, medical record, education record, membership list, electoral roll
    • Is identified in a databank, including unpublished human research data, e.g. an analysis of existing unpublished data collected by another researcher or collected for another research project

    This section “Do not start your research without ethics approval” is also worth a read.

  33. Carrick, if you can be both a human subject and a researcher, that would address my reservations. It sounds a little weird, but it would make sense. Would a researcher who is a human subject have to sign the same releases as the rest? That’d seem silly, but I’m not sure if consent would be implied by them authoring the paper. Similarly, I’m not sure if an ethics approval would be needed to basically do work on yourself.

    Actually, I can easily see ethics approval being needed for work where the human subjects are also the researchers. It certainly arises in medical testing.

  34. Brandon, you definitely can be a researcher as well as a participant—I have papers that include acoustics measurements obtained from myself.

    If it is your project, you don’t need to sign a consent form, because you are supervising it. But you still need to conform to the protocols that were established for the other subjects. If you have grad students participating, since they are not supervising it, you do need to get a signed consent form from them. There are also ethical issues involving ensuring their voluntary participation, so if you are planning on including your students in the subject pool, that needs to be spelled out when you’re requesting IRB approval for the research

    Note that subjects can voluntarily reveal their participation in a project (unless they are under a non-disclosure agreement). And they can give consent to have their data discussed in a non-anonymized fashion (normally via a signed statement). Thus, you, as principal investigator, can nearly always discuss data you collected on yourself.

  35. Gotcha. That makes sense.

    Follow-up question. What kind of waiver do you need before sharing the identity of a human subject? I’m thinking specifically of the fact raters could see who else had rated papers. Disclosure of one’s identity by posting in a forum may be voluntary, but a number of the raters never did that. Their identities were disclosed to the other raters* by the system John Cook implemented. Am I right in thinking ethical obligations mean they had to specifically agree to such?

    *I’m not sure whether or not one needed special permission to start rating papers. It sounded like anyone could rate papers if their account had permissions for the forum, but I can’t be sure. If that is the case, it’d appear 200+ people had access to all the rater identities (or at least, their handles).

  36. Brandon:

    What kind of waiver do you need before sharing the identity of a human subject?

    It varies, but this can be included in the consent form. Beyond that, it depends on what you were “promised” in the consent form itself.

    Am I right in thinking ethical obligations mean they had to specifically agree to such?

    If it is protected data, then yes. I think that covers virtually any scientifically interesting measurements, though.

    UQ spelled out pretty well the case for maintaing the confidentiality of the subjects, though I thought they focussed unduly on the repercussions for the University (and overstated the possible damage to the University). Your first obligation is protecting subjects from harm.

    With highly politicized topics, I can imagine subjects being harassed for participating in a study that found a “politically inconvenient” result. Keeping the subject’s identities anonymous amounts to protecting the subjects from harm in that case.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s