Stupidity Proves You’re Smart

After uploading my last post criticizing work by Dan M. Kahan, I saw he had responded to me in a comment to address my concerns. This led to me finding a quote from him which is mind-boggling:

Indeed, while such an outcome is unlikely, an item could be valid even if the response scored as “correct” is indisputably wrong, so long as test takers with the relevant comprehension capacity are more likely to select that response.

Kahan believes it is okay to measure people’s scientific intelligence by taking absolutely incorrect answers as “right.” Why? Because if scientifically literate people all make the same mistake, that mistake shows they are scientifically literate!

So remember folks. It doesn’t matter if tons of people believe in something stupid. It doesn’t matter if the stupid “97% consensus” meme on global warming is only spread by people who refuse to look at or think about what the data collected actually shows. It doesn’t matter if Michael Mann’s hockey stick is only defended by people who willfully avoid thinking about basic truths apparent to anyone who made any effort to understand his work.

None of that matters. As long as the people saying and believing stupid things give the right answers to other questions, they’re the scientifically literate ones, and you’re the idiots for refusing to believe in incredibly stupid things.

Advertisements

18 comments

  1. ==> “Kahan believes it is okay to measure people’s scientific intelligence by taking absolutely incorrect answers as “right.” .

    Heh. Brandon – I would suggest that next time, before writing something like that, you bother to take the time to investigate what someone says before you characterize what they say. It might help prevent you from making mischaracterizations.

    Then again, having seen your past efforts, it might not.

    It’s worth a shot, however.

  2. Brandon, let me try another tack.

    If the kind of test one is administering is designed to certify retention or assimilation of some canonical set of “facts,” then the questions have to be ones that *have* right answers & have to be scored as “correct” when someone gives the “right” answer.

    But if one is measuring a kind of reasoning disposition, then questions have to be *valid.* That means that they have to be such that one or the other response correlates with the disposition in question. Being “right” or “wrong” is not logically related to being *valid* in this sense: that is, answers to questions that are “right” might not predict or diagnose or measure the disposition of interest. Likewise, it is at least theoretically possible that an answer that is “wrong” could. That’s not very likely, but it could happen! There are some sorts of mistakes that only people who have high math proficiency are likely to make; in that case, when someone gives the signature “wrong” answer, that is actually some evidence that the person has a high math proficiency.

    I’m pretty sure we agree that a science comprehension test *shouldn’t* be the first sort of test — one that certifies the absorption of some set of “facts” that the test designer has designated as “right.” So it continues to puzzle me that you keep making me out as subscribing to that view.

    But others can judge for themselves easily enough: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2466715

  3. DGH, I didn’t include a link to his blog in the first post because the post I commented on wasn’t really about the survey. I had meant to include a link to the documentation notes for the survey. I’m not sure how I forgot to do that.

    I did include a link to his blog in this post though. It’s in the second sentence. I included that one because the post I linked to was directly topical to what I was discussing here. I regret not finding a similar post to link to in the last one.

  4. dmk38, there are a couple problems with this:

    But if one is measuring a kind of reasoning disposition, then questions have to be *valid.* That means that they have to be such that one or the other response correlates with the disposition in question. Being “right” or “wrong” is not logically related to being *valid* in this sense: that is, answers to questions that are “right” might not predict or diagnose or measure the disposition of interest. Likewise, it is at least theoretically possible that an answer that is “wrong” could. That’s not very likely, but it could happen!

    First, this is not how you depict your results. You clearly labeled certain answers as “correct.” You can’t label an answer as “correct” then turn around and say it doesn’t matter whether or not the answer is actually correct. If you tell everybody an answer is “correct,” the answer needs to be correct. If it isn’t, you’ve messed up.

    Second, if whether or not a response is correct is irrelevant, you obviously cannot be measuring genuine knowledge. You may be measuring something, but what? You can’t say. You can’t tell us what the latent variable your scale measures actually is. You’re not measuring scientific intelligence if you say we can disregard the correctness of the answers.

    As for you being puzzled:

    I’m pretty sure we agree that a science comprehension test *shouldn’t* be the first sort of test — one that certifies the absorption of some set of “facts” that the test designer has designated as “right.” So it continues to puzzle me that you keep making me out as subscribing to that view.

    You specifically labeled answers as “correct.” That means you designated the “facts” represented by your chosen answers as “right.” The reason I keep making you out as subscribing to the view these answers are “facts” you have designated as “right” is that’s exactly what you did in your text.

    If you don’t want people to interpret your test as indicating people’s ability to absorb a certain set of facts, you shouldn’t write an entire document describing it that way.

  5. Joshua, there are a few other questions from this study by Kahan that boggle my mind.

    From here, there is this one:

    “Human-caused global warming has increased the number and severity of hurricanes around the world in recent decades”.

    Wat?

    The correct answer is “we don’t know”. It’s very difficult to make attribution to any variability with hurricanes due to the short duration of the measurement period giving competing effects from natural variability and due to the fact you are trying to measure phenomena that exist in the tail of the probability distribution.

    Neither True nor False. “We don’t know” is the only right answer.

    Hurricanes are a complex phenomenon. It is not easy to come up with definitive numerical model based predictions of whether we’ll get more hurricanes, or whether they’ll become more intense. The argument that people make about surface temperature increasing is a rule of thumb not a model at all; the rate of temperature increase (for well known reasons involving differences between sensible and latent heat) are much less in the tropics (so any putative effect is smaller in the tropics than the global value); and if you have more hurricanes, this makes more powerful ones less likely (stirring up of surface ocean waters by a hurricane reduces the potential of further storms that follow the same or similar storm track from becoming as energetic as it might otherwise become). Fewer hurricanes probably translates into a few powerful ones. Many hurricanes, especially in the Atlantic with the formation zone off the Horn of Africa, implies more hurricanes, but less destructive ones.

    Then this one: “human-caused global warming will result in flooding of many coast regions”.

    Given that the changes expected over the next multiple centuries amounts perhaps to 1-m, you can say it will “contribute to flooding”, but not so-much that it causes it. So again, IMO, neither true nor false.

    Further, it’s worth noting that you need to be careful to say what you mean when you say “flooding”. Do you mean gradual inundation or do you mean sudden inundation? From an economic perspective, the risks of flooding from a gradual 1-m rise in oceans are mild compared to the damaging effects of removal of natural barriers to flooding like mangrove and cyprus swamps. We saw this very clearly with the 2004 Indonesian tsunami.

    So it is fair to say increasing ocean surface levels does increase the risk of flooding. It’s not fair to say this is even close to the most significant human activity that increases flooding. In addition to natural barrier removal, humans are also concentrating their populations on shorelines. It’s easy to have a boogieman like AGW to point the finger at, when the problem is with e.g. US East Coast real estate developers trying to make a killing off of ocean view properties. It is also fair to say that there are other much less costly things we could do to reduce the risk of flooding besides decreasing anthropogenic CO2 emissions.

  6. Carrick, remember, those questions are prefaced with, “Climate scientists believe….” That means people aren’t asking about the truth, but what other people think the truth is.

    Which kind of makes things worse. I have no doubt some climate scientists believe global warming has increased the intensity and frequency of hurricanes. I’ve seen some say so. At the same time, I have no doubt other climate scientists believe differently. I’ve seen some say that as well.

    Would I be expected to pick “Yes” and “No”? “I don’t know”? “This is a stupid question”? Could people just skip questions like this?

    I’d say we should ask Dan Kahan, but he’s apparently decided making things up about what his critics say then running away is the best tactic.

  7. If you are careful to include the proviso “Climate scientists believe..” there are going to be few statements that are definitively “true” or “false” questions. These are both contentious questions (the hurricane one more than the flooding one, though the second if you bring in “policy relevance”.)

    I think Dan Kahan really thinks that people who are physical scientists really think exactly the way the media portrays them as thinking.

    To make it even messier, people like Trenberth sometimes make arguments more definitive sounding than they really are, because they are trying to influence policy. Thus, “Some climate scientists say…” would be true. Whether Trenberth believes his own shtick, well it’s hard to take him seriously, IMO, when you know he’s wearing an activist hat as often as he is that of a scientist.

  8. Yup. This is something of a mess.

    In other news, Dan Kahan may have banned me. I’ve had two comments disappear into moderation despite passing their CAPTCHA filter. I’ll try a test comment later, but it looks like I was at least put in pre-moderation.

    I know I wasn’t nice in my criticisms, but it’d be silly to do this sort of thing, especially without notifying people. It’s especially bad since he repeatedly made things up about what I’ve said and ignored it when I pointed that out.

  9. Carrick –

    I missed you comment earlier. I hope that you’ll see this.

    Yes, I had similar criticisms w/r/t each of the items that you discussed. I have had a number of discussions with Dan, where I have talked about why I don’t think that the “scientific literacy”: assessments are valid (in the sense commonly used in testing, where valid means measures what they purport to measure.” I find such tests to be mostly based on a rather empty rational that because you can assign a number to something that you have therefor measured what you think that number measures.

    If you read what NiV has written in a number of Dan’s posts about those assessments, you will see him, also, raise questions about their validity.

    But the larger point is that Dan recognizes the significant limitations of those assessments. That is a large part of his point. actually. Brandon’s line of argumentation is based in a fundamental misunderstanding of Dan’s focus – and that’s why he misrepresents what Dan says (i.e., that Dan says that the batteries measure “intelligence.”)

    At any rate, they are far from perfect. But no matter their problems and the legitimate questions related to specific items, his underlying point does, IMO, stand – that the tests serve to measure who someone is much more than what they know about climate science.

  10. I concur with Brandon that the Y-axis label “correct answers” should have been written “consensus answers” or even just “DK’s opinions”. For the most part it won’t make that big a difference until page 14 when the author clearly presumes upon a thing that is not universally regarded as “correct” or even consensus.

    From the paper, the obvious-conclusion-of-the-day:

    “Factor analysis also supports the inference that global warming beliefs and risk perceptions are more convincingly treated as indicators of a latent identity that features a left-right political orientation.”

    Duh. How many times must society pay for someone to “discover” this? Global warming is now almost exclusively a leftwing phenomenon.

  11. The first set of charts is slightly confusing but interesting. It differentiates whether people themselves believe in evolution or big bang, versus whether they believe in (or at least are aware of) the *definition* of evolution.

    It is clear that many people know the definition but do not personally subscribe to it. This will doubtless be true of human caused global warming as well, with important nuances about how much the “A” actually influences the “GW”.

  12. Same blog — different sentence:

    “But an adroit reader of this blog—perhaps a climate scientist or maybe just a well educated nonexpert—objected that in fact floating sea ice has slightly less salinity than sea water, and as a result of some interesting mechanism or another displaces a teeny tiny bit less water than it would add if melted.”

    the “interesting mechanism” is simply the volume of the iceberg above the waterline which is NOT displacing sea water. The weight or mass of seawater displaced matches the weight of the iceberg, but that portion that floats *above* the sea is no longer displacing the sea.

    (Okay, where’s my science comprehension gold star?)

  13. Michael 2, I don’t think you can get any gold star for that comment. The “interesting mechanism” being referred to in that quote is the one which allows floating ice that melts to raise water levels, contrary to what one might normally expect (and what you refer to). Namely, the greater the salinity of water, the greater its density. If ice which melts has less salt in it than the water it is floating in, the water level will rise.

  14. Hmm, on second thought, perhaps because ice has *expanded* slightly it does actually displace the same volume despite some of that expansion now being above water. So I’ll agree that melting a frozen distilled ice water cube in a glass of distilled water won’t raise the level but I can see where salt water is more dense and consequently will raise freshwater ice to a higher level (with less displacement as a result).

    Now THAT is science comprehension. I would never have thought of that (mostly because I don’t need to think about it).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s