Indeed, while such an outcome is unlikely, an item could be valid even if the response scored as “correct” is indisputably wrong, so long as test takers with the relevant comprehension capacity are more likely to select that response.
Kahan believes it is okay to measure people’s scientific intelligence by taking absolutely incorrect answers as “right.” Why? Because if scientifically literate people all make the same mistake, that mistake shows they are scientifically literate!
So remember folks. It doesn’t matter if tons of people believe in something stupid. It doesn’t matter if the stupid “97% consensus” meme on global warming is only spread by people who refuse to look at or think about what the data collected actually shows. It doesn’t matter if Michael Mann’s hockey stick is only defended by people who willfully avoid thinking about basic truths apparent to anyone who made any effort to understand his work.
None of that matters. As long as the people saying and believing stupid things give the right answers to other questions, they’re the scientifically literate ones, and you’re the idiots for refusing to believe in incredibly stupid things.