Much ado lately about a pair of articles in The Economist that gather the growing evidence that science is getting an awfully lot wrong these days, and may be less reliable than we think (and certainly far less than we wish.)
Perhaps the best summary of this I’ve read is from Jerry Coyne. Here’s the nut:
As I read these pieces, I did so captiously, really wanting to find some flaws with their conclusions. I don’t like to think that there are so many problems with my profession. But the authors have done their homework and present a pretty convincing case that science, especially given the fierce competition to get jobs and succeed in them, is not doing a bang-up job. That doesn’t mean it is completely flawed, for if that were true we’d make no advances at all, and we do know that many discoveries in recent years (dinosaurs evolving into birds, the Higgs boson, black matter, DNA sequences, and so on) seem solid.
I see five ways that a reported scientific result may be wrong:
- The work could be shoddy and the results therefore untrustworthy.
- There could be duplicity, either deliberate fraud or a “tweaking” of results in one’s favor, which might even be unconscious.
- The statistical analysis could be wrong in several ways. For example, under standard criteria you will reject a correct “null” hypothesis and accept an alternative but incorrect hypothesis 5% of the time, which means that something like 1 in 20 “positive” results—rejection of the null hypothesis—could be wrong. Alternatively, you could accept a false null hypothesis if you don’t have sufficient statistical power to discriminate between it and an alternative true hypothesis. Further, as the Economist notes, many scientists simply aren’t using the right statistics, particularly when analyzing large datasets.
- There could be a peculiarity in one’s material, so that your conclusions apply just to a particular animal, group of animals, species, or ecosystem. I often think this might be the case in evolutionary biology and ecology, in which studies are conducted in particular places at particular times, and are often not replicated in different locations or years. Is a study of bird behavior in, say, California, going to give the same results as a similar study of the same species in Utah? Nature is complicated, with many factors differing among locations and times (food abundance, parasites, predators, weather, etc.), and these could lead to results that can’t be generalized across an entire species. I myself have failed to replicate at least three published results by other people in my field. (Happily, I’m not aware that anyone has failed to replicate any of my published results.)
- There could be “craft skills”—technical proficiency gained by experience that isn’t or can’t be reported in a paper’s “materials and methods,” that make a given result irreproducible by other investigators.
If you read the Economist pieces, all of these are mentioned save #4 (peculiarity of one’s material). And the findings are disturbing.
Coyne follows with a particularly lucid commentary on these articles and their implications. Highly recommended.
Find it at Science is in bad shape « Why Evolution Is True.