Rethinking the scientific method | The New Yorker (subscription required, unfortunately)
Extremely disturbing article about well-intentioned scientists’ experiments and conclusions gone awry. Most upsetting is the lack of reproducibility in many important studies and the likelihood that those works don’t get published.
The disturbing implication of his study is that a lot of extraordinary scientific data is nothing but noise. This suggests that the decline effect is actually a decline of illusion. Many scientific theories continue to be considered true even after failing numerous experimental tests. The decline effect is troubling because it reminds us how difficult it is to prove anything.
The article is about medicine, but I wonder how big of a problem this is in Astronomy. I posed this question to my Buzz followers, and Gus Muench pointed out this article on the distances the LMC. The author describes an unsettling cluster of distance around the HST Key Project (HSTKP) value.
Indeed, these measures cluster too tightly around the HSTKP value, with 68% of the measures being within 0.5-sigma of 18.50 mag. A Kolmogorov-Smirnov test proves that this concentration deviates from the expected Gaussian distribution at a >3-sigma probability level. This concentration is a symptom of a worrisome problem. Interpretations considered include correlations between papers, widespread over-estimation of error bars, and band-wagon effects. The purpose of this paper is to alert workers in the field that this is a serious problem that should be addressed.
Do you know of other examples of band-wagon effects that we should be aware (and wary) of? Are there astronomical questions that are more susceptible to these types of biases than others?