Rethinking the scientific method | The New Yorker (subscription required, unfortunately)
Extremely disturbing article about well-intentioned scientists’ experiments and conclusions gone awry. Most upsetting is the lack of reproducibility in many important studies and the likelihood that those works don’t get published.
The disturbing implication of his study is that a lot of extraordinary scientific data is nothing but noise. This suggests that the decline effect is actually a decline of illusion. Many scientific theories continue to be considered true even after failing numerous experimental tests. The decline effect is troubling because it reminds us how difficult it is to prove anything.
The article is about medicine, but I wonder how big of a problem this is in Astronomy. I posed this question to my Buzz followers, and Gus Muench pointed out this article on the distances the LMC. The author describes an unsettling cluster of distance around the HST Key Project (HSTKP) value.
Indeed, these measures cluster too tightly around the HSTKP value, with 68% of the measures being within 0.5-sigma of 18.50 mag. A Kolmogorov-Smirnov test proves that this concentration deviates from the expected Gaussian distribution at a >3-sigma probability level. This concentration is a symptom of a worrisome problem. Interpretations considered include correlations between papers, widespread over-estimation of error bars, and band-wagon effects. The purpose of this paper is to alert workers in the field that this is a serious problem that should be addressed.
Do you know of other examples of band-wagon effects that we should be aware (and wary) of? Are there astronomical questions that are more susceptible to these types of biases than others?
The same thing occurred for years in studies of the Hubble Constant, in which the values clustered around the “near” or “far” scales (both of which were off from the currently-favored value). After the Key Project value was published, nearly all successive studies (although notably not all, e.g., Sandage) found a value consistent with the Key Project value.
I think it’s likely that things like this happen all the time.
One example that astrobetter has posted on in the past is the term “flat rotation curve” in spiral galaxies.
Besides the distance scale problems already named, I have the impression that a number of times group-think has affected stellar evolution theory, where over the years the limitations of an approximation or assumption can sometimes be forgotten. Certainly we should keep in mind that anything calculated with a mixing length theory is using a very questionable approximation, even if there are practical reasons to use it. I believe better opacity tables have sometimes resolved long-standing controversies, problems or paradoxes. I bet there are many examples in ages of clusters: Personally, I have the impression that even today ages of the young moving group/associations (beta Pic; TWA) are given too consistently and too precisely, though if I could prove it I’d be writing a paper not a comment.
Still, many cases scientists may just be doing the best they can with limited knowledge. In the 1990s, my impression was that there were many who believed the universe was flat and therefore omega=1; and many others that believed that omega=0.3 and the universe was open. Oddly enough, they were both right and wrong. (Similar issues were going on with the Hubble Constant and globular cluster ages.) Or think of the case of the case of the solar neutrino problem which wasn’t resolved for decades: If the problem had been with the Solar models, which was certainly a plausible explanation, then all of the stellar calculations throughout astrophysics potentially had big problems. Perhaps there were New Yorker articles pointing this out! As it turned out, those who pressed ahead with stellar calculations were correct since it was the neutrinos. (Whether those physicists who correctly thought neutrinos had nonzero masses, contrary to the Standard Model, did well or not I couldn’t say!)
This is also a well know issue in particle physics (e.g. “Bandwagon effects and error bars in particle physics”, Jeng, 2008, NIMA). One should blind the analysis until the very end of the procedure, and/or use training/validation samples.
I am glad to have read this post! I have been working on such band-wagon effect about the distance of the closest microquasars. After almost two years of investigation, I published a complete “review” paper that show, to the best I could, the problems associated with the fact that the distance of a well known microquasar was nothing but smoke, and never really measured, even if 5 Nature papers were published on that object. I resolved every little tiny reference relationships I could find in Nature and regular papers, and IAU Circulars, Telegrams and so on, and discovered many cross-references etc etc. ADS can tell me my paper is often read. But very few cite it. A former lab colleague even told me “it’s useless”… Have a read here:
What is the closest black hole to the Sun?
http://adsabs.harvard.edu/abs/2009NewA…14..674F
After such study, I felt stoned. Nobody cares (not especially about my paper, but about what it means). Since then, I left research (for different reasons, this one is on the top-3 list).
Yes… although be aware that the original New Yorker article you link to also has much ado about nothing, and some completely ridiculous assertions (read the last paragraph!). For a look by someone actually in medical science, see http://www.sciencebasedmedicine.org/?p=8987 . There are also a couple physics examples the New Yorker article, one which I’m not qualified to discuss, but the other is just completely ridiculous, and demonstrates a complete lack of understanding of how science works on the part of Jonah Lehrer (the New Yorker author).
I’m not saying there’s no bandwagon effect. It can be particularly pernicious when theorists cite observers and observers cite theorists if neither can judge at all the reliability of the other’s work. But the New Yorker article is pretty crappy, and the medical examples (the core of the article) aren’t discussed properly.