Failure to Replicate Famous Study Causes Furor

Researchers recently tried, and failed, to replicate a classic psychological study. Discover’s Ed Yong (Not Exactly Rocket Science) summarized the original study this way:

John Bargh and his colleagues found that infusing people’s minds with the concept of age could slow their movements. The volunteers in the study had to pick the odd word from a group of scrambled ones. When this word related to being old, the volunteers walked more slowly when they left the laboratory. They apparently didn’t notice anything untoward about the words, but their behaviour changed nonetheless.

Stéphane Doyen and colleagues attempted to reproduce this finding, making only a few changes in the methodology — they used infrared sensors rather than stopwatches to time the participants — and found no effect of age on the subjects’ speed. Their conclusion was that the original result may have been caused by experimental bias: “Perhaps they themselves moved more slowly if they expected the volunteer to do so. Maybe they spoke more languidly… Maybe they were responsible for creating the very behaviour they expected to see.”

Last week, Bargh, a psychologist at Yale, published a scathing attack on the authors of the study, the journal that published it (PLoS ONE), and Yong (for “swallow[ing] the conclusions whole”). I linked to that post at the time, but it since has caused its own furor. Bargh called PLoS ONE a “for-profit” operation that disdains peer review and whose articles should be viewed as “essentially self-published.” He suggested that anyone with $1,350 can get his or her article published. In fact, as the publisher of PLoS ONE wrote in the comments section, the online journal is peer reviewed, non-profit, rejects 31% of articles submitted —and the fee structure resembles that of other open-access journals. Fees, which are waived when necessary, help substitute for the high subscription fees that it does not charge.

You might raise eyebrows at the article-acceptance rate of 69%, but much of Bargh’s claims about PLoS ONE seemed to evaporate on inspection. More substantively, he said that there was no way that the experimenters could have shaped the results in his study, because they were “blind” to the conditions — but commenters, citing the original study, have raised questions about just how absolute that blindness was. Other charges Bargh made about differences in the design of the two studies are also in dispute.

(An indication of the tenor of the argument: The Doyen paper was titled “Behavioral Priming: It’s All in the Mind, but Whose Mind?”; Bargh titled his response, “Nothing in Their Heads.”)

And Yong, who says he gave Bargh a chance to comment on the study, in advance, which was declined, says he basically stands by his reporting — adding that he perhaps should have stressed that a single non-replication does not invalidate an article.

At Marginal Revolution, Alex Tabarrok calls Bargh’s post “a model of how not to respond” to a paper criticizing your work. In any case, this is yet another example of the ways in which questions about methodology and the statistical robustness of findings bedevil psychology.

P.S. Edited to fix my odd slip into Brit-English (“furore”).


Open all references in tabs: [1 - 6]

Leave a Reply