As many readers of this blog may know, in the last few years a “replication crisis” has caused intense soul searching in psychology – and particularly in social psychology. This crisis was sparked when several widely cited findings in psychology subsequently failed to replicate when tested by independent researchers (for some background see Earp and Trafimow’s paper here). This is – of course – a substantial problem because it severely limits the confidence we can have in psychological findings. For social psychological research this problem may be even graver, given that many of the topics studied – e.g. attitudes, prejudice, political voting – have direct implications for policy making. If we cannot have confidence in our findings, social psychology is in very hot water indeed.
To take an example from my own research field of social/moral psychology, in a highly influential study published in Science, Zhong and Liljenquist (2006) reported evidence of a “Macbeth Effect” whereby a threat to people’s moral purity leads them to seek, literally, to cleanse themselves. Seeking to expand upon these findings and explore whether they would be observed in different domains, my colleagues Brian Earp, Elizabeth Madva, and Kiley Hamlin attempted to replicate and extend this work. To their surprise, they were unable to replicate the original results. They then turned to me, and together we conducted a series of direct replications of Study 2 from ZL’s seminal report. We used ZL’s original materials and methods, investigated samples that were more representative of the general population, investigated samples from different countries and cultures, and substantially increased the power of our statistical tests. Despite our multiple good-faith efforts, however, we were unable to detect a “Macbeth Effect” in any of our experiments. We then tried to publish these findings, facing much difficulty in the process. Put simply, replication efforts are often not viewed favorably in the publication process. Eventually, however, we were able to publish our findings in the journal Basic and Applied Social Psychology (see here for an open access version of the paper).
This experience caused Brian Earp and I to reflect upon the replication crisis in psychology. In a new opinion piece, we suggest that this “crisis” can be interpreted as a disciplinary social dilemma, with the problem facing early-career researchers being especially acute. Social dilemmas – situations in which collective interests are at odds with private interests – are an enduring feature of the modern world, and have two fundamental characteristics: first, that each individual receives a higher payoff for defecting from what is in the collective interest (e.g., using all of the available resources for one’s own advantage) than for cooperating, regardless of what other individuals do; and second, that all individuals are better off if they all cooperate than if they all defect. We interpret the crisis through the lens of a social dilemma, arguing that while it is in everyone’s interest that high-quality, direct replications of key studies in the field are conducted (so that it is possible for the scientific community to know what degree of confidence to place in previous findings from the literature), it is not typically in any particular researcher’s interest to spend her time conducting such replications. This is for a number of reasons: (1) such replications may be time-consuming; (2) they are likely to take energy and resources directly away from other projects that reflect one’s own original thinking; (3) they are generally harder to publish; (4) even if they are published, they are likely to be seen as “bricklaying” exercises, rather than as major contributions to the field; (5) they (accordingly) bring less recognition and reward, including grant money, to their authors—and so on.
Brian and I propose a new a structural solution to this collective problem:
“as a condition of receiving their PhD from any accredited institution, graduate students in psychology should be required to conduct, write up, and submit for publication a high-quality replication attempt of at least one key finding from the literature, focusing on the area of their doctoral research.” (p.1).
As we note in the paper, before any formal implementation at a university level, the basic soundness of the idea would have to have been established—which means subjecting it to critical scrutiny. And so, in both the paper and on this blog, we present our idea in a public forum, and invite discussion and constructive feedback from our colleagues. Could PhD students resolve the tragedy of the academic commons?
Everett, J.A.C.**, Earp, B.D.** (2015). A Tragedy of the (Academic) Commons: Interpreting the Replication Crisis in Psychology as a Social Dilemma for Early-Career Researchers. Frontiers in Psychology.
Open all references in tabs: [1 - 9]