It’s time for psychologists to put their house in order

In 2005, the epidemiologist John Ioannidis provocatively claimed that "most published research findings are false". In the field of psychology – where negative results rarely see the light of day – we have a related problem: there is the very real possibility that many unpublished, negative findings are true.

Psychologists have an aversion to some essential aspects of science that they perceive to be unexciting or less valuable. Historically, the discipline has done almost nothing to ensure the reliability of findings through the publication of repeat studies and negative ("null") findings.

Psychologists find significant statistical support for their hypotheses more frequently than any other science, and this is not a new phenomenon. More than 30 years ago, it was reported that psychology researchers are eight times as likely to submit manuscripts for publication when the results are positive rather than negative.

Unpublished, "failed" replications and negative findings stay in the file-drawer and therefore remain unknown to future investigators, who may independently replicate the null-finding (each also unpublished) - until by chance, a spuriously significant effect turns up.

It is this study that is published. Such findings typically emerge with large effect sizes (usually being tested with small samples), and then shrivel as time passes and replications fail to document the purported phenomenon. If the unreliability of the effect is eventually recognised, it occurs with little fanfare.

From where does the bias stem? The publication bias is pervasive and systemic, afflicting researchers, reviewers and editors – all of whom seem symbiotically wed to journals pursuing greater impact from ever more glamorous or curious findings. Unsurprisingly, this relationship encourages the spinning of findings by authors, over-egged claims and outright fraud to add to publication bias in psychology.

Indeed, in a survey of nearly 6,000 American psychologists, the majority admitted being guilty of selectively reporting studies that "worked" (67%), failing to report all dependent measures (74%), continuing to collect data to reach a significant result (71%), reporting unexpected findings as expected (54%), and excluding data post-hoc (58%).

Remarkably, 35% indicated that they had doubts about the integrity of their own research on at least one occasion and 1.7% admitted to having faked their data.

Has psychology cast itself adrift from other sciences, having lost faith in the links between the believability and the replicability of psychological phenomena? Back in 1965, the psychologist Ward Edwards stated: "If a hypothesis is preposterous to start with, why test it?" Such "anti-science" views should be discouraged and any empirically testable hypothesis is worthy of investigation.

In this context, we might raise the spectre of Daryl Bem's recent paper documenting evidence of precognition. Interestingly, it contains multiple replications of a phenomenon that we just don't believe. Other psychological phenomena are less readily replicated (eg. bystander apathy) and yet we firmly believe in them.

No doubt a key factor is whether the replication occurs within the same lab or across different labs. Nonetheless, as the psychologist Arina Bones humorously remarked, since so few psychology papers state hypotheses that are later disproved, scientific hypothesising in psychology may be the most reliable form of precognition.

Last year, Harold Pashler and Eric-Jan Wagenmakers noted that psychology is becoming " … the public face for the replicability problems of science in the early 21st century, [and] psychological science has the opportunity to rise to the occasion and provide leadership in finding better ways to overcome bias and error in science generally."

Although some individuals are culpable, psychology's problems are largely systemic. We cannot rectify every issue immediately, but a great start would be to value replication and negative findings by offering researchers the space to publish such findings – not in a dedicated journal for unloved findings but at the heart of mainstream psychology journals.

With the launch of BMC Psychology on Wednesday, our journal policy is to offer a dedicated open access forum for psychologists to publish work deemed by peer reviewers " … to be a coherent and sound addition to scientific knowledge and to put less emphasis on interest levels."

This remit unquestionably includes null results and replications and the more central role they must play within the discipline. We cannot avoid the conclusion that psychologists, editors and reviewers have conspired to deny the rightful place of negative results and the importance of replication – psychology's dirty little secrets. We must change.

Keith Laws is professor of cognitive neuropsychology at the University of Hertfordshire, and a section editor of BMC Psychology

Open bundled references in tabs:

Leave a Reply