Precognition studies and the curse of the failed replications | Professor …

Professor Daryl Bem of Cornell University is a well-respected psychologist. The Journal of Personality and Social Psychology is a well-respected journal, published by the American Psychological Association. It has a high impact factor and a high rejection rate for submissions. It is clearly one of the top journals in the field. It was not surprising, therefore, that when Bem last year published the results of a series of nine experiments appearing to suggest that precognition – or the ability to "feel the future" – is real, the story received a great deal of coverage from mainstream science media around the world.

Bem used a variety of techniques but the general approach was to "time reverse" established psychological effects. For example, the experiment that produced the largest effect size (experiment 9) took as its starting point the trivial observation that memory for words is better if one is allowed to rehearse the words as opposed to being exposed to them just once. Of course, this usually involves rehearsing the words before one's memory for them is tested.

The astonishing claim made by Bem – apparently supported by his experimental data – was that memory for words is improved even if the rehearsal does not take place until after recall has been tested. The effect was dubbed the "retroactive facilitation of recall".

To his credit, in his paper Bem encouraged other psychologists to attempt replications of his findings and even offered to provide appropriate software to run the studies. In collaboration with Stuart Ritchie at the University of Edinburgh, Professor Richard Wiseman at the University of Hertfordshire, and members of my own group at the Anomalistic Psychology Research Unit at Goldsmiths, University of London, decided to do just that.

It was agreed that a replication attempt would take place at each of the three institutions. All three attempts would follow the same procedures as those used by Bem, including using the same number of participants, and the experiments would be pre-registered. Regardless of outcome, we would write up our results and submit them for publication.

As can be seen from our published report in PLoS ONE, none of us produced results that supported the effect reported by Bem (neither did Eric Robinson in a paper published in July 2011 in the Journal of the Society for Psychical Research). Our failure to replicate Bem's results will, no doubt, not come as a surprise to many readers as they will have assumed from the outset that the alleged paranormal effect was not real. Indeed, many commentators strongly criticised the Journal of Personality and Social Psychology for publishing Bem's paper in the first place, though it had been put through the same peer review process as other submissions.

I do not share their view. Once we think we know in advance which effects are real and which are illusory, true scientific objectivity flies out of the window. Having said that, my personal opinion is that retroactive facilitation of recall is not a real effect.

I also have my doubts about the other effects reported by Bem. As would be expected given the controversial nature of Bem's claims, a number of critics have gone through the original paper with a fine-toothed comb and highlighted evidence of flawed methodology and inappropriate statistical analyses.

Even so, I find myself in agreement with Tal Yarkoni's comment on his excellent blog: "It's important to note that none of these concerns is really terrible individually. Sure, it's bad to peek at your data but data peeking alone probably isn't going to produce nine different false positives. Nor is using one-tailed tests, or constructing measures on the fly, etc. But when you combine data-peeking, liberal thresholds, study recombination, flexible hypotheses, and selective measures, you have a perfect recipe for spurious results."

Whether or not any of the effects reported by Bem are real will ultimately depend upon the outcome of further replication attempts. But at least as interesting as that question are the issues that were raised when we attempted to publish our failed replications.

Given that the Journal of Personality and Social Psychology was responsible for publishing the controversial claims in the first and that Bem's paper included an explicit appeal to other psychologists to attempt replications, we figured that this journal was the obvious choice to target in terms of publishing our own findings. The editor of the journal, however, did not agree and rejected our paper without even sending it out for peer review on the grounds that his journal "does not publish replications".

We then submitted it to Science Brevia and received the same response. The same thing happened when we submitted it to Psychological Science. Our failure to even get our paper sent for peer review became something of a story in itself. It was covered, for example, by Ben Goldacre in his Bad Science column, as well as by New Scientist and The Psychologist.

When we submitted it to the British Journal of Psychology, it was finally sent for peer review. One referee was very positive about it but the second had reservations and the editor rejected the paper. We were pretty sure that the second referee was, in fact, none other than Daryl Bem himself, a suspicion that the good professor kindly confirmed for us. It struck us that he might possibly have a conflict of interest with respect to our submission. Furthermore, we did not agree with the criticisms and suggested that a third referee be brought in to adjudicate. The editor rejected our appeal.

We were determined to publish the paper in a high impact journal. Although parapsychology journals have a much more enlightened policy with respect to publishing negative findings (that is to say, they do actually publish them), we were pretty sure that the mainstream science media would not report the results if we did so. After all, did you see any reports in the newspapers about Eric Robinson's study? We were therefore very pleased when the paper was accepted by PLoS ONE, an open-access journal with a high impact factor.

It was published in PLoS ONE on Wednesday.

This whole saga raises important questions. Although we are always being told that "replication is the cornerstone of science", the truth is that the "top" journals are simply not interested in straight replications – especially failed replications. They only want to report findings that are new and positive.

Most scientists are aware of this bias and will rarely bother with straight replications. But straight replication attempts are often exactly what is required, especially when dealing with controversial claims. For example, parapsychologists are typically happy to accept the findings of a new study if it replicates a previously reported paranormal effect. However, if it fails to do so, they are likely to blame any deviation from the original procedure, no matter how minor. It was for this reason that we chose to follow Bem's procedure as closely as possible (apart from a minor methodological improvement).

Given the high cost of paper publications and the high submission rejection rate of "top" journals, it might be argued that rejecting replication studies was defensible in the pre-internet era. But what would prevent such journals from adopting a policy of sending reports of replications, failed or otherwise, for full peer review and, if accepted, publishing the abstract of the paper in the journal and the full version online? Otherwise, publication bias looks set to remain a major problem in psychology and science in general.

There is one final twist in this tale. PLoS ONE has recently found itself at the centre of another controversy over a failed replication. Professor John Bargh of Yale University wrote an angry blog for Psychology Today in response to an article in PLoS ONE by Stéphane Doyen and colleagues of the Free University of Brussels. These scientists had failed to replicate a classic experiment by Bargh and colleagues published in 1996 in the Journal of Personality and Social Psychology suggesting that behaviour can be significantly affected by unconscious priming.

Bargh lambasted the researchers and the papers' reviewers, and claimed the journal "does not receive the usual high scientific journal standards of peer-review scrutiny". Just for good measure, Bargh also laid into bloggers who reported the failed replication, describing their efforts as "superficial online science journalism". In fact, much of what Bargh has to say appears to be unjustified, as ably argued by Ed Yong and the publisher of PLoS ONE, Peter Binfield.

I will limit myself to just one additional point regarding the Bargh controversy. Bargh strongly criticises Doyen et al for what he sees as serious deviations from his original procedure. He writes, "When researchers attempt (in good faith) to replicate another lab's findings, they are supposed to follow the original procedure as closely as possible." As I hope the above shows, this is not the best advice if you actually want to get your work published quickly in a "top" journal.

Chris French is a professor of psychology at Goldsmiths, University of London, and heads the Anomalistic Psychology Research Unit. He edits The Skeptic magazine

Open bundled references in tabs:

Leave a Reply