Life is too short for wasting time probing every instance of professional organizations promoting bad science when they have an established record of doing just that.
There were lots of indicators that’s what we were dealing with in the Association for Psychological Science (APS) recent campaign for the now discredited and retracted ‘sadness prevents us from seeing blue’ article.
A quick assessment of the press release should have led us to dismiss the claims being presented and convinced us to move on.
Readers can skip my introductory material by jumping down this blog post to [*} to see my analysis of the APS press release.
Readers can also still access the original press release, which has now disappeared from the web, here . Some may want to read the press release and from their own opinions before proceeding into this blog post.
What, I’ve stopped talking about the PACE trial? Yup, at least at Mind the Brain, for now. But you can go here for the latest in my continued discussion of the PACE trial of CBT for chronic fatigue syndrome, in which I moved from critical observer to activist a while ago.
Before we were so rudely interrupted by the bad science and bad media coverage of the PACE trial, I was focusing on how readers can learn to make quick assessments of hyped media coverage of dubious scientific studies.
In “Sex and the single amygdala” I asked:
Can skeptics who are not specialists, but who are science-minded and have some basic skills, learn to quickly screen and detect questionable science in the journals and its media coverage?
The counter argument of course is Chris Mooney telling us “You Have No Business Challenging Scientific Experts”. He cites
“Jenny McCarthy, who once remarked that she began her autism research at the “University of Google.”
But while we are on the topic of autism, how about the counter example of The Lancet’s coverage of the link between vaccines and autism? This nonsense continues to take its toll on American children whose parents – often higher income and educated than the rest – refused to vaccinate them on the basis of a story that started in The Lancet. Editor Richard Horton had to concede
If we accept Chris Mooney‘s position, we are left at the mercy of press releases cranked out by the likes of professional organizations like Association for Psychological Science (APS) that repeatedly demand that we revise our thinking about human nature and behavior, as well as change our behavior if we want to extend our lives and live happier, all on the basis of a single “breakthrough” study. Rarely do APS press releases have any follow-up as to the fate of a study they promoted. One has to hope that PubPeer or PubMed Commons pick up on the article touted in the press release and see what a jury of post-publication peers decides.
As we have seen in my past Mind the Brain posts, there are constant demands on our attention from press releases generated from professional organizations, university press officers, and even NIH alerting us to supposed breakthroughs in psychological and brain science. Few such breakthroughs hold up over time.
Are there no alternatives?
Are there no alternatives to our simply deferring to the expertise being offered or taking the time to investigate for ourselves claims that are likely to prove exaggerated or simply false?
We should approach press releases from the APS – or from its rival American Psychological Association – using prior probabilities to set our expectations. The Open Science Collaboration: Psychology (OSC) article in Science presented results of a systematic attempt to replicate 100 findings from prestigious psychological journals, including APS’ s Psychological Science and APA’s Journal of Personality and Social Psychology. Less than half of the findings were replicated. Findings from the APS and APA journals fared worse than the others.
So, our prior probabilities are that declarations of newsworthy, breakthrough findings trumpeted in press releases from psychological organizations are likely to be false or exaggerated – unless we assume that the publicity machines prefer the trustworthy over the exciting and newsworthy in the article they selected to promote.
I will guide readers through a quick assessment of APS press release which I started on this post before getting swept up into the PACE controversy. However, in the intervening time, there have been some extraordinary developments, which I will then briefly discuss. We can use these developments to validate my and your evaluation of the press release available earlier. Surprisingly, there is little overlap between the issues I note in the press release and what concerned post-publication commentators.
*A running commentary based on screening the press release
What once was a link to the“feeling blue and seeing blue” article now takes one only to
Fortunately, the original press release can still be reached here. The original article is preserved here.
My skepticism was already high after I read the opening two paragraphs of the press release
The world might seem a little grayer than usual when we’re down in the dumps and we often talk about “feeling blue” — new research suggests that the associations we make between emotion and color go beyond mere metaphor. The results of two studies indicate that feeling sadness may actually change how we perceive color. Specifically, researchers found that participants who were induced to feel sad were less accurate in identifying colors on the blue-yellow axis than those who were led to feel amused or emotionally neutral.
Our results show that mood and emotion can affect how we see the world around us,” says psychology researcher Christopher Thorstenson of the University of Rochester, first author on the research. “Our work advances the study of perception by showing that sadness specifically impairs basic visual processes that are involved in perceiving color.”
What Anglocentric nonsense. First, blue as a metaphor for sad does not occur across most languages other than English and Serbian. In German, to call someone blue is suggesting the person is drunk. In Russian, you are suggesting that the person is gay. In Arabic, if you say you are having a blue day, it is a bad one. But if you say in Portuguese that “everything is blue”, it suggests everything is fine.
In Indian culture, blue is more associated with happiness than sadness, probably traceable to the blue-blooded Krishna being associated with divine and human love in Hinduism. In Catholicism, the Virgin Mary is often wearing blue and so the color has come to be associated with calmness and truth.
We are off to a bad start. Going to the authors’ description of the first study, we learn:
In one study, the researchers had 127 undergraduate participants watch an emotional film clip and then complete a visual judgment task. The participants were randomly assigned to watch an animated film clip intended to induce sadness or a standup comedy clip intended to induce amusement. The emotional effects of the two clips had been validated in previous studies and the researchers confirmed that they produced the intended emotions for participants in this study.
Oh no! This is not a study of clinical depression, but another study of normal college students “made sad” with a mood induction.
So-called mood induction tasks don’t necessarily change actual mood state, but they do convey to research participants what is expected of them and how they are supposed to act. In one of the earliest studies I ever did, we described a mood induction procedure to subjects without actually subjecting them to it. We then asked them to respond as if they received it. Their responses were indistinguishable and so we concluded that we could not rule out that what were considered effects of a mood induction task were simply demand characteristics, what research participants perceive as instructions as to how they should behave.
It was fashionable way back then for psychology researchers who were isolated in departments that did not have access to clinically depressed patients to claim that they were nonetheless conducting analog studies of depression. Subjecting students to unsolvable anagram task or uncontrollable loud noises was seen as inducing learned helplessness, thereby allowing investigators an analog study of depression. We demonstrated a problem with that idea.. If students believe that the next task that they were administered was part of the same experiment, they performed poorly, as if they were in a state of learned helplessness or depression. However, if they believed that the second task was unrelated to the first, they would show no such deficits.
But the experiment could also be seen as a priming experiment. The research participants were primed by the film clip and their response to a color naming task was then examined.
It is fascinating that neither the press release nor the article itself ever mentioned the word priming. It was only a few years ago that APS press releases were crowing about priming studies. For instance, a 2011 entitled “Life is one big priming experiment…” declared:
One of the most robust ideas to come out of cognitive psychology in recent years is priming. Scientists have shown again and again that they can very subtly cue people’s unconscious minds to think and act certain ways. These cues might be concepts—like cold or fast or elderly—or they might be goals like professional success; either way, these signals shape our behavior, often without any awareness that we are being manipulated.
Whoever wrote that press release should be embarrassed today. In the interim, priming effects have not proven robust. Priming studies that cannot be replicated have figured heavily in the assessment that the psychological literature is untrustworthy. Priming studies also figure heavily in the 56 retracted studies of fraudster psychologist Diederik Stapel. He claims that he turned to inventing data when his experiments failed to demonstrate priming effects that he knew were there. Yet, once he resorted to publishing studies with fabricated data, others claimed to replicate his work.
I made up research, and wrote papers about it. My peers and the journal editors cast a critical eye over it, and it was published. I would often discover, a few months or years later, that another team of researchers, in another city or another country, had done more or less the same experiment, and found the same effects. My fantasy research had been replicated. What seemed logical was true, once I’d faked it.
We have an APS press release reporting a study that assumes that the association between sadness and the color blue is so hardwired and culturally universal that is reflected in basic visual processes. Yet the study does not involve clinical depression, only an analog mood induction and a closer look reveals that once again APS is pushing a priming study. I think it’s time to move on. But let’s read on:
The results cannot be explained by differences in participants’ level of effort, attention, or engagement with the task, as color perception was only impaired on the blue-yellow axis.
“We were surprised by how specific the effect was, that color was only impaired along the blue-yellow axis,” says Thorstenson. “We did not predict this specific finding, although it might give us a clue to the reason for the effect in neurotransmitter functioning.”
The researchers note that previous work has specifically linked color perception on the blue-yellow axis with the neurotransmitter dopamine.
The press release tells us that the finding is very specific, occurring only on the blue-yellow axis, not the red-green axes and the differences between are not found in level of effort, attention, or engagement of the task. The researchers did not expect such a specific finding, they were surprised.
The press release wants to convince us of an exciting story of novelty and breakthrough. A skeptic sees it differently: This is an isolated finding that is unanticipated by the researchers. See, we should’ve moved on.
The evidence with which the press release wants to convince us is exciting because it is specific and novel is what The researchers are celebrating the specificity of their finding, but the blue-yellow axis finding may be the only one statistically significant because it is due to chance or an artifact.
And bringing up unmeasured “neurotransmitter functioning” is pretentious and unwise. I challenge the researchers to show that effects of watching a brief movie clip registers in measurable changes in neurotransmitters. I’m skeptical even if persons drawn from the community or outpatient samples reliably differ from non-depressed persons in measures of the neurotransmitter dopamine.
This is new work and we need to take time to determine the robustness and generalizability of this phenomenon before making links to application,” he concludes.
Claims in APS press releases are not known for their “robustness and generalizability”? I don’t think this particular claim should prompt an effort at independent replication when scientists have so many more useful things to keep them busy.
Maybe, these investigators should have checked robustness and generalizability before rushing into print. Maybe APS should stop pestering us with findings that surprise researchers and that have not been replicated.
A flying machine in pieces on the ground
Sadness impairs color perception was sent soaring high, lifted by an APS press release now removed from the web, but still available here. The press release was initially uncritically echoed, usually cut-and-paste or outright churnaled in over two dozen media mentions.
But, alas, Sadness impairs color perception is now a flying machine in pieces on the ground
The article’s problems seem to have started with some chatter of skeptically-minded individuals on Twitter that led to comments at PubPeer . What followed was a wonderful demonstration of crowdsourced post-publication peer review in action. Lesson: PubPeer rocks and can overcome the failures of pre-publication peer review.
You can follow the thread of comments at PubPeer.
- An anonymous skeptic started off by pointing out an apparent lack of a significant statistical effect where one was claimed.
- There was an immediate call for a retraction, but it seemed premature.
- Soon re-analyses of the data from the paper were being reported, confirming the lack of a significant statistical effect when appropriate analyses were conducted.
- Then doubts were expressed about the integrity of the data.
- The data set for the article was mysteriously changed after it had been uploaded.
- The data disappeared.
- There was an announcement of a retraction.
The retraction notice indicated that the researchers were still confident of the validity of their hypothesis, despite deciding to retract their paper.
We remain confident in the proposition that sadness impairs color perception, but would like to acquire clearer evidence before making this conclusion in a journal the caliber of Psychological Science.
The retraction note also carries a curious Editors note:
Although I believe it is already clear, I would like to add an explicit statement that this retraction is entirely due to honest mistakes on the part of the authors.
Since then, doubts about express whether retraction was a sufficient response or whether something more is needed. Some of the participants in the PubPeer discussion drafted a letter to the editor incorporating their reanalyses and prepared to submit it to Psychological Science. Unfortunately, having succeeded in getting the bad science retracted, these would be consideration of their submission. As of this date, their fascinating account remains unpublished but available on the web.
Postscript
Next time you see an APS or APA press release, what will be your starting probabilities about the trustworthiness of the article being promoted? Do you agree with Chris Mooney that you should simply defer to the expertise of the professional organization?
Why would professional organizations risk embarrassment with these kinds of press releases? Apparently they are worth the risk. Such press releases can echo through the conventional and social media and attract early attention to an article. The game is increasing the impact factor of the journal [JIFs].
Although it is unclear precisely how journal impact factors are calculated, the number reflects the average number of citations an article obtains within two years of publication. However, press releases can promote “early releases” of articles they can acquire citations before the clock starts ticking for the two years. APS and APA are in intense competition for prestige of their journals and membership. It matters greatly to them who can claim the most prestigious journals, as demonstrated by their JIFs. So press releases become important from garnering early attention. Apparently breakthroughs, innovations, and “first ever” mattered more than trustworthiness. In the professional organizations hope we won’t remember the fate of past claims.