Don’t miss an important article in this week’s Nature about how psychologists are facing up to problems with unreplicated studies in the wake of several high profiles controversies.
Positive results in psychology can behave like rumours: easy to release but hard to dispel. They dominate most journals, which strive to present new, exciting research. Meanwhile, attempts to replicate those studies, especially when the findings are negative, go unpublished, languishing in personal file drawers or circulating in conversations around the water cooler…
One reason for the excess in positive results for psychology is an emphasis on “slightly freak-show-ish” results, says Chris Chambers, an experimental psychologist at Cardiff University, UK. “High-impact journals often regard psychology as a sort of parlour-trick area,” he says. Results need to be exciting, eye-catching, even implausible. Simmons says that the blame lies partly in the review process. “When we review papers, we’re often making authors prove that their findings are novel or interesting,” he says. “We’re not often making them prove that their findings are true.”
It’s perhaps worth noting that clinical psychology suffers somewhat less from this problem, as treatment studies tend to get replicated by competing groups and negative studies are valued just as highly.
However, it would be interesting to see whether the “freak-show-ish” performing pony studies are less likely to replicate than specialist and not very catchy cognitive science (dual-process theory of recognition, I’m looking at you).
As a great complement to the Nature article, this month’s The Psychologist has an extended look at the problem of replication [pdf] and talks to a whole range of people affected by the problem, from journalists to research experts.
But I honestly don’t know where this ‘conceptual replication’ thing came from – where you test the general conclusion of a study in another form – as this just seems to be a test of the theory with another study.
It’s like saying your kebab is a ‘conceptual replication’ of the pizza you made last night. Close, but no neopolitana.