Psychologist Paul Bloom has made a fantastic video for Big Think that introduces pretty much the whole of psychology in 48 minutes.
It’s a brilliant and engaging introduction to the science of mind. Highly recommended.
Link to ‘Paul Bloom: The Psychology of Everything’ on YouTube.
I’ve got a piece in today’s Observer about how psychological biases can affect DNA testing from crime scenes.
It seems counter-intuitive, but that’s largely because we’ve come to accept the idea that DNA is a sort of individual genetic ‘serial number’ that just needs to be ‘read off’ from a biological sample – but the reality is far more complex.
Despite this, the psychological power of DNA evidence is huge and has misled several investigations that have privileged mistaken DNA results above everything else – including the case of a shadowy transsexual serial killer that led the German police astray.
The piece riffs on the work of psychologist Itiel Dror who was the first to show that the identification of people by their fingerprints could be biased by extraneous information and he’s now found the same with certain types of DNA analysis.
More at the link below.
Link to Observer article on the psychology of forensic identification.
Science writer Ed Yong has just posted the audio of a fantastic talk on problems in psychology research and how to fix them.
The talk was delivered at Bristol University but is remarkably direct and he pulls no punches in pointing out psychology’s scientific flaws.
Interestingly, Yong makes the point that this is not a problem of psychology specifically, because many of the problems – like publication bias and selective reporting – appear across the scientific board, but that psychology is a hot topic because the field is trying to do something about it.
Yong has been doing some fantastic work not only highlighting these difficulties but getting a public debate going about solutions to these problems of research culture.
His talk is an excellent round-up of his own work and the state of play in the fight to change science culture.
Link to post with audio of Ed Yong’s talk.
My latest column for BBC Future. The original is here. I was hesitant to write this at first, since nobody loves a problemmatiser, but I figured that something in support of team “I think you’ll find its a bit more complicated than that” couldn’t hurt, and there’s an important general point about the way facts about behaviour are built from data in the final paragraphs, and why theory is important.
Recent reports say curved glasses make you drink beer quicker. But, we must be cautious about drawing simple conclusions from single studies.
We all love a neat science story, but even rock solid facts can be less revealing than they seem. Let’s take an example of a piece of psychology research reported recently: the idea that people drink faster from curved glasses.
Hundreds of news sources around the globe covered the findings, many of them changing the story slightly to report that people drink more (rather than faster) from a curved glass. At first it seems like a straightforward piece of psychology research, with clear implications: curved glasses will make pacing yourself harder, so you’ll end up drinking more than you should. Commentators agreed with the research (funded by Alcohol Research UK) – beverage manufacturers were probably onto this before, and will now be rushing to make us take our favourite tipple out of a curved glass.
But before we change our drinking habits or restock our glass collections, let’s look at what the scientists actually did.
Luckily for us the team of researchers from the University of Bristol, UK, published their paper in an open access journal, which means the research details are free for all to read.
The Bristol team invited participants into the lab and asked them to drink lager (or lemonade) from a straight class or a curved one, while watching a nature documentary (a BBC one, I’m happy to report). They also asked their volunteers to judge when the glass was half full. The results of both were clear, participants finished their drink of lager sooner in the curved glass. They also judged the halfway point as being lower down the curved glass than the straight glass – suggesting a reason for the faster drinking: if people thought the glass was fuller than it really was when then they would underestimate the rate at which they were drinking.
Human factor
Now this is all well and good, but there are many reasons why the results don’t mean that we can make people drink more by changing the shape of their glass. Importantly, none of these reasons would have to do with this research being wrong or inexpertly done. I’m absolutely certain that if we did the study ourselves we’d find exactly the same thing.
No, the reasons you can’t jump to conclusions from this kind of study is because, inevitably, a single study can only test one aspect of the world, under one set of circumstances. This makes it hard to draw general conclusions of the sort that get reported. Notice how the psychologists measured one thing (rate of drinking lager), for just two different glasses, over a single drink, for one set of people (volunteers in Bristol, in 2012), and yet a generalised truth stating that “people drink more from curved glasses” emerged from a specific set of circumstances.
Now obviously, the aim of science is to come up with answers to questions that become generalised truths, but psychology is a domain in which it is fiendishly hard to establish them. If you are studying a simple system, then cause and effect is relatively easy to establish. For instance, the harder you throw a rock, the further it tends to travel in distance. The relation between the force you put in and the acceleration of the rock you get out is straightforward. Add a human factor into the equation, however, and such simple relations begin to disappear. (Please don’t experiment by throwing rocks at people.)
To see how this limits the conclusions that can be drawn from the drinks study, think of even the most trivial factor that could change these results. Would you get the same result if people drank ale rather than lager? Probably. If they drank two pints rather than one? Maybe. If they drank in groups rather than watching TV (arguably closer to the circumstances of most drinking)? Who knows! It seems to me perfectly plausible that a social situation would produce different effects than a solo-drinking experience.
We could carry on. Would the effect be the same if we tried it in Minneapolis? In Lagos? In Kuala Lumpur, Reykjavik or Alice Springs? Most psychology studies are carried out on urban, affluent, students of the western world – a culturally unusual group, if you take a global or historical perspective. All the subjects studied were “social drinkers”, presumably with some learnt associations about curved and straight glasses. Maybe the Brits had learnt that expensive beer came in curved glasses. If this is the case, the result might be true for everyone who has a history of drinking from straight glasses in the UK, but not for other cultures where alcohol isn’t drunk like that.
Little things, big effect
Software entrepreur Jim Manzi calls the rate at which small changes can have surprising effects on outcomes, and the consequent difficulty in drawing general conclusions, “causal density”. It’s because human psychology and social life is so causally dense that we can’t simply take straight reports that X affects Y and apply them across the board. But there are hundreds of these relationships reported all the time from the annals of psychology: glass shape affects drinking time, taller men are better paid, holding a hot drink makes you like someone, and so on. Surface effects like these are vulnerable to small changes in circumstances that might remove, or even reverse, the effect you’re relying on.
Psychology researchers know all these arguments, and that’s why they’re cautious about drawing simple conclusions from single studies. The challenge of psychology is to track down those results that actually do generalise across different situations.
The way to do this is to report findings that are about theories, not just about effects. The Bristol researchers show the way in their paper: as well as testing drinking speed, they relate it to people’s ability to estimate how full a glass is. They could have just measured drinking speed, but they knew they had to relate it to a theory about what people really believed to come up with a strong conclusion.
If we can find the right principles that affect people’s actions, then we can draw conclusions that cut across situations. Unless we know the reasons why someone does something, we’ll be tricked time and time again when we try to infer from what they do.
The Lancet has a powerful essay on children born from rape and the social and psychological consequences for mother, child and community.
I’ll let the article speak for itself as it carefully articulates how the relationship between mother and child can be affected by these tragic events.
There is one point worth highlighting, however. The piece notes that when affected women do have contact with healthcare professionals, clinicians often avoid tackling problems with childcare because they are denied or ignored by the mothers who, understandably, find it difficult to address problems linked to such a violent and painful event.
The article notes that the wellbeing of the child is often not well addressed as “Many practitioners who care for women who have been raped maintain this silence because either their focus is on the well-being of the mother or they genuinely believe that the interests of the mother and child are not served by articulating relational difficulties”.
Mental health professionals rightly identify avoidance as one of the key factors that maintain problematic behaviours. It’s a strategy that places short-term comfort above longer-term well-being and we all use it, but when we rely it to manage serious emotional or behavioural difficulties it can mean we never recover.
But what is less admitted is that healthcare professionals also suffer from avoidance. We don’t like making people distressed, even when it is necessary to overcome serious difficulties. Consequently, we also avoid addressing painful issues, which is something that can also help maintain the problem in the person we are working with.
Ironically, it is very difficult to get healthcare professionals to recognise that they themselves are affected by this. We are much more comfortable when the problems are safely situated in the patient.
Link to Lancet article ‘Child in the shadowlands’ (via @EvaAlisic)
One of the most interesting commentaries I’ve ever read on Thomas Szasz, the long-time critic of psychiatry who recently passed away, has been left as a comment in the obituary we recently published.
The comment is by ‘Aporeticist’ and he or she is clearly a fierce critic of modern psychiatry (to the point of indulging in sweeping generalisations at times) but the analysis of Szasz is remarkably insightful and cuts to the core of both his triumphs and failings.
Many of Szasz’s early critics have over the years quietly come around to some of his basic views. (Karl Menninger was one of his colleagues who acknowledged his change of heart.) The notion that the great majority of people with mental illness should never be hospitalized against their will (even when they are troublesome to those around them) has become common sense. It remains one of the great injustices of history that the psychiatric establishment continues to refuse to credit Szasz with being the first member of his profession who, in the mid 1960s, stated on record — against the unanimous opinion of his colleagues — the revolutionary contention that homosexuality was not a disease, and that it didn’t warrant “treatment” of any kind.
The classical liberal notion of “live and let live” resonates closely with the “first, do no harm” of the Hippocratic Oath that Thomas Szasz took as a young medical doctor. For better or worse, Szasz remained consistently faithful to these principles of negative freedom his whole life. Those, however, who believe that, as individuals and as a society, we have a moral obligation to (somehow) assist the mentally ill even when they don’t reach out for support, would regard Szasz’s characterization of psychiatric paternalism as “cruel compassion” as equally descriptive of his own apparent lack of concern for the welfare of those labelled mentally ill. Szasz tirelessly defended the autonomy of even the most severely disturbed mental patients (so long as they didn’t violate the law), yet seemed to care little whether they live or die if no one infringed on their sacred negative rights.
Recommended. Thanks Aporeticist.
Link to commentary on Szasz’s legacy by Aporeticist.
Oxford neuropsychologist Dorothy Bishop has given a fantastic video lecture about how neuroscience can be misinterpreted and how it can be misleading.
If you check out nothing else, do read the summary on the Neurobonkers blog, which highlights Bishop’s four main criticisms of how neuroscience is misused.
But if you have the time, sit back and see the lecture in full.
The key is that these are not slip-ups only restricted to the popular press and self-help books – they are exactly the sort of poor reasoning about neuroscience that affects many scientists as well.
Essentially, if you get the Bishop’s four main points of how ‘neurosciency stuff leads to a loss of critical faculties’, you’re on fine form to separate the wheat from the chaff in the world of cognitive neuroscience.
Excellent stuff.
Link to coverage on the Neurobonkers blog.
Link to streamed video of the lecture.
The New York Times discusses Argentina’s love affair with psychoanalysis. A country that has more psychologists – the majority Freudian – than any other nation on Earth.
Argentina is genuinely unique with regard to psychology. Even in Latin America, where Freudian ideas remain relatively strong, Argentina remains a stronghold of the undiluted classic schools of psychoanalysis.
It is also unique in terms of the access people have to the practice. In the majority of the world, psychoanalysis is the reserve of the upper middle classes and aristocracy – both in terms of the analysts and the patients.
While the watered-down (some would say made sensible) psychodynamic psychotherapy is more widely available, psychoanalytic training and therapy is extremely expensive. You could easily spend a couple of thousand US dollars a month on therapy alone.
As trainees have to be taught, supervised and be in constant treatment themselves (although the latter usually at a discounted rate) it remains a practice by and for a very narrow group from society. If you want to see this for yourself, training institutes often have open evenings, which I highly recommend as an interesting anthropological field trip.
This elitism is much less the case in Argentina, however, meaning that people from all walks of life see psychoanalysts and Freudian-inspired commentary is an integral part of popular culture.
The NYT article is a little puzzled as to why psychoanalysis has gained such a foothold in the country. Of course, it received a great many psychoanalyst émigrés in the years surrounding the Second World War, as many were Jewish, but in covering similar ground myself, I wondered whether there are good psychological reasons for its continued popularity.
Link to NYT piece on psychoanalysis in Argentina.
Link to earlier piece by me on the same.
Posttraumatic stress disorder is one of the defining disorders of modern psychiatry. Although first officially accepted as a diagnosis in the early 1980s, many believe that it has always been with us, but two new studies suggest that this unlikely to be the case – it may be a genuinely modern reaction to trauma.
The diagnosis of PTSD involves having a traumatic experience and then being affected by a month of symptoms of three main groups: intrusive memories, hyper-arousal, and avoidance of reminders or emotional numbing.
It was originally called ‘post-Vietnam syndrome’ and was promoted by anti-war psychiatrists who felt that the Vietnam war was having a unique effect on the mental health of American soldiers, but the concept was demilitarised and turned into a civilian diagnosis concerning the chronic effects of trauma.
Since then there has been a popular belief that PTSD has been experienced throughout history but simply wasn’t properly recognised. Previous labels, it is claimed, like ‘shell shock’ or ‘combat fatigue’, were just early descriptions of the same universal reaction.
But until now, few studies have systematically looked for PTSD or post-trauma reactions in the older historical record. Two recent studies have done exactly this, however, and found no evidence for a historical syndrome equivalent to PTSD.
A study just published in the Journal of Anxiety Disorders looked at the extensive medical records for soldiers in the American Civil War, whose mortality rate was about 50-80 greater than modern soldiers fighting in Iraq and Afghanistan.
In other words, there would have been many more having terrifying experiences but despite the higher rates of trauma and mentions of other mental problems, there is virtually no mention of anything like the intrusive thoughts or flashbacks of PTSD.
In a commentary, psychologist Richard McNally makes the point that often these symptoms have to be asked about specifically to be detected, but even so, he too admits that the fact that PTSD-like symptoms virtually make no appearance in hundreds of thousands of medical records suggests that PTSD is unlikely to be a ‘universal timeless disorder’.
Taking an even longer view, a study published in Stress and Health looked at historical accounts of traumatic experiences from antiquity to the 16th century.
The researchers found that although psychological trauma has been recognised throughout history, with difficult events potentially leading to mental disorder in some, there were no consistent effects that resembled the classic PTSD syndrome.
Various symptoms would be mentioned at various times, some now associated with the modern diagnosis, some not, but it was simply not possible to find ‘historical accounts of PTSD’.
The concept of PTSD is clearly grounded in a particular time and culture, but even from a modern diagnostic perspective it is important to recognise that we tend to over-focus on PTSD as the outcome of horrendous events.
Perhaps the best scientific paper yet published on the diversity of trauma was an article authored by George Bonanno and colleagues in 2011. You can read the full-text online as a pdf.
It notes that the single most common outcome after a traumatic event is recovery without intervention, and for those who do remain affected, depression and substance abuse problems are equally, if not more likely, than a diagnosis of posttraumatic stress disorder.
Link to locked study on trauma in Civil War soldiers.
Link to locked study on trauma through history.
The New York Times has an excellent article about the fallacy of assuming that a brain-based explanation of behaviour automatically implies that the person is less responsible for their actions.
The piece is by two psychologists, John Monterosso and Barry Schwartz, who discuss their research on how attributions of blame can be altered simply by giving psychological or neurological explanations for the same behaviour.
The fallacy comes in, of course, because psychology and neuroscience are just different tools we use to describe, in this case, the same behaviour.
A brain characteristic that was even weakly associated with violence led people to exonerate the protagonist more than a psychological factor that was strongly associated with violent acts….
We labeled this pattern of responses “naïve dualism.” This is the belief that acts are brought about either by intentions or by the physical laws that govern our brains and that those two types of causes — psychological and biological — are categorically distinct. People are responsible for actions resulting from one but not the other. (In citing neuroscience, the Supreme Court may have been guilty of naïve dualism: did it really need brain evidence to conclude that adolescents are immature?)
Naïve dualism is misguided. “Was the cause psychological or biological?” is the wrong question when assigning responsibility for an action. All psychological states are also biological ones.
A better question is “how strong was the relation between the cause (whatever it happened to be) and the effect?”
In light of the Aurora shootings and the prematurely and already misfiring debate about the shooter’s ‘brain state’, this is well worth checking out.
Link to NYT piece ‘Did Your Brain Make You Do It?’ (via @TheNeuroTimes)
Somatosphere has a fantastic account of the debates rocking the world of global mental health – the still nascent field that aims to make mental health a world priority.
The idea itself is sound in the general sense, but there is still a lot of argument about what it means to promote mental health and much discussion about whether ‘global mental health’ is just a means of exporting Western ideas and diagnoses in a sort of 21st century globalisation of the mind.
I am always a little struck by the fact that the ‘global mental health’ movement seems mainly to focus on Asia and Africa.
For example, the lack of participation of Latin American mental health professionals and advocates is striking in both the headline-making publications and the key conferences.
This is a pity as Latin America has developed a unique perspective on mental health that, by reading the debates covered by Somatosphere, would be very relevant.
If you want to get your head into the space of this particular Latin American approach, have a think about this analogy.
How would you react if instead of supporting the American civil rights movement in the 1960s, you were told the major problem was that people were being affected by a mental illness called ‘post-discrimination stress disorder’?
I’m sure it would be possible to come up with a valid and reliable ‘PDSD’ diagnosis that could be agreed upon and would genuinely predict behavioural and psychological distress and impairment – the experience of racism is known to predict mental health problems and the discrimination of the civil rights era was extreme.
Arguing for more resources to be put into treating ‘post-discrimination stress disorder’ when the civil rights movement was almost at breaking point in the 1940s and 50 would lay you open to accusations of ‘putting up a smoke screen’ and ‘making a distraction’ when what was needed was social change, not an attempt to pathologise black people.
The question that you may be asking, and many Latin American psychologists have asked, is whether we should be instead focussing on inequality and violence to improve mental health.
The Western focus on disorders, they argue, can distract and blind us to societal problems. Instead of preventing oppression, we pathologise its victims.
This approach was born out of a field called liberation psychology that made exactly this point.
One of the founders was a remarkable chap called Ignacio Martín-Baró who was a Spanish priest who trained as a psychologist and worked in El Salvador during the Salvadoran civil war.
He was eventually murdered by a government death squad because of his theories, which at least shows their power if not their popularity.
Unfortunately, liberation psychology has become heavily politicised and you often hear variations of “Martín-Baró’s work means you must support my left-wing views” from proponents.
This is a shame because Martín-Baró’s work was often making a more profound and over-arching point – that there is no such thing as an apolitical act in mental health, and, indeed, in health care in general.
For example, the West’s understanding of the victims of war, torture and displacement in terms of PTSD and other diagnostic labels is largely due to the experience of treating refugees who have fled these horrible situations.
In this context, PTSD makes sense in the West because it has the implicit assumption that the person is now safe (after all, it’s post-traumatic stress disorder) and that the experiences and reactions described in the diagnosis are, therefore, inappropriate.
However, if you live in a war zone, intrusive thoughts, feeling on edge and avoiding reminders of danger could be considered quite a reasonable reaction to the constant experience of death and violence.
When you meet people who do live in war zones, who would clearly meet the criteria for PTSD, they rarely complain about their mental state. They’re usually more concerned about the actual dangers.
They’re concerned about torture, not intrusive thoughts about being tortured – the threat of rape, not rape-related anxiety.
So, the hard question becomes: are we really helping by sending professionals and training locals to recognise and treat people with, for example, PTSD?
And this is where Martín-Baró drew his inspiration from. The way we understand and treat mental health problems, he argued, is always political. There is no absolute neutrality in how we understand distress and those that think so are usually just blind to their own biases.
And this is what the global mental health movement is wrestling with. And needless to say, there are plenty of biases to overcome.
Big Pharma pushes theories as adverts for its medication. Western mental health professionals can see themselves as healers of people who don’t necessarily need healing.
Researchers see an untapped gold mine of data and local scientists see a way out of what seems like a limiting and unglamorous academic life distant from the shining lights of Northern Hemisphere High Science.
So when we talk about ‘mental health literacy’ are we talking education or propaganda? It’s not an easy question to answer or, for many, to even think about.
The Somatosphere piece is one of the best guides to this debate I’ve yet read. Essential reading.
Link to Somatosphere on ‘Global Mental Health and its Discontents’
The latest issue of The Psychologist has hit the shelves and it has a freely available and suprisingly thought-provoking article about bathroom psychology.
If you’re thinking it’s an excuse for cheap jokes you’d be mistaken as takes a genuine and inquisitive look at why so little psychology, Freud excepted, has been concerned with one of our most important bodily functions.
This part, on the history of theories regarding graffiti found in toilets, is as curious is it is bizarre.
Toilet graffiti, dubbed ‘latrinalia’ by one scholar, has drawn attention from many researchers and theorists over the years. Many of them have focused on gender, using public lavatories as laboratories for studying sex differences in the content and form of these scribblings. Alfred Kinsey was one of the first researchers to enter the field, surveying the walls of more than 300 public toilets in the early 1950s and finding more erotic content in men’s and more romantic content in women’s. Later research has found that men’s graffiti also tend to be more scatological, insulting, prejudiced, and image-based, and less likely to offer advice or otherwise respond to previous remarks.
Theorists have struggled to explain differences such as these. True to his time, Kinsey ascribed them to women’s supposedly greater regard for social conventions and lesser sexual responsiveness. Psychoanalytic writers proposed that graffiti writing was a form of ‘phallic expression’ or that men pursued it out of an unconscious envy of women’s capacity for childbirth. Semioticians argued that men’s toilet graffiti signify and express political dominance, whereas women’s respond to their subordination. Social identity theorists proposed that gender differences in latrinalia reflect the salience of gender in segregated public bathrooms: rather than merely revealing their real, underlying differences, women and men polarise their behaviour in these gender-marked settings so as to exaggerate their femaleness or maleness.
The article looks at many other curious episodes in the bashful psychology of the bathroom.
Link to The Psychologit on ‘toilet psychology’
Yesterday’s ‘troubles with fMRI’ article has caused lots of debate so I thought I’d post the original answers given to me by neuroimagers Russ Poldrack and Tal Yarkoni from which I quoted.
Poldrack and Yarkoni have been at the forefront of finding, fixing and fine-tuning fMRI and its difficulties. I asked them about current challenges but could only include small quotes in The Observer article. Their full answers, included below with their permission, are important and revealing, so well worth checking out.
First, however, a quick note about the reactions the piece has received from the neuroimaging community. They tend to be split into “well said” and “why are you saying fMRI is flawed?”
Because of this, it’s worth saying that I don’t think fMRI or other imaging methods are flawed in themselves. However, it is true that we have discovered that a significant proportion of the past research has been based on potentially misleading methods.
Although it is true that these methods have largely been abandoned there still remain some important and ongoing uncertainties around how we should interpret neuroimaging data.
As a result of these issues, and genuinely due to the fact that brain scans are often enchantingly beautiful, I think neuroimaging results are currently given too much weight as we are trying to understand the brain but that we shouldn’t undervalue neuroimaging as a science.
Despite having our confidence shaken in past studies, neuroimaging will clearly come out better and stronger as a result of current debates about problems with analysis and interpretation.
At the moment, the science is at a fascinating point of transition, so it’s a great time to be interested in cognitive neuroscience and I think this is made crystal clear from Russ and Tal’s answers below.
Russ Poldrack from the University of Texas Austin
What’s the most pressing problem fMRI research needs to address at the moment?
I think that biggest fundamental problem is the great flexibility of analytic methods that one can bring to bear on any particular dataset; the ironic thing is that this is also one of fMRI’s greatest strengths, i.e., that it allows us to ask so many different questions in many different ways. The problem comes about when researchers search across many different analysis approaches for a result, without the realization that this induces an increase in the ultimate likelihood of finding a false positive. I think that another problem that interacts with this is the prevalence of relatively underpowered studies, which are often analyzed using methods that are not stringent enough to control the level of false positives. The flexibility that I mentioned above also includes methods that are known by experts to be invalid, but unfortunately these still get into top journals, which only helps perpetuate them further.
Someone online asked the question “How Much of the Neuroimaging Literature Should We Discard?” How do you think should we consider past fMRI studies that used problematic methodology?
I think that replication is the ultimate answer. For example, the methods that we used in our 1999 Neuroimage paper that examined semantic versus phonological processing seem pretty abominable by today’s standards, but the general finding of that paper has been replicated many times since then. There are many other findings from the early days that have stood the test of time, while others have failed to replicate. So I would say that if a published study used problematic methods, then one really wants to see some kind of replication before buying the result.
Tal Yarkoni from the University of Colorado at Boulder
What’s the most pressing problem fMRI research needs to address at the moment?
My own feeling (which I’m sure many people would disagree with) is that the biggest problem isn’t methodological laxness so much as skewed incentives. As in most areas of science, researchers have a big incentive to come up with exciting new findings that make a splash. What’s particularly problematic about fMRI research–as opposed to, say, cognitive psychology–is the amount of flexibility researchers have when performing their analyses. There simply isn’t any single standard way of analyzing fMRI data (and it’s not clear there should there be); as a result, it’s virtually impossible to assess the plausibility of many if not most fMRI findings simply because you have no idea how many things the researchers tried before they got something to work.
The other very serious and closely related problem is what I’ve talked about in my critique of Friston’s paper [on methods in fMRI analysis] as well as other papers (e.g., I wrote a commentary on the Vul et al “voodoo correlations” paper to the same effect): in the real world, most effects are weak and diffuse. In other words, we expect complicated psychological states or processes–e.g., decoding speech, experiencing love, or maintaining multiple pieces of information in mind–to depend on neural circuitry widely distributed throughout the brain, most of which are probably going to play a relatively minor role. The problem is that when we conduct fMRI studies with small samples at very stringent statistical thresholds, we’re strongly biased to detect only a small fraction of the ‘true’ effects, and because of the bias, the effects we do detect will seem much stronger than they actually are in the real world. The result is that fMRI studies will paradoxically tend to produce *less* interesting results as the sample size gets bigger. Which means your odds of getting a paper into a journal like Science or Nature are, in many cases, much higher if you only collect data from 20 subjects than if you collect data from 200.
The net result is that we have hundreds of very small studies in the literature that report very exciting results but are unlikely to ever be directly replicated, because researchers don’t have much of an incentive to collect the large samples needed to get a really good picture of what’s going on.
Someone online asked the question “How Much of the Neuroimaging Literature Should We Discard?” How do you think should we consider past fMRI studies that used problematic methodology?
This is a very difficult question to answer in a paragraph or two. I guess my most general feeling is that our default attitude to any new and interesting fMRI finding should be skepticism–instead of accepting findings at face value until we discover a good reason to discount them, we should incline toward disbelief until a finding has been replicated and extended. Personally I’d say I don’t really believe about 95% of what gets published. That’s not to say I think 95% of the literature is flat-out wrong; I think there’s probably a kernel of truth to most findings that get published. But the real problem in my view is a disconnect between what we should really conclude from any given finding and what researchers take license to say in their papers. To take just one example, I think claims of “selective” activation are almost without exception completely baseless (because very few studies really have the statistical power to confidently claim that absence of evidence is evidence of absence).
For example, suppose someone publishes a paper reporting that romantic love selectively activates region X, and that activation in that region explains a very large proportion of the variance in some behavior (this kind of thing happens all the time). My view is that the appropriate response is to say, “well, look, there probably is a real effect in region X, but if you had had a much larger sample, you would realize that the effect in region X is much smaller than you think it is, and moreover, there are literally dozens of other regions that show similarly-sized effects.” The argument is basically that much of the novelty of fMRI findings stems directly from the fact that most studies are grossly underpowered. So really I think the root problem is not that researchers aren’t careful to guard against methodological problems X, Y, and Z when doing their analyses; it’s that our mental model of what most fMRI studies can tell us is fundamentally wrong in most cases. A statistical map of brain activity is *not* in any sense an accurate window into how the brain supports cognition; it’s more like a funhouse mirror that heavily distorts the true image, and to understand the underlying reality, you also have to take into account the distortion introduced by the measurement. The latter part is where I think we have a systemic problem in fMRI research.
I’ve written a piece for The Observer about ‘the trouble with brain scans’ that discusses how past fMRI studies may have been based on problematic assumptions.
For years the media has misrepresented brain scan studies (“Brain centre for liking cheese discovered!”) but we are now at an interesting point where neuroscientists are starting to seriously look for problems in their own methods of analysis.
In fact, many of these problems have now been corrected, but we still have 100s or 1000s of previous studies that have been based on methods that have now been abandoned.
In part, the piece was inspired by a post on the Neurocritic blog entitled “How Much of the Neuroimaging Literature Should We Discard?” that was prompted by growing concerns among neuroscientists.
The fact is, fMRI is a relatively new science – it just celebrated it’s 20th birthday – and it is still evolving.
I suspect it will be revised and reconsidered many times yet.
Link to Observer article ‘The Trouble With Brain Scans’
I’ve written an article for the Discover Magazine’s blog The Crux on what the DSM diagnostic manual is supposed to do.
This is quite an interesting question when you think about it. In other words, it asks – how do we define mental illness – both in theory and in practice?
The article tackles how you decide what a mental illness is in the first place and then how you go about classifying mental states that, by definition, can only be experienced by one person. It turns out, classifying mental illness is a lot like classifying literature.
It also discusses the old and possibly futile quest for ‘biological tests for mental illness’ as if there is a perfect mapping between how we classify mental states and how the brain actually works at the neurobiological level.
So if you want to know the thinking and, indeed, problems behind one of the central and often unquestioned assumptions of psychiatry, this should be a good place to start.
Link to ‘What Is the “Bible of Psychiatry” Supposed to Do?’
Don’t miss an important article in this week’s Nature about how psychologists are facing up to problems with unreplicated studies in the wake of several high profiles controversies.
Positive results in psychology can behave like rumours: easy to release but hard to dispel. They dominate most journals, which strive to present new, exciting research. Meanwhile, attempts to replicate those studies, especially when the findings are negative, go unpublished, languishing in personal file drawers or circulating in conversations around the water cooler…
One reason for the excess in positive results for psychology is an emphasis on “slightly freak-show-ish” results, says Chris Chambers, an experimental psychologist at Cardiff University, UK. “High-impact journals often regard psychology as a sort of parlour-trick area,” he says. Results need to be exciting, eye-catching, even implausible. Simmons says that the blame lies partly in the review process. “When we review papers, we’re often making authors prove that their findings are novel or interesting,” he says. “We’re not often making them prove that their findings are true.”
It’s perhaps worth noting that clinical psychology suffers somewhat less from this problem, as treatment studies tend to get replicated by competing groups and negative studies are valued just as highly.
However, it would be interesting to see whether the “freak-show-ish” performing pony studies are less likely to replicate than specialist and not very catchy cognitive science (dual-process theory of recognition, I’m looking at you).
As a great complement to the Nature article, this month’s The Psychologist has an extended look at the problem of replication [pdf] and talks to a whole range of people affected by the problem, from journalists to research experts.
But I honestly don’t know where this ‘conceptual replication’ thing came from – where you test the general conclusion of a study in another form – as this just seems to be a test of the theory with another study.
It’s like saying your kebab is a ‘conceptual replication’ of the pizza you made last night. Close, but no neopolitana.
Link to Nature article on psychology and replication.
pdf of Psychologist article ‘Replication, replication, replication’
Mind Hacks 