Voodoo correlations in social brain studies

Image by Ben Mathis: poopinmymouth.comI’ve just come across a bombshell of a paper that looked at numerous headline studies on the cognitive neuroscience of social interaction and found that many contained statistically impossible or spurious correlations between behaviour and brain activity.

The article is currently ‘in press’ for the journal Perspectives on Psychological Science but the preprint is available online as a pdf file.

Social cognitive neuroscience is a hot new area and many of the headline studies use fMRI brain imaging to look at how activity in the brain is correlated with social decision-making or perception.

This new analysis, led by neuroscientist Edward Vul, was inspired by the fact that some of these correlations seem to good to be true, and so the research team investigated. The abstract of their study is below, and it’s powerful stuff – indicating that many of the results are due to flawed analyses.

If you’re not familiar with neuroimaging research it might be useful to know that what a ‘voxel‘ is before reading the abstract.

Essentially, brain scanners digitally divide the scanned area into a block of tiny boxes and each one of these is called a voxel (think 3D pixel).

This allows the scans to be analysed by comparing the activity or tissue density in each voxel to another measure – which could be the same voxel during another scan, or it could be something entirely different, such as a measure of emotion or social decision-making.

The newly emerging field of Social Neuroscience has drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between behavioral and self-report measures of personality or emotion and measures of brain activation obtained using fMRI. We show that these correlations often exceed what is statistically possible assuming the (evidently rather limited) reliability of both fMRI and personality/emotion measures. The implausibly high correlations are all the more puzzling because social-neuroscience method sections rarely contain sufficient detail to ascertain how these correlations were obtained.

We surveyed authors of 54 articles that reported findings of this kind to determine the details of their analyses. More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases.

We outline how the data from these studies could be reanalyzed with unbiased methods to provide the field with accurate estimates of the correlations in question. We urge authors to perform such reanalyses and to correct the scientific record.

The paper notes that some of the most widely-reported studies in recent years contain this flaw and this new paper has the potential to really shake up the world of social cognitive neuroscience.

pdf of preprint of ‘Voodoo Correlations in Social Neuroscience’.

8 Comments

  1. hermit
    Posted December 31, 2008 at 12:15 am | Permalink

    about time

    • Posted June 21, 2012 at 12:24 pm | Permalink

      My 36 yr old girlfriend is laying in a hosp bed about 2 die.The doctor’s cannot find out what is wrong with her,her husband say’s is some type of psychosis,she told me what it is,she traveled 2 Jamaica 2 visit a man she met online,To make a long story short,The family of this man hated her because she was american,and did not want her 2 return 2 Jamaica 2 marry the brother- son.She returned back here 2 CT,and she fell out on the floor in excruciating pain,went 2 the doc,they couldnt find anything wrong w/her.She ended up in a wheelchair not being able 2 walk.Wich means no travling back 2 Jamaica.All of her organs are failing ,she is pretty much paralyzed and she is on life support.They still dont know how this happened.She told me they put roots on her,she knows this for a fact.Is thier anything I can do 2 remove these root’s or voodoo from her body?Please help,my number is 860-477-7175Thank You Lisa,She doesnt have much longer maby a few days.

  2. alex
    Posted January 15, 2009 at 3:15 pm | Permalink

    i work in the field though am not involved directly – and i’m a fan of mind hacks. but the issues aren’t as clear cut as the authors of ‘voodoo correlations’ suggest – a response from some of the ‘red list’ authors is now online at http://www.bcn-nic.nl/replyVul.pdf. there is more as well at http://www.nature.com/news/2009/090114/pdf/457245a.pdf. i hope the owner as well as readers of the site will have a look.

  3. Dale
    Posted January 26, 2009 at 6:15 pm | Permalink

    I agree with the point made by Alex. The issues are definitely not as clear cut as implied by the authors of “voodoo correlations”. The response from the red list authors that Alex posted above makes at least as compelling of a case in favor of their approach. Moreover, an actual read through of the studies that the authors of “voodoo correlations” attack persuaded me that the approach taken by many of the red list studies was actually quite sound and definitely not nearly as inappropriate as one might gather from only reading the “voodoo” paper. The “voodoo” paper may be stretching it a bit. So take it with a grain of salt.

  4. Dale
    Posted January 27, 2009 at 10:49 pm | Permalink

    Another response paper to the voodoo correlation issue was recently posted on the web by some of the criticized authors. I definitely recommend checking it out as it makes a strong case that there were some flaws with the Vul et al methods, analysis, and conclusions. This is further evidence that some of the studies may have been misconstrued or unjustly criticized by Vul and colleagues. The response can be found at:
    http://www.scn.ucla.edu/pdf/LiebermanBerkmanWager(invitedreply).pdf

  5. Johnny Blaze
    Posted October 5, 2009 at 9:24 pm | Permalink

    We had done some research using MRI brain imaging as a means of tracking Nicotine addiction when developing our electronic cigarette product. We wanted to test how the brain interpreted the use of an electronic cigarette versus that of a traditional cigarette. The technology is unbelievable and what you call voxels, we actually referred to as cubix. We actually found that there was no major difference brain activity between e-cigarettes and tobacco cigs when conducting our studies. Very cool technology that wasn’t even a thought 10 years ago.
    Johnny B
    Director Of Product Development
    Halo Electronic Cigarette Company

  6. Posted August 5, 2010 at 12:46 am | Permalink

    Why is there more evidence that some of the studies have been misconstrued or unjustly criticized by Vul and colleagues


Post a Comment

Required fields are marked *
*
*

Follow

Get every new post delivered to your Inbox.

Join 2,569 other followers