This seems to lead to another problem: the closed circle of fMRI researchers vetting each others’ work. I presume not many people outside of fMRI work have used something like MVPA enough to understand it and ensure it’s properly employed. While I don’t mean to imply there’s some kind of fMRI old boys’ club, I can understand Tal Yarkoni’s skepticism since it’s probably counterproductive for an fMRI-using reviewer to tear apart a submission to a high tier journal.

I feel like a spammer at the moment because whenever these kinds of discussions crop up I always post the following link:
http://neuroskeptic.blogspot.co.uk/2012/04/fixing-science-systems-and-politics.html

Which is where I argue that we ought to pre-register scientific studies so that the original methods and analyses are publically available, and post hoc “fishing” can be distinguished from hypothesis-driven a priori stuff.

It’s interesting that both Tal & Russell point to the flexibility of fMRI analyses as the primary problem with the field – this flexibility could be a very good thing (it means the methods are versatile) but it easily becomes a problem when people are able to selectively publish only the significant findings.

Preregistration would help mitigate this – although it’s not a panacea and replication would still be important.

Re Tom Michael’s comment about brain injury studies, it’s always good to look at a scientific issue from multiple angles, and brain injury studies are a useful example here – but carry their own problems (damage not limited to area under study, brain adapting to damage in unpredictable ways, unknown extra hidden damage, e.g. in white matter tracts, and so on). Neuroimaging, especially fMRI, is an incredibly useful, powerful method potentially, but can only live up to this potential if various things improve, including better training, more valid application of techniques, more replication, and generally more rigour at all stages.

Just to flag up that I wrote a long blogpost about these problems, along with potential solutions (with an extensive comment discussion, including from Russ Poldrack and Tal Yarkoni) here:
http://www.danielbor.com/dilemma-weak-neuroimaging/

And there was a great special issue of this month’s The Psychologist magazine, with many articles generally discussing the problem of replication in psychology, with one article (by me) specifically describing problems and potential solutions in neuroimaging. You can access this issue here:
http://issuu.com/thepsychologist/docs/0512?mode=embed&layout=http%3A%2F%2Fskin.issuu.com%2Fv%2Fdark%2Flayout.xml&showFlipBtn=true&proShowSidebar=true

the coloured blobs represent a “pleasure centre”, an “art centre” or perhaps a “love centre”

But rather a basic misrepresentations of causality, presumably because of the pressure to over-simplify for catchy sound bites [for a nice discussion of such nefarious pressures on science, see: http://www.guardian.co.uk/science/blog/2011/feb/16/thinking-caps-pseudoscience-neuroscience%5D.

Yes, stats are very important, but we should not forget about logic along the way.

For some more discussion related to Vaughan’s article: http://the-brain-box.blogspot.co.uk/2012/05/truth-before-beauty-making-sense-of.html

The power problem (i.e. that there are not enough participants to be sure that we are not getting false positive, or false negative results) is not unique to fMRI. As we psychologists are aware, the vast majority of psychological studies have very small sample sizes. The general public may not be aware of this however. In this respect, some of the best psychology studies I’ve seen have been carried out by the dating website OK Cupid – with sample sizes of 100,000+

The other problem with fMRI as I see it is the association-cause problem. Yes, BOLD signals may be a good proxy for cognition, but just because there is activation in brain area X when we engage in behaviour Y doesn’t mean that brain area X causes cognition Z. For this reason, I tend to trust interpretations of double-dissociation studies with brain injured people much more than imaging methods.

However, the power problem is sometimes even worse in brain injury studies (I am using qualitative data and case studies for part of my PhD for example). Also, one thing that fMRI has going for it is that it is truly experimental, particularly if methods like TMS are used in conjunction with fMRI. In comparison, double-dissociation brain injury studies are by their nature pseudo-experimental, as we have to find brain injured participants who have been unfortunate enough to have suffered a brain injury of the type we are interested in studying. This is another reason why our sample sizes are often quite small.