Mind Hacks

There is a clich√© in media stories where figures for a disease or condition are quoted followed by a statement that “the true figures may be higher”. Sampling errors mean that initial figures are equally as likely to be under-estimates as over-estimates but we only ever seem to be told that the condition is under-detected.

For example, this is from a recent (actually pretty good) New Scientist article about gender identity disorder (GID) in children, a condition where children who are biologically male feel female and vice versa:

It is unclear how common GID is among children, but many transsexual adults say they felt they were “in the wrong body” from an early age. The incidence of adult transsexualism has been estimated at about 1 in 12,000 for male-to-females, and around in 1 in 30,000 for female-to-males, although transsexual lobby groups say the true figures may be far higher.

These estimates are usually drawn from prevalence studies where a maybe a few hundred or thousand people are tested. The researchers extrapolate from the number of cases to make an estimate of how many people in the population as a whole will have the condition.

These estimates are made with statistical tests which give a margin of error, meaning that within a certain range, typically described by confidence intervals, the real figure is likely to be between a range which equally includes both higher and lower values than the quoted amount.

For any individual study you can validly say that you think the estimate is too low, or indeed, too high, and give reasons for that. For instance, you might say that your sample was mainly young people who tend to be healthier than the general public, or maybe that the diagnostic tools are known to miss some true cases.

But when we look at reporting as a whole, it almost always says the condition is likely to be much more common than the estimate.

For example, have a look at the results of this Google search:

“the true number may be higher” 20,300 hits

“the true number may be lower” 3 hits

You can try variations on the phrasing and see the same sort of pattern emerges. I’m curious as to why this bias occurs, or whether there’s another explanation for it.

ABC Radio National’s All in the Mind has just had an excellent programme on ‘the singularity‘, the idea that at some point in the future computer power will outstrip the ability of the human brain and then humanity will be better off in some sort of vague and unspecified way.

The idea, is of course, ludicrous and is based on a naive notion that intelligence can measured as a type of unitary ‘power’ which we can adequately compare between computer and humans. The discussion on All in the Mind is a solid critical exploration of this wildly left-field notion as well as the community from whence it comes.

It’s a popular theme among transhumanists who, despite seeming to have a mortal fear of human limitations, I quite like.

Transhumanists are like the eccentric uncle of the cognitive science community. Not the sort of eccentric uncle who gets drunk at family parties and makes inappropriate comments about your kid sister (that would be drug reps), but the sort that your disapproving parents thinks is a bit peculiar but is full of fascinating stories and interesting ideas.

They occasionally take themselves too seriously and it’s the sort of sci-fi philosophy that has few practical implications but it’s enormously good fun and is great for making you re-evaluate your assumptions.

By the way, there’s loads of extras on the AITM blog, so do check it out.

Link to All in the Mind on ‘the singularity’.
Link to extras on AITM blog.

I’ve just a read a fantastic New York Times article from last year on the ongoing $1,000,000 Netflix challenge to create an algorithm that will predict what unseen films customers will liked based on their past preferences.

As well as an interesting insight into how companies are trying to guess our shopping preferences it is also a great guide to one of the central problems in scientific psychology: how we can reconcile numerical data with human thought and behaviour.

The Netflix prize teams have a bunch of data from customers who have rated films they’ve already seen and they have been challenged to write software that predicts future ratings.

Part of this process is hypothesis testing, essentially an experimental approach to find out what might be important in the decision process. For example, a team might guess that women will rate musicals higher than men. They can then test this prediction out on the data, making further predictions based on past conclusions, theories or even just hunches.

The other approach is to use mathematical techniques that look for patterns in the data. To use the jargon, these procedures look for ‘higher order properties’ – in other words, patterns in the patterns of data.

Think of it like looking at the relationship between different forests rather than thinking of everything as individual trees.

The trouble is, is that these mathematical procedures can sometimes find reliable high level patterns when it isn’t obvious to us what they represent. For example, the article discusses the use of a technique called singular value decomposition (SVD) to categorise movies based on their ratings;

There’s a sort of unsettling, alien quality to their computers’ results. When the teams examine the ways that singular value decomposition is slotting movies into categories, sometimes it makes sense to them — as when the computer highlights what appears to be some essence of nerdiness in a bunch of sci-fi movies. But many categorizations are now so obscure that they cannot see the reasoning behind them. Possibly the algorithms are finding connections so deep and subconscious that customers themselves wouldn’t even recognize them.

At one point, Chabbert showed me a list of movies that his algorithm had discovered share some ineffable similarity; it includes a historical movie, “Joan of Arc,” a wrestling video, “W.W.E.: SummerSlam 2004,” the comedy “It Had to Be You” and a version of Charles Dickens’s “Bleak House.” For the life of me, I can’t figure out what possible connection they have, but Chabbert assures me that this singular value decomposition scored 4 percent higher than Cinematch — so it must be doing something right. As Volinsky surmised, “They’re able to tease out all of these things that we would never, ever think of ourselves.” The machine may be understanding something about us that we do not understand ourselves.

In these cases, it’s tempting to think there’s some deeply psychological property of the film that’s been captured by the analysis. Maybe all trigger a wistful nostalgia, or perhaps each represents the same unconscious fantasy.

It could also be that each is under 90 minutes, or comes with free popcorn. It could even be that the grouping is entirely spurious and represents nothing significant. Importantly, the answer to these questions is not in the data to be discovered, we have to make the interpretation ourselves.

Experimental methods go from meaning to data, while exploratory methods go from data to meaning. Somewhere in the middle is our mind.

The Netflix challenge is this problem on steroids and the NYT piece brilliantly explores the practical problems in making sense of it all.

Link to NYT piece ‘If You Liked This, You‚Äôre Sure to Love That’

Neuroskeptic covers a hilarious new study that involved brain scanning a dead salmon and finding activation in the brain as it ‘looked’ at photos of human faces.

The authors are not genuinely arguing that dead fish have brain activity but have run the experiment to show that some common statistical methods used in fMRI research will give false positives if they’re not adequately controlled for.

The research, led by neuroscientist Craig Bennett, was presented as a poster at a recent conference and has the brilliant title of “Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction” and is available online as a jpg.

I’d say that this research was justified on comedic grounds alone, but they were also making an important scientific point. The (fish-)bone of contention here is multiple comparisons correction. The “multiple comparisons problem” is simply the fact that if you do a lot of different statistical tests, some of them will, just by chance, give interesting results.

Most statistics used in psychology, and indeed brain imaging, are based on calculating a p value.

Usually, a p value of less than 0.05 is considered significant and this means that if there was genuinely no difference in the things you were comparing, you would get a false positive less than 5% of the time.

But your average fMRI brain scan analysis can involve 40,000 comparisons, so even if there’s nothing going on, some bits of the brain are going to seem active just through falsely detecting noise and measurement error as real effect.

To help prevent this, you can correct for multiple comparisons by reducing the 5% cut-off to a smaller amount. Unfortunately, some of the standard methods of doing this can be so strict as to create false negatives, when genuine differences are dismissed as statistical noise.

There is no hard and fast rule about which methods to use, but our salmon neuroscientists have graphically illustrated how misleading results can occur if we naively assume that not correcting accounting ‘multiple comparisons problem’ will give us an accurate picture of brain function.

Kudos to the Neuroskeptic blog for picking up on this and for some excellent coverage of this study.

Link to Neuroskeptic on dead salmon study.
jpg of conference poster.