The Social Priming Studies in “Thinking Fast and Slow” are not very replicable

train_wreck_at_montparnasse_1895In Daniel Kahneman’s “Thinking Fast and Slow” he introduces research on social priming – the idea that subtle cues in the environment may have significant, reliable effects on behaviour. In that book, published in 2011, Kahneman writes “disbelief is not an option” about these results. Since then, the evidence against the reliability of social priming research has been mounting.

In a new analysis, ‘Reconstruction of a Train Wreck: How Priming Research Went off the Rails‘, Ulrich Schimmack, Moritz Heene, and Kamini Kesavan review chapter 4 of Thinking Fast and Slow, picking out the references which provide evidence for social priming and calculating how statistically reliable they:

Their conclusion:

The results are eye-opening and jaw-dropping.  The chapter cites 12 articles and 11 of the 12 articles have an R-Index below 50.  The combined analysis of 31 studies reported in the 12 articles shows 100% significant results with average (median) observed power of 57% and an inflation rate of 43%.  …readers of… “Thinking Fast and Slow” should not consider the presented studies as scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness.

The argument is that the pattern of 100% significant results is near to impossible, even if the effects known were true, given the weak statistical power of the studies to detect true effects.

Remarkably, Kahneman responds in the comments:

What the blog gets absolutely right is that I placed too much faith in underpowered studies. …I have changed my views about the size of behavioral priming effects – they cannot be as large and as robust as my chapter suggested.

The original analysis, and Kahneman’s response are worth reading in full. Together they give a potted history of the replication crisis, and a summary of some of the prime causes (e.g. file draw effects), as well as showing off how mature psychological scientists can make, and respond to critique.

Original analysis: ‘Reconstruction of a Train Wreck: How Priming Research Went off the Rails‘, Ulrich Schimmack, Moritz Heene, and Kamini Kesavan. (Is it a paper? Is it a blogpost? Who knows?!)

Kahneman’s response

Sex differences in cognition are small

Lately I’ve been thinking about sex differences in brain and cognition. There are undeniable differences in the physical size of the brain, and different brain areas, even if there are no ‘female’ and ‘male’ brains categorically. These physical differences do not translate directly into commensurate differences in cognition. Indeed, there is support for a ‘gender similarities hypothesis’ which asserts that on most measures there is no difference between men and women.

Most, but maybe not all. There are a few areas of fundamental cognitive ability where gender differences seem to persist – mental rotation, vocabulary and maybe maths. But these differences are small. To see how small, I put them on the same chart with the physical differences and a few other behavioural differences for perspective.

Standardised mean differences (Cohen’s d effect size) for various gender differences in brain, behaviour and cognition:

gender_effectsReferences and calculations at the end of this post, below the fold. And if you need a primer on what is meant by standardised difference then go here.

Even with these, small, observed differences in cognition, we don’t know what proportion is due to contingent facts, such as the different experience and expectations men and women encounter in their lifetimes, and what proportion is immutable consequence of genetic difference in sex.

One possibility for why there is a mismatch between physical differences in the brain and cognitive differences is the possibility that structural differences between male and female brains may actually serve to support functional similarity, not difference.

For more, so much more, on this, see the special issue of Journal of Neuroscience Research (January/February 2017) on An Issue Whose Time Has Come: Sex/Gender Influences on Nervous System Function.

Includes: Grabowska, A. (2017). Sex on the brain: Are gender‐dependent structural and functional differences associated with behavior?. Journal of Neuroscience Research, 95(1-2), 200-212.

Previously: Gender brain blogging

Continue reading “Sex differences in cognition are small”

The gender similarities hypothesis

cubeThere is a popular notion that men and women are very different in their cognitive abilities. The evidence for this may be weaker than you expect. Janet Hyde advances what she calls the ‘gender similarities hypothesis‘, ‘which holds that males and females are similar on most, but not all, psychological variables’. In a 2016 review she states:

According to meta-analyses, however, among both children and adults, females perform equally to males on mathematics assessments. The gender difference in verbal skills is small and varies depending on the type of skill assessed (e.g., vocabulary, essay writing). The gender difference in 3D mental rotation shows a moderate advantage for males.

So from three celebrated examples of differences in ability only two actually show a moderate gender difference. Other abilities show no or negligible gender differences, Hyde concludes. Gender differences in ability may be overinflated in the popular imagination.

Worth noting is that the name of the game here isn’t to find gender differences in behaviour. That’s too easy. Women wear more make-up for example, men are more likely to wear trousers. The game is to find a measure which reflects some more fundamental aspect of mental capacity. Hence the focus on vocabulary size, mental rotation ability, maths ability and the like. These may be less subject to the vagaries of exactly what is expected of each gender, but that’s a shaky assumption. Indeed, it would be weird if different roles and expectations for men vs women didn’t produce different motivations and opportunities for practice of cognitive abilities such as these.

The real challenge is to find immutable gender differences, or to track differences in how abilities develop under different conditions. Without this evidence, we’re not going to be sure which gender differences are immutable, and which are contingent on the specific psychological history of particular men and particular women living in our particular societies.

One way of addressing this challenge is to look at how gender differences change across different socities, or across time as society changes. A 2014 study, ‘The changing face of cognitive gender differences in Europe‘ did just that, showing that less gender-restricted educational opportunities tended to decrease some gender differences but not others. In other words, increasing equality in educational attainment magnified some differences between the sexes.

You can read my take on this in this piece for The Conversation : Are women and men forever destined to think differently?

The Gender Similarities Hypothesis: Hyde, J. S. (2005). The gender similarities hypothesis. American psychologist, 60(6), 581-592

2016 update: Hyde, J. S. (2016). Sex and cognition: gender and cognitive functions. Current opinion in neurobiology, 38, 53-56.

Previously: Gender brain blogging: Sex differences in brain size, no male and female brain types.

no male and female brain types

What would it mean for there to be a “male brain” or a “female brain”? Human genitals are mostly easy to categorise just by sight as either male or female. It makes sense to talk about there being different male and female types of genitals. What would it mean for the same to be true of brains? Daphna Joel and colleagues, in a 2015 paper Sex beyond the genitalia: The human brain mosaic have a proposal on what needs to hold for us to be able to say there are distinct male and female varieties of brains:

1. particular brain features must be highly dimorphic (i.e., little overlap between the forms of these features in males and females).
and
2. those features which are dimorphic must be consistent for each brain (i.e. a brain has only “male” or only “female” features).

They analyse MRI scans of 1400 human brains and find that these conditions don’t hold. There is extensive overlap, so that categorical brains, defined like this, just don’t exist. They write:

…analyses of internal consistency reveal that brains with features that are consistently at one end of the “maleness-femaleness” continuum are rare. Rather, most brains are comprised of unique “mosaics” of features, some more common in females compared with males, some more common in males compared with females, and some common in both females and males…Our study demonstrates that, although there are sex/gender differences in the brain, human brains do not belong to one of two distinct categories: male brain/female brain.

So the easy gender categorisation we can do on the genitals doesn’t translate to the (usually-unseen) anatomy of the brain. The ‘male/female brain’ doesn’t exist in the same way as the male/female sex organs.

Context for this is that there are differences between the average male and average female brain (for overall size, at least, these differences are large). Although there may not be categorical types, a follow up analysis showed that it is possible to classify the brains used in the Joel paper as belonging to a man or a women at somewhere between 69%-77% accuracy. A related study, on a different data set, claimed 93% classification accuracy.

Paper: Joel, D., Berman, Z., Tavor, I., Wexler, N., Gaber, O., Stein, Y., … & Liem, F. (2015). Sex beyond the genitalia: The human brain mosaic. Proceedings of the National Academy of Sciences, 112(50), 15468-15473.

Responses: Del Giudice, M., Lippa, R. A., Puts, D. A., Bailey, D. H., Bailey, J. M., & Schmitt, D. P. (2016). Joel et al.’s method systematically fails to detect large, consistent sex differences. Proceedings of the National Academy of Sciences, 113(14), E1965-E1965.

Chekroud, A. M., Ward, E. J., Rosenberg, M. D., & Holmes, A. J. (2016). Patterns in the human brain mosaic discriminate males from females. Proceedings of the National Academy of Sciences, 113(14), E1968-E1968.

The responses are linked to in Debra Soh’s LA Times article Are gender feminists and transgender activists undermining science?

Betteridge’s Law

Previously: gender brain blogging

Sex differences in brain size

Next time someone asks you “Are men and women’s brains different?”, you can answer, without hesitation, “Yes”. Not only do they tend to be found in different types of bodies, but they are different sizes. Men’s are typically larger by something like 130 cubic centimeters.

Not only are they actually larger, but they are larger even once you take into account body size (i.e. men’s brains are bigger even when accounting for the fact that heavier and/or taller people will tend to have bigger heads and brains, and than men tend to be heavier and taller than women). And this is despite the fact that there is no difference in size of brain at birth – the sex difference in brain volume development seems to begin around age two. (Side note: no difference in brain volume between male and female cats).

But is this difference in brain volume a lot? There’s substantial variation between individuals, as well as across the individuals of each sex. What does ~130cc mean in the context of this variation? One way of thinking about it is in terms of standardised effect size, which measures the size of a difference between the two population averages in standard units based on the variation within those populations.

Here’s a good example – we all know that men are taller than women. Not all men are taller than all women, but men tend to be taller. With the effect size, we can precisely express this vague idea of ‘tend to be’. The (Cohen’s d) effect size statistic of the height difference between men and women is ~1.72.

What this means is that the distribution of heights in the two populations can be visualised like this:

mf_heightsWith this spread of heights, the average man is taller than 95.7% of women.

Estimates of the effect size of total brain volume vary, but a reasonable value is about ~1.3, which looks like this:

mf_brainsThis means that the average man has a larger brain, by volume, than 90% of the female population.

For reference, psychology experiments typically look at phenomena with effet sizes of the order ~0.4 , which looks like this:

mf_0p4And which means that the average of group A exceeds 65.5% of group B.

In this context, human sexual dimorphism in brain volume is an extremely large effect.

So when they ask “Are men and women’s brains different?”, you can unhesitatingly say, “yes”. And when they ask “And what does that mean for differences in how they think” you can say “Ah, now that’s a different issue”.

Link: meta-analysis of male-female differences in brain structure:

Kristoffer Magnusson’s awesome interactive effect size visualisation

Previously: gendered brain blogging

Edit 8/2/17: Andy Fugard pointed out that there are many different measures of effect size, and I only discuss/use one: the Cohen’s d effect size. I’ve edited the text to make this clearer.

Edit 2 (8/2/17): Kevin Mitchell points out this paper that claims sex differences in brain size are already apparent in neonates

How to overcome bias

How do you persuade somebody of the facts? Asking them to be fair, impartial and unbiased is not enough. To explain why, psychologist Tom Stafford analyses a classic scientific study.

One of the tricks our mind plays is to highlight evidence which confirms what we already believe. If we hear gossip about a rival we tend to think “I knew he was a nasty piece of work”; if we hear the same about our best friend we’re more likely to say “that’s just a rumour”. If you don’t trust the government then a change of policy is evidence of their weakness; if you do trust them the same change of policy can be evidence of their inherent reasonableness.

Once you learn about this mental habit – called confirmation bias – you start seeing it everywhere.

This matters when we want to make better decisions. Confirmation bias is OK as long as we’re right, but all too often we’re wrong, and we only pay attention to the deciding evidence when it’s too late.

How we should to protect our decisions from confirmation bias depends on why, psychologically, confirmation bias happens. There are, broadly, two possible accounts and a classic experiment from researchers at Princeton University pits the two against each other, revealing in the process a method for overcoming bias.

The first theory of confirmation bias is the most common. It’s the one you can detect in expressions like “You just believe what you want to believe”, or “He would say that, wouldn’t he?” or when the someone is accused of seeing things a particular way because of who they are, what their job is or which friends they have. Let’s call this the motivational theory of confirmation bias. It has a clear prescription for correcting the bias: change people’s motivations and they’ll stop being biased.

The alternative theory of confirmation bias is more subtle. The bias doesn’t exist because we only believe what we want to believe, but instead because we fail to ask the correct questions about new information and our own beliefs. This is a less neat theory, because there could be one hundred reasons why we reason incorrectly – everything from limitations of memory to inherent faults of logic. One possibility is that we simply have a blindspot in our imagination for the ways the world could be different from how we first assume it is. Under this account the way to correct confirmation bias is to give people a strategy to adjust their thinking. We assume people are already motivated to find out the truth, they just need a better method. Let’s call this the cognition theory of confirmation bias.

Thirty years ago, Charles Lord and colleagues published a classic experiment which pitted these two methods against each other. Their study used a persuasion experiment which previously had shown a kind of confirmation bias they called ‘biased assimilation’. Here, participants were recruited who had strong pro- or anti-death penalty views and were presented with evidence that seemed to support the continuation or abolition of the death penalty. Obviously, depending on what you already believe, this evidence is either confirmatory or disconfirmatory. Their original finding showed that the nature of the evidence didn’t matter as much as what people started out believing. Confirmatory evidence strengthened people’s views, as you’d expect, but so did disconfirmatory evidence. That’s right, anti-death penalty people became more anti-death penalty when shown pro-death penalty evidence (and vice versa). A clear example of biased reasoning.

For their follow-up study, Lord and colleagues re-ran the biased assimilation experiment, but testing two types of instructions for assimilating evidence about the effectiveness of the death penalty as a deterrent for murder. The motivational instructions told participants to be “as objective and unbiased as possible”, to consider themselves “as a judge or juror asked to weigh all of the evidence in a fair and impartial manner”. The alternative, cognition-focused, instructions were silent on the desired outcome of the participants’ consideration, instead focusing only on the strategy to employ: “Ask yourself at each step whether you would have made the same high or low evaluations had exactly the same study produced results on the other side of the issue.” So, for example, if presented with a piece of research that suggested the death penalty lowered murder rates, the participants were asked to analyse the study’s methodology and imagine the results pointed the opposite way.

They called this the “consider the opposite” strategy, and the results were striking. Instructed to be fair and impartial, participants showed the exact same biases when weighing the evidence as in the original experiment. Pro-death penalty participants thought the evidence supported the death penalty. Anti-death penalty participants thought it supported abolition. Wanting to make unbiased decisions wasn’t enough. The “consider the opposite” participants, on the other hand, completely overcame the biased assimilation effect – they weren’t driven to rate the studies which agreed with their preconceptions as better than the ones that disagreed, and didn’t become more extreme in their views regardless of which evidence they read.

The finding is good news for our faith in human nature. It isn’t that we don’t want to discover the truth, at least in the microcosm of reasoning tested in the experiment. All people needed was a strategy which helped them overcome the natural human short-sightedness to alternatives.

The moral for making better decisions is clear: wanting to be fair and objective alone isn’t enough. What’s needed are practical methods for correcting our limited reasoning – and a major limitation is our imagination for how else things might be. If we’re lucky, someone else will point out these alternatives, but if we’re on our own we can still take advantage of crutches for the mind like the “consider the opposite” strategy.

This is my BBC Future column from last week. You can read the original here. My ebook For argument’s sake: Evidence that reason can change minds is out now.

Can boy monkeys throw?

180px-cebus_albifrons_editAimed throwing is a gendered activity – men are typically better at it than women (by about 1 standard deviation, some studies claim). Obviously this could be due to differential practice, which is in turn due to cultural bias in what men vs women are expected to be a good at and enjoy (some say “not so” to this practice-effect explanation).

Monkeys are interesting because they are close evolutionary relatives, but don’t have human gender expectations. So we note with interest this 2000 study which claims no difference in throwing accuracy between male and female Capuchin monkeys. In fact, the female monkeys were (non-significantly) more accurate than the males (perhaps due to throwing as part of Capuchin female sexual displays?).

Elsewhere, a review of cross-species gender differences in spatial ability finds “most of the hypotheses [that male mammals have better spatial ability than females] are either logically flawed or, as yet, have no substantial support. Few of the data exclusively support or exclude any current hypotheses“.

Chimps are closer relatives to humans than monkeys, but although there is a literature on gendered differences in object use/preference among chimps, I couldn’t immediately find anything on gendered differences in throwing among chimps. Possibly because few scientists want to get near a chimp when it is flinging sh*t around.

Cite: Westergaard, G. C., Liv, C., Haynie, M. K., & Suomi, S. J. (2000). A comparative study of aimed throwing by monkeys and humans. Neuropsychologia, 38(11), 1511-1517.

Previously: gendered brain blogging