Category: Theory

Frontiers in Psychology has just published an article on ‘Fifty psychological and psychiatric terms to avoid’. These sorts of “here’s how to talk about” articles are popular but themselves can often be misleading, and the same applies to this one.

The article supposedly contains 50 “inaccurate, misleading, misused, ambiguous, and logically confused words and phrases”.

The first thing to say is that by recommending that people avoid certain words or phrases, the article is violating its own recommendations. That may seem like a trivial point but it isn’t when you’re giving advice about how to use language in scientific discussion.

It’s fine to use even plainly wrong terms to discuss how they’re used, the multiple meanings and misconceptions behind them. In fact, a lot of scientific writing does exactly this. When there are misconceptions that may cloud people’s understanding, it’s best to address them head on rather than avoid them.

Sometimes following the recommendations for ‘phrases to avoid’ would actually hinder this process.

For example, the piece recommends you avoid the term ‘autism epidemic’ as there is no good evidence that there is an actual epidemic. But this is not advice about language, it’s just an empirical point. According to this list, all the research that has used the term, to discuss the actual evidence in contrary to the popular idea, should have avoided the term and presumably referred to it as ‘the concept that shall not be named’.

The article also recommends against using ‘ambiguous’ words but this recommendation would basically kill the English language as many words have multiple meanings – like the word ‘meaning’ for example – but that doesn’t mean you should avoid them.

If you’re a fan of pedantry you may want to go through the article and highlight where the authors have used other ambiguous psychological phrases (starter for 10, “memory”) and post it to some obscure corner of the internet.

Many of the recommendations also rely on you agreeing with the narrow definition and limits of use that the authors premise their argument on. Do you agree that “antidepressant medication” means that the medication has a selective and specific effect on depression and no other conditions – as the authors suggest? Or do you think this just describes a property of the medication? This is exactly how medication description works throughout medicine. Aspirin is an analgesic medication and an anti-inflammatory medication, as well as having other properties. No banning needed here.

And in fact, this sort of naming is just a property of language. If you talk about an ‘off-road vehicle’, and someone pipes up to tell you “actually, off-road vehicles can also go on-road so I recommend you avoid that description” I recommend you ignore them.

The same applies to many of the definitions in this list. The ‘chemical imbalance’ theory of depression is not empirically supported, so don’t claim it is, but feel free to use the phrase if you want to discuss this misconception. Some conditions genuinely do involve a chemical imbalance though – like the accumulation of copper in Wilson’s disease, so you can use the phrase accurately in this case, being aware of how its misused in other contexts. Don’t avoid it, just use it clearly.

With ‘Lie detector test’ no accurate test has ever been devised to detect lies. But you may be writing about research which is trying to develop one or research that has tested the idea. ‘No difference between groups’ is fine if there is genuinely no difference in your measure between the groups (i.e. they both score exactly the same).

Some of the recommendations are essentially based on the premise that you ‘shouldn’t use the term except for how it was first defined or defined where we think is the authoritative source’. This is just daft advice. Terms evolve over time. Definitions shift and change. The article recommends against using ‘Fetish’ except for in its DSM-5 definition, despite the fact this is different to how it’s used commonly and how it’s widely used in other academic literature. ‘Splitting’ is widely used in a form to mean ‘team splitting’ which the article says is ‘wrong’. It isn’t wrong – the term has just evolved.

I think philosophers would be surprised to hear ‘reductionism’ is a term to be avoided – given the massive literature on reductionism. Similarly, sociologists might be a bit baffled by ‘medical model’ being a banned phrase, given the debates over it and, unsurprisingly, its meaning.

Some of the advice is just plain wrong. Don’t use “Prevalence of trait X” says the article because apparently prevalence only applies to things that are either present or absent and “not dimensionally distributed in the population, such as personality traits and intelligence”. Many traits are defined by cut-off scores along dimensionally defined constructs, making them categorical. If you couldn’t talk about the prevalence in this way, we’d be unable to talk about prevalence of intellectual disability (widely defined as involving an IQ of less than 70) or dementia – which is diagnosed by a cut-off score on dimensionally varying neuropsychological test performance.

Some of the recommended terms to avoid are probably best avoided in most contexts (“hard-wired”, “love molecule”) and some are inherently self-contradictory (“Observable symptom”, “Hierarchical stepwise regression”) but again, use them if you want to discuss how they’re used.

I have to say, the piece reminds me of Stephen Pinker’s criticism of ‘language mavens’ who have come up with rules for their particular version of English which they decide others must follow.

To be honest, I think the Frontiers in Psychology article is well-worth reading. It’s a great guide to how some concepts are used in different ways, but it’s not good advice for things to avoid.

The best advice is probably: communicate clearly, bearing in mind that terms and concepts can have multiple meanings and your audience may not be aware of which you want to communicate, so make an effort to clarify where needed.
 

Link to Frontiers in Psychology article.

Online testing is sure to play a large part in the future of Psychology. Using Mechanical Turk or other crowdsourcing sites for research, psychologists can quickly and easily gather data for any study where the responses can be provided online. One concern, however, is that online samples may be less motivated to pay attention to the tasks they are participating in. Not only is nobody watching how they do these online experiments, they whole experience is framed as a work-for-cash gig, so there is pressure to complete any activity as quickly and with as low effort as possible. To the extent that the online participants are satisficing or skimping on their attention, can we trust the data?

A newly submitted paper uses data from the Many Labs 3 project, which recruited over 3000 participants from both online and University campus samples, to test the idea that online samples are different from the traditional offline samples used by academic psychologists:

The findings strike a note of optimism, if you’re into online testing (perhaps less so if you use traditional university samples):

Mechanical Turk workers report paying more attention and exerting more effort than undergraduate students. Mechanical Turk workers were also more likely to pass an instructional manipulation check than undergraduate students. Based on these results, it appears that concerns over participant inattentiveness may be more applicable to samples recruited from traditional university participant pools than from Mechanical Turk

This fits with previous reports showing high consistency when classic effects are tested online, and with reports that satisficing may have been very high in offline samples, we just weren’t testing for it.

However, an issue I haven’t seen discussed is whether, because of the relatively small pool of participants taking experiments on MTurk, online participants have an opportunity to get familiar with typical instructional manipulation checks (AKA ‘catch questions’, which are designed to check if you are paying attention). If online participants adapt to our manipulation checks, then the very experiments which set out to test if they are paying more attention may not be reliable.

Link: new paperGraduating from Undergrads: Are Mechanical Turk Workers More Attentive than Undergraduate Participants?

This paper provides a useful overview: Conducting perception research over the internet: a tutorial review

“Face It,” says psychologist Gary Marcus in The New York Times, “Your Brain is a Computer”. The op-ed argues for understanding the brain in terms of computation which opens up to the interesting question – what does it mean for a brain to compute?

Marcus makes a clear distinction between thinking that the brain is built along the same lines as modern computer hardware, which is clearly false, while arguing that its purpose is to calculate and compute. “The sooner we can figure out what kind of computer the brain is,” he says, “the better.”

In this line of thinking, the mind is considered to be the brain’s computations at work and should be able to be described in terms of formal mathematics.

The idea that the mind and brain can be described in terms of information processing is the main contention of cognitive science but this raises a key but little asked question – is the brain a computer or is computation just a convenient way of describing its function?

Here’s an example if the distinction isn’t clear. If you throw a stone you can describe its trajectory using calculus. Here we could ask a similar question: is the stone ‘computing’ the answer to a calculus equation that describes its flight, or is calculus just a convenient way of describing its trajectory?

In one sense the stone is ‘computing’. The physical properties of the stone and its interaction with gravity produce the same outcome as the equation. But in another sense, it isn’t, because we don’t really see the stone as inherently ‘computing’ anything.

This may seem like a trivial example but there are in fact a whole series of analog computers that use the physical properties of one system to give the answer to an entirely different problem. If analog computers are ‘really’ computing, why not our stone?

If this is the case, what makes brains any more or less of a computer than flying rocks, chemical reactions, or the path of radio waves? Here the question just dissolves into dust. Brains may be computers but then so is everything, so asking the question doesn’t tell us anything specific about the nature of brains.

One counter-point to this is to say that brains need to algorithmically adjust to a changing environment to aid survival which is why neurons encode properties (such as patterns of light stimulation) in another form (such as neuronal firing) which perhaps makes them a computer in a way that flying stones aren’t.

But this definition would also include plants that also encode physical properties through chemical signalling to allow them to adapt to their environment.

It is worth noting that there are other philosophical objections to the idea that brains are computers, largely based on the the hard problem of consciousness (in brief – could maths ever feel?).

And then there are arguments based on the boundaries of computation. If the brain is a computer based on its physical properties and the blood is part of that system, does the blood also compute? Does the body compute? Does the ecosystem?

Psychologists drawing on the tradition of ecological psychology and JJ Gibson suggest that much of what is thought of as ‘information processing’ is actually done through the evolutionary adaptation of the body to the environment.

So are brains computers? They can be if you want them to be. The concept of computation is a tool. Probably the most useful one we have, but if you say the brain is a computer and nothing else, you may be limiting the way you can understand it.
 

Link to ‘Face It, Your Brain Is a Computer’ in The NYT.

Understanding statistical power is essential if you want to avoid wasting your time in psychology. The power of an experiment is its sensitivity – the likelihood that, if the effect tested for is real, your experiment will be able to detect it.

Statistical power is determined by the type of statistical test you are doing, the number of people you test and the effect size. The effect size is, in turn, determined by the reliability of the thing you are measuring, and how much it is pushed around by whatever you are manipulating.

Since it is a common test, I’ve been doing a power analysis for a two-sample (two-sided) t-test, for small, medium and large effects (as conventionally defined). The results should worry you.

This graph shows you how many people you need in each group for your test to have 80% power (a standard desirable level of power – meaning that if your effect is real you’ve an 80% chance of detecting it).

Things to note:

• even for a large (0.8) effect you need close to 30 people (total n = 60) to have 80% power
• for a medium effect (0.5) this is more like 70 people (total n = 140)
• the required sample size increases drammatically as effect size drops
• for small effects, the sample required for 80% is around 400 in each group (total n = 800).

What this means is that if you don’t have a large effect, studies with between groups analysis and an n of less than 60 aren’t worth running. Even if you are studying a real phenomenon you aren’t using a statistical lens with enough sensitivity to be able to tell. You’ll get to the end and won’t know if the phenomenon you are looking for isn’t real or if you just got unlucky with who you tested.

Implications for anyone planning an experiment:

• Is your effect very strong? If so, you may rely on a smaller sample (For illustrative purposes the effect size of male-female heigh difference is ~1.7, so large enough to detect with small sample. But if your effect is this obvious, why do you need an experiment?)
• You really should prefer within-sample analysis, whenever possible (power analysis of this left as an exercise)
• You can get away with smaller samples if you make your measure more reliable, or if you make your manipulation more impactful. Both of these will increase your effect size, the first by narrowing the variance within each group, the second by increasing the distance between them

Technical note: I did this cribbing code from Rob Kabacoff’s helpful page on power analysis. Code for the graph shown here is here. I use and recommend Rstudio.

Cross-posted from www.tomstafford.staff.shef.ac.uk where I irregularly blog things I think will be useful for undergraduate Psychology students.

A paper in the high-profile journal Science has been alleged to be based on fraudulent data, with the PI calling for it to be retracted. The original paper purported to use survey data to show that people being asked about gay marriage changed their attitudes if they were asked the survey questions by someone who was gay themselves. That may still be true, but the work of a team that set out to replicate the original study seems to show that the data reported in that paper was never collected in the way reported, and at least partly fabricated.

The document containing these accusations is interesting for a number of reasons. It contains a detailed timeline showing how the authors were originally impressed with study and set out to replicate it, gradually uncovering more and more elements that concerned them and let them to investigate how the original data was generated. The document also reports the exemplary way in which they shared their concerns with the authors of the original paper, and the way the senior author responded. The speed of all this is notable – the investigators only started work on this paper in January, and did most of the analysis substantiating their concerns this month.

As we examined the study’s data in planning our own studies, two features surprised us: voters’ survey responses exhibit much higher test-retest reliabilities than we have observed in any other panel survey data, and the response and reinterview rates of the panel survey were significantly higher than we expected. We set aside our doubts about the study and awaited the launch of our pilot extension to see if we could manage the same parameters. LaCour and Green were both responsive to requests for advice about design details when queried.

So on the one hand this is a triumph for open science, and self-correction in scholarship. The irony being that any dishonesty that led to publication in a high-impact journal, also attracted people with the desire and smarts to check if what was reported holds up. But the tragedy is the circumstances that led the junior author of the original study, himself a graduate student at the time, to do what he did. No statement from him is available at this point, as far as I’m aware.

The original: When contact changes minds: An experiment on transmission of support for gay equality

The accusations and retraction request: Irregularities in LaCour (2014)

As the UK election results roll in, one of the big shocks is the discrepancy between the pre-election polls and the results. All the pollsters agreed that it would be incredibly close, and they were all wrong. What gives?

Some essential psych 101 concepts come in useful here. Polls rely on sampling – the basic idea being that you don’t have to ask everyone to get a rough idea of how things are going to go. How rough that idea is depends on how many you ask. This is the issue of sampling error. We understand sampling error – you can estimate it, so as well as reducing this error by taking larger samples there are also principled ways of working out when you’ve asked enough people to get a reliable estimate (which is why polls of a country with a population of 70 million can still be accurate with samples in the thousands).

But, as Tim Harford points out in in this excellent article on sampling problems big data, with every sample there are two sources of unreliability. Sampling error, as I’ve mentioned, but also sampling bias.

sampling error has a far more dangerous friend: sampling bias. Sampling error is when a randomly chosen sample doesn’t reflect the underlying population purely by chance; sampling bias is when the sample isn’t randomly chosen at all.

The problem with sample bias is that, when you don’t know the ground truth, there is no principled way of knowing if your sample is biased. If your sample has some systematic bias in it, you can make a reliable estimate (minimising sample error), but you are still left with the sample bias – a bias you don’t know how big it is until you find out the truth. That’s my guess at what happened with the UK election. The polls converged, minimising the error, but the bias remained – a ‘shy tory‘ effect where many voters were not admitting (or not aware) that they would end up voting for the Conservative party.

The exit polls predicted the real result with surprising accuracy not because they minimised sampling error, but because they avoided the sample bias. By asking the people who actually turned up to vote how they actually voted, their sample lacked the bias of the pre-election polls.

Start the week with one of the founding fathers of cognitive science: in ‘The scientist as problem solver‘, Herb Simon (1916-2001) gives a short retrospective of his scientific career.

To tell the story of the research he has done, he advances a thesis: “The Scientist is a problem solver. If the thesis is true, then we can dispense with a theory of scientific discovery – the processes of discovery are just applications of the processes of problem solving.”. Quite aside from the usefulness of this perspective, the paper is an reminder of intoxicating possibility of integration across the physical, biological and social sciences: Simon worked on economics, management theory, complex systems and artificial intelligence as well as what we’d call now cognitive psychology.

He uses his own work on designing problem solving algorithms to reflect on how he – and other scientists – can and should make scientific progress. Towards the end he expresses what would be regarded as heresy in many experimentally orientated psychology departments. He suggests that many of his most productive investigations lacked a contrast between experimental and control conditions. Did this mean they were worthless, he asks. No:

…You can test theoretical models without contrasting an experimental with a control condition. And apart from testing models, you can often make surprising observations that give you ideas for new or improved models…

Perhaps it is not our methodology that needs revising so much as the standard textbook methodology, which perversely warns us against running an experiment until precise hypotheses have been formulated and experimental and control conditions defined. How do such experiments ever create surprise – not just the all-too-common surprise of having our hypotheses refuted by facts, but the delight-provoking surprise of encountering a wholly unexpected phenomenon? Perhaps we need to add to the textbooks a chapter, or several chapters, describing how basic scientific discoveries can be made by observing the world intently, in the laboratory or outside it, with controls or without them, heavy with hypotheses or innocent of them.

REFERENCE
Simon, H. A. (1989). The scientist as problem solver. Complex information processing: The impact of Herbert A. Simon, 375-398.

“You can’t play 20 questions with nature and win” is the title of Allen Newell‘s 1973 paper, a classic in cognitive science. In the paper he confesses that although he sees many excellent psychology experiments, all making undeniable scientific contributions, he can’t imagine them cohering into progress for the field as a whole. He describes the state of psychology as focussed on individual phenomena – mental rotation, chunking in memory, subitizing, etc – studied in a way to resolve binary questions – issues such as nature vs nature, conscious vs unconscious, serial vs parallel processing.

There is, I submit, a view of the scientific endeavor that is implicit (and sometimes explicit) in the picture I have presented above. Science advances by playing twenty questions with nature. The proper tactic is to frame a general question, hopefully binary, that can be attacked experimentally. Having settled that bits-worth, one can proceed to the next. The policy appears optimal – one never risks much, there is feedback from nature at every step, and progress is inevitable. Unfortunately, the questions never seem to be really answered, the strategy does not seem to work.

As I considered the issues raised (single code versus multiple code, continuous versus discrete representation, etc.) I found myself conjuring up this model of the current scientific process in psychology- of phenomena to be explored and their explanation by essentially oppositional concepts. And I couldn’t convince myself that it would add up, even in thirty more years of trying, even if one had another 300 papers of similar, excellent ilk.

His diagnosis for one reason that phenomena can generate an endless excellent papers without endless progress is that people can do the same task in different ways. Lots of experiments dissect how people are doing the task, without constraining sufficiently the things Newell says are essential to predict behaviour (the person’s goals and the structure of the task environment), and thus providing no insight into the ultimate target of investigation, the invariant structure of the mind’s processing mechanisms. As a minimum, we must know the method participants are using, never averaging over different methods, he concludes. But this may not be enough:

That the same human subject can adopt many (radically different) methods for the same basic task, depending on goal, background knowledge, and minor details of payoff structure and task texture — all this — implies that the “normal” means of science may not suffice.

As a prognosis for how to make real progress in understanding the mind he proposes three possible courses of action:

1. Develop complete processing models – i.e. simulations which are competent to perform the task and include a specification of the way in which different subfunctions (called ‘methods’ by Newell) are deployed.
2. Analyse a complex task, completely, ‘to force studies into intimate relation with each other’, the idea being that giving a full account of a single task, any task, will force contradictions between theories of different aspects of the task into the open.
3. ‘One program for many tasks’ – construct a general purpose system which can perform all mental tasks, in other words an artificial intelligence.

It was this last strategy which preoccupied a lot of Newell’s subsequent attention. He developed a general problem solving architecture he called SOAR, which he presented as a unified theory of cognition, and which he worked on until his death in 1992.

The paper is over forty years old, but still full of useful thoughts for anyone interested in the sciences of the mind.

Reference and link:
Newell, A. You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. in Chase, W. G. (Ed.). (1973). Visual Information Processing: Proceedings of the Eighth Annual Carnegie Symposium on Cognition, Held at the Carnegie-Mellon University, Pittsburgh, Pennsylvania, May 19, 1972. Academic Press.

See a nice picture of Newell from the Computer History Museum

In the latest edition of The Psychologist I’m involved in a debate with John Cromby about whether our understanding of mental illness is mired in the past.

He thinks it is, I think it isn’t, and we kick off from there.

The article is readable online with a free registration but I’ve put the unrestricted version online as a pdf if you want to read it straight away.

Much of the debate is over the role of biological explanations in understanding mental distress which I think is widely understood by many.

Hopefully, amid the knockabout, the debate gets to clarify some of that.

Either way, I hope it raises a few useful reflections.
 

Link to ‘Are understandings of mental illness mired in the past?’ (free reg).
pdf of full debate.

The Times Higher Education has an article on post-publication peer review, and whether it will survive legal challenges

The legal action launched by a US scientist who claims that anonymous comments questioning his science cost him a lucrative job offer has raised further questions about the potential for post-publication peer review to replace pre-publication review.

The article chimes with comments made by several prominent Psychologists who have been at the centre of controversies and decried the way their work has been discussed outside of the normal channels of the academic journals.

Earlier this year the head of a clinical trial of Tamiflu wrote to the British Medical Journal to protest that a BMJ journalist had solicited independent critique of the stats used in his work – “going beyond the reasonable response to a press release”.

John Bargh (Yale University) in his now infamous ‘nothing in their heads’ blogpost accused the open access journal PLoS of lacking “the usual high scientific journal standards of peer-review scrutiny”, and accussed Ed Yong – laughably – of “superficial online science journalism”. He concluded:

“I am not so much worried about the impact on science of essentially self-published failures to replicate as much as I’m worried about your ability to trust supposedly reputable online media sources for accurate information on psychological science.”

Simone Schnall (University of Cambridge) is a social psychologist whose work has also been at the centre of the discussion about replication (backstory, independent replication of her work recently reported). She has recently written that ‘no critical discussion is possible’ on social media, where ‘judgments are made quickly nowadays in social psychology and definitively’.

See also this comment from a scientist when a controversial paper which suggested that many correlations in fMRI studies of social psychological constructs were impossibly high was widely discussed before publication: . “I was shocked, this is not the way that scientific discourse should take place.”

The common theme is a lack of faith in the uncontrolled scientific discussion that now happens in public, before and after publication in the journal-sanctioned official record. Coupled, perhaps, with a lack of faith in other people to understand – let alone run – psychological research. Scientific discussion has always been uncontrolled, of course, the differences now are in how open the discussion is, and who takes part. Pre social media, ‘insider’ discussions of specialist topics took place inside psychology departments, and at conference dinners and other social gatherings of researchers. My optimistic take is that social media allows access to people who would not normally have it due to constraints on geography, finance or privilege. Social media means that if you’re in the wrong institution, aren’t funded, or if you have someone to look after at home that means you can’t fly to the conference, you can still experience and contribute to specialist discussions – that’s a massive and positive change and one we should protect as we work out how scientific discussion should take place in the 21st century.

Link: Simone Schnall’s comments in full: blog, video

Previously: Stafford, T., & Bell, V. (2012). Brain network: social media and the cognitive scientist. Trends in Cognitive Sciences, 16(10), 489–490. doi:10.1016/j.tics.2012.08.001

Previously What Jason Mitchell’s ‘On the emptiness of failed replications’ gets right, which includes some less optimistic notes on the current digital disruption of scholarly ways of working

I’ve been puzzling over this tweet from Jeff Rouder:

Surely, I thought, psychology is built out of effects. What could be wrong with focussing on testing which ones are reliable?

But I think I’ve got it now. The thing about effects is that they show you – an experimental psychologist – can construct a situation where some factor you are interested in is important, relative to all the other factors (which you have managed to hold constant).

To see why this might be a problem, consider this paper by Tsay (2013): “Sight over sound in the judgment of music performance”. This was a study which asked people to select the winners of a classical music competition from 6 second clips of them performing. Some participants got the audio, so they could only hear the performance; others got the video, so they could only see the performance; and some got both audio and video. Only those participants who watched the video, without sound, could select the actual competition winners at above chance level. This demonstrates a significant bias effect of sight in judgements of music performance.

To understand the limited importance of this effect, contrast with the overclaims made by the paper: “people actually depend primarily on visual information when making judgments about music performance” (in the abstract) and “[Musicians] relegate the sound of music to the role of noise” (the concluding line). Contrary to these claims the study doesn’t show that looks dominate sound in how we assess music. It isn’t the case that our musical taste is mostly determined by how musicians look.

The Tsay studies took the 3 finalists from classical music competitions – the best of the best of expert musicians – and used brief clips of their performances as stimuli. By my reckoning, this scenario removes almost all differences in quality of the musical performance. Evidence in support for this is that Tsay didn’t find any difference in performance between non-expert participants and professional musicians. This fact strongly suggests that she has designed a task in which it is impossible to bring any musical knowledge to bear. musical knowledge isn’t an important factor.

This is why it isn’t reasonable to conclude that people are making judgments about musical performance in general. The clips don’t let you judge relative musical quality, but – for these equally almost equally matched performances – they do let you reflect the same biases as the judges, biases which include an influence of appearance as well as sound. The bias matters, not least because it obviously affects who won, but proving it exists is completely separate from the matter of whether the overall judgements of music, is affected more by sight or sound.

Further, there’s every reason to think that the conclusion from the study of the bias effect gives the opposite conclusion to the study of overall importance. In these experiments sight dominates sound, because differences due to sound have been controlled out. In most situations where we decide our music preferences, sounds is obviously massively more important.

Many psychological effects are impressive tribute to the skill of experimenters in designing situations where most factors are held equal, allowing us to highlight the role of subtle psychological factors. But we shouldn’t let this blind us to the fact that the existence of an effect due to a psychological factor isn’t the same as showing how important this factor is relative to all others, nor is it the same as showing that our effect will hold when all these other factors start varying.

Link: Are classical music competitions judged on looks? – critique of Tsay (2013) written for The Conversation

Link: A good twitter thread on the related issue of effect size – and yah-boo to anyone who says you can’t have a substantive discussion on social media

UPDATE: The paper does give evidence that the sound stimuli used do influence people’s judgements systemmatically – it was incorrect of me to say that differences due to sound have been removed. I have corrected the post to reflect what I believe the study shows: that differences due to sound have been minimised, so that differences in looks are emphasised.

The New Yorker has an interesting article about a lack of political diversity in social psychology and how that may be leading to a climate of bias against conservative researchers, ideas and the evidence that might support them.

Some of the evidence for a bias against conservative thinking in social psychology goes back some years, and the article gives a good account of the empirical work as well as the debate.

However, the issue was recently raised again by morality researcher Jonathan Haidt leading to a renewed reflection on the extent of the problem.

There is a case to be made that, despite the imbalance, no formal changes need to be made, and that, on the whole, despite its problems, social psychology continues to function remarkably well and regularly produces high-quality research. Controversial work gets done. Even studies that directly challenge the field—like Haidt’s—are publicized and inspire healthy debate…

And yet the evidence for more substantial bias, against both individuals and research topics and directions, is hard to dismiss—and the hostility that some social psychologists have expressed toward the data suggests that self-correction may not be an adequate remedy.

A timely reminder of the eternal truth that bias is entirely non-partisan, and if you’ve not heard it before, a pointer to a great BBC Radio documentary that outlines how it works equally across people of every political stripe.

 
Link to ‘Is Social Psychology Biased Against Republicans?’

I have a new paper out in Frontiers in Psychology: The perspectival shift: how experiments on unconscious processing don’t justify the claims made for them. There has been ongoing consternation about the reliability of some psychology research, particularly studies which make claims about unconscious (social) priming. However, even if we assume that the empirical results are reliable, the question remains whether the claims made for the power of the unconscious make any sense. I argue that they often don’t.

Here’s something from the intro:

In this commentary I draw attention to certain limitations on the inferences which can be drawn about participant’s awareness from the experimental methods which are routine in social priming research. Specifically, I argue that (1) a widely employed definition of unconscious processing, promoted by John Bargh is incoherent (2) many experiments involve a perspectival sleight of hand taking factors identified from comparison of average group performance and inappropriately ascribing them to the reasoning of individual participants.

The problem, I claim, is that many studies on ‘unconscious processing’, follow John Bargh in defining unconscious as meaning “not reported at the time”. This means that experimenters over-diagnose unconscious influence, when the possibility remains that participants were completely conscious of the influence of the stimili, but may not be reporting them because they have forgotten, worry about sounding silly or because the importance of the stimuli is genuinely trivial compared to other factors.

It is this last point which makes up the ‘perspectival shift’ of the title. Experiments on social priming usually work by comparing some measure (e.g. walking speed or reaction time) across two groups. My argument is that the factors which make up the total behaviour for each individual will be many and various. The single factor which the experimenter is interested in may have a non-zero effect, yet can still justifiably escape report by the majority of participants. To make this point concrete: if I ask you to judge how likeable someone is on the 1 to 7 scale, your judgement will be influenced by many factors, such as if they are like you, if you are in a good mood, the content of your interaction with the person, if they really are likeable and so on. Can we really expect participants to report an effect due to something that only the experimenter sees variation in, such as whether they are holding a hot drink or a cold drink at the time of judgement? We might as well expect them to report the effect due to them growing up in Europe rather than Asia, or being born in 1988 not 1938 (both surely non-zero effects in my hypothetical experiment).

More on this argument, and what I think it means, in the paper:

Stafford, T. (2014) The perspectival shift: how experiments on unconscious processing don’t justify the claims made for them. Frontiers in Psychology, 5, 1067. doi:10.3389/fpsyg.2014.01067

I originally started writing this commentary as a response to this paper by Julie Huang and John Bargh, which I believe is severely careless with the language it uses to discuss unconscious processing (and so a good example of the conceptual trouble you can get into if you start believing the hype around social priming).

Full disclosure: I am funded by the Leverhulme Trust to work on a project looking at the philosophy and psychology of implicit bias. This post is cross-posted on the project blog.