In which I suggest applying the methods of experimental psychology to a longstanding question in political science.
Many people feel that there is no “real difference” between political parties (for example, Labour vs Conservatives in the UK). Politians are all the same, right? At least superficially, mainsteam parties will all echo commitments to values such as “community” and “education” and positions such as “tough on crime” and “for a strong economy”.
In perceptual psychology we have a number of methods of calculating how accurate and sensitive a sense, like sight or hearing, is. Using these ‘psychophysical’ methods you can come up with a number which allows you to compare across different senses, or across different people. So, for example, we could show that vision is more sensitive than hearing, or that your vision is more sensitive than mine (or even that your vision is more sensitive than my hearing). These methods account for things like base-rate biases in people’s responding (so, for example, it could account for the fact that you might be more likely to say you can see something when you are in doubt, while I might be more likely to say that I can’t when I am in doubt). This sensitivity statistic I am thinking of is called d’ (“d prime”) by psychologists.
I’ve been considering whether these methods from perceptual psychology could be used to address the question of how similar the positions of political parties are. My way of testing and tracking the difference in the stated policy positions of the parties would work like this: you take a standard public expression of party positions (election manifestos?) and sample policy statements (size of sample to be decided, somewhere between individual sentences and paragraphs). Then, after coding the statements for their year and origin, you anonymise them and ask voters to say which party they think the statements come from. With a few psychophysical calculations we can then come up with a sensitivity statistics which reveals how easy voters find it to distinguish the policy positions of the two parties, and we can then compare how this changes over time, or in different policy areas.
Friend and political scientist Will Jennings, told me that – of course – political scientists already look at this topic. The British Election Study has been asking voters since 1964 how close the parties are. Projects such as the Comparative Manifestos Project have coded party manifestos from around the world, using techniques such as automated coding of text and expert surveys (i.e. asking academics what they think).
The problem with asking voters how close the parties are, or to code the parties as more “left-wing” or more “right-wing” is that you deal with opinions of voters, not their actual ability to discriminate between the positions of the parties. The problem with coding the manifestos is that it puts a layer of intepretation (as to what counts as left-wing, or converservative, or whatever) before you can judge one manifesto as closer or further away from another.
My psychophysics approach tests directly the ability of voters to discriminate between stated policy positions. We do this by presenting many small fragments of the manifestos and asking a participant to judge which party they are from. By gathering many many judgements we can get a sense of how likely they are to name each particular party (i.e. their bias) and get a sense for how likely they are to be correct (i.e. their sensitivity). We combine these, accounting for any bias towards naming a particular party, to get an estimate of their ability to discriminate between the parties based on their stated policy positions. You can average this index across people, removing random variation in sensitivity between people, to get an estimate of how discriminable two stated positions truly are.
Cross-posted, with some informed comment, at The Monkey Cage
Mind Hacks 
Given politicians track record with following up on the things they said before the election I don’t think it makes much sense to say that they represent the true position of a party.
I agree with ChristianKI. Do not use political statements, but the decisions that the political party takes when in power.
I depends whether you want to understand the parties or people’s perceptions of the parties. I am interested in people’s perceptions. This is what drives their voting behaviour. And is more tractable than the reality of any party’s position
There is a necessary difference between the aspirational statements of a party and its actions: the statements are intended to distinguish those within the party, while legislative actions usually must seek consensus that embrace those outside the party, either by broadening the action or offering reciprocal benefit to those outside.
Those outside the sausage factory perceive that behavior as selling out principles or politics as usual.
Another layer of complexity is the motivational context of the person perceiving these statements. Is it affected by need to justify their voting past, their intended vote, or their decision not to vote?
I’ve always been fascinated by the story that aboriginal hunter-gathers around the Artic circle don’t just perceive snow, they perceive and understand its many different flavors because the outcomes require it.
Similarly, it would be interesting to see whether the perceptual responses of participants in the political process are different from those who leave it.
I also understand that people generally revile lawyers, yet have much more positive beliefs about their own lawyer.
It would interesting to see if the perceptions of the political are influenced by familiarity with the local narrative, ie. politicians who “name a political action” are unprincipled and corrupt, but when my politician takes that action, he does so good reason (or at least, I think he does).
Great post. I expect a d’ of ~0 for the experiment you suggest! It might be interesting to, instead of measuring differences in statements and positions, to measure based on actual voting records.
The idea that “manifestos” are an actual crystallization of a party’s political views seems very naive to me. The media-savvy operators who craft the “official” views that go up on the websites very shrewdly water them down – tactically its better to put anodyne pleasantries in the “manifestos” than divisive statements. There are usually two readers of manifestos – people who are already on board and are looking for confirmation and then hostile enemies. Those looking for confirmation will be easily mollified by vacuous but pleasant language, the latter group will find no ammunition.
Text selection bias seems to be a perpetual problem for those seeking to those seeking to subject such a complex human behavior to the limited tools of psychological research.
“When the only tool you have is a hammer, every problem looks like a nail”
The idea that you can’t believe anything politians say, or that what is written in manifestos is meaningless seems to me to be a position if cynicism which is politically neutering. Very convenient for politicians who might want to ignore you!
@naigewron good point about the text bias – it’s hard to ignore an existing and already digitised trace of behaviour!
I like the idea; but you have the two related problems about being able to adequately sample a space, which you have been able to define clearly enough. What’s the distance metric along your ‘left-right’ axis, for instance?
There may be solutions, and it’s certainly worth a go.
You write, “We do this by presenting many small fragments of the manifestos and asking a participant to judge which party they are from. By gathering many many judgements …”
I’m surprised to see no reference. I mean … you have a link to Wikipedia for “Receiver Operating Characteristic” but nothing for this lovely method.
Do you have something published?
@bentrem
p.s. “we could show that vision is more sensitive than hearing” … oh my. We can certainly compare vision in subject A with subject B, or against some sort of norm or average … but … vision more sensitive than hearing? That makes no sense to me.
–ben
Good points by commenters regarding the difference between political statements and track records, but I disagree that track records are necessarily the more appropriate choice for this research question. The research is concerned with how people perceive a party’s views, and I think that for most people, most of this perception is based on stated political positions more so than prior track records. Furthermore, statements of political position are how a party chooses to convey its views to the public. Whether people can really discriminate between these statements is therefore an entirely relevant question.
@Jake yes, agreed – that’s why m. Greene’s suspicion would be so interesting if confirmed! Finding the d’ would address the assumption that people are voting according to what politician’s say, rather than what they’ve done (or other factors like tribal loyalty)
@ben google the Weber fraction!
Yes, right … will google.
My question was more about your work, the work you describe in the item, why your stuff is self-referential but not re-entrant.
re: “would address the assumption that people are voting according to what politician’s say, rather than what they’ve done” … this ship sailed years ago.
Measuring an audience during a Ronal Reagan speech it was found that individuals experienced a positive reaction even when the “great communicator” was describing policies with which they disagreed.
No, “weber fraction” doesn’t apply. I was doing psycho-acoustics in the 70s, FWIW … the early days of FM Stereo.
It seems you totally ignored one question and then totally mis-read the other.
Yes, 1 dB is the least perceptible change.
But what I questioned was your comparing sensitivity of hearing to sensitivity of site … cross-sense comparison.
It seems to have plenty of time for supporters, for those who flatter. But you don’t seem to have any time at all for substantial questions.
You wrote ““We do this by presenting many small fragments of the manifestos and asking a participant to judge which party they are from. By gathering many many judgements …” and I ask where your work has been published, if at all. No reply.
Where was this work done? In what lab? With what organization?
This is legitimately interesting. But I think you just made it up.
Gedanken?
p.s. linking to Wikipedia but not to your work … then writing “Google Weber fraction” … that’s contempt. Yes, I know, most folk couldn’t care less. But some of us do.
My problem here is with taking the opinions seriously in the first place, and not seeing them for what they: facile, half-baked, off-the-cuff (one hopes) and unthought-out comments.
Maybe this is the problem with questionnare-type data; I for one refuse them wherever possible because I cannot answer them as they are intended to be answered.
Or am I being facile etc etc? I suspect myself of a little knee-jerkiness, here.
Political scientists already have a measure that does what you want. It’s called “political sophistication”, and it’s a simple measurement that addresses the degree to which people’s ideas about a given party’s platform actually FIT that platform. If you’ve access to JSTOR, go there and do a keyword search.
@lester – thanks for the keyword. I’ll look it up. I still suspect that there is something special in measuring people’s actual ability to distinguish manifesto statements from different parties, rather than having an expert or some such code different manifestos for ‘true’ position.
@mrm this relates to your point, I’m suggesting taking judgements from people, not their opinions via a questionnaire – so maybe it would get around some of the suspicion of the answers people give to questionnaires
@ben I think you’ve got the wrong end of the stick. I haven’t published anything about this research. It is a suggestion.
As for comparing between the senses, I promise that the Weber Fraction will do this for you. It is a measure of the change in stimulus intensity needed to make a stimulus just noticeably different. Weber (1795 – 1878) found that this amount is a percentage of the current strength of a stimulus – so you notice a small increase in light in a dim room, but not a small increase of light in a bright room (although the change in the stimulus is constant, the proportional change in the second case is much smaller and hence now below the Weber fraction for being a perceptible difference). Incidentally, this is why you can see the stars at night (on their dark background), but you can’t see the stars during the daytime (on their bright background). They are still “there”, just as luminous, but not visible because the proportion different from the background is not large enough during the day.
By comparing Weber fractions, say for intensity of white light with intensity of white noise, we can get a comparison of sensitivity between different senses. Now it isn’t the only way of comparing, but it does allow you to put them on the same scale.