Open Science Essentials: The Open Science Framework

Open science essentials in 2 minutes, part 2

The Open Science Framework (osf.io) is a website designed for the complete life-cycle of your research project – designing projects; collaborating; collecting, storing and sharing data; sharing analysis scripts, stimuli, results and publishing results.

You can read more about the rationale for the site here.

Open Science is fast becoming the new standard for science. As I see it, there are two major drivers of this:

1. Distributing your results via a slim journal article dates from the 17th century. Constraints on the timing, speed and volume of scholarly communication no longer apply. In short, now there is no reason not to share your full materials, data, and analysis scripts.

2. The Replicability crisis means that how people interpret research is changing. Obviously sharing your work doesn’t automatically make it reliable, but since it is a costly signal, it is a good sign that you take the reliability of your work seriously.

You could share aspects of your work in many ways, but the OSF has many benefits

  • the OSF is backed by serious money & institutional support, so the online side of your project will be live many years after you publish the link
  • It integrates with various other platform (github, dropbox, the PsyArXiv preprint server)
  • Totally free, run for scientists by scientists as a non-profit

All this, and the OSF also makes easy things like version control and pre-registration.

Good science is open science. And the fringe benefit is that making materials open forces you to properly document everything, which makes you a better collaborator with your number one research partner – your future self.

Cross-posted at tomstafford.staff.shef.ac.uk.  Part of a series aimed at graduate students in psychology. Part 1: pre-registration.

 

Open Science Essentials: pre-registration

Open Science essentials in 2 minutes, part 1

The Problem

As a scholarly community we allowed ourselves to forget the distinction between exploratory vs confirmatory research, presenting exploratory results as confirmatory, presenting post-hoc rationales as predictions. As well as being dishonest, this makes for unreliable science.

Flexibility in how you analyse your data (“researcher degrees of freedom“) can invalidate statistical inferences.

Importantly, you can employ questionable research practices like this (“p-hacking“) without knowing you are doing it. Decide to stop an analysis because the results are significant? Measure 3 dependent variables and use the one that “works”? Exclude participants who don’t respond to your manipulation? All justified in exploratory research, but mean you are exploring a garden of forking paths in the space of possible analysis – when you arrive at a significant result, you won’t be sure you got there because of the data, or your choices.

The solution

There is a solution – pre-registration. Declare in advance the details of your method and your analysis: sample size, exclusion conditions, dependent variables, directional predictions.

You can do this

Pre-registration is easy. There is no single, universally accepted, way to do it.

  • you could write your data collection and analysis plan down and post it on your blog.
  • you can use the Open Science Framework to timestamp and archive a pre-registration, so you can prove you made a prediction ahead of time.
  • you can visit AsPredicted.org which provides a form to complete, which will help you structure your pre-registration (making sure you include all relevant information).
  • Registered Reports“: more and more journals are committing to published pre-registered studies. They review the method and analysis plan before data collection and agree to publish once the results are in (however they turn out).

You should do this

Why do this?

  • credibility – other researchers (and journals) will know you predicted the results before you got them.
  • you can still do exploratory analysis, it just makes it clear which is which.
  • forces you to think about the analysis before collecting the data (a great benefit).
  • more confidence in your results.

Further reading

 

Addendum 14/11/17

As luck would have it, I stumbled across a bunch of useful extra resources in the days after publishing this post

Cross-posted on at tomstafford.staff.shef.ac.uk.  Part of a series aimed at graduate students in psychology. Part 2: The Open Science Framework

Don’t speculate on others’ mental health from afar

In The Guardian, Nick Davis makes a clear and timely case for affirming The Goldwater Rule. The Rule, which binds members of the American Psychiatric Association, forbids giving an opinion on the mental state of someone you have not examined.

The US president’s behaviour has brought the rule back into the public eye, but Davis argues that we shouldn’t lose sight of the importance of the Rule, and how it protects us all from speculation about our mental health – speculation which is often flavoured by simple prejudice:

Read the article here: The Goldwater rule: why commenting on mental health from a distance is unhelpful 

The Enigma of Reason (review)

The Enigma of Reason: A New Theory of Human Understanding, by Hugo Mercier and Dan Sperber was published in April, and I have a review in this week’s Times Higher Education.

The books follows on and expands on their landmark ‘Why do humans reason? Arguments for an argumentative theory‘, published in 2011 in Behavioral and Brain Sciences.

The core of the argumentative theory is this (quoting my review):

reasoning is primarily a social, rather than an individual, tool. Here the purpose of reasoning is not inquisitive, but instead justificatory – we provide reasons to other people, and we evaluate the reasons provided by other people. The niche of reasoning is in the highly social world of human cooperative groups, a niche where it is highly advantageous to be able to transfer information and trust between individuals who are not kin

You can read the full review on the THE site, but I highly recommend checking out the book. It’s a fantastic example of a book which has both theoretical depth and reach, connecting fundamental theoretical perspectives across cognitive science to give a provocative and satisfying account of the nature of human reasoning.

You can also check out Hugo Mercier’s pages about the argumentative theory, which has links to experiments suggested by the theory (which have by and large confirmed predictions it makes).

What triggers that feeling of being watched?

You feel somebody is looking at you, but you don’t know why. The explanation lies in some intriguing neuroscience and the study of a strange form of brain injury.

Something makes you turn and see someone watching you. Perhaps on a busy train, or at night, or when you’re strolling through the park. How did you know you were being watched? It can feel like an intuition which is separate from your senses, but really it demonstrates that your senses – particularly vision – can work in mysterious ways.

Intuitively, many of us might imagine that when you look at something with your eyes, signals travel to your visual cortex and then you have the conscious experience of seeing it, but the reality is far weirder.

Once information leaves our eyes it travels to at least 10 distinct brain areas, each with their own specialised functions. Many of us have heard of the visual cortex, a large region at the back of the brain which gets most attention from neuroscientists. The visual cortex supports our conscious vision, processing colour and fine detail to help produce the rich impression of the world we enjoy. But other parts of our brain are also processing different pieces of information, and these can be working away even when we don’t – or can’t – consciously perceive something.

The survivors of neural injury can cast some light on these mechanisms. When an accident damages the visual cortex, your vision is affected. If you lose all of your visual cortex you will lose all conscious vision, becoming what neurologists call ‘cortically blind’. But, unlike if you lose your eyes, cortically blind is only mostly blind – the non-cortical visual areas can still operate. Although you can’t have the subjective impression of seeing anything without a visual cortex, you can respond to things captured by your eyes that are processed by these other brain areas.

In 1974 a researcher called Larry Weiskrantz coined the term ‘blindsight’ for the phenomenon of patients who were still able to respond to visual stimuli despite losing all conscious vision due to destruction of the visual cortex. Patients like this can’t read or watch films or anything requiring processing of detail, but they are – if asked to guess – able to locate bright lights in front of them better than mere chance. Although they don’t feel like they can see anything, their ‘guesses’ have a surprising accuracy. Other visual brain areas are able to detect the light and provide information on the location, despite the lack of a visual cortex. Other studies show that people with this condition can detect emotions on faces and looming movements.

More recently, a dramatic study with a blindsight patient has shown how we might be able feel that we are being looked at, without even consciously seeing the watchers’ face. Alan J Pegna at Geneva University Hospital, Switzerland, and team worked with a man called TD (patients are always referred to by initials only in scientific studies, to preserve anonymity). TD is a doctor who suffered a stroke which destroyed his visual cortex, leaving him cortically blind.

People with this condition are rare, so TD has taken part in a string of studies to investigate exactly what someone can and can’t do without a visual cortex. The study involved looking at pictures of faces which had their eyes directed forward, looking directly at the viewer, or which had their eyes averted to the side, looking away from the viewer. TD did this task in an fMRI scanner which measured brain activity during the task, and also tried to guess which kind of face he was seeing. Obviously for anyone with normal vision, this task would be trivial – you would have a clear conscious visual impression of the face you were looking at at any one time, but recall that TD has no conscious visual impression. He feels blind.

The scanning results showed that our brains can be sensitive to what our conscious awareness isn’t. An area called the amygdala, thought to be responsible for processing emotions and information about faces, was more active when TD was looking at the faces with direct, rather than averted, gaze. When TD was being watched, his amygdala responded, even though he didn’t know it. (Interestingly, TD’s guesses as to where he was being watched weren’t above chance, and the researchers put this down to his reluctance to guess.)

Cortical, conscious vision, is still king. If you want to recognise individuals, watch films or read articles like this you are relying on your visual cortex. But research like this shows that certain functions are simpler and maybe more fundamental to survival, and exist separately from our conscious visual awareness.

Specifically, this study showed that we can detect that people are looking at us within our field of view – perhaps in the corner of our eye – even if we haven’t consciously noticed. It shows the brain basis for that subtle feeling that tells us we are being watched.

So when you’re walking that dark road and turn and notice someone standing there, or look up on the train to see someone staring at you, it may be your nonconscious visual system monitoring your environment while you’re conscious attention was on something else. It may not be supernatural, but it certainly shows the brain works in mysterious ways.

This is my BBC Future column from last week. The original is here.

This map shows what white Europeans associate with race – and it makes for uncomfortable reading

File 20170425 12650 16jxfww
The ConversationThis new map shows how easily white Europeans associate black faces with negative ideas. The ConversationSince 2002, hundreds of thousands of people around the world have logged onto a website run by Harvard University called Project Implicit and taken an “implicit association test” (IAT), a rapid-response task which measures how easily you can pair items from different categories.To create this new map, we used data from a version of the test which presents white or black faces and positive or negative words. The result shows how easily our minds automatically make the link between the categories – what psychologists call an “implicit racial attitude”.Each country on the map is coloured according to the average score of test takers from that country. Redder countries show higher average bias, bluer countries show lower average bias, as the scale on the top of the map shows.Like a similar map which had been made for US states, our map shows variation in the extent of racial bias – but all European countries are racially biased when comparing blacks versus whites.

In every country in Europe, people are slower to associate blackness with positive words such as “good” or “nice” and faster to associate blackness with negative concepts such as “bad” or “evil”. But they are quicker to make the link between blackness and negative concepts in the Czech Republic or Lithuania than they are in Slovenia, the UK or Ireland.

No country had an average score below zero, which would reflect positive associations with blackness. In fact, none had an average score that was even close to zero, which would reflect neither positive nor negative racial associations.

A screeshot from the online IAT test.
IAT, Project Implict

Implicit bias

Overall, we have scores for 288,076 white Europeans, collected between 2002 and 2015, with sample sizes for each country shown on the left-hand side.

Because of the design of the test it is very difficult to deliberately control your score. Many people, including those who sincerely hold non-racist or even anti-racist beliefs, demonstrate positive implicit bias on the test. The exact meaning of implicit attitudes, and the IAT, are controversial, but we believe they reflect the automatic associations we hold in our minds, associations that develop over years of immersion in the social world.

Although we, as individuals, may not hold racist beliefs, the ideas we associate with race may be constructed by a culture which describes people of different ethnicities in consistent ways, and ways which are consistently more or less positive. Looked at like this, the IAT – which at best is a weak measure of individual psychology – may be most useful if individuals’ scores are aggregated to provide a reflection on the collective social world we inhabit.

The results shown in this map give detail to what we already expected – that across Europe racial attitudes are not neutral. Blackness has negative associations for white Europeans, and there are some interesting patterns in how the strength of these negative associations varies across the continent.

North and west Europe, on average, have less strong anti-black associations, although they still have anti-black associations on average. As you move south and east the strength of negative associations tends to increase – but not everywhere. The Balkans look like an exception, compared to surrounding countries. Is this because of some quirk about how people in the Balkans heard about Project Implicit, or because their prejudices aren’t orientated around a white-black axis? For now, we can only speculate.

Open questions

When interpreting the map there are at least two important qualifications to bear in mind.

The first is that the scores only reflect racial attitudes in one dimension: pairing white/black with goodness/badness. Our feelings about ethnicity have many more dimensions which aren’t captured by this measure.

The second is that the data comes from Europeans who visit the the US Project Implicit website, which is in English. We can be certain that the sample reflects a subset of the European population which are more internet-savvy than is typical. They are probably also younger, and more cosmopolitan. These factors are likely to underweight the extent of implicit racism in each country, so that the true levels of implicit racism are probably higher than shown on this map.

This new map is possible because Project Implicit release their data via the Open Science Framework. This site allows scientists to share the raw materials and data from their experiments, allowing anyone to check their working, or re-analyse the data, as we have done here. I believe that open tools and publishing methods like these are necessary to make science better and more reliable.

This article was originally published on The Conversation. Read the original article.

Edit 4/5/17. The colour scale chosen for this map emphasises the differences between countries. While that’s most important for working out what drives IAT scores, the main take-away from the map is that all of Europe is considerably not neutral. That conclusion is supported by a continuous colour scale, as used in this version of the map here

neurotransmitter fashion

A graph of scientific articles published per year which mention four major neurotransmitters in their title:

What I take from this is

  • Dopamine is king! And with great popularity, comes great misrepresentation.
  • What happened to glutamate research in the mid 1990s?
  • The recent hype about oxytocin doesn’t seem to be driven by a spike in the primary literature.
  • Nor does the hype about serotonin. Yes, publications increase on this neurotransmitter, but not compared to glutamate. And most people haven’t heard about glutamate, despite it being more abundant.

Technical note: I scraped the data from google scholar using scholar.py by Christian Kriebich

Update: here’s the raw data, should you want it