Month: November 2017

Open science essentials in 2 minutes, part 2

The Open Science Framework (osf.io) is a website designed for the complete life-cycle of your research project – designing projects; collaborating; collecting, storing and sharing data; sharing analysis scripts, stimuli, results and publishing results.

You can read more about the rationale for the site here.

Open Science is fast becoming the new standard for science. As I see it, there are two major drivers of this:

1. Distributing your results via a slim journal article dates from the 17th century. Constraints on the timing, speed and volume of scholarly communication no longer apply. In short, now there is no reason not to share your full materials, data, and analysis scripts.

2. The Replicability crisis means that how people interpret research is changing. Obviously sharing your work doesn’t automatically make it reliable, but since it is a costly signal, it is a good sign that you take the reliability of your work seriously.

You could share aspects of your work in many ways, but the OSF has many benefits

• the OSF is backed by serious money & institutional support, so the online side of your project will be live many years after you publish the link
• It integrates with various other platform (github, dropbox, the PsyArXiv preprint server)
• Totally free, run for scientists by scientists as a non-profit

All this, and the OSF also makes easy things like version control and pre-registration.

Good science is open science. And the fringe benefit is that making materials open forces you to properly document everything, which makes you a better collaborator with your number one research partner – your future self.

Cross-posted at tomstafford.staff.shef.ac.uk.  Part of a series aimed at graduate students in psychology. Part 1: pre-registration.

 

Open Science essentials in 2 minutes, part 1

The Problem

As a scholarly community we allowed ourselves to forget the distinction between exploratory vs confirmatory research, presenting exploratory results as confirmatory, presenting post-hoc rationales as predictions. As well as being dishonest, this makes for unreliable science.

Flexibility in how you analyse your data (“researcher degrees of freedom“) can invalidate statistical inferences.

Importantly, you can employ questionable research practices like this (“p-hacking“) without knowing you are doing it. Decide to stop an analysis because the results are significant? Measure 3 dependent variables and use the one that “works”? Exclude participants who don’t respond to your manipulation? All justified in exploratory research, but mean you are exploring a garden of forking paths in the space of possible analysis – when you arrive at a significant result, you won’t be sure you got there because of the data, or your choices.

The solution

There is a solution – pre-registration. Declare in advance the details of your method and your analysis: sample size, exclusion conditions, dependent variables, directional predictions.

You can do this

Pre-registration is easy. There is no single, universally accepted, way to do it.

• you could write your data collection and analysis plan down and post it on your blog.
• you can use the Open Science Framework to timestamp and archive a pre-registration, so you can prove you made a prediction ahead of time.
• you can visit AsPredicted.org which provides a form to complete, which will help you structure your pre-registration (making sure you include all relevant information).
• “Registered Reports“: more and more journals are committing to published pre-registered studies. They review the method and analysis plan before data collection and agree to publish once the results are in (however they turn out).

You should do this

Why do this?

• credibility – other researchers (and journals) will know you predicted the results before you got them.
• you can still do exploratory analysis, it just makes it clear which is which.
• forces you to think about the analysis before collecting the data (a great benefit).
• more confidence in your results.

Further reading

• A pre-registration primer by Tom Hardwicke: https://osf.io/8uz2g/ (see also for background references on the state of psychology and p-hacking)
• Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., … & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
• Psychology’s ‘registration revolution’, Chris Chambers in The Guardian

 

Addendum 14/11/17

As luck would have it, I stumbled across a bunch of useful extra resources in the days after publishing this post

• Another template for pre-registration, via @openstatslab
• A Preregistration Coaching Network from the Center for Open Science
• How to properly pre-register a study from the people who brought you AsPredicted.org
• Two examples of pre-registration. 1. (by us) via OSF, 2. via Sheffield ORDA
• Slides from SfN talk on pre-registration from COS’s David Mellor
• A consortium pre-registration for undergraduate projects: letter in The Psychologist, slides from Kate Button at Oxford Reproducibility Lectures

Cross-posted on at tomstafford.staff.shef.ac.uk.  Part of a series aimed at graduate students in psychology. Part 2: The Open Science Framework