Preregistration, a Boring Ass Word for a very Important Proposal

In response to open letter from more than 80 scientists in the Guardian last month, Sophie Scott (deputy director of the Institute of Cognitive Neuroscience at University College London) has a post in the Times Higher Education. She argues that preregistration would threaten scientific progress by limiting exploratory and speculative research, imposing a sort of methodological tyranny (my word, not hers) requiring researchers to adhere to a strict hypothesis-testing format. She also claims that preregistration could lock researchers into publishing in a journal in which they have preregistered a study, even when surprising results may be a better fit for another venue. She closes by suggesting that preregistration could stifle the kind of genius exhibited by scientists like Newton, whose lesser known pursuits included forays into alchemy.

Newton! Galileo! Science just wants to be free! Fight the man, right?

Well, no. The mistake that Scott makes in this column is so common from people who oppose preregistration, that it’s frankly getting a little tiresome. Preregistration would not suppress exploratory research. It would not press the jackboot against the throat of scientific serendipity. It doesn’t aim to root out speculation and force all investigations into the mold of radical empiricism. Why? Because preregistration is not exclusive.

Yes, I suppose that some people have talked about journals and funding agencies requiring preregistration in the same way that it is required for clinical trials, but there are several reasons to believe such proposals would not bring the death of creative research. First of all, nothing about these plans demands that you publish only results based on preregistered hypotheses. Spontaneous observations, chance findings, novel analyses all can reasonably appear in a preregistered study. Should I repeat that?  It doesn’t seem to be getting through to the critics. Again, preregistration doesn’t exclude unanticipated results, it simply requires that you label them as such. Also, despite the huge groundswell of public support [crickets], and the hegemonic domination of science publishing by preregistration advocates (really?), there will always be journals that will publish observational and speculative studies. If those studies are good, they will be cited.

Really, what’s mind-boggling about Scott’s defense of the status quo is the failure to even acknowledge what’s wrong with current statistical practices in the sciences. As pointed out skillfully in Slate by Andrew Gelman, researchers are able to instantaneously test huge numbers of hypotheses essentially unconsciously while ‘exploring’ their data, and yet these multiple comparisons remain uncorrected in the final analysis. Instead, the paper is written as if the researchers had the specific comparisons in mind from the beginning: p<0.05, BAM! Science Accomplished!

Is this just the inevitable Kuhnian lurching of science, as Scott suggests, or is it a systemic problem for which we can create institutional incentives for better practices?  First of all, it’s completely ridiculous to turn the observation that scientific progress has historically been paroxysmal (and politically contentious) into an argument that we don’t need to worry about the way we do hypothesis testing. Preregistration is not about excluding exploratory or speculative work, it is about setting up an infrastructure to reward researchers for using practices that are more likely to generate results which are meaningful, reliable and reproducible. What could be wrong with that?

So maybe the problem is branding. Instead of “preregistration”, perhaps we should call it “honest science labeling.”

Photo Credit: From flickr user judy_breck under Creative Commons.

Update: Minor edit about 15 minutes after posting to clarify Sophie Scott’s affiliation.

~ by nucamb on July 25, 2013.