Preregistration, a Boring Ass Word for a very Important Proposal

science

In response to open letter from more than 80 scientists in the Guardian last month, Sophie Scott (deputy director of the Institute of Cognitive Neuroscience at University College London) has a post in the Times Higher Education. She argues that preregistration would threaten scientific progress by limiting exploratory and speculative research, imposing a sort of methodological tyranny (my word, not hers) requiring researchers to adhere to a strict hypothesis-testing format. She also claims that preregistration could lock researchers into publishing in a journal in which they have preregistered a study, even when surprising results may be a better fit for another venue. She closes by suggesting that preregistration could stifle the kind of genius exhibited by scientists like Newton, whose lesser known pursuits included forays into alchemy.

Newton! Galileo! Science just wants to be free! Fight the man, right?

Well, no. The mistake that Scott makes in this column is so common from people who oppose preregistration, that it’s frankly getting a little tiresome. Preregistration would not suppress exploratory research. It would not press the jackboot against the throat of scientific serendipity. It doesn’t aim to root out speculation and force all investigations into the mold of radical empiricism. Why? Because preregistration is not exclusive.

Yes, I suppose that some people have talked about journals and funding agencies requiring preregistration in the same way that it is required for clinical trials, but there are several reasons to believe such proposals would not bring the death of creative research. First of all, nothing about these plans demands that you publish only results based on preregistered hypotheses. Spontaneous observations, chance findings, novel analyses all can reasonably appear in a preregistered study. Should I repeat that?  It doesn’t seem to be getting through to the critics. Again, preregistration doesn’t exclude unanticipated results, it simply requires that you label them as such. Also, despite the huge groundswell of public support [crickets], and the hegemonic domination of science publishing by preregistration advocates (really?), there will always be journals that will publish observational and speculative studies. If those studies are good, they will be cited.

Really, what’s mind-boggling about Scott’s defense of the status quo is the failure to even acknowledge what’s wrong with current statistical practices in the sciences. As pointed out skillfully in Slate by Andrew Gelman, researchers are able to instantaneously test huge numbers of hypotheses essentially unconsciously while ‘exploring’ their data, and yet these multiple comparisons remain uncorrected in the final analysis. Instead, the paper is written as if the researchers had the specific comparisons in mind from the beginning: p<0.05, BAM! Science Accomplished!

Is this just the inevitable Kuhnian lurching of science, as Scott suggests, or is it a systemic problem for which we can create institutional incentives for better practices?  First of all, it’s completely ridiculous to turn the observation that scientific progress has historically been paroxysmal (and politically contentious) into an argument that we don’t need to worry about the way we do hypothesis testing. Preregistration is not about excluding exploratory or speculative work, it is about setting up an infrastructure to reward researchers for using practices that are more likely to generate results which are meaningful, reliable and reproducible. What could be wrong with that?

So maybe the problem is branding. Instead of “preregistration”, perhaps we should call it “honest science labeling.”

Photo Credit: From flickr user judy_breck under Creative Commons.

Update: Minor edit about 15 minutes after posting to clarify Sophie Scott’s affiliation.

~ by nucamb on July 25, 2013.

23 Responses to “Preregistration, a Boring Ass Word for a very Important Proposal”

  1. There’d be a lot more contentious findings published with preregistration – negative ones mostly (though not exclusively).

    It might turn out that many of our cherished assumptions are wrong – and they only seemed right for so long because people were able to (and pressured into) p-hack support for them from any old data.

    I welcome the end to that conservativism: let’s look the truth in the face. Science must cast off its chains.

  2. “Cast off them chains indeed !!”

    Nice piece, thanks for writing it.

  3. It’s not just a branding issue. And it’s not that Sophie’s not understood the idea behind pre-registration. Instead Sophie’s point is completely legitimate, but possibly based on a different model of how peer review works. You believe pre-registration will work efficiently and that reviewers won’t just kill everything that’s observational. I think the opposite, and that pre-registration will make lots of interesting science harder to publish by playing to the prejudices of some reviewers. Similar overzealous rejecting of papers currently happens based on reviewers mis-applying statistical or methodological criticisms, for example, motion in neuroimaging, multiple comparisons or double dipping (e.g., http://www.sciencedirect.com/science/article/pii/S1053811912003990).

    Either way. It’s not a theoretical question as to whether pre-registration is a good approach or not. It’s a practical question, it’s about how science is in practice and will it improve science or not. You and others think it will. I think it’ll make it worse. So, why not put it to the empirical test. Set up a pre-registered trial to test whether pre-registration works, and show that we’re wrong.

    • I definitely agree that is an empirical question, but let me get clear on your argument. I tried to make the point that no pre-reg system would be exclusive, but you are suggesting that pre-reg studies would become so dominant that observational/exploratory studies would be essentially stigmatized and marginalized out of the mainstream. Is that it?

      You and Sophie have this scenario where a creative and intelligent scientists submits an elegant observational study for pre-reg and are denied because they don’t have a sufficiently hypothesis-driven approach (even if that criticism is purely a fiction of an antagonistic reviewer). What I’m saying is that I don’t know what is to prevent that scientist from submitting the same paper as a non-pre-reg paper to the same journal (or some other journal) once it’s done.

      My point is that in this era when computational methods and visualizations allow us to implicitly test thousands of hypotheses without really thinking about it (and certainly without accounting for it in the resulting paper), we should build a mechanism by which researchers can get credit for not dredging their data. This is really just a question for research which attempts to use a p-value or a confidence interval to establish the likely non-spuriousness of their results.If such a modest proposal is really so dangerous to the viability of other methods, perhaps we have more fundamental disagreements about the scientific method in general.

      • “I definitely agree that is an empirical question… I tried to make the point that no pre-reg system would be exclusive, but you are suggesting that pre-reg studies would become so dominant that observational/exploratory studies would be essentially stigmatized and marginalized out of the mainstream. Is that it?”
        I believe there is a substantial danger of down-weighting of non-pre-reg studies. And if it that does happen, the result may be to slow down the development of neuroscience.

        I think of it like this:
        We all know there are: (1) many studies out there that report some effects that are not true (in that some of the results are due to noise). (2) There are also many studies which are true, in the sense that their results are not due to chance, but which don’t really progress things much. Both of these types of studies will get swept away and become insignificant and ignored when the next big paradigm shift tsunami washes over us. Finally there are (3), the smaller number of much more influential studies that radically change things. Now, what we, as a field, really want to do is encourage (3), they carry much more scientific weight in terms of “truth”. I suspect pre-registration will very successfully eliminate (1), I suspect it will increase (2). But I think the collateral damage will be to reduce (3) because of a cultural shift towards established methodology/interpretations and away from risk taking and exploratory science. And that would be a bad thing. Getting rid of studies of type (1) (studies which will be ignored in the long run) would not make up for it.

  4. “Instead Sophie’s point is completely legitimate,”

    I have a hard time grasping what the point is, except “this is not appropriate for all scientific research” (but I don’t think that’s what the call for pre-registration is about)

    “I think the opposite, and that pre-registration will make lots of interesting science harder to publish by playing to the prejudices of some reviewers”

    Isn’t this the case in the current system? (i.e. journals publish stuff that their editors view as “interesting” or “groundbreaking”)

    “It’s not a theoretical question as to whether pre-registration is a good approach or not”

    Isn’t it theoretical in the sense that such a model would solve some issues concerning QRP’s for instance, just from a reasoning-perspective? (i.e. you would pre-register all analyses you would do, and therefore you can’t leave any out when you write up your article?)

    “So, why not put it to the empirical test. Set up a pre-registered trial to test whether pre-registration works, and show that we’re wrong.”

    I would be very interested in this. Compare replications for pre-registered studies with replications for non-pre-registered studies. Great !!

  5. “Finally there are (3), the smaller number of much more influential studies that radically change things. Now, what we, as a field, really want to do is encourage (3)”

    I find this an interesting argument, and also never understand it. Here is an attempt to explain why:

    I reason that findings/studies will have the most chance of “radically changing things” when they are true in some way. For me that would then entail that the 1st part of your no. 2 “There are also many studies which are true, in the sense that their results are not due to chance, [but which don’t really progress things much]” would hold true. You seem to agree that no. 2 (so also the 1st part of it) could possibly happen when more studies would be pre-registered.

    Reasoning from that perspective, I note 2 things:

    1. If studies which radically change things would as a characteristic have that their results are not due to chance, then I would reason that pre-registration could then also be useful for no. 3 (i.c. the 1st part of your no.2 would also be necessary for no. 3 and pre-registration could help with that)

    2. I would think that not spending money, time, and effort on no. 1 studies (“(1) many studies out there that report some effects that are not true (in that some of the results are due to noise)”) would be possibly useful in not slowing down science. You seem to agree that no. 1 studies could possibly be decreased when using pre-registration.

    Taken together, I wonder if pre-registration could in fact speed up scientific progress.

  6. I think that the proponents of pre-registration should lead the way. They should pre-register their studies and show everybody that this is a sustainable model for scientific studies.

    Right now, 2,5 months after the creation of registered reports in Cortex, I haven’t heard about any successful submission. Would the articles already be published at the IPA (in-principle acceptance) stage?

    At this stage, I’m intrigued by the idea of registering my studies but there are no example available for this new process. So the 80 scientists who signed the Guardian article should register their studies. They should also provide an honest account of the process (time required for the reviews, usefulness of the review, example of the IPA document, etc.)

  7. P.C.,
    Thanks for those links (above; can’t embed replies more than 3 deep in this theme). I would really like the anti-prereg folks to engage with the criticisms of current practice. Do they believe that data-dredging and hidden post-hoc analysis is not a problem, or simply that the pre-reg cure is worse than the disease?

    • I think most opponents of pre-reg accept that publication bias is out there and fairly widespread, but think that the cure is worse than the disease. Although if you believe in publication bias, then there is very likely a publication bias affecting studies looking a publication bias (almost a paradox), so the problem may be less bad than we fear.

      • Publication bias is only part of the problem. It’s also about what the Slate piece calls ‘researcher degrees of freedom’, or undisclosed post hoc comparisons that creep into the analysis process simply because computational methods have made it so easy to try so many things. Of course as researchers we try to restrain ourselves from too much data dredging, and as reviewers we try to call out statistical methods that seem too convoluted and ad hoc (at least against the effect sizes), but it’s still a problem.

  8. In response to “I don’t think it can get any worse”, it definitely can: have a look at: http://en.wikipedia.org/wiki/Trofim_Lysenko
    Or else, read up on the corrosive effects of Chomsky on psychology and linguistics research.

    • Are you saying that it can get worse in the sense that you can have an entire scientific discipline that it completely and fundamentally wrong, or is it something about state control and ideologically driven science? If it’s the former, then well yes, but I’m not sure how that relates specifically to pre-reg. If it is the latter, then I guess I was right to make fun (above) of Scott’s implication that prereg was some kind of tyrannical zealotry; I thought I was exaggerating for comic effect.

      Also, I would be interested to hear more about this Chomsky critique. Since he is reasonably credited with founding much of modern linguistics and wrenching psychology from the dark ages of behaviorism (in just 32 pages of text), I’m not sure I know what corrosive effects you are talking about (though I am familiar with many, and agree with some, critiques of Chomsky). Can you explain this line of work in a little more detail so I can check it out?

      • I was just responding to the poster above, who used a really irritating rhetorical flourish to say it can’t get worse. Science can get much worse along many dimensions. I didn’t mean that pre-reg will cause it to get worse in that way.

        Chomsky certainly didn’t wrench psychology from behaviourism – Vygotsky and Piaget predate him. As does cognitive work of people like Cherry, and all the AI stuff of people like Herbert Simon etc.

    • Yes, there were antecedents to Chomsky’s work, but he has also been extremely influential. I would still like to hear in detail about how his work has been “corrosive.” As I say, I’ve heard many critiques of his work, but I don’t know specifically what line of reasoning you are referring to (or how it can be used as a symbol of science gone bad).

  9. “I was just responding to the poster above, who used a really irritating rhetorical flourish to say it can’t get worse”

    I apologize for this comment then. It was/is not my intention to be irritating. It was just something I thought about when reading the comment.

    I have the greatest respect for scientists who try and think critically, try and come up with ideas which can possibly help improve science, etc. I valso ery much appreciate the willingness of scientists (e.g. through blogs like this, and commenting on things, etc.) to engage in talking about these very diffcult issues.

    Please know I did not mean to be irritating, and I apologize if I have been/ if it came across that way.

    • I would also like to add that I think you are absolutely correct in that things can get worse. I merely took those articles as possibly pointing to specific issues like the file-drawer problem, HARK-ing, undisclosed data-analysis-issues, etc. for which pre-registration could help (which is also why I thought the pre-registration idea was so interesting). Reasoning only from that perspective, in combination with the linked articles, I made the comment.

      Again, I apologize if I have been irritating. I have the greatest respect for people’s willingness to talk about these issues, and share their thoughts. Just from that perspective alone, please know I have the greatest respect for this, and did not mean to be irritating, and I apologize if I have been.

  10. […] Michael Collins […]

  11. […] the debate on Twitter Branch Conversation Neuroskeptic James Kilner | (follow-up) Daniel Lakens Michael Collins Ase Kvist Innes-Ker Suzi Gage Dorothy Bishop July 2013 | Jan 2013 Nate Kornell Martin Larsson […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 45 other followers

%d bloggers like this: