Preregistration: More Promises than Pitfalls

Or, maybe the real pitfalls are different from what we usually think

Ok, I admit it; I went to a scientific meeting on preregistration. Two full days, the only topic was preregistration. When I confessed this to my friends and colleagues who are not engrossed in the worlds of metascience, open science, and scientific reform, it seemed absurd, comical even. How could there be that much to discuss about preregistration? Some—which may include some of you readers—had no idea what “preregistration” even is!

It turns out that there is plenty to discuss, as the two days were filled with boisterous talks and rousing discussions. On the other hand, if you give a bunch of academics two days to talk about a topic—any topic—I am certain that they will go right ahead and do so, all the while treating it as very serious and important. Whether such discussions are actually useful is another matter.

For those not acquainted, preregistration refers to posting a time-stamped, discoverable study plan that outlines the research questions, hypotheses, methods, and planned analyses prior to data collection and/or analysis (Nosek et al., 2018). It has been proposed as one of many reforms to our research workflow that has the potential to help improve research quality. Preregistration has become increasingly prevalent across many disciplines, but has not been without its critics. Disagreements about the practice set the stage for the meeting, “The promises and pitfalls of preregistration” at The Royal Society in London.

The purpose of this post is to reflect on the meeting¹ and synthesize my observations with other ongoing discussions in the field. Although inspired by the meeting, I am providing a broader discussion, so this post should be useful even if you did not attend the meeting.

It is risky to say this kind of thing publicly, but I consider myself to be well-acquainted with the practices and perspectives on preregistration. That is not to say that I believe I know everything there is to know. That would be foolish. Rather, what I mean is that I have closely followed discussions and innovations in preregistration over the past 12 years, use them, write about them, teach about them, consult with others about them, dream about them, and so on.

From this vantage point, I continually see criticisms about preregistration that reveal that the criticizer is not fully informed on the subject. Although I’m sure people could find plenty of personal violations of this belief in my writings and soapboxing, I strongly believe that one should have a firm understanding of something before they engage in public criticism. With preregistration, and open science practices in general, folks seem to feel free to fire off half-baked criticisms, raising issues that have already been fully addressed. This was on full display at the meeting, with comments about how preregistration prohibits exploratory analysis (false), that researchers are “locked in” to their preregistration plan (false), that preregistration prohibits optional stopping of data collection within a frequentist framework (false), that preregistration may not be appropriate for public datasets (false), that preregistration’s primary use is for replication studies (false), and so on. These are all just false claims, and it is disappointing to see them trotted out again and again.

A maxim that I have adopted is that if I have some kind of criticism about an existing practice, not only is it highly likely that someone else has already raised that criticism, but it is also likely that someone else has already developed an effective solution to the problem. Or, at the very least, I should probably look to see whether or not someone has. This is true for many different subjects, but especially preregistration.  

Coming into the meeting, I held the position that the promises of preregistration far outweigh their pitfalls, and that position remained strong after the meeting concluded. The meeting reaffirmed that the critiques of preregistration simply don’t stand up to scrutiny.

There are, however, pitfalls to preregistration, but these pitfalls concern how we think about and discuss preregistration, and are not about preregistration itself. Think of these as pitfalls in the discourse of preregistration. These pitfalls are major barriers to having productive conversations and generating useful solutions. I came up with five of them, but feel free to add your own.

Discourse Pitfall 1: Criticisms That Fail to Separate the Idea from the Implementation

The support for preregistration that I just declared does not mean that I believe all is peachy. Far from it. One of the barriers to effective critical discussions about preregistration is not adequately separating out the idea of preregistration from its implementation. In my view, the criticisms of the idea of preregistration have failed repeatedly and are without merit. In contrast, there are many valid criticisms of the implementation, or more generally the practice of preregistration.

My own presentation at the meeting reported on some data indicating extremely low rates of editors and reviewers indicating that they actually compared the preregistration document to the reporting in the paper (Syed, 2024). This is particularly important given the evidence on how often papers include undisclosed deviations from their preregistration plans (Claesen et al., 2021; van den Akker, 2023).

Summary of frequencies in which editors/reviewers mentioned preregistration, indicate accessing the preregistration, and compared the preregistration plan to the reporting in the manuscript. Panel A shows data for the article level (n = 201) and Panel B for the editor/reviewer level (n = 689). Reprinted from https://osf.io/preprints/psyarxiv/nh7qw

There are plenty of other implementation problems, including people believing that the mere presence of preregistration indicates that the study was high quality (false) and that some people will game the system by generating several preregistrations that cover all possible outcomes, reporting only the one that is consistent with their findings². These are all important issues that are in great need of productive conversations about the problems with the practice of preregistration.

The problem is that some folks seem to take the fact of these implementation problems as an indictment of the idea of preregistration itself. This is both wrong-headed and counterproductive.

Having presenters with considerable expertise on registrations for clinical trials was instructive. They highlighted how the clinical trial registration system has been far from ideal, both from the author side and journal side. Nevertheless, neither they nor any other credible scientist would use these limitations as the basis for arguments that minimize the importance of trial registration, or that the system should be eliminated altogether. That would be absurd.

Discourse Pitfall 2: Assuming Researchers Act without Bias

There was a stark difference at the meeting between people who did and did not consider the possibility of researcher bias. This difference seems to underlie what I find to otherwise be the most compelling (relatively speaking) argument against the idea of preregistration, namely that the quality of an explanation is what is most critical, and if an explanation for a phenomenon is a good one, it does not matter when in time that explanation is made. That is, there is no epistemic difference between an explanation that is provided ante hoc via a preregistered hypothesis and an explanation made post hoc based on the observed data. This issue is a whole can of worms, as the merits of prediction vs. accommodation have a long and contentious history. Nevertheless, the relative merits of the two are likely to be weighted quite differently depending on your views on researcher bias.

At this point, I would say we have plenty of evidence that researchers will act in ways that are biased towards their preferred results, and thus their data-dependent explanations cannot be fully trusted, nor can their unverifiable claims of data-independent prediction. For this reason, yes, I very much do care when in time an explanation is provided and I very much do care whether or not researchers can produce evidence of their predictions (e.g., via preregistrations). This point does not rely on assigning value to supposedly confirmatory vs. exploratory analyses, a distinction that I do not personally believe is useful, meaningful, or helpful. Rather, I assign greater value to making a claim about what would constitute a fair test of a hypothesis prior to knowing the results, over a claim of prediction made after knowing the results (Lakens, 2019)³. This should not be controversial, and yet.

Discourse Pitfall 3: Overly Individualistic Solutions

Somehow, I continue to be surprised when people will overly emphasize individual-level solutions to organizational and structural problems. Several presenters pointed to publication bias as the core problem that preregistration, in a sort of roundabout way, is meant to address. Scientific journals have long selected for statistically significant or otherwise “positive” results, and thus much of the bad behavior of scientists has been fomented by this dysfunctional definition of success. If journals and funders did not reward positive results, there would be little motivation⁴ for researchers to engage in selective reporting and other questionable research practices. I am not big on appealing to the incentives (read Yarkoni on this matter), but here we can identify incentives and then change them. If journals declared tomorrow that all submitted studies must be preregistered, and that they have assigned dedicated paid staff⁵ to evaluate the preregistrations relative to the reporting in the paper, we would see major changes with a quickness. I am not claiming this is the correct intervention (see below), but if journals caused this problem, journals need to take the lead in fixing it. To be clear, individual researcher behavior matters and should be discussed, but we cannot let journals off the hook.

Discourse Pitfall 4: Physics Envy was the Guest of Honor

I shit you not, I lost count of how many times speakers referenced physics. Mind you, I do not believe there was a single physicist in the room. Physics envy is rampant and toxic across the sciences, but especially in psychology. People really need to rethink whether physics is the proper comparison for the work we are doing in the behavioral sciences and perhaps consider just taking our work on its own terms. Basing your argument against preregistration on the fact that Einstein didn’t need it to make a prediction about the orbit of Mercury based on his General Theory of Relativity is not the argument you think it is.  

Discourse Pitfall 5: Registered Reports Snuck in the Back Door and Ate all the Good Food

As the meeting carried on, we talked less about preregistration and more about Registered Reports. Registered Reports are related to preregistration, but have important differences. Rather than a researcher practice, Registered Reports represent a change to the peer review process and, critically, a change to the criteria used for publication selection (see Chambers & Tzavella, 2021).

With Registered Reports, the peer review process is separated into two stages. At Stage 1, authors submit the Introduction, Method, and Planned Analysis section, all prior to beginning the study and/or conducting any data analysis. This Stage 1 proposal is peer reviewed as normal, with the ultimate outcome being an in-principle acceptance, which is a guarantee from the journal that they will publish the full paper, regardless of the results, so long as the researchers do as they specified and do it competently. Following the in-principle acceptance, the researchers conduct the study, write-up the remaining part of the paper, and submit that Stage 2 manuscript for final review and approval.

Registered Reports shift the publication section criteria from the nature of results to the quality of the study conceptualization and study design. It incentives researchers to conduct studies that would be informative regardless of the results, rather than the current system that incentives them to chase down positive findings. Accordingly, at present I view Registered Reports as the most promising available intervention to improve our science.

Not only that, but they address nearly all of the implementation problems of preregistration and make nearly all objections about preregistration irrelevant. I’m not claiming that Registered Reports are perfect—they are most certainly a work in progress—but the shift in the discussion towards them did make me wonder why we were wasting so much time discussing the pros and cons of preregistration when a vastly superior alternative was staring us right in the face.

Of course, I was not actually wondering that, because I know that the collective appetite for Registered Reports is currently much, much lower than for preregistration. Nevertheless, I couldn’t help think that one day preregistration will be akin to the compact disc—a once-prominent but ultimately inferior product that will be scantly remembered several years from now.

Nevertheless, Preregistration Is Good (For Now)

Look, preregistration is not a magic solution that will fix what is wrong with science. Rather, it is a helpful and promising tool that we can put to good use. I have noticed a major change in my research group since we started preregistering our work. We are more thoughtful, careful, and detailed than we ever were before. But it takes work. Just filling out a preregistration template and affixing a badge to our articles does not mean we are suddenly doing Good Science^TM. We have to actually want to make the improvements to our research that preregistration is supposed to facilitate. Nothing else will substitute.

1. I made the decision to not “name names” in this post, for both points I agreed with and those that I didn’t, as I wanted to focus here on the ideas vs. the people.

2. I doubt this is particularly common, but anyone who does this is unquestioningly engaging in academic misconduct.

3. Important to note here that I see great value in using preregistration far beyond the confines of hypothesis testing (see Syed, 2024).

4. I intentionally use “little motivation” rather than “no motivation” because there are obviously other reasons why researchers might be biased towards positive results.

5. All of the major for-profit publishers could absolutely afford to do this if they valued it.