When National Institutes of Health Director Jay Bhattacharya started laying out his plans in 2025, he emphasized replication as part of the gold standard for science. In March, he launched the $850,000 Replication Prize for replicating high-impact areas of biomedical research. During his confirmation hearing, health secretary Robert F. Kennedy Jr. suggested a replication effort worth 20% of the NIH’s $47 billion budget.
However, these plans have since been dramatically scaled down to $174 million for “efforts to make rigorous research methods and replicating experiments a routine part of scientists’ work,” following a report from NIH’s Council of Councils that raised two concerns about the original plans: 1) being prohibitively expensive and 2) unfairly undermining confidence in individual scientists’ work.
Both of these concerns fall short. While cost may be an issue worth discussing, providing scientists with enough support and motivation to pursue replication projects is still the best path forward.
Bhattacharya himself has time and again correctly emphasized that when another scientist attempts to replicate your work, it should be seen “as an honor rather than a threat” because it means “your paper is worth the attention of other scientists.” (I’ve linked to a Reddit post here because the original article from the NIH Record is no longer available; the publication stopped publishing in January and the archives were removed.)
More than this though, in scientific research, committing a Type 1 error — incorrectly believing there is sufficient evidence for a new claim when the claim is actually wrong — does not reflect poorly on the quality of the science done.
Rather, it reflects the limits of practice when dealing with data and uncertainty. If all research is done at a significance level of 5%, common among many fields, then among all endeavors where there is nothing there to find, 5% will nevertheless produce evidence suggesting the contrary, even when the research has been conducted according to the highest scientific standards.
We could, of course, require all studies to use a 1% significance threshold instead. However, doing so reduces the statistical power to detect evidence for a claim when that claim is actually true, such that we would need larger sample sizes to get reasonably sufficient power, making research costlier and more difficult to conduct.
But whether we want 5% or 1%, the point is that there is always risk for Type 1 error, even when everything is done right. This is why replication studies are critical. While the risk of a Type 1 error for a single study is 5%, the probability of obtaining false positive evidence for the same claim in two independent studies, each conducted at 5% risk, is 0.05^2 = 0.0025, or 0.25%, a 20-fold reduction in risk.
That’s what the NIH would have been buying with a strong replication infrastructure, and it has nothing to do with “checking” a scientist’s work for mistakes or malpractice. It has, instead, everything to do with simply how numbers in statistical inference work. It is unfortunate that people are quick to associate replication rates with quality, and my research has shown how relying on this as a heuristic can be considerably misleading.
Now, the issue about cost is real. It would be prohibitively expensive to encourage replication for replication’s sake.
But there is no reason to build policy that way. The beauty of replication is that we have considerably more information when conducting a study a second time. The cost of a study has a lot to do with how much statistical power we want it to have, and calculating power often requires assumptions about quantities such as the expected effect size and variability.
Those assumptions can be quite uncertain when designing an entirely new study. However, with a replication, we have an existing study to use as a starting point. We have empirical information about the effect size that was observed, the variability in the data, and other features of the original study that can inform the replication design. For the goal of determining whether the original finding can be reproduced, these available details provide a much more concrete basis for determining how large the replication needs to be.
In other words, replication does not necessarily mean blindly repeating an expensive study. It gives us information that can make the next study more targeted and more efficient.
Finally, what makes supporting replication efforts so fundamentally necessary is the realities of human behavior and the incentives surrounding scientific discovery. Despite researchers’ best intentions, there will always be a tendency to push the boundaries of the evidence beyond conservative limits, and with that comes greater risk of making claims that ultimately fail to generalize. A scientific enterprise that is rightly motivated to push the frontiers of knowledge will, in aggregate, inevitably produce more false positives, even when everyone is playing honestly.
The only real way to determine which important findings are robust is to invest the money and effort required for replication itself.
Mark Louie F. Ramos, Ph.D., is an assistant research professor in the Department of Health Policy and Administration at Pennsylvania State University and has been teaching statistics for more than 15 years.