The Post-Hoc Problem: Why Bayesian Pre-Specification Matters More Than the Philosophy

The Bayesian vs. frequentist debate usually centers on philosophy. Priors are subjective! No, they formalize existing knowledge! You're smuggling in assumptions! You're ignoring relevant information!

It's a fun argument. It's also the wrong one, at least for regulatory decision-making.

The real value of Bayesian frameworks in drug development isn't the philosophy. It's the discipline. Specifically: forcing sponsors to commit to their decision rules before outcomes are known.

The pivot problem

Consider a simplified scenario that plays out more often than we'd like to admit:

A sponsor designs a trial with a functional clinical endpoint as the primary outcome. They also measure a biomarker they believe predicts clinical benefit. The trial runs. The functional endpoint fails to show a statistically significant effect. But the biomarker looks striking: large effect size, tight confidence intervals, biologically plausible mechanism.

Now what?

The sponsor pivots. The submission emphasizes the biomarker. The narrative shifts from "we designed this trial to show functional benefit" to "the biomarker is really the right measure here, and the functional endpoint was always going to be noisy in this population."

Biomarkers are just one example; similar pivots happen with endpoints, estimands, subpopulations, and analysis choices.

Maybe they're right. Maybe the biomarker genuinely is more meaningful than the clinical endpoint. Maybe approving based on that signal is the correct decision for patients.

But here's the problem: we can't know whether the sponsor would have made that argument if the results were reversed. If the functional endpoint had succeeded and the biomarker had been equivocal, would they have submitted based on the biomarker? Of course not.

This is the post-hoc pivot problem, and it's fundamentally untestable. The problem isn't that sponsors are wrong; it's that the evidentiary standard itself becomes contingent on the observed outcome.

What pre-specified Bayesian frameworks could solve

Imagine a different regulatory world. At the design stage, sponsors must declare:

1. A prior distribution on biomarker surrogacy. How confident are you that this biomarker predicts clinical benefit? If you're highly confident, say so, and quantify it. If you're uncertain, that uncertainty should be reflected in how much weight the biomarker gets in your decision framework.
2. A loss function for approval and denial. What's the cost of approving a drug that doesn't work? What's the cost of denying a drug that does? In a fatal disease with no alternatives, these aren't symmetric. Make sponsors commit to that asymmetry explicitly.
3. A decision threshold. At what posterior probability of benefit do you approve? 80%? 90%? This can't be decided after you know where your results fall.

Now replay the scenario. The sponsor has pre-specified a modest prior on biomarker surrogacy; they weren't sure NfL reduction would translate to functional benefit. They've also specified a loss function that weights the cost of denial heavily in a fatal disease.

When the functional endpoint fails and the biomarker succeeds, the framework does the work. Either the pre-specified prior and loss function lead to approval under the observed data, or they don't. The sponsor doesn't get to suddenly discover they had high confidence in the biomarker all along.

A skeptic might ask: don't statistical analysis plans already pre-specify endpoints and analyses? Yes, but there's an important distinction. Frequentist pre-specification fixes procedures. Bayesian pre-specification fixes decision rules. The former tells you how to calculate a p-value; the latter tells you what to do with the evidence.

The discipline isn't in the Bayesian math. It's in the commitment.

The hard questions

I'm not claiming this solves everything. Pre-specified Bayesian frameworks raise their own challenges:

Who sets the priors? Sponsor-proposed? FDA-negotiated? If sponsors choose, do they just game the priors at design stage instead of gaming the narrative at submission? Probably some of that happens. But at least the gaming is visible, documented, and subject to regulatory pushback before the trial runs.

What if science evolves mid-trial? New surrogacy data might emerge. Should sponsors be locked into priors that no longer reflect current knowledge? This argues for pre-specified sensitivity analyses and clear rules for when prior updating is legitimate.

Does this just shift the problem earlier? Maybe. But "gaming the design" is at least auditable in a way that "gaming the narrative" isn't. A pre-specified framework creates a paper trail. A post-hoc narrative doesn't.

Can you really quantify loss functions for regulatory decisions? This is hard. But FDA already makes these tradeoffs implicitly through accelerated approval, surrogate acceptance, and post-marketing requirements. Loss functions simply make those judgments explicit. Forcing explicit quantification, even if imperfect, surfaces assumptions that currently stay hidden.

Why this matters now

The FDA's recent Bayesian guidance for clinical trials emphasizes pre-specification. Sponsors must commit to their priors, their analysis methods, and their sensitivity approaches upfront.

This isn't a philosophical endorsement of Bayesian inference over frequentist methods. It's a practical recognition that Bayesian frameworks, properly constrained, can impose discipline that the current system lacks.

The guidance says, in effect: you want to use priors? Fine. But you have to commit to them before you see the data, and you have to show what happens under different assumptions.

That's not about philosophy. It's about trust.

The bottom line

I've written before about cases where FDA faced difficult decisions after trials delivered ambiguous results. Reasonable people disagree about whether those decisions were correct.

But here's what I've come to believe: we shouldn't need to rely on case-by-case regulatory judgment calls made under pressure after trials fail their pre-specified endpoints. That's not a scalable system. It's not a transparent system. And it puts FDA in an impossible position.

Pre-specified Bayesian frameworks offer a path forward. Not because Bayesian math is superior to frequentist math; that debate can continue in the journals. But because forcing sponsors to commit to their decision rules upfront creates accountability that post-hoc narrative construction never can.

Pre-specification doesn't eliminate judgment. It relocates judgment to a moment when it can be debated calmly, documented, and stress-tested.

The philosophy is interesting. The discipline is what matters.

This post was sparked by a discussion with readers about a recent case study. The critique that sponsors shouldn't be able to pivot their success criteria after seeing results is valid. My argument is that this critique actually strengthens the case for Bayesian pre-specification, not undermines it.

📬 For more essays on experimental design, regulatory evidence, and statistical decision-making across domains, subscribe to the Evidence in the Wild newsletter or browse the archive.