Pre-register an A/B test before it starts collecting data

I am about to run an online experiment. Write the pre-registration for it: everything that has to be decided before any data exists, so the result is readable when it arrives.

The change under test: {{what the treatment does differently from control, concretely}}
Who is exposed to it: {{which users, on which surface, and at what point in their journey}}
Baseline for the metric I care about: {{current rate or average, and how it varies week to week}}
Traffic available: {{eligible users or sessions per day reaching the exposure point}}
Randomization unit available: {{user, session, device, account, or cluster — say which the platform actually supports}}
When I need a decision: {{date, and what happens if the test is not conclusive by then}}
Platform and statistics available: {{e.g. Statsig, GrowthBook, an in-house splitter with a t-test, CUPED available or not}}
What I would do if the change wins, and what I would do if it loses: {{answer honestly, including "ship it anyway"}}

Start with the last input. If I would ship the change regardless of the outcome, say so immediately and tell me not to run the experiment, or to run it as a monitored rollout with guardrails instead. That conversation is worth more than a correct power calculation.

Then work through the following, and put the whole thing in a form I can paste into a ticket before launch.

**One primary metric.** Give the exact definition: numerator, denominator, the window it is measured over, and which users are in the denominator. If the change plausibly moves two things I care about equally, say which one is primary and write down the trade you are accepting by demoting the other.

**Guardrails.** The metrics that would stop the launch even on a clear win: latency, error rate, unsubscribes, support contacts, revenue per user, whatever a change of this shape puts at risk. For each, the threshold that constitutes a stop rather than a shrug.

**The effect worth detecting, argued from the business.** Ask what improvement would justify building and maintaining this, and derive the minimum detectable effect from that answer rather than from what the traffic can resolve. Then say plainly whether the traffic can resolve it. Anchor my expectations with real magnitudes: effects at large consumer products are routinely fractions of a percent, and a design that can only detect a 10% lift will call almost everything null.

**Sample size and runtime.** Show the inputs you used — baseline, variance or conversion rate, significance level, power — and the resulting sample per arm and days to reach it at my traffic. Run to whole weeks so that day-of-week composition matches across arms. If the runtime exceeds my decision date, do not quietly shrink the effect size to make it fit: tell me the test cannot answer this question in the time available, and give me the options, which include narrowing the population, choosing a more sensitive metric closer to the change, using variance reduction if my platform has it, or accepting a lower confidence level with that cost stated.

**Exposure and randomization.** The unit, when a user first counts as exposed, and how control users who never reach the surface are kept out of the denominator. Name the ways assignment could leak here: shared accounts, the same person across devices, both arms visible to one user, network or household effects, and caching that serves one arm to everyone.

**The stopping rule, chosen now.** Fixed horizon or sequential, stated before launch. If fixed, say who is allowed to look at results before the end and what they are allowed to do with what they see. Checking a fixed-horizon test repeatedly and stopping at the first significant reading is the ordinary way experiments produce false wins, and the inflation is not marginal. If my platform supports sequential testing, say so and use it, since it is built for continuous monitoring rather than violated by it.

**Validity checks to run on day one and at the end.** Sample ratio mismatch against the intended split, exposure counts against expectation, the metric computed on the pre-period where both arms should be identical, and instrumentation firing on both arms. Give the observation that means stop and investigate rather than continue.

**The decision table, filled in before launch.**

| Outcome | What it means | What I do |
|---|---|---|
| Primary up, significant, guardrails clean | | |
| Primary up, not significant | | |
| Flat, adequately powered | | |
| Primary down, significant | | |
| Guardrail breached, primary up | | |

Fill every row with a specific action. The flat row is the one that gets rewritten after the fact, so make it unambiguous now.

**Finally, the case against running it.** One short paragraph on what makes this experiment weak: whether the change is too small for the traffic, whether the metric is too far downstream to move, whether a two-week look at existing data would answer it for free.

Tip: The honest answer to "what would I do if it loses" decides whether the rest of this is worth doing. Bring the baseline rate and the daily eligible traffic as real numbers; a power calculation on a guessed baseline produces a runtime that is wrong by weeks. Reading the result once it arrives is a separate prompt.