A/B Test Setup Coach

---
name: ab-test-setup-coach
description: Use when designing an A/B test or experiment. Triggers on "A/B test", "split test", "experiment setup", "test hypothesis".
---

# A/B Test Setup Coach

A/B tests fail in predictable ways: underpowered, ambiguous metric, leaky variants, novelty effect. Designing for these upfront beats discovering them at week 4.

## Required inputs

1. **What you're testing** — the specific change between control and variant
2. **Hypothesis** — what you predict and why (not "it'll improve things")
3. **Success metric** — the ONE primary metric you'll judge by
4. **Guardrail metrics** — things that shouldn't get worse
5. **Current baseline** — conversion rate, sample volume per week

## Output

### Sample size calculation
- Detect a Y% relative lift in metric X
- At Z% statistical power
- With confidence interval of W%
- → need N conversions per variant

If N exceeds what you can get in 4-6 weeks, you can't run this test as designed. Options: pick a bigger effect to detect, lower the power requirement, or pick a different test.

### Test design
- **Sample allocation** — 50/50 typically; 90/10 if risk of broken variant is high
- **Duration** — minimum 2 weekly cycles, prefer 4 weeks for B2B
- **Randomization unit** — user (most common) vs. session vs. account
- **Stopping criteria** — fixed duration (no peeking) or sequential testing

### Guardrails
Variables that shouldn't get worse even if primary improves:
- Revenue per user
- Day-7 retention
- Support ticket volume
- Specific segment performance

### Analysis plan (write BEFORE running)
- Primary metric pass/fail criteria
- How you'll handle small-but-significant results
- How you'll handle null results
- What you'll do if guardrails are breached
- How long after test ends before you make the call (to let lagging metrics surface)

### Common failure modes to pre-empt
- **Novelty effect** — variant wins in week 1, fades by week 4
- **Selection bias** — variants weren't randomly assigned
- **SRM** (sample ratio mismatch) — split came out 55/45 instead of 50/50, suggests data plumbing issue
- **Cherry-picking segments** — "it worked for enterprise" post-hoc

## Anti-patterns
- Multiple primary metrics (now everything is "directionally positive")
- "Inconclusive" framing for null results (null IS a result)
- Peeking and stopping early when one variant is ahead
- Testing 5 things at once and not knowing which moved the metric
- Shipping the winner without first checking if it has subgroup harm

## When to skip A/B testing
- Effect would have to be huge to matter; just ship the better option
- Sample size will never be enough; do qualitative research instead
- You'd ship the variant regardless of result (then save the cycles)