7 A/B Testing Statistical Mistakes That Invalidate Your Results
Seven A/B test design and analysis checks covering stopping rules, sample size, metric selection, assignment, and changes during an experiment.
Statistical errors in experiment design and analysis can make an A/B test misleading, even when assignment and measurement appear to work.
The seven checks below cover stopping rules, sample size, metric selection, assignment, and changes during a test.
Seven Statistical Mistakes to Check
1. Peeking at results and stopping early
A team launches a test, checks results daily, and stops as soon as it sees a significant p-value.
Repeatedly checking results and stopping when significance appears increases the false positive rate above the threshold specified for a fixed-horizon test.
The fix: decide your stopping date before the experiment launches. Do not look at results until that date. If you need to peek, use a sequential testing method (like always-valid p-values) designed for interim analysis.
2. Not computing sample size before launch
A team runs a test for two weeks, sees a 3% lift, and calls it a winner. The confidence interval spans from -4% to +10%. The result is noise.
Calculate sample size before launch using the baseline rate, minimum detectable effect, power, and significance threshold. If the required sample is unavailable, reconsider the duration or effect size worth detecting. Narrowing the audience also reduces available traffic and needs a new calculation.
An underpowered experiment may leave the team unable to distinguish a useful effect from noise.
3. Testing multiple metrics and declaring a winner on the best-looking one
If you run an experiment and check 20 metrics, the probability that at least one will appear significant by chance alone is very high — even if your variant had zero real effect. This is called multiple comparisons inflation.
Designate a primary metric before launch and predefine guardrails. If decisions depend on multiple outcomes, choose an analysis method that explicitly addresses multiple comparisons; using Bayesian analysis alone does not settle that design choice.
4. Sample ratio mismatch (SRM)
You allocated 50/50 between control and variant. After two weeks, control has 12,000 users and variant has 9,400. Something went wrong with assignment.
Sample ratio mismatch (SRM) is an often-missed validity check. When the observed split differs significantly from the expected split, your data is contaminated. Common causes: bots being filtered post-assignment in only one variant, caching serving stale pages to control users, or a bug in the assignment logic itself.
Run a statistical SRM check before interpreting treatment effects. A fixed percentage difference is insufficient because sample size matters. Microsoft's SRM guidance describes testing observed counts against the planned assignment ratio.
5. Novelty and primacy effects
Users behave differently toward new things. A redesigned interface may spike engagement in week one because users are exploring it, not because it is genuinely better. Conversely, users who are deeply habituated to an existing UI may initially perform worse with a change even if the change is correct long-term.
Neither effect is stable. If you run a test for only a few days, you may be measuring novelty rather than real preference.
The fix: run experiments for at least one full business cycle, typically a minimum of seven days, with two weeks preferred for anything that touches user habits.
6. Network effects and interference between users
Standard A/B testing assumes that each user's behavior is independent of other users. This assumption breaks in products with social features, referral programs, shared workspaces, or marketplaces.
If a treated user can influence a control user — by inviting them, by affecting inventory they both see, or by changing their own behavior in a way that ripples outward — your experiment is measuring a contaminated signal.
For products where interference is a real risk, cluster-based randomization (assigning entire cohorts or social groups to the same variant) is the appropriate method, at the cost of requiring significantly more traffic.
7. Changing the experiment while it runs
Resizing images, changing copy in both variants, modifying the audience targeting, or pushing a backend change that affects all users mid-experiment invalidates the test.
Once an experiment is live, treat it as frozen infrastructure. If a critical bug requires a change, stop the experiment, fix the issue, and restart with fresh assignment. Do not try to salvage data from a contaminated run.
How to Build Cleaner Experimentation Practice
Most of these mistakes are preventable with a pre-launch review process. Before any experiment goes live, verify:
- Primary metric is defined and logged
- Sample size calculation is documented
- Stopping date is set
- SRM check is scheduled for day two
- No dependent experiments are running on the same population
Document these decisions before the results can influence them.