CUPED Explained: Variance Reduction for Faster A/B Tests

When experiment traffic is limited, reducing metric variance can improve sensitivity without increasing the audience. CUPED uses pre-experiment behavior as a covariate for that adjustment.

CUPED stands for Controlled-experiment Using Pre-Experiment Data. The benefit depends on how well the pre-experiment covariate predicts the outcome being measured.

What CUPED Does and Why It Works

The core problem in A/B testing is that the metric you are measuring has noise. Users vary enormously in their baseline behavior — a power user and a new user will both be assigned to your experiment, and their baseline differences inflate the variance in your treatment effect estimate. This forces you to collect more data before the signal clears the noise.

CUPED uses information you already have — each user's pre-experiment behavior — to adjust your metric and remove the portion of variance explained by baseline differences.

The idea is simple: if you know a user's conversion behavior before the experiment started, you can subtract out the baseline variation that was going to happen anyway, regardless of which variant they were assigned to. What remains is a cleaner estimate of the treatment effect.

Mathematically, the CUPED-adjusted metric is:

Y_adjusted = Y - theta * X

Where:

• Y is the user-level outcome metric during the experiment
• X is the same metric (or a correlated covariate) from a pre-experiment window
• theta is the regression coefficient that captures how strongly the covariate predicts the outcome

The key property: the expected value of Y_adjusted is the same as Y (so the estimate is unbiased), but the variance is lower because you have removed the portion explained by X.

When CUPED Delivers the Most Benefit

CUPED works best when:

The covariate is strongly correlated with the outcome. If a user's conversion rate in the 30 days before the experiment predicts their conversion rate during the experiment, CUPED can remove a large share of variance. If the correlation is weak, the benefit is limited.

You have enough pre-experiment history. For CUPED to be effective, you need covariate data covering a window long enough to capture stable user behavior — typically 7-30 days depending on the product and metric.

Missing history is handled explicitly. New users lack pre-experiment observations. Define and validate how the estimator treats missing covariates; do not remove users from the experiment simply because they lack history.

You are measuring user-level metrics. CUPED requires a user-level covariate. Metrics that cannot be attributed to individual users (like aggregate page load time) are less straightforwardly handled.

Implementing CUPED in Practice

The implementation steps are:

1. Define the covariate. Most commonly, this is the same metric you are testing, measured over a lookback window ending at experiment start. If your primary metric is 7-day revenue per user, your covariate might be 30-day revenue per user in the pre-experiment period.

2. Compute the covariate for every assigned user. For new users with no history, you typically assign a covariate of zero or the population mean.

3. Estimate theta. Regress the outcome on the covariate using pooled data across all variants. This regression is run after the experiment concludes, using data from all users — it does not depend on variant assignment.

4. Compute the adjusted outcome per user. Subtract theta * (X - mean(X)) from each user's Y, matching the centered adjustment shown above.

5. Run your standard t-test or z-test on the adjusted metric. The analysis proceeds exactly as normal — the only change is you are now testing Y_adjusted instead of Y.

With the usual linear adjustment, the remaining variance is proportional to 1 - R²; the reduction is R². The benefit depends on the covariate and estimator. Microsoft's variance-reduction explanation.

CUPED vs Other Variance Reduction Methods

CUPED is not the only option. Other approaches include:

• Stratified sampling — ensure balance across known high-variance subgroups (device type, geography, account tier) at assignment time. Can be combined with CUPED.
• Post-stratification — similar to CUPED but uses categorical variables instead of continuous covariates. Simpler to implement, generally less powerful.
• Delta method adjustments — used for ratio metrics (conversion rate, revenue per session) where naive variance estimates are biased.

Estimate the relationship between the covariate and outcome before deciding whether CUPED is useful for your metric.

What CUPED Does Not Do

CUPED can improve power by reducing variance. It does not repair biased assignment, sample-ratio mismatch, or an invalid stopping rule.

Use CUPED alongside a defined analysis plan, valid assignment, and appropriate stopping rules.

Discuss experiment analysis infrastructure with LinkIT.