Minimum Detectable Effect in A/B Testing: What It Is and Why It Matters

Before calculating an A/B test’s sample size, decide how small a change the test needs to detect. That choice affects the traffic and runtime required.

The minimum detectable effect, or MDE, expresses that target effect size for the test design.

What Is Minimum Detectable Effect?

The minimum detectable effect is the smallest relative (or absolute) change in your primary metric that your experiment is designed to reliably detect, given your chosen statistical power and significance threshold.

It is not the effect you expect to see. It is the floor below which your experiment cannot distinguish signal from noise.

If you set your MDE to 5% and your variant produces a real 3% lift, your experiment will not detect it. The test will come back inconclusive — not because nothing happened, but because you designed the experiment to be blind to changes that small.

Why MDE Is a Business Decision, Not a Statistics Decision

Choose MDE with the business decision in mind:

"How large does this effect need to be to justify shipping it?"

Consider your context:

• If your current checkout conversion rate is 4%, a 5% relative lift is 0.2 percentage points. If your volume is high enough, this is substantial. If your volume is low, this may not move a business metric you can measure.
• If a variant requires significant ongoing engineering maintenance, you may only want to ship it if the lift is large enough to justify the overhead.
• If a variant is zero marginal cost to ship (a copy change, a color, a headline), you might be willing to ship a 2% lift.

The MDE should reflect the minimum effect that makes the decision economically meaningful. Do not choose it by asking "what can we detect?" — choose it by asking "what would change our decision to ship?"

How MDE Connects to Sample Size and Test Duration

MDE, sample size, statistical power, and significance threshold are mathematically linked. Given any three, you can solve for the fourth.

The practical relationship:

• Smaller MDE requires a larger sample size to detect
• Larger MDE requires a smaller sample size
• Higher power (e.g., 90% vs 80%) requires a larger sample size
• Lower significance threshold (e.g., 99% vs 95%) requires a larger sample size

This means that choosing an aggressive MDE (wanting to detect small effects) is only feasible if you have the traffic to support the required sample size within a reasonable time window.

Check whether the available traffic can support the chosen effect size within the experiment window. The answer depends on the baseline rate, test allocation, power, and significance threshold.

Common MDE Mistakes

Setting MDE too small without checking traffic. Teams want to detect every possible improvement, so they set MDE at 1-2%. For most products at early or mid-stage scale, this requires sample sizes that would take months to accumulate. The test either never reaches significance or gets stopped early.

Setting MDE to whatever size the calculator accepts. Some teams work backward from traffic to find the MDE they can detect and treat that as the answer. This inverts the decision: you should set the MDE based on business value, then decide if you have enough traffic to run the test.

Using the same MDE for every experiment. A pricing page experiment and an error message experiment have completely different business implications. The minimum lift worth shipping is not a constant across your product.

Confusing absolute and relative MDE. A 5% relative lift on a 10% conversion rate is 10.5% — an absolute change of 0.5 percentage points. A 5% absolute change on a 10% baseline is 15%. Make sure your calculator and your team are using the same definition.

Choosing an MDE in Practice

A reasonable starting approach:

1. Look at the expected business impact of the change if it succeeds. What is a 10% lift worth in revenue or activation over the next year?
2. Set your MDE at the level where the business impact justifies the engineering and analysis cost of the experiment.
3. Calculate the required sample size. Check whether your traffic can support it in a 2-4 week window.
4. If it cannot, either increase the MDE or accept that you will run this experiment over a longer window — and plan for that explicitly.

Choose an effect size that can support the decision you need to make with the traffic available.

Discuss experiment design with LinkIT.