The formula we run
requiredSampleSize in worker/src/lib/stats.ts, returning visitors
per variation:
PHI_INV is inverseNormalCdf, a Beasley-Springer-Moro approximation
accurate to about 1e-9 across (0, 1).
Note the two variance terms. The zAlpha term uses the pooled variance
under the null, where both arms share pAvg. The zBeta term uses the
unpooled variance under the alternative, where the arms genuinely
differ. Formulas that use a single variance for both terms return a
different number, sometimes by a few percent, and the difference is not
a rounding artifact. Ours matches the test we actually run, which
pools.
The function returns Infinity, meaning “cannot be sized”, when the
baseline rate is not strictly inside (0, 1), when the MDE is zero or
negative, or when the implied variant rate reaches 1.
Worked example
A 4 % baseline, hunting a 10 % relative lift, at 95 % confidence and 80 % power.What our own function returns
Every cell below isrequiredSampleSize(baseline, mde) evaluated on the
shipped code at the default 95 % confidence and 80 % power. Visitors per
variation.
Two things worth reading off that grid.
Lower baselines cost more for the same relative MDE. A rare event
carries more relative noise, so a 1 % checkout completion needs about
25 times the traffic of a 20 % newsletter signup to resolve the same
percentage lift.
The scaling is close to inverse square but not exactly it. Dividing the
5 % column by the 20 % column gives 14.9 at a 1 % baseline rising to
15.2 at a 20 % baseline, against the 16 a clean inverse square would
predict. The gap is real:
p2 moves with the MDE, so the variance
terms in the numerator move too, and they move in the direction that
slightly favours the larger effect. Anyone quoting a flat “quartering
the effect costs sixteen times the traffic” is quoting the limit, not
the formula.
Choosing the number
Work backwards from the decision, not from the calendar. The question is what lift would be large enough to justify shipping and maintaining this change. If a 3 % lift on this page would be worth having, 3 % is your MDE and the traffic bill is what it is. If only a 15 % lift would change anything, powering for 3 % buys sensitivity you will never spend. The common failure runs the other way. A team wants the test finished this month, so the MDE goes up until the sample size looks comfortable, and the test launches underpowered for the effect it is actually likely to produce. It then returns inconclusive on a change that genuinely helped, and the team records that as evidence the change did nothing. An underpowered test converts a guess into a confident-looking null. If the honest MDE demands traffic you do not have, the available moves are to test further up the funnel, to test a bolder change, or to accept a longer run. Raising the MDE on paper is not one of them. Before you commit to any of them, the public A/B test calculator will price the choice from your own baseline. It runs the samerequiredSampleSize function this page describes.
Where MDE lives in ABTestly
The experiment editor carries a Minimum detectable effect field inside the traffic and split section. It accepts 0.5 % to 50 % in 0.5 % steps and defaults to 10 %. The value is stored in relative basis points on the experiment (mde_bps, so 10 % is 1000).
It feeds two things and blocks nothing.
The projected runway under the verdict on the results page divides the
remaining sample by recent traffic. See
test duration for that calculation.
The collection-health banner uses the same requirement. When the
projection exceeds 8 weeks the experiment reads “This experiment may
take a while to reach a confident result”, prints the MDE as a
percentage, and links back to the field so you can reconsider it.
What the MDE does not tell you
It is not a prediction of the lift you will get, and it is not a threshold the result has to clear. A test powered for 10 % can return a significant 4 % lift; it just will not do so reliably. It is also not a promise. Power of 80 % means that if the true effect is exactly your MDE, you catch it about four times in five, and miss it the fifth.Sources
The power calculation on this page is standard, and the guidance on choosing an effect size rather than accepting one is not ours.- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press. Chapter 17 covers statistical power and the practice of fixing the smallest effect worth detecting before a test starts.
Related
Test duration
Sample size divided by traffic, and the statuses the estimator can return.
The peeking problem
Why the sample size only protects you if you wait for it.
Reading a result
What the results page shows once the visitors arrive.
A/B test calculator
A public port of
requiredSampleSize, pinned to worker/src/lib/stats.ts by a generated vector suite.Already testing somewhere else
These pages assume you are deciding how to run a test. If you are already running them in another tool, the quickest way to judge this one is to rebuild a single live experiment here instead of starting from an empty account.Send us one live experiment
If you already run experiments in Convert, VWO, Optimizely, AB Tasty or PostHog, send us one that is live today and we rebuild it in ABTestly with you, free. Within two business days you get back three lists: what carries across as it is, what has to be re authored, and what we cannot reproduce. We never ask for a login to your current tool.