1. The conversion rate of one variation
Wilson score interval,wilsonCI in dashboard/src/lib/wilson.ts,
at z = 1.96.
2. The relative lift against control
relLiftCI in dashboard/src/lib/report-export.ts. A normal
approximation on the difference of two proportions, divided through by
the control rate.
liftConfidenceInterval, is what the
public A/B test calculator
runs.
3. The absolute difference, at your confidence level
The first two intervals are display objects, fixed at 95 %. The verdict carries its own interval, and that one honours the experiment’s configuration. FromfrequentistVerdict in
worker/src/lib/stats-engine.ts:
4. Revenue per visitor
Revenue is not a proportion, so none of the above applies. FrommeanConfidenceInterval in worker/src/lib/stats.ts, working from
running moments rather than stored observations:
n is visitors, not orders, because the metric is revenue per visitor.
The function returns nothing when n < 2: one observation carries no
dispersion, so any interval built from it would be invented. The
variance is clamped at zero so catastrophic cancellation and genuinely
degenerate arms both collapse to an honest zero-width interval instead
of a NaN. The bounds themselves are not clamped, and a negative lower
bound on a noisy revenue arm is a true statement about your data.
The difference of two revenue arms uses a Welch (unpooled) standard
error in meanDifferenceCI:
A documented simplification. When an experiment has refunds, the
revenue interval is centred on the refund-adjusted revenue per visitor
that the card displays, while its width still comes from the raw
order-value moments. Refund variance is not modelled. It is a
second-order effect on the width, and we would rather name it than let
you assume otherwise.
A worked results row
Control converts 384 of 12,000. The variation converts 432 of 12,000.Why the denominator is always next to the interval
The interval is a deterministic function of the counts. Print it without itsn and a reader cannot tell a wide interval caused by a small
sample from a wide interval caused by a genuinely noisy metric, and
those call for opposite responses. The results table therefore carries
visitors, conversions, rate, rate CI, lift, lift CI and confidence on
one row, and the CSV export carries the same columns.
Two floors back that up. The significance verdict is withheld unless
both the variation and its control have at least 100 visitors and at
least 5 conversions, so the confidence column stops quoting a percentage
off a handful of visitors. The rate, interval and lift are never
withheld, because they are honest at any sample size and the interval is
the thing that says so. The revenue-per-visitor interval uses the same
100-visitor and 5-conversion bar, since a revenue interval is at its
noisiest exactly where that bar is not met.
What a confidence interval is not
It is not the probability that the true value lies inside it. Under the frequentist reading, the true value is fixed and the interval is what is random. A 95 % interval is a procedure that brackets the truth in 95 % of repetitions. The sentence people actually want, of the form “there is a such-and-such probability that this variation is better”, is a posterior probability with a credible interval beside it, and that comes from the Bayesian engine. It is also not a guard against bias. An interval describes sampling noise and nothing else. A test with a sample ratio mismatch, a goal that only fires on one variation, or an instrumentation gap will produce intervals that are perfectly well formed and centred on the wrong number.Sources
The interval construction here is textbook. The reason we show the denominator beside every interval is not.- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press. The standard treatment of intervals, variance and what a confidence interval does and does not claim.
- Deng, A., Xu, Y., Kohavi, R. and Walker, T. (2013), Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, WSDM ‘13. On tightening intervals with pre-experiment data. ABTestly does not implement CUPED today; it is included because it is the usual next question once the width of an interval starts to matter.
Related
Sample ratio mismatch
The precondition an interval cannot check for you.
The three engines
Confidence interval, confidence sequence and credible interval are three different objects.
Reading a result
Where each of these lands on the page.
A/B test calculator
The same lift interval and the same z-test, on your own numbers.