The test behind a frequentist verdict
twoProportionZTest in worker/src/lib/stats.ts. A two-tailed
Z-test on two independent proportions, pooled standard error.
PHI is normalCdf, an Abramowitz and Stegun 26.2.17 polynomial
approximation with a maximum absolute error of about 7.5e-8 across all
real z. When either arm has no visitors, or when the pooled standard
error is zero, the function returns p = 1 rather than dividing by
zero.
One look, worked through
Control converts 384 of 12,000 visitors. The variation converts 432 of 12,000.What repeated looks actually cost
A running test’s p-value wanders. Conversions arrive in clumps, the gap between arms opens and closes, and the p-value tracks it. Every time you look you give that wandering line another chance to cross 0.05, and a stopping rule that acts on the first crossing collects all of those chances into one decision. Rather than quote a figure from the literature, we measured it against the function we ship.Method. 20,000 simulated A/A experiments. Both arms drawn from
Bernoulli(0.032), 20,000 visitors per arm, accrued evenly across 28
days. Daily counts are exact binomial draws from a seeded mulberry32
generator (seed 20260906), not a normal approximation. Every look is
evaluated by the shipped
twoProportionZTest, and a run counts as a
false positive if any scheduled look reads p < 0.05. There is no real
effect in any of these runs.
The code is published. These figures were produced by
scripts/peeking-simulation.mjs,
which imports the shipped twoProportionZTest rather than reimplementing
it, draws exact binomial counts by CDF inversion, and runs from a seeded
generator so the output is reproducible. An independent implementation of
the same method, with a different random stream, returns 4.8 %, 12.8 %,
21.7 % and 27.7 % for the four rows above. Agreement across two streams is
a stronger check than a bit-identical rerun would be.The second problem: the size you report is wrong too
A test that crosses the threshold early crosses it on an upward swing, because that is what a crossing is. The measured lift at the moment of crossing is therefore biased away from zero. You ship the variation, the swing decays toward the true effect, and the win never shows up in the quarter’s numbers. That much is standard. The size of it is not usually published, so we measured it with the same simulation, changing one thing: the variant now has a real 10 % lift. Every run below contains a genuine effect, so nothing here is a false positive. The only question is what number you report if you stop at the first significant read.
Stopping early on a real 10 % lift makes you report roughly 20 %. Not a
rounding error, and not in a direction that corrects itself: you will
have shipped the change, banked a doubled number, and be waiting on a
quarter that never arrives.
The engine that removes the penalty
The sequential engine (seq-bern-betamix-union-1) is built for
continuous reading. It is a beta-binomial mixture confidence sequence:
for a Bernoulli mean with S successes and F failures, a Beta(a0, b0)
mixing distribution gives a test martingale for a point null p0,
B is the Beta function. M_t is a nonnegative martingale with
expectation 1 under the null, so by Ville’s inequality the probability
that it ever exceeds 1/a, at any stopping time you choose, is at most
a. That “ever” is what a fixed-horizon test cannot offer. We pin the
Jeffreys mixture Beta(1/2, 1/2); coverage is distribution-free over any
proper mixture, so the choice affects tightness, never validity.
The interval is a deterministic function of the counts (S, F), the
mixture parameters and a. It does not depend on the order events
arrived in, so it cannot be tuned by how you look at it.
Coverage, measured. The repository carries an A/A coverage battery
as a ship gate (
worker/src/lib/__tests__/stats-sequential-coverage.test.ts).
300 independent A/A streams per cell, evaluated at every one of 600
peeks per arm, for true rates of 0.05, 0.20 and 0.50, at family error
0.05 and 0.01, on a seeded generator. On the current code every cell
reports 0 false positives out of 300. Ville’s bound predicts the
mixture is conservative, and that is what the run shows.What we enforce, and what we do not
We suppress a confidence verdict below a floor. A variation’s p-value, its 95 % and 99 % flags, and the engine’s evidence flag are all withheld unless that variation and its control each have at least 100 visitors and at least 5 conversions (MIN_VISITORS_FOR_CONFIDENCE and
MIN_CONVERSIONS_FOR_CONFIDENCE in worker/src/lib/confidence-gate.ts).
Rate, confidence interval and lift are still shown. Only the verdict is
held back. Below the floor the results page reads
“Still collecting, no significant difference yet”.
We do not enforce your planned sample size. The frequentist verdict
turns significant the moment the gated p-value clears the threshold,
whether that is on day 3 or day 30, and nothing in the product stops
you acting on it. If you want the guarantee, either hold the horizon
yourself or pick the sequential engine, which is the one that makes
early reading safe. Holding the horizon means fixing it before the
first visitor arrives, which is what the public
A/B test calculator does
from your baseline and MDE. Anyone claiming a fixed-horizon tool
protects you from your own stopping rule is describing a product
feature we did not build, because the honest version of it is a
decision, not a control.
Sources
The inflation this page describes is not an ABTestly result. It is a published one, and the arithmetic above is a restatement of it.- Armitage, P., McPherson, C. K. and Rowe, B. C. (1969), Repeated Significance Tests on Accumulating Data, Journal of the Royal Statistical Society Series A, 132(2). The original treatment of what repeated looks at accumulating data do to a fixed significance threshold.
- Johari, R., Pekelis, L. and Walsh, D. J., Always Valid Inference: Bringing Sequential Analysis to A/B Testing. The modern sequential formulation, and the reason an anytime valid engine can be read continuously without the penalty.
Related
Minimum detectable effect
How the smallest lift worth catching sets the sample size you should be holding out for.
Test duration
Turning that sample size into a date, and how we project the runway.
The three engines
Frequentist, sequential, Bayesian. What each one reads as, and what it costs.
A/B test calculator
The same
requiredSampleSize function this product runs, in a public page.Already testing somewhere else
These pages assume you are deciding how to run a test. If you are already running them in another tool, the quickest way to judge this one is to rebuild a single live experiment here instead of starting from an empty account.Send us one live experiment
If you already run experiments in Convert, VWO, Optimizely, AB Tasty or PostHog, send us one that is live today and we rebuild it in ABTestly with you, free. Within two business days you get back three lists: what carries across as it is, what has to be re authored, and what we cannot reproduce. We never ask for a login to your current tool.