> ## Documentation Index > Fetch the complete documentation index at: https://docs.abtestly.com/llms.txt > Use this file to discover all available pages before exploring further. # The peeking problem > Why a fixed-horizon significance test loses its guarantee when you read it repeatedly. The exact test we run, a false-positive rate we measured against our own z-test, and the engine that removes the penalty. A p-value answers one question: if the two arms were identical, how often would random noise produce a gap at least this large? The answer is only valid for the number of times you asked. Ask every morning for four weeks and stop at the first reassuring answer, and the 5 % you think you are running at is not the 5 % you are getting. This page shows the test we run, what repeated reading does to it, and what we do about it. ## The test behind a frequentist verdict `twoProportionZTest` in `worker/src/lib/stats.ts`. A two-tailed Z-test on two independent proportions, pooled standard error. ``` p_pooled = (p1 * n1 + p2 * n2) / (n1 + n2) SE = sqrt( p_pooled * (1 - p_pooled) * (1/n1 + 1/n2) ) z = (p1 - p2) / SE p = 2 * (1 - PHI(|z|)) ``` `PHI` is `normalCdf`, an Abramowitz and Stegun 26.2.17 polynomial approximation with a maximum absolute error of about 7.5e-8 across all real z. When either arm has no visitors, or when the pooled standard error is zero, the function returns `p = 1` rather than dividing by zero. ### One look, worked through Control converts 384 of 12,000 visitors. The variation converts 432 of 12,000. ``` p1 = 432 / 12000 = 0.036 p2 = 384 / 12000 = 0.032 p_pooled = (0.036 * 12000 + 0.032 * 12000) / 24000 = 0.034 SE = sqrt(0.034 * 0.966 * (1/12000 + 1/12000)) = 0.00233966 z = (0.036 - 0.032) / 0.00233966 = 1.70965 p = 2 * (1 - PHI(1.70965)) = 0.0873 ``` A relative lift of 12.5 %, and a p-value of 0.087. Not significant at 95 %. That 0.087 is a correct answer to exactly one question, asked once. ## What repeated looks actually cost A running test's p-value wanders. Conversions arrive in clumps, the gap between arms opens and closes, and the p-value tracks it. Every time you look you give that wandering line another chance to cross 0.05, and a stopping rule that acts on the first crossing collects all of those chances into one decision. Rather than quote a figure from the literature, we measured it against the function we ship. **Method.** 20,000 simulated A/A experiments. Both arms drawn from Bernoulli(0.032), 20,000 visitors per arm, accrued evenly across 28 days. Daily counts are exact binomial draws from a seeded mulberry32 generator (seed 20260906), not a normal approximation. Every look is evaluated by the shipped `twoProportionZTest`, and a run counts as a false positive if any scheduled look reads p \< 0.05. There is no real effect in any of these runs. | Reading schedule | Looks | Measured false-positive rate | | ------------------------ | ----- | ---------------------------- | | Once, at the planned end | 1 | 5.1 % | | Weekly | 4 | 12.9 % | | Every other day | 14 | 22.5 % | | Daily | 28 | 28.0 % | False positive rate by reading schedule: 4.8 % at one look, rising to 27.7 % at twenty eight looks, against a nominal 5 % line Each figure carries a simulation margin of roughly 0.3 to 0.6 percentage points at 95 %. The single-look row landing on 5.1 % is the control on the experiment: it tells you the simulation and the test agree with the nominal rate when the rule is honoured. Every row below it is the cost of the stopping rule, not of the test. Reading the dashboard is not what breaks this. Acting on it is. A team that looks daily and always waits for the planned end is still running at 5 %. **The code is published.** These figures were produced by [`scripts/peeking-simulation.mjs`](https://github.com/maaislam/abtestly/blob/main/marketing/scripts/peeking-simulation.mjs), which imports the shipped `twoProportionZTest` rather than reimplementing it, draws exact binomial counts by CDF inversion, and runs from a seeded generator so the output is reproducible. An independent implementation of the same method, with a different random stream, returns 4.8 %, 12.8 %, 21.7 % and 27.7 % for the four rows above. Agreement across two streams is a stronger check than a bit-identical rerun would be. ## The second problem: the size you report is wrong too A test that crosses the threshold early crosses it on an upward swing, because that is what a crossing is. The measured lift at the moment of crossing is therefore biased away from zero. You ship the variation, the swing decays toward the true effect, and the win never shows up in the quarter's numbers. That much is standard. The size of it is not usually published, so we measured it with the same simulation, changing one thing: the variant now has a **real 10 % lift**. Every run below contains a genuine effect, so nothing here is a false positive. The only question is what number you report if you stop at the first significant read. | Reading schedule | Looks | Stopped early | Median reported lift | Overstated by | | ---------------- | ----- | ------------- | -------------------- | ------------- | | Weekly | 4 | 41.2 % | 20.2 % | 2.02x | | Every other day | 14 | 56.9 % | 20.4 % | 2.04x | | Daily | 28 | 61.8 % | 21.1 % | 2.11x | Stopping early on a real 10 % lift makes you report roughly 20 %. Not a rounding error, and not in a direction that corrects itself: you will have shipped the change, banked a doubled number, and be waiting on a quarter that never arrives. **This test is underpowered on purpose, and that bounds the result.** Detecting a 10 % lift on a 3.2 % baseline needs about 49,800 visitors per arm. The simulation uses 20,000, roughly two and a half times short. That is deliberate: teams peek precisely because a test is slow and underpowered, so this is the regime where early stopping actually happens. A properly powered test crosses later and closer to the truth, and the inflation shrinks. Read the table as what peeking costs when you are tempted to peek, not as a universal constant. ## The engine that removes the penalty The sequential engine (`seq-bern-betamix-union-1`) is built for continuous reading. It is a beta-binomial mixture confidence sequence: for a Bernoulli mean with S successes and F failures, a Beta(a0, b0) mixing distribution gives a test martingale for a point null p0, ``` M_t(p0) = [ B(a0+S, b0+F) / B(a0, b0) ] / [ p0^S * (1-p0)^F ] CS_a = { p0 in (0,1) : M_t(p0) < 1/a } ``` where `B` is the Beta function. `M_t` is a nonnegative martingale with expectation 1 under the null, so by Ville's inequality the probability that it ever exceeds `1/a`, at any stopping time you choose, is at most `a`. That "ever" is what a fixed-horizon test cannot offer. We pin the Jeffreys mixture Beta(1/2, 1/2); coverage is distribution-free over any proper mixture, so the choice affects tightness, never validity. The interval is a deterministic function of the counts (S, F), the mixture parameters and `a`. It does not depend on the order events arrived in, so it cannot be tuned by how you look at it. **Coverage, measured.** The repository carries an A/A coverage battery as a ship gate (`worker/src/lib/__tests__/stats-sequential-coverage.test.ts`). 300 independent A/A streams per cell, evaluated at every one of 600 peeks per arm, for true rates of 0.05, 0.20 and 0.50, at family error 0.05 and 0.01, on a seeded generator. On the current code every cell reports 0 false positives out of 300. Ville's bound predicts the mixture is conservative, and that is what the run shows. The trade is real. At any fixed sample size the sequential interval is wider than the fixed-horizon one, so you pay for the right to look in power. See [the engines page](/results/engines) for how to pick, and note that an experiment's engine and method version are locked when it starts. ## What we enforce, and what we do not We suppress a confidence verdict below a floor. A variation's p-value, its 95 % and 99 % flags, and the engine's evidence flag are all withheld unless that variation and its control each have at least 100 visitors and at least 5 conversions (`MIN_VISITORS_FOR_CONFIDENCE` and `MIN_CONVERSIONS_FOR_CONFIDENCE` in `worker/src/lib/confidence-gate.ts`). Rate, confidence interval and lift are still shown. Only the verdict is held back. Below the floor the results page reads "Still collecting, no significant difference yet". We do not enforce your planned sample size. The frequentist verdict turns significant the moment the gated p-value clears the threshold, whether that is on day 3 or day 30, and nothing in the product stops you acting on it. If you want the guarantee, either hold the horizon yourself or pick the sequential engine, which is the one that makes early reading safe. Holding the horizon means fixing it before the first visitor arrives, which is what the public [A/B test calculator](https://abtestly.com/ab-test-calculator) does from your baseline and MDE. Anyone claiming a fixed-horizon tool protects you from your own stopping rule is describing a product feature we did not build, because the honest version of it is a decision, not a control. ## Sources The inflation this page describes is not an ABTestly result. It is a published one, and the arithmetic above is a restatement of it. * [Armitage, P., McPherson, C. K. and Rowe, B. C. (1969), *Repeated Significance Tests on Accumulating Data*, Journal of the Royal Statistical Society Series A, 132(2)](https://doi.org/10.2307/2343787). The original treatment of what repeated looks at accumulating data do to a fixed significance threshold. * [Johari, R., Pekelis, L. and Walsh, D. J., *Always Valid Inference: Bringing Sequential Analysis to A/B Testing*](https://arxiv.org/abs/1512.04922). The modern sequential formulation, and the reason an anytime valid engine can be read continuously without the penalty. Every link above was checked on 6 September 2026. ## Related How the smallest lift worth catching sets the sample size you should be holding out for. Turning that sample size into a date, and how we project the runway. Frequentist, sequential, Bayesian. What each one reads as, and what it costs. The same `requiredSampleSize` function this product runs, in a public page. *** ## Already testing somewhere else These pages assume you are deciding how to run a test. If you are already running them in another tool, the quickest way to judge this one is to [rebuild a single live experiment here](https://abtestly.com/switch-one-experiment) instead of starting from an empty account. If you already run experiments in Convert, VWO, Optimizely, AB Tasty or PostHog, send us one that is live today and we rebuild it in ABTestly with you, free. Within two business days you get back three lists: what carries across as it is, what has to be re authored, and what we cannot reproduce. We never ask for a login to your current tool.