worker/src/lib/stats.ts or the
dashboard module named alongside it. Where a page states a constant, the
constant is quoted from the file that defines it. Where a page gives a
number, it was produced by running the function, not by rederiving it by
hand.
The same core is what the public
A/B test calculator runs, so
a sample size you compute there is the sample size the product will
project against.
The pages
Minimum detectable effect
The sample-size formula, worked to the visitor, and what our own function returns across common baselines.
Test duration
Required sample divided by accrual, the four statuses the projection can return, and what it does not know.
The peeking problem
A measured false-positive rate for four reading schedules, and the anytime-valid engine that removes the penalty.
Confidence intervals
Four intervals, four formulas, and why every one of them is printed next to its denominator.
Sample ratio mismatch
The chi-square check on the assumption everything else depends on, and which of its outputs reach a screen.
A reading order
If you are planning a test, start with minimum detectable effect, then test duration. Those two turn a business question into a date you can commit to. If a test is running, read the peeking problem before you act on an early green, and sample ratio mismatch if a warning has appeared. If a test has finished and someone is asking how confident you are, confidence intervals is the page that answers it, and reading a result is where those numbers sit on the screen.What we will not tell you
Two things are worth stating up front, because these pages keep coming back to them. We will not tell you a number is trustworthy when the method says otherwise. Below the sample floor there is no confidence percentage. Under a flagged sample ratio mismatch there is a banner instead of a verdict. On write-sampled data the SRM check publishes counts and no pass or fail. We will not describe a screen that does not exist. Several values on these pages are computed and returned on the API without any dashboard surface, and each page says so where it applies rather than implying a panel you would go looking for.Underneath the statistics
Sources
The methodology in this section is standard published statistics applied to one product. These are the works the individual pages cite, collected in one place so the reading order is obvious.- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press. The field’s standard text. If you read one thing here, read this.
- Armitage, P., McPherson, C. K. and Rowe, B. C. (1969), Repeated Significance Tests on Accumulating Data, Journal of the Royal Statistical Society Series A, 132(2). Why checking a running test repeatedly breaks the threshold you chose.
- Johari, R., Pekelis, L. and Walsh, D. J., Always Valid Inference: Bringing Sequential Analysis to A/B Testing. How a sequential engine removes that penalty.
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. and Dmitriev, P. (2019), Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners, KDD ‘19. Diagnosing a broken split before trusting anything else.
- Deng, A., Xu, Y., Kohavi, R. and Walker, T. (2013), Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, WSDM ‘13. Variance reduction, for when interval width becomes the constraint.
The exact experiment ledger
The canonical store the final numbers are computed against.
Bucketing model
How a visitor becomes a variant, and how to re-derive any assignment.
The three engines
Frequentist, sequential and Bayesian, locked per experiment at start.
Goal firing health
The failure mode no amount of statistics will rescue you from.
Already testing somewhere else
These pages assume you are deciding how to run a test. If you are already running them in another tool, the quickest way to judge this one is to rebuild a single live experiment here instead of starting from an empty account.Send us one live experiment
If you already run experiments in Convert, VWO, Optimizely, AB Tasty or PostHog, send us one that is live today and we rebuild it in ABTestly with you, free. Within two business days you get back three lists: what carries across as it is, what has to be re authored, and what we cannot reproduce. We never ask for a login to your current tool.