> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abtestly.com/llms.txt
> Use this file to discover all available pages before exploring further.
# The three statistics engines
> Frequentist for classical CRO, sequential for peeking without penalty, Bayesian for probability-of-being-better. Pick per experiment. The math is locked once you start.
ABTestly ships three statistics engines. You pick one per experiment; the
verdict text and the confidence chart are computed from the engine you
picked. Once an experiment starts, its engine and the specific method
version are **locked**, so an upgrade to the engine code cannot change a
running test's math.
## Which one for what
A two-proportion Z-test. Classical CRO. Best-known verdict language,
fixed sample-size decisions.
An anytime-valid confidence sequence. Peek as often as you want
without inflating false positives. Best for "call it early when you
can".
A Beta-Binomial posterior. Reports probability that variant is better
than control, and expected loss. Best for stakeholders who want
probabilities.
## Frequentist
Method id: **`freq-z-1`**.
The two-proportion Z-test. For conversion goals this compares the
conversion rate of each variant against control. For revenue goals it
compares mean revenue per visitor with a mean-difference confidence
interval.
**Reads as:** "Variant B beats control by X %, p \< 0.05, 95 % CI \[a, b]."
**Peeking penalty:** yes. Deciding based on early looks inflates your
false-positive rate. If you cannot resist looking, use Sequential instead.
## Sequential
Method id: **`seq-bern-betamix-union-1`** (conversion),
**`seq-rev-acs-1`** (revenue).
An **anytime-valid** confidence sequence. The confidence bound is designed
so you can look as often as you like, including automated dashboards, and
the false-positive guarantee still holds.
**Reads as:** "As of today, we can rule out any effect smaller than X %."
The bound tightens over time.
**Trade-off:** slightly less power at any given sample size than
frequentist, but you buy the right to peek. If you plan to run a test
until you can call it and then move on, this is a very good default.
## Bayesian
Method id: **`bayes-beta-1`** (conversion),
**`bayes-rev-normal-1`** (revenue).
A Beta-Binomial posterior. Under the hood it is a deterministic quadrature
over the beta posterior, no MCMC, so the answer is the same every time
you recompute.
**Reports:**
* **Probability variant beats control.** A number, 0 to 100 %.
* **Expected loss** if you pick variant and you are wrong.
* **Credible interval** for the true difference.
**Reads as:** "There is a 92 % probability that variant beats control,
and if you are wrong the expected loss is about 0.3 conversions per 100
visitors."
**Trade-off:** the priors matter. ABTestly uses a weakly-informative prior
that matches typical CRO effect sizes; if your test is unusually
large-effect or unusually small-sample, that will show up in the number
sooner than in a frequentist read.
## How you pick
Per site there is a **default** engine, set under
[Settings → General](/settings/general). Per experiment, you can override
that default.
Once the experiment **starts**, the engine and its method version are
locked into the experiment's epoch. If we ship a v2 of the frequentist
engine, your running tests keep using v1, the epoch is a record of what
math was applied to your numbers.
If you [reset data](/results/reading-a-result), a new epoch begins and
the engine can be picked again.
## What appears on the results page
Regardless of engine, the results page shows:
* **A verdict written in plain English.** No confidence numbers naked; the
copy translates them.
* **A cumulative confidence chart.** Confidence per goal over the run of
the experiment, plotted daily.
* **Per-variant metrics.** Visitors, conversions (or revenue), and lift
with a confidence or credible interval.
## Revenue is its own thing
Revenue metrics (revenue per visitor, average order value, orders per
visitor) each have their own math. See
[Revenue metrics](/results/revenue-metrics).
## Confidence gate
There is a minimum-visitors gate on the verdict. It exists so a test with
five visitors on each arm cannot proudly declare a winner because two of
them clicked something. When you are below the gate, the verdict text is
explicit that we are not calling the test yet regardless of what the
confidence number says.
## The math, in one paragraph, for anyone who wants it
Two-proportion Z with pooled variance under H0, standard implementation.
Sequential is a mixture-based confidence sequence on a Bernoulli mean.
Howard, Ramdas, McAuliffe, Sekhon (2021) is the reference. Bayesian is a
Beta(1, 1) prior updated to a Beta posterior with deterministic quadrature
for `P(B > A)`, expected loss, and credible interval. Revenue sequential
is the asymptotic confidence sequence for a mean (Waudby-Smith and
Ramdas). Revenue Bayesian is a normal-CLT posterior on RPV. Every method
is unit-tested against reference implementations; see the source under
`worker/src/lib/stats*.ts`.