Which one for what
Frequentist
A two-proportion Z-test. Classical CRO. Best-known verdict language,
fixed sample-size decisions.
Sequential
An anytime-valid confidence sequence. Peek as often as you want
without inflating false positives. Best for “call it early when you
can”.
Bayesian
A Beta-Binomial posterior. Reports probability that variant is better
than control, and expected loss. Best for stakeholders who want
probabilities.
Frequentist
Method id:freq-z-1.
The two-proportion Z-test. For conversion goals this compares the
conversion rate of each variant against control. For revenue goals it
compares mean revenue per visitor with a mean-difference confidence
interval.
Reads as: “Variant B beats control by X %, p < 0.05, 95 % CI [a, b].”
Peeking penalty: yes. Deciding based on early looks inflates your
false-positive rate. If you cannot resist looking, use Sequential instead.
Sequential
Method id:seq-bern-betamix-union-1 (conversion),
seq-rev-acs-1 (revenue).
An anytime-valid confidence sequence. The confidence bound is designed
so you can look as often as you like, including automated dashboards, and
the false-positive guarantee still holds.
Reads as: “As of today, we can rule out any effect smaller than X %.”
The bound tightens over time.
Trade-off: slightly less power at any given sample size than
frequentist, but you buy the right to peek. If you plan to run a test
until you can call it and then move on, this is a very good default.
Bayesian
Method id:bayes-beta-1 (conversion),
bayes-rev-normal-1 (revenue).
A Beta-Binomial posterior. Under the hood it is a deterministic quadrature
over the beta posterior, no MCMC, so the answer is the same every time
you recompute.
Reports:
- Probability variant beats control. A number, 0 to 100 %.
- Expected loss if you pick variant and you are wrong.
- Credible interval for the true difference.
How you pick
Per site there is a default engine, set under Settings → General. Per experiment, you can override that default. Once the experiment starts, the engine and its method version are locked into the experiment’s epoch. If we ship a v2 of the frequentist engine, your running tests keep using v1, the epoch is a record of what math was applied to your numbers. If you reset data, a new epoch begins and the engine can be picked again.What appears on the results page
Regardless of engine, the results page shows:- A verdict written in plain English. No confidence numbers naked; the copy translates them.
- A cumulative confidence chart. Confidence per goal over the run of the experiment, plotted daily.
- Per-variant metrics. Visitors, conversions (or revenue), and lift with a confidence or credible interval.
Results page. Verdict copy, meta bar, and overview.
Revenue is its own thing
Revenue metrics (revenue per visitor, average order value, orders per visitor) each have their own math. See Revenue metrics.Confidence gate
There is a minimum-visitors gate on the verdict. It exists so a test with five visitors on each arm cannot proudly declare a winner because two of them clicked something. When you are below the gate, the verdict text is explicit that we are not calling the test yet regardless of what the confidence number says.The math, in one paragraph, for anyone who wants it
Two-proportion Z with pooled variance under H0, standard implementation. Sequential is a mixture-based confidence sequence on a Bernoulli mean. Howard, Ramdas, McAuliffe, Sekhon (2021) is the reference. Bayesian is a Beta(1, 1) prior updated to a Beta posterior with deterministic quadrature forP(B > A), expected loss, and credible interval. Revenue sequential
is the asymptotic confidence sequence for a mean (Waudby-Smith and
Ramdas). Revenue Bayesian is a normal-CLT posterior on RPV. Every method
is unit-tested against reference implementations; see the source under
worker/src/lib/stats*.ts.