The statistic
Pearson chi-square goodness of fit on per-variation distinct user counts.srmCheck in worker/src/lib/stats.ts.
Where the p-value comes from
There is no scipy in a Worker, so the chi-square survival function is computed directly. Using the standard relation between the chi-square CDF and the regularized lower incomplete gamma,chiSquareSurvival evaluates Q, the regularized upper incomplete
gamma, by the Numerical Recipes 6.2 split: the series expansion gser
when x < a + 1, the continued fraction gcf otherwise, so each branch
stays in its fast-converging regime. Both cut off at 200 iterations or a
relative epsilon of 3e-7. The log-gamma is a Lanczos approximation good
to about 15 digits. Four-digit precision on the p-value is more than the
decision needs, since anything under 0.001 collapses to the same
verdict.
A chi2 of zero or less returns 1. A non-positive dof throws, because
a caller that gets there has already made a mistake upstream.
The three constants
The 500 floor and the expected-cell rule are both about the chi-square
approximation rather than about caution. The asymptotic distribution is
a poor fit for small expected counts, and a skewed split such as 95/5
keeps the small arm’s expected count low long after the total looks
healthy. Below either floor no verdict is published at all, which is why
a brand new test never shows the warning however lopsided its counts
look.
Why 0.001 and not 0.05
The results page re-evaluates on every load, and there is no alpha spending and no correction for repeated looks. At 0.05 a continuously-watched healthy test would trip the warning regularly, the warning would stop meaning anything, and people would learn to click past it. The 0.001 bar is the standard published bar for exactly this reason, and it is strict enough that continuous re-evaluation rarely reaches it. It is not free: a test watched for weeks still has more chances to trip than a test looked at once.Three worked splits
All three are a configured 50/50, sodof is 1 and each expected count
is half the total.
A 20,000-visitor test that split 10,100 / 9,900:
ok. A 50.5 / 49.5 split at this size is ordinary noise.
The same test splitting 10,214 / 9,786:
ok. This is the case worth sitting with. A 51.1 / 48.9 split on
20,000 visitors would clear a conventional 0.05 bar comfortably, and it
is very probably a real bug. We do not flag it, because a bar loose
enough to catch it is a bar that fires on healthy tests too. The check
is deliberately tuned to be quiet, and to be believed when it speaks.
A 10,000-visitor test splitting 5,218 / 4,782:
flagged. Half the sample of the previous example and a wider
proportional gap, and the p-value falls by two orders of magnitude.
You can run this check on your own counts without an account. The
public A/B test calculator
takes the visitors per arm and the configured split and applies the
same test at the same 0.001 threshold.
What the orchestration adds
srmCheck is pure arithmetic. computeSrm in worker/src/lib/srm.ts
is the single place that turns exposure rows into a verdict, and it is
what the results route calls.
The rows come from a dedicated query, srmExposureSql. It counts
distinct users per variation bounded only by the data-reset floor and
the experiment’s generation clause. It structurally cannot take a date
range or a segment filter, which means the SRM verdict does not change
when you narrow the results view. That was a deliberate correction: a
filtered SRM check answers a question nobody asked.
Variations weighted to zero are dropped from the chi-square rather than
carried with an expected count of zero, which would trip the
expected-cell guard and pin the experiment at collecting forever. Any
zero-weight arm still drawing visitors is reported separately, since a
paused arm receiving traffic is its own bug.
Exposure rows whose variation key no longer exists, typically a
variation deleted mid-run, are excluded from the total by construction
and reported separately, so a deletion cannot quietly absorb an
imbalance.
If any row came back write-sampled, the verdict is suppressed entirely
and the status becomes sampled. Sampling drops whole visitor rows with
a bias toward busier arms, so the observed split is skewed by the
sampling itself and any pass or fail would be an artifact of the
pipeline. We publish the observed counts and no verdict.
What actually reaches a screen
This matters more than it sounds, because a computed value is not a feature. Aflagged status renders a red banner above the results, carrying the
per-variation observed and expected counts, the p-value, and a link
here. An experiment carrying a flag also shows a warning chip in the
experiments list, and is disqualified from business-impact reporting.
ok and collecting render nothing at all. There is no green tick, no
“SRM healthy” panel, and no p-value on a passing test. If you want the
number on a healthy experiment, you will not find it in the dashboard.
The zero-weight and orphan-key diagnostics are computed on every results
request and returned on the API response as srmDiagnostics. They have
no dashboard surface yet.
The evaluation window runs from the data-reset floor forward. A weight
change mid-run is a genuine confound, because visitors bucketed under
the old weights are tested against the new ones, and we do not correct
for it. A floor that starts at the last weight change is planned and not
shipped. If you change weights on a live test, reset the data.
Two failures SRM cannot see
The exposure is recorded before any variation code runs. That ordering is deliberate, because it means the split cannot be bent by what a variation does to the page. The cost is that a variation which breaks the page after it has been applied does not move the ratio at all. That class of failure surfaces in goal firing health. The check also sees one experiment at a time. On a single-page app, a variation in experiment A can change navigation and therefore change which visitors ever reach experiment B. Inside B the split stays correct, so no chi-square will flag it, even though B’s population has been reshaped. Put tests that can steer each other’s traffic in a mutual exclusion group.Sources
The taxonomy of causes and the practitioner thresholds this page uses come from the standard reference on the subject.- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. and Dmitriev, P. (2019), Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners, KDD ‘19. Co-authored across several large experimentation programmes, and the source of the rule of thumb that an SRM invalidates the experiment rather than merely flagging it.
Related
Reading an SRM warning
Causes, an investigation order, and what to do after you find one.
Bucketing model
The assignment the split is supposed to produce, and how to re-derive it.
Confidence intervals
What the numbers underneath a flagged banner would have meant.
A/B test calculator
Runs the same
srmCheck and the same three constants as the product.