> ## Documentation Index > Fetch the complete documentation index at: https://docs.abtestly.com/llms.txt > Use this file to discover all available pages before exploring further. # Sample ratio mismatch > The chi-square goodness-of-fit test we run on the observed traffic split, the incomplete-gamma machinery behind its p-value, why the bar is 0.001 rather than 0.05, and exactly which of its outputs reach a screen. Every conversion-rate comparison rests on one assumption: the two groups differ only in which variation they saw. Sample ratio mismatch is the check on that assumption. It compares the split you configured against the split that actually arrived, and when the gap is too large to be chance, the comparison sitting on top of it is not measuring your change any more. This page is the method. For what to do when the warning fires, see [detecting sample ratio mismatch](/results/srm). ## The statistic Pearson chi-square goodness of fit on per-variation distinct user counts. `srmCheck` in `worker/src/lib/stats.ts`. ``` total = sum(observed) E_i = total * weight_i / sum(weights) chi2 = sum over i of (observed_i - E_i)^2 / E_i dof = number of compared variations - 1 p = P(X >= chi2 | dof) ``` Weights are variation weights in basis points, not traffic allocation. Allocation decides what share of visitors enter the experiment at all; weights decide the split among those who did. SRM tests the second. The expected counts renormalise over whatever weights are handed in, so excluding a variation from the comparison automatically rescales the rest. The counted unit is distinct users, not impressions. A visitor who reloads twenty times contributes one. ## Where the p-value comes from There is no scipy in a Worker, so the chi-square survival function is computed directly. Using the standard relation between the chi-square CDF and the regularized lower incomplete gamma, ``` F(x; k) = P(k/2, x/2) P(X >= x) = Q(k/2, x/2) ``` `chiSquareSurvival` evaluates `Q`, the regularized upper incomplete gamma, by the Numerical Recipes 6.2 split: the series expansion `gser` when `x < a + 1`, the continued fraction `gcf` otherwise, so each branch stays in its fast-converging regime. Both cut off at 200 iterations or a relative epsilon of 3e-7. The log-gamma is a Lanczos approximation good to about 15 digits. Four-digit precision on the p-value is more than the decision needs, since anything under 0.001 collapses to the same verdict. A `chi2` of zero or less returns 1. A non-positive `dof` throws, because a caller that gets there has already made a mistake upstream. ## The three constants | Constant | Value | Effect | | --------------------------- | ----- | ---------------------------------------------------- | | `SRM_P_THRESHOLD` | 0.001 | Below this p, status is `flagged` | | `SRM_MIN_SAMPLE_TOTAL` | 500 | Below this total, status is `collecting` | | `SRM_MIN_EXPECTED_PER_CELL` | 5 | Any expected cell below this, status is `collecting` | The 500 floor and the expected-cell rule are both about the chi-square approximation rather than about caution. The asymptotic distribution is a poor fit for small expected counts, and a skewed split such as 95/5 keeps the small arm's expected count low long after the total looks healthy. Below either floor no verdict is published at all, which is why a brand new test never shows the warning however lopsided its counts look. ## Why 0.001 and not 0.05 The results page re-evaluates on every load, and there is no alpha spending and no correction for repeated looks. At 0.05 a continuously-watched healthy test would trip the warning regularly, the warning would stop meaning anything, and people would learn to click past it. The 0.001 bar is the standard published bar for exactly this reason, and it is strict enough that continuous re-evaluation rarely reaches it. It is not free: a test watched for weeks still has more chances to trip than a test looked at once. ## Three worked splits All three are a configured 50/50, so `dof` is 1 and each expected count is half the total. A 20,000-visitor test that split 10,100 / 9,900: ``` E = 10,000 each chi2 = (100^2 / 10000) + (100^2 / 10000) = 1 + 1 = 2 p = Q(0.5, 1) = 0.1573 ``` Status `ok`. A 50.5 / 49.5 split at this size is ordinary noise. The same test splitting 10,214 / 9,786: ``` E = 10,000 each chi2 = (214^2 / 10000) * 2 = 4.5796 * 2 = 9.1592 p = Q(0.5, 4.5796) = 0.002475 ``` Still `ok`. This is the case worth sitting with. A 51.1 / 48.9 split on 20,000 visitors would clear a conventional 0.05 bar comfortably, and it is very probably a real bug. We do not flag it, because a bar loose enough to catch it is a bar that fires on healthy tests too. The check is deliberately tuned to be quiet, and to be believed when it speaks. A 10,000-visitor test splitting 5,218 / 4,782: ``` E = 5,000 each chi2 = (218^2 / 5000) * 2 = 9.5048 * 2 = 19.0096 p = Q(0.5, 9.5048) = 0.0000130 ``` Status `flagged`. Half the sample of the previous example and a wider proportional gap, and the p-value falls by two orders of magnitude. You can run this check on your own counts without an account. The public [A/B test calculator](https://abtestly.com/ab-test-calculator) takes the visitors per arm and the configured split and applies the same test at the same 0.001 threshold. ## What the orchestration adds `srmCheck` is pure arithmetic. `computeSrm` in `worker/src/lib/srm.ts` is the single place that turns exposure rows into a verdict, and it is what the results route calls. The rows come from a dedicated query, `srmExposureSql`. It counts distinct users per variation bounded only by the data-reset floor and the experiment's generation clause. It structurally cannot take a date range or a segment filter, which means the SRM verdict does not change when you narrow the results view. That was a deliberate correction: a filtered SRM check answers a question nobody asked. Variations weighted to zero are dropped from the chi-square rather than carried with an expected count of zero, which would trip the expected-cell guard and pin the experiment at `collecting` forever. Any zero-weight arm still drawing visitors is reported separately, since a paused arm receiving traffic is its own bug. Exposure rows whose variation key no longer exists, typically a variation deleted mid-run, are excluded from the total by construction and reported separately, so a deletion cannot quietly absorb an imbalance. If any row came back write-sampled, the verdict is suppressed entirely and the status becomes `sampled`. Sampling drops whole visitor rows with a bias toward busier arms, so the observed split is skewed by the sampling itself and any pass or fail would be an artifact of the pipeline. We publish the observed counts and no verdict. ## What actually reaches a screen This matters more than it sounds, because a computed value is not a feature. A `flagged` status renders a red banner above the results, carrying the per-variation observed and expected counts, the p-value, and a link here. An experiment carrying a flag also shows a warning chip in the experiments list, and is disqualified from business-impact reporting. `ok` and `collecting` render nothing at all. There is no green tick, no "SRM healthy" panel, and no p-value on a passing test. If you want the number on a healthy experiment, you will not find it in the dashboard. The zero-weight and orphan-key diagnostics are computed on every results request and returned on the API response as `srmDiagnostics`. They have no dashboard surface yet. The evaluation window runs from the data-reset floor forward. A weight change mid-run is a genuine confound, because visitors bucketed under the old weights are tested against the new ones, and we do not correct for it. A floor that starts at the last weight change is planned and not shipped. If you change weights on a live test, reset the data. ## Two failures SRM cannot see The exposure is recorded before any variation code runs. That ordering is deliberate, because it means the split cannot be bent by what a variation does to the page. The cost is that a variation which breaks the page after it has been applied does not move the ratio at all. That class of failure surfaces in [goal firing health](/goals/health). The check also sees one experiment at a time. On a single-page app, a variation in experiment A can change navigation and therefore change which visitors ever reach experiment B. Inside B the split stays correct, so no chi-square will flag it, even though B's population has been reshaped. Put tests that can steer each other's traffic in a [mutual exclusion group](/exclusion-groups/overview). ## Sources The taxonomy of causes and the practitioner thresholds this page uses come from the standard reference on the subject. * [Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. and Dmitriev, P. (2019), *Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners*, KDD '19](https://exp-platform.com/Documents/2019_KDDFabijanGupchupFuptaOmhoverVermeerDmitriev.pdf). Co-authored across several large experimentation programmes, and the source of the rule of thumb that an SRM invalidates the experiment rather than merely flagging it. Every link above was checked on 6 September 2026. ## Related Causes, an investigation order, and what to do after you find one. The assignment the split is supposed to produce, and how to re-derive it. What the numbers underneath a flagged banner would have meant. Runs the same `srmCheck` and the same three constants as the product.