Skip to main content
If you’ve landed here from a warning on your experiment results, this page explains what’s happening and what to do about it.

​
What it means in plain terms

When you set up an A/B test, you tell ABTestly how to split traffic between variants, usually 50/50, sometimes 70/30 or 80/20. Sample ratio mismatch means the actual split of visitors who entered the test no longer matches what you configured, by more than chance could explain. A normal 50/50 test won’t land on exactly 50/50, random variation means you might see 49.4/50.6 after a few thousand visitors, and that’s fine. But if you see, say, 54/46 on a 50/50 test with tens of thousands of visitors, that’s not random. Something in the way visitors are being counted, routed, or bucketed is producing a skewed split. ABTestly only fires the warning when the imbalance is statistically extreme, well beyond the level where chance is a plausible explanation.

​
What you should do with the results

Don’t trust the numbers on a test that’s showing SRM. The two variant groups aren’t comparable anymore, one group has been filtered or shaped differently than the other, so any difference in conversion rate could be caused by that filtering instead of by the variant. We still show you the conversion-rate table when SRM fires (hiding it would be worse, you’d lose context). But we mark the results as potentially invalid, and the right move is almost always:
  1. Find the cause.
  2. Fix it.
  3. Restart the test from a clean slate.

​
The test behind the warning

ABTestly publishes the method so you can judge the warning yourself. Below either minimum the experiment reads as collecting and no verdict is published. That is why a very new test never shows the warning, however lopsided its counts look. A few details worth knowing:
  • Weights, not traffic allocation. The check tests the split among the visitors who entered the experiment. Excluding a share of your traffic from the test entirely does not affect it.
  • Arms weighted to zero are excluded from the chi square and the remaining weights are renormalised. If a zero weight arm is still receiving visitors, that is reported separately, because it is its own bug.
  • Variants deleted mid run are excluded from the total and reported separately, so a deletion cannot quietly hide an imbalance.
  • The window is the current run. The check reads only the visitors counted since the last data reset and under the current experiment generation. It does not compare all historical traffic against a newer allocation.
  • Changing weights mid run is a confound we do not correct for. The check uses whatever weights are configured now, and tests every visitor counted in the window against them, including visitors bucketed under the old weights. If you change weights on a live test, reset the data.
  • Every results page load re evaluates. There is no alpha spending and no correction for repeated looks. The p below 0.001 threshold is set far stricter than a conventional 0.05 partly for that reason, but a test watched continuously for weeks still has more opportunities to trip it than a test looked at once.

​
Common causes

SRM is almost always a bug somewhere in the data pipeline, not a real imbalance in how interesting the variants are. The usual culprits:
  • The snippet not running at all on some pages, browsers or devices. ABTestly counts the exposure before it applies any variation code, so a variation that breaks the page after it has been applied does not skew the ratio. What does skew it is a visitor never reaching the counting step: a route where the snippet is missing, a browser where it throws before bucketing, or a page that unloads before the beacon leaves.
  • Redirects or routing that skip one variant. A redirect rule, a CDN config, or a single-page-app navigation that doesn’t pass through the bucketing step for one of the variants.
  • A variation that delays the snippet itself. Bucketing happens early, but a page that takes long enough to reach the snippet loses visitors who leave first. If that delay differs by variation, counted exposures skew toward the faster one.
  • Targeting or bucketing setup that changed mid-run. If you change variant weights, targeting rules, or paused/resumed traffic during a test, the historical split may not match the current expected split.
  • Bot or crawler traffic hitting variants unevenly. Less common, but possible if one variant has a different URL pattern.
  • Cache or localStorage issues. A cache hit that serves the wrong variant, or a localStorage write that fails on one variant, produces uneven counted exposures.

​
How to investigate

Work through the list above against your specific test:
  1. Open both variants in a fresh incognito window and walk through the user flow. Look for JavaScript errors, broken layouts, redirects that drop you out of the test, or unusual load times.
  2. Check your targeting rules, did anything change after the test started? URL conditions, audience filters, traffic allocation?
  3. Look at the per-variant counts on the results page. Which variant is over- or under-represented? If one variant has way fewer exposures than expected, that variant is most likely the one with the bug.
  4. Check the variant code itself, anything that throws an error, blocks the page, or replaces analytics/tracking can cause SRM.
  5. If you can’t find the cause, write us at support@abtestly.com with the experiment URL, we’ll help you investigate.

​
What to do after you find it

Once you’ve fixed the underlying cause, restart the test rather than continuing. The pre-fix data is contaminated by the imbalance. combining it with post-fix data won’t give you a clean answer either. A fresh run on clean conditions is the only reliable path to a trustworthy result. You don’t need to delete the old data, ABTestly’s “Reset data” button on the experiment settings page keeps the experiment configuration but wipes the result counters, so you can restart cleanly without rebuilding the test from scratch.

​
Why we surface this at all

Most A/B-testing tools quietly publish whatever results come out of their data pipeline, including from tests where the underlying split was broken. We’d rather tell you “this test is showing a sample ratio mismatch, the result may not be trustworthy” than let you ship a change based on an invalid comparison. The warning is a guardrail protecting the validity of decisions you make from these tools, we’d rather be honest about a broken test than show you a confident-looking but misleading number. The bar for firing the warning is set very strictly, far stricter than a conventional significance threshold, so an SRM warning is a real signal worth taking seriously. Two limitations are worth knowing. The check compares the visitors who were counted, and ABTestly counts the exposure before it applies any variation code. That ordering is deliberate: it means the split cannot be bent by what a variation does to the page. The tradeoff is that a variation which breaks after being applied does not shift the ratio at all, so SRM will not catch it. That class of failure shows up in goal firing health instead. The check also only sees one experiment at a time. On a single page app, a variation in experiment A can change navigation and so change which visitors ever reach experiment B’s page. Inside B the split stays correct, so no chi square will flag it, even though B’s population has been reshaped. Put tests that can steer each other’s traffic into a mutual exclusion group.

​
Start testing

You can start a 14 day trial on monthly Starter, Pro or Business and run this against your own traffic. Annual billing and Enterprise are not eligible, and the trial page explains why.

Start a 14 day ABTestly trial

Fourteen days on monthly Starter, Pro or Business, at your own traffic. A card is required when you start the trial and the first charge lands on day 15. The snippet is served from Cloudflare’s edge, and every result carries a confidence interval.
Last modified on September 6, 2026