Sample ratio mismatch (SRM)
If you’ve landed here from a warning on your experiment results, this page explains what’s happening and what to do about it.What it means in plain terms
When you set up an A/B test, you tell ABTestly how to split traffic between variants — usually 50/50, sometimes 70/30 or 80/20. Sample ratio mismatch means the actual split of visitors who entered the test no longer matches what you configured, by more than chance could explain. A normal 50/50 test won’t land on exactly 50/50 — random variation means you might see 49.4/50.6 after a few thousand visitors, and that’s fine. But if you see, say, 54/46 on a 50/50 test with tens of thousands of visitors, that’s not random. Something in the way visitors are being counted, routed, or bucketed is producing a skewed split. ABTestly only fires the warning when the imbalance is statistically extreme — well beyond the level where chance is a plausible explanation.What you should do with the results
Don’t trust the numbers on a test that’s showing SRM. The two variant groups aren’t comparable anymore — one group has been filtered or shaped differently than the other, so any difference in conversion rate could be caused by that filtering instead of by the variant. We still show you the conversion-rate table when SRM fires (hiding it would be worse — you’d lose context). But we mark the results as potentially invalid, and the right move is almost always:- Find the cause.
- Fix it.
- Restart the test from a clean slate.
Common causes
SRM is almost always a bug somewhere in the data pipeline, not a real imbalance in how interesting the variants are. The usual culprits:- A variant that breaks or crashes on certain pages, browsers, or devices. Visitors in the broken variant bounce before being recorded, while visitors in the working variant aren’t bounced, so the counted-variant ratio drifts.
- Redirects or routing that skip one variant. A redirect rule, a CDN config, or a single-page-app navigation that doesn’t pass through the bucketing step for one of the variants.
- A variant that’s much slower than the other. If one variant adds noticeable load time, more visitors bail mid-load before being recorded — counted exposures skew toward the faster variant.
- Targeting or bucketing setup that changed mid-run. If you change variant weights, targeting rules, or paused/resumed traffic during a test, the historical split may not match the current expected split.
- Bot or crawler traffic hitting variants unevenly. Less common, but possible if one variant has a different URL pattern.
- Cache or
localStorageissues. A cache hit that serves the wrong variant, or alocalStoragewrite that fails on one variant, produces uneven counted exposures.
How to investigate
Work through the list above against your specific test:- Open both variants in a fresh incognito window and walk through the user flow. Look for JavaScript errors, broken layouts, redirects that drop you out of the test, or unusual load times.
- Check your targeting rules — did anything change after the test started? URL conditions, audience filters, traffic allocation?
- Look at the per-variant counts on the results page. Which variant is over- or under-represented? If one variant has way fewer exposures than expected, that variant is most likely the one with the bug.
- Check the variant code itself — anything that throws an error, blocks the page, or replaces analytics/tracking can cause SRM.
- If you can’t find the cause, write us at support@abtestly.com with the experiment URL — we’ll help you investigate.