Skip to main content
Weird results are almost always a data issue, not a statistics one. Walk this list before rejecting a verdict.

​
Conversions higher than visitors

You see 200 visitors and 250 conversions on one variant. Cause: the goal counts fires instead of unique converters. That is usually a revenue goal (revenue is summed across fires) or a custom event set to “count all fires”. Fix: check the goal’s definition. Switch the goal type if you want unique converters.

​
Two arms have wildly different sizes

Split is 50/50 but the results show 60/40 or worse. Almost always a Sample Ratio Mismatch. See SRM for the full write-up. Short version:
  • A click activation with a selector that only exists in one variant.
  • A redirect variant whose exposure was refused by the browser. The runtime dispatches it before it navigates, so it is no longer a race against the page unloading, but a blocked sendBeacon still loses that exposure and the loss falls on one variation only.
  • Uneven bot or crawler traffic reaching one variant more than the other.

​
Lift flip-flops daily

Yesterday Variant B was +2 %. Today Variant A is +1 %. Almost always too small a sample. The CI is wide; the point estimate is bouncing around inside it. A converging test stops flip-flopping. Fix: let the test run until your chosen method reaches its decision rule or a planned stopping point. A wide interval means high uncertainty, not proof that the true effect is small.

​
Verdict says “no effect” but I see a real change on-site

Two possibilities:
  1. The change is not what the primary goal measures. You redesigned the hero, but the primary goal is checkout completion six pages downstream. Downstream goals move slowly and are hard to attribute.
  2. The change genuinely does not move the primary. A pretty change that does not affect behavior. Happens more than people admit.

​
Revenue does not match my analytics

GA4 shows 10,000inthesameperiod.ABTestlyshows10,000 in the same period. ABTestly shows 8,000. Neither is wrong; they measure different things.
  • ABTestly counts revenue tagged to an experiment’s exposed visitors. Non-bucketed visitors are not in the total.
  • GA4 counts all revenue, regardless of experiment membership.
Use ABTestly to compare tracked revenue between the variants of an experiment, and GA4 for site wide revenue. The ABTestly figure is the revenue observed for exposed visitors under its goal, dedup, and currency rules; it is not a measure of revenue caused by the experiment. Differences from GA4 can come from population, identity, attribution window, consent, and currency handling.

​
Results page shows “Processing”

The Exact Ledger is catching up on a burst of events. Dashboard live counters are current; the ledger-driven verdict is held while the queue drains. Usually seconds. Fix: wait a minute. If it does not clear in ten minutes, email support.

​
Everything looks fine but I do not believe the winner

Sensible skepticism:
  1. Check the CI width vs the effect size. A “9 % lift” with a CI of [-4 %, +22 %] could easily be zero.
  2. Check the guardrails. A win with a broken guardrail is usually not a real win.
  3. Run it again. Winners that survive replication are stronger evidence. If a replication differs, compare both tests’ intervals, implementation, and populations before concluding why, rather than assuming the first result was not real.
Last modified on September 6, 2026