Skip to main content
Weird results are almost always a data issue, not a statistics one. Walk this list before rejecting a verdict.

​
Conversions higher than visitors

You see 200 visitors and 250 conversions on one variant. Cause: the goal counts fires instead of unique converters. That is usually a revenue goal (revenue is summed across fires) or a custom event set to “count all fires”. Fix: check the goal’s definition. Switch the goal type if you want unique converters.

​
Two arms have wildly different sizes

Split is 50/50 but the results show 60/40 or worse. Almost always a Sample Ratio Mismatch. See SRM for the full write-up. Short version:
  • Preview links poisoning the data.
  • A click activation with a selector that only exists in one variant.
  • A redirect variant firing an exposure then navigating away before the beacon lands.
  • An analytics adapter blocking the beacon on one arm’s page.

​
Lift flip-flops daily

Yesterday Variant B was +2 %. Today Variant A is +1 %. Almost always too small a sample. The CI is wide; the point estimate is bouncing around inside it. A converging test stops flip-flopping. Fix: wait longer, or accept that the true effect is very small and probably not worth calling.

​
Verdict says “no effect” but I see a real change on-site

Two possibilities:
  1. The change is not what the primary goal measures. You redesigned the hero, but the primary goal is checkout completion six pages downstream. Downstream goals move slowly and are hard to attribute.
  2. The change genuinely does not move the primary. A pretty change that does not affect behavior. Happens more than people admit.

​
Revenue does not match my analytics

GA4 shows 10,000inthesameperiod.ABTestlyshows10,000 in the same period. ABTestly shows 8,000. Neither is wrong; they measure different things.
  • ABTestly counts revenue tagged to an experiment’s exposed visitors. Non-bucketed visitors are not in the total.
  • GA4 counts all revenue, regardless of experiment membership.
Trust ABTestly for experiment-attributable revenue and GA4 for total revenue.

​
Results page shows “Processing”

The Exact Ledger is catching up on a burst of events. Dashboard live counters are current; the ledger-driven verdict is held while the queue drains. Usually seconds. Fix: wait a minute. If it does not clear in ten minutes, email support.

​
Everything looks fine but I do not believe the winner

Sensible skepticism:
  1. Check the CI width vs the effect size. A “9 % lift” with a CI of [-4 %, +22 %] could easily be zero.
  2. Check the guardrails. A win with a broken guardrail is usually not a real win.
  3. Re-run the test. Winners that survive replication are winners; ones that do not, were not.