A result is only worth as much as the chain it sits on: research, hypothesis, implementation, measurement, analysis, decision. What breaks at each link, what we do about it, and the experiment on our own homepage that ran for a month without recording a conversion.
Our homepage test had been live since 8 August. The experiments list showed
it collecting. A month later it still had no result to show.The statistics were never the problem. The snippet on our own marketing site
is registered in our cookie banner’s analytics category, so it does not load
until a visitor accepts cookies. The experiment was assigned, configured and
running, and the only visitors it could ever reach were the ones who clicked
Accept.Our own dashboard was no help either. The list said Collecting while the
experiment page said Not collecting, one click apart. Both were correct.
The list meant “running, no result yet”; the detail page meant “we have not
seen this snippet load in 24 hours”. Same word, two meanings, and the screen
that knew was not the screen you looked at first.We fixed both this week. But the month is gone, and no amount of statistical
sophistication would have bought it back, because nothing upstream of the
statistics was working.
Six links, and every one inherits the one before it:If the research is weak, the hypothesis is arbitrary. If the implementation
is broken, the measurement is misleading. If the exposure or goal event is
wrong, no engine on the results page can rescue it. And if nobody agreed what
to do before launch, the decision becomes a negotiation around whichever
number looks best.This is why experiment quality is not only a statistics problem. The rest of
this section covers the statistics in detail. This page covers what has to be
true before they mean anything.
01 Research: start with a problem, not a variation
The fastest way to build a bad test is to begin with the change. Make the
button bigger. Add a testimonial. Rewrite the headline. Those can become
useful experiments, but none of them is a hypothesis yet, because none says
what is currently going wrong.A stronger starting point names the failure:
New visitors do not understand the difference between our plans. If the
comparison is clearer, more of them should pick one, without more refunds
or support contacts.
Now there is something to check before building anything. Are visitors
reaching the pricing section at all? Are they clicking things that are not
clickable? Does GA4 show a drop between plan views and checkout starts?Signals exists for this step. It reads page structure,
your GA4 funnel where you have connected it, and the first party heatmap
where one is available, then scores each idea against an
evidence contract and labels it Test ready,
Needs evidence, Below threshold or Unverified.Be clear about what that does and does not mean. It is not a prediction of
which variation wins. An idea marked Needs evidence is not a bad idea, it is
an idea whose supporting evidence has not cleared its gate, and the label
names the cheapest thing that would raise it.
A hypothesis you cannot lose is not a hypothesis. Before building, write down
the primary metric, the guardrails you refuse to damage, the minimum
detectable effect and, separately, the smallest improvement actually worth
shipping. Those last two are different numbers and both matter. See
minimum detectable effect.Write the decision rule at the same time. A rule agreed after the data
arrives is not a rule.
This is the link that can fail without anyone noticing.Assignment and application are two separate events. A visitor can be assigned
to variant B and never see it, because the JavaScript threw, a selector was
missing, or the component had not rendered yet. The assignment already
happened.ABTestly records it anyway, deliberately. Every variation’s code runs inside
its own try/catch, and a throw does not retract the exposure. That sounds
wrong until you ask who it protects: control has no variation JavaScript, so
control can never throw. Suppress the broken assignments and the loss lands
on one arm only, and you have manufactured a
sample ratio mismatch out of your own
error handling. The tidier number would be the dishonest one.Optimizely counts its impression on the decision as well, and its
documentation says impressions are processed when they are sent, rather than
when they occur. Assignment is the event. Whether the page changed is a
separate question, and until recently nobody we looked at answered it, ourselves included.We now do. The results page reports how many page loads a variation threw on,
as a count and never a percentage, because the
error is deduplicated per session while exposures are recorded per route
change, and a rate across those two granularities would understate the
problem on a single page app without limit.Before launch, check the things that break quietly:
Does assignment hold across route changes, not just first load?
Does the exposure fire only when the experience is genuinely shown?
Do the primary goal and guardrails record on both arms?
Does the original return cleanly when the test is paused?
One argument worth heading off starts with “GA4 does not match the
platform, so the test cannot be trusted”. That conclusion is too quick.The two are not counting the same population. The platform starts from
visitors who were eligible and assigned. GA4 starts from events that were
collected, processed and attributed. Consent, identity, assignment timing and
metric scope all move those apart, and none of that means either number is
broken.ABTestly keeps accepted exposures and goals in an
exact experiment ledger, and the snippet can push
the same experience_impression event into your existing dataLayer and GTM
setup. That makes the comparison useful, because both sides start from one
shared event instead of two loosely related reports.The question worth asking is why the populations differ. Not which dashboard
to make look right.
05 Analysis: significant and worth doing are different tests
A result can be significant and commercially pointless. It can also be
inconclusive and still tell you where to aim a better powered follow up.A winning primary metric is not enough on its own if revenue per visitor
falls, if the split is uneven, or if one unusually large order is carrying
the arm. That last one is real enough that revenue analysis applies an order
value cap, and the cap excludes rather than winsorises, so a single outlier
cannot quietly set the result.One deliberate design choice worth knowing: the verdict on the results page
is confidence only. Guardrails, sample ratio mismatch and delivery health are
not folded into it. They sit beside it. A trust signal that silently
moves a winner is a trust signal you cannot audit, so we show them together
and leave the judgement with you.
If the team has not agreed what a result means, the meeting decides instead
of the experiment.Record the decision and, more importantly, the transferable part: what you
now believe about your visitors that you did not believe last month. It is also
the easiest step to skip, because nothing forces it.
AI is genuinely useful across this chain. It can summarise research, draft an
implementation, suggest segments worth investigating and propose alternative
explanations for a result.It is also very good at making thin evidence sound confident. It will offer a
reason a result happened, and the reason will be fluent whether or not it is
true.So the useful test of an AI feature in experimentation is not how convincing
its output reads. It is whether it shows you what evidence it used, what it
assumed, what it could not verify, and what experiment would settle it. An
assistant that hands you a conclusion has moved the judgement call somewhere
you cannot inspect.
If this test returns a result tomorrow, will we know what it means, who it
applies to, and what we will do about it?If the answer is no, more traffic will not fix it. Somewhere upstream, a link
in the chain is weak, and the result will inherit it.Our homepage test is the version of this we learned the hard way. It ran for
a month, on our own site, built by the people who make the tool, and the
weak link was not the statistics. It was the cookie banner deciding who
the test could reach.
