Underneath the statistics

Our homepage test had been live since 8 August. The experiments list showed it collecting. A month later it still had no result to show. The statistics were never the problem. The snippet on our own marketing site is registered in our cookie banner’s analytics category, so it does not load until a visitor accepts cookies. The experiment was assigned, configured and running, and the only visitors it could ever reach were the ones who clicked Accept. Our own dashboard was no help either. The list said Collecting while the experiment page said Not collecting, one click apart. Both were correct. The list meant “running, no result yet”; the detail page meant “we have not seen this snippet load in 24 hours”. Same word, two meanings, and the screen that knew was not the screen you looked at first. We fixed both this week. But the month is gone, and no amount of statistical sophistication would have bought it back, because nothing upstream of the statistics was working.

​

A result is a chain

Six links, and every one inherits the one before it: If the research is weak, the hypothesis is arbitrary. If the implementation is broken, the measurement is misleading. If the exposure or goal event is wrong, no engine on the results page can rescue it. And if nobody agreed what to do before launch, the decision becomes a negotiation around whichever number looks best. This is why experiment quality is not only a statistics problem. The rest of this section covers the statistics in detail. This page covers what has to be true before they mean anything.

​

01 Research: start with a problem, not a variation

The fastest way to build a bad test is to begin with the change. Make the button bigger. Add a testimonial. Rewrite the headline. Those can become useful experiments, but none of them is a hypothesis yet, because none says what is currently going wrong. A stronger starting point names the failure:

New visitors do not understand the difference between our plans. If the comparison is clearer, more of them should pick one, without more refunds or support contacts.

Now there is something to check before building anything. Are visitors reaching the pricing section at all? Are they clicking things that are not clickable? Does GA4 show a drop between plan views and checkout starts? Signals exists for this step. It reads page structure, your GA4 funnel where you have connected it, and the first party heatmap where one is available, then scores each idea against an evidence contract and labels it Test ready, Needs evidence, Below threshold or Unverified. Be clear about what that does and does not mean. It is not a prediction of which variation wins. An idea marked Needs evidence is not a bad idea, it is an idea whose supporting evidence has not cleared its gate, and the label names the cheapest thing that would raise it.

​

02 Hypothesis: something a result can contradict

A hypothesis you cannot lose is not a hypothesis. Before building, write down the primary metric, the guardrails you refuse to damage, the minimum detectable effect and, separately, the smallest improvement actually worth shipping. Those last two are different numbers and both matter. See minimum detectable effect. Write the decision rule at the same time. A rule agreed after the data arrives is not a rule.

​

03 Implementation: assignment is not application

This is the link that can fail without anyone noticing. Assignment and application are two separate events. A visitor can be assigned to variant B and never see it, because the JavaScript threw, a selector was missing, or the component had not rendered yet. The assignment already happened. ABTestly records it anyway, deliberately. Every variation’s code runs inside its own try/catch, and a throw does not retract the exposure. That sounds wrong until you ask who it protects: control has no variation JavaScript, so control can never throw. Suppress the broken assignments and the loss lands on one arm only, and you have manufactured a sample ratio mismatch out of your own error handling. The tidier number would be the dishonest one. Optimizely counts its impression on the decision as well, and its documentation says impressions are processed when they are sent, rather than when they occur. Assignment is the event. Whether the page changed is a separate question, and until recently nobody we looked at answered it, ourselves included. We now do. The results page reports how many page loads a variation threw on, as a count and never a percentage, because the error is deduplicated per session while exposures are recorded per route change, and a rate across those two granularities would understate the problem on a single page app without limit. Before launch, check the things that break quietly:

• Does assignment hold across route changes, not just first load?
• Does the exposure fire only when the experience is genuinely shown?
• Do the primary goal and guardrails record on both arms?
• Does the original return cleanly when the test is paused?

​

04 Measurement: do not force two systems to agree

One argument worth heading off starts with “GA4 does not match the platform, so the test cannot be trusted”. That conclusion is too quick. The two are not counting the same population. The platform starts from visitors who were eligible and assigned. GA4 starts from events that were collected, processed and attributed. Consent, identity, assignment timing and metric scope all move those apart, and none of that means either number is broken. ABTestly keeps accepted exposures and goals in an exact experiment ledger, and the snippet can push the same experience_impression event into your existing dataLayer and GTM setup. That makes the comparison useful, because both sides start from one shared event instead of two loosely related reports. The question worth asking is why the populations differ. Not which dashboard to make look right.

​

05 Analysis: significant and worth doing are different tests

A result can be significant and commercially pointless. It can also be inconclusive and still tell you where to aim a better powered follow up. A winning primary metric is not enough on its own if revenue per visitor falls, if the split is uneven, or if one unusually large order is carrying the arm. That last one is real enough that revenue analysis applies an order value cap, and the cap excludes rather than winsorises, so a single outlier cannot quietly set the result. One deliberate design choice worth knowing: the verdict on the results page is confidence only. Guardrails, sample ratio mismatch and delivery health are not folded into it. They sit beside it. A trust signal that silently moves a winner is a trust signal you cannot audit, so we show them together and leave the judgement with you.

​

06 Decision: written down before the data

If the team has not agreed what a result means, the meeting decides instead of the experiment. Record the decision and, more importantly, the transferable part: what you now believe about your visitors that you did not believe last month. It is also the easiest step to skip, because nothing forces it.

​

Where AI helps, and where it does not

AI is genuinely useful across this chain. It can summarise research, draft an implementation, suggest segments worth investigating and propose alternative explanations for a result. It is also very good at making thin evidence sound confident. It will offer a reason a result happened, and the reason will be fluent whether or not it is true. So the useful test of an AI feature in experimentation is not how convincing its output reads. It is whether it shows you what evidence it used, what it assumed, what it could not verify, and what experiment would settle it. An assistant that hands you a conclusion has moved the judgement call somewhere you cannot inspect.

​

The question worth asking before launch

If this test returns a result tomorrow, will we know what it means, who it applies to, and what we will do about it? If the answer is no, more traffic will not fix it. Somewhere upstream, a link in the chain is weak, and the result will inherit it. Our homepage test is the version of this we learned the hard way. It ran for a month, on our own site, built by the people who make the tool, and the weak link was not the statistics. It was the cookie banner deciding who the test could reach.