Skip to main content
Two quiet failure modes account for most tests that “did not work” even though the code was right: the goal never fires, or it only fires on one variation. ABTestly checks the primary goal for both and shows a banner on the results page when either applies.
These checks run on the primary goal only. Guardrail and secondary goals are not checked this way; a guardrail with too little evidence shows as insufficient in the guardrail list instead.

​
When the check runs

The banner needs enough traffic before it will judge anything: Below 200 total visitors, ABTestly says nothing. It is too early to tell a broken selector from a slow start.

​
Your primary goal is not firing

Shown when: total visitors are at or above 200 and the primary goal has zero conversions on every variation. Almost always one of:
  • Selector typo. The click goal targets .hero_cta but the button is .hero-cta. The preview modal’s QA overlay shows which selectors matched what on the current page.
  • The goal fires on a page the experiment does not cover. A goal for /checkout on an experiment whose Location is only /pricing cannot fire. On a normal site the snippet never runs the experiment on /checkout at all; on a site with SPA support the route gate withdraws the fire.
  • The goal really has not fired yet. For a rare conversion on a low traffic page, silence past 200 visitors is still possible.
The one goal this does not apply to is the system Engaged session goal, which is credited across pages regardless of Location. See System defaults.
Widening the Location is one fix, and it is not always the right one. Adding /checkout to the Location means your variation code will run on checkout pages. If the variation was never meant to touch checkout, move the goal or split the test instead.

​
Your primary goal is only converting on one variation

Shown when: at least two variations have 100 or more visitors each, and exactly one variation has any conversions at all. Two explanations, and the banner does not pick between them:
  • The goal cannot fire on the other variation. The selector matches an element that variation deletes, renames or moves. This is the common cause. Adjust the goal to match a stable element, or add a hook that every variation keeps.
  • A real effect. If the element genuinely only exists in one variation, one sided firing is exactly what you should see, and the banner says so.

​
What the banner does and does not do

The banner is advisory. It does not block the verdict, hide the results, or stop a winner being declared. If your engine’s evidence threshold is met, the results page will still call the test. That is deliberate. A permanent veto on one sided firing would also veto a legitimate zero versus positive result, which is a real outcome for a variation that adds a control the baseline does not have. Treat the banner as a mandatory verification step: confirm the goal can fire on every variation, then trust the engine. The verdict itself is gated on evidence, not on these checks. A variation needs at least 100 visitors and at least 5 conversions, and so does its control, before ABTestly quotes a confidence number for it at all. See reading a result.

​
When to act

  • Banner right after a fresh publish: wait. Cache propagation and traffic building up explain most of it.
  • Banner that persists past a day of normal traffic: investigate. Open the preview modal, verify the goal fires when you click the target, and adjust.
Last modified on August 28, 2026