Availability. The two system guardrails
attach automatically on every plan, including Free. Attaching your
own goals as guardrails is listed from the Starter plan upward on
the pricing page.On every plan, an experiment has exactly one primary goal. ABTestly
never lets two goals decide one verdict, so there is no combination
rule, because there is never anything to combine.
Primary
The primary goal decides the verdict. When the results page says “Variant B wins”, it means “wins on the primary goal”. Exactly one goal is primary; pick it in the experiment editor under What to measure. Promoting a goal to primary clears its guardrail flag automatically, so a goal is never both at once. Pick the primary goal that maps to the thing you actually want to change. For a checkout test, primary is usually a purchase or a checkout completion event. For a landing page test, it is usually the CTA click that leads into the funnel, not the eventual purchase, which happens far downstream and adds noise. Rule of thumb: the primary should fire on a large enough share of visitors that the confidence interval closes in a reasonable time. Very downstream events with tiny fire rates take forever to call.Guardrails
Guardrails are goals you do not want to win, you want them to not break. Once the primary goal has a winner, that same variation is checked against every guardrail. Common guardrails:- Increase Engagement and Engaged session, attached automatically. See System defaults.
- Revenue per visitor, for tests near the funnel, to catch a “clicks more but buys less” pattern.
- A downstream conversion, for early funnel tests, to catch the same pattern one step later.
How a guardrail is judged
A guardrail resolves to one of four states, evaluated only after the primary goal has a winner:
Guardrails use the same statistics engine as the primary. Frequentist,
sequential, or Bayesian, whichever engine you picked for the
experiment applies to all of its goals. A breach requires both that the
engine’s evidence threshold is met and that the effect direction is
downward.
What to do with a guardrail breaching win
Three options:- Ship anyway if you understand the tradeoff. A test that lifts revenue by 5 % but drops engagement by 1 % might still be net positive, depending on what engagement means for your business.
- Iterate. Take the winning variation, fix the guardrail problem, re-test.
- Do not ship. The lift did not survive contact with a guardrail that matters.
Do I always need guardrails?
For a first pass, no. The two system guardrails auto attach and cover the two most common failure modes. Add task specific guardrails when you know a specific downstream metric is what actually matters: a purchase for a hero test, a return rate for a shipping information test, a support ticket rate for a documentation change.Detaching the system guardrails
You can detach them per experiment (they do not disappear from the library). Detach when you have a good reason:- A test on a page that is intentionally a dead end. A confirmation page is designed to end the visit.
- A single page app where the “second page” concept does not fit. The ten second condition still works, but you may prefer a goal that reflects your app’s own definition of engagement.