What it means in plain terms
Every variant’s JavaScript and CSS runs inside its own try/catch. ABTestly wraps it deliberately, so a mistake in one variation’s code cannot break the page, cannot break another experiment running on the same page, and cannot break ABTestly’s own tracking. If a variation throws, the visitor keeps browsing, the page keeps working, and nothing else on it is touched. What ABTestly does with that throw is report it. The affected page loads count on your results is the number of page loads on which that variation’s code threw at least once. It is reported once per experiment per page load: a page that navigates many times inside a single page app still counts once for that experiment, however many times the underlying error actually fired. That is the whole definition. It is a count, not a rate. There is no percentage attached to it and no share of traffic it corresponds to, because there is no denominator behind it. It does not tell you what fraction of your visitors this happened to, only how many page loads it happened on.Why the exposure still counts
When a variation throws, ABTestly still records the exposure, meaning that visitor is still counted as having entered the test under that variation. This is deliberate, and it’s worth explaining why, because the alternative sounds more appealing at first glance. Suppose ABTestly instead discarded the exposure whenever a variation’s code threw. The number on your results page would look tidier. But think about what that discard would do to the two arms of your test. The control arm has no variation JavaScript at all, so it can never throw and never has anything discarded. Only the treatment arm can lose visitors this way. Quietly dropping them would shrink one arm relative to the other for a reason that has nothing to do with which variant is better, and that is exactly the shape of a sample ratio mismatch: the actual split of visitors no longer matches what you configured. It would happen on every test where the variation code has a bug, and it would happen silently. So ABTestly keeps the visitor in the count instead. The tidier number would have been the dishonest one, quietly asymmetric in a way that has nothing to do with the question your test is asking. What you get in its place is a true, if unflattering, count of how many page loads ran into a problem, sitting next to a result whose sample you can still trust.This is also why the count sits beside your results rather than inside
them. It’s a fact about how the data was collected, not an adjustment to it.
What the count is not
- Not a percentage. There’s no total it’s divided by.
- Not a share of traffic. It says nothing about what fraction of your visitors this happened to.
- Not a count of visitors. The same visitor loading the page more than once, on separate occasions, can appear in the count more than once.
- Not a correction to the statistics. The result isn’t reweighted or adjusted to exclude the affected page loads. The count is informational: it tells you the result includes people who didn’t receive the variation as intended, so you can judge it yourself.
The visitor may still have seen part of the treatment
A throw doesn’t mean the visitor saw nothing. A variation can inject CSS, mutate part of the page, and only then reach the line that throws. Whatever ran before the error already applied and is still sitting on the page. So an affected page load isn’t the same as a control page load with a label on it, treat it as a page load where the intended change may be incomplete, not one where it’s absent. Don’t assume from this count alone that the visitor never saw the treatment.How the error text is handled
Open the detail behind an affected page load and the text has already been reduced before it ever left the visitor’s browser. URLs keep their origin and path but lose their query string and fragment. Email addresses, long tokens, and long runs of digits are redacted. This is a privacy measure: a stack trace pointing at a URL that happened to carry a session token or an address in its query string doesn’t carry that value into your dashboard.What this does not do
- It does not tell you what share of your traffic was affected. There’s no denominator behind it, so there’s no rate to compare between variants.
- It does not mean the visitor saw none of the treatment. Code can run, and CSS can apply, before the line that throws.
- It does not correct or reweight your results. The verdict and the confidence interval are computed from the data as collected.
- It does not stop counting the exposure. The visitor stays in the sample precisely so the two arms of your test stay comparable.
Related
- Reading a result for how the rest of the results page is laid out.
- Editing a running test for the other kind of notice that can sit on a result you’re reading.
- Sample ratio mismatch for what happens when the visitor split breaks for other reasons.