Long-term holdouts

The single most valuable use of an exclusion group is not to prevent tests from colliding. It is to hold out a slice of traffic that sees the original forever. Why? Because A/B tests measure short-term lift on one primary goal. They do not measure:

• Long-tail retention effects. A change that lifts signup 5 % might tank six-month retention 10 %. That is a net loss no primary goal test can catch.
• Cumulative effects across many tests. You shipped ten small wins this year. Did they combine into a big lift, or did they cancel each other out? A holdout tells you.
• Novelty vs sustained lift. A test that wins in week one may regress in week eight when the change stops feeling new. Only a long-running holdout catches that.

​

The recipe

Create an exclusion group. Give it a memorable name like “Homepage holdout” or “Q3 2026 holdout”. Add every relevant experiment to that group as it launches. The hero test, the CTA test, the pricing card test, whichever ones run on the pages the holdout covers. Leave a percentage unallocated. 20 % is a common choice. Enough statistical power to detect a meaningful cumulative effect without sacrificing too much test traffic. For a group with two experiments and a 20 % holdout: The 20 % that lands in unallocated space sees the original hero and the original CTA. They are your baseline for measuring the combined effect of everything else in the group.

​

Reading holdout results

Holdout analysis is not built into the dashboard’s per-experiment results page, it needs the ledger. Two ways to answer “how are holdout visitors doing compared to everyone else?”: Raw event export. Pull the full event stream via Export → Raw events, filter on the group’s generation stamp (gn field), and compare aggregates between “member = null” (unallocated) and “member = anything”. Custom BigQuery query if you use the GA4 BigQuery integration. Join exposures with your own conversion events and split by experiment_id presence, no exposure to any group member means the visitor is in the holdout. We are working on a first-class Holdout Analysis view in the dashboard. Until it ships, the ledger export is the honest answer.

​

What a holdout is really telling you

Numbers over time look like:

• Week 1, no signal. Sample too small, effects too subtle.
• Week 4, a faint pattern. Holdout visitors’ conversion rate drifts slightly against non-holdout. Could be noise.
• Week 12, a real signal. Non-holdout is 3 % ahead of holdout in cumulative revenue per visitor. That is the sum of every test you shipped this quarter.

A positive cumulative effect validates your program. A null effect suggests your wins are cancelling each other out, you are running tests but net-lifting nothing. A negative effect is the scariest and most valuable finding you can make; it means the things you thought were wins were not. Most teams never look, so they never find out. Setting up a holdout once and letting it run for a quarter is the cheapest way to avoid that outcome.

​

Rotating holdouts

Some teams rotate the holdout every quarter, retire the old group, create a new one with a fresh salt and a fresh generation, start over. This resets the comparison so you are always measuring “the last quarter’s changes” not “everything since the beginning of time”. Other teams keep one holdout running for years. Both work; the right cadence depends on how much your product and site change.

​

Cost

The cost of a holdout is the traffic you spend on it. A 20 % holdout on your homepage means 20 % of homepage visitors are seeing the original instead of your latest optimizations. That is real. The value is knowing whether those optimizations actually compounded, which is the difference between “we ran 40 tests this year” and “we ran 40 tests this year and lifted the top-line 8 %”. For most teams the value is worth the cost. But it is a real tradeoff and worth talking about with your team before setting one up.

​

When not to hold out

• New site, tiny traffic. If you have 10,000 visitors a month, a 20 % holdout is 2,000 visitors, probably too small to measure any long-tail effect anyway. Wait until you have more traffic.
• Single high-stakes flow. A checkout page with a hundred sales a month is not a great holdout candidate; the sample size cannot detect subtle effects and you cannot afford to spend 20 % on no-test. Reserve holdouts for higher-traffic surfaces.
• Very short campaigns. A one-week Black Friday hero test does not need a holdout; the test itself is short enough to just look at the result.

​

Related