Skip to main content
If you landed here from a Results include a configuration change notice, skip to the notice. Tests get edited while they run. A goal gets promoted to primary on day four. The audience gets widened because traffic was slower than expected. Someone pushes a fix to the variation code. Each change is reasonable on its own. The problem is what the change does to the number underneath it. Visitors who arrived before the edit were measured one way and visitors who arrived after were measured another, and the results page adds them together. Nothing looks broken afterwards. The chart still renders and confidence still climbs. ABTestly handles this in two ways: it asks you to decide at the moment of the edit, and it records what you decided so the results page cannot forget.

​
Which settings count

Six settings are treated as material, meaning a change to any of them can alter what the experiment is measuring: Everything else stays freely editable at any time, including the experiment name, its description, tags, the minimum detectable effect, and variant names. None of those change what is being measured.

​
The choice at the moment of the edit

Save a material change on an experiment that has collected any data since its last reset and ABTestly stops with:
Reset experiment data? You changed the following while this test was collecting data.
The list underneath names each change. You get three options. Reset and continue discards the data collected so far and starts a clean baseline from the new setup. The test keeps running. Use this when the change means the earlier data is no longer answering your question. Keep data and continue applies the change and keeps every event already collected. The change is recorded and surfaces on your results, described below. Use this when the change is genuinely cosmetic, or when losing the data costs more than the mixing does and you are willing to say so on the readout. Cancel abandons the edit. The same prompt appears when you resume a paused experiment that was edited while it was paused, so pausing is not a way around the question.

​
The notice on your results

Choose Keep data and the results page carries a standing notice:
Results include a configuration change. These settings changed after data collection started; the numbers below mix traffic from before and after the change.
Underneath it is every recorded change, with a timestamp, in the wording ABTestly generates from the before and after state:
  • 'Add to cart' set as primary
  • Goal 'Newsletter signup' removed
  • Traffic 50/50 → 70/30
  • Traffic exposure 100% → 50%
  • Variation 1 code edited
  • Audience / targeting changed
  • Host scope changed
The wording is produced on the server by comparing a snapshot of the experiment before the change to one after it. It is not typed by whoever made the edit, and it cannot be edited afterwards. The same warning appears against the experiment on your experiments list, so you see it before you open the result.
This notice does not clear itself. It is attached to the data rather than to the current numbers, so it comes back on every visit until a data reset. That is deliberate. A notice that disappears once you have seen it is a notice that stops doing its job the moment it matters, which is when a colleague reads the result three weeks later.
To clear it, reset the experiment data. That advances the baseline, so every change recorded before that point falls outside the window. Nothing is deleted. The change events stay in the table, they just stop being in scope. A reset is not a way of dismissing the warning. It also discards the counts collected so far and rotates the experiment’s bucketing salt, so visitors are assigned to variants again from scratch and someone who saw one variation may now see another. You cannot keep the numbers and lose the notice about them.

​
The lock at 1,000 visitors

Once a running experiment has 1,000 visitors since its last reset, the six material settings become read only in the editor:
Locked after 1,000 visitors. Pause the experiment to edit.
Pause the experiment and they open up again, still subject to the reset or keep question above. The threshold is 1,000 visitors on every plan. The friction is the point. Below 1,000 visitors a change is cheap because there is little data to spoil. Above it, changing the setup is a decision worth making deliberately rather than mid scroll.

​
What this does not do

  • It does not block you. Keep data and continue is always available. The goal is an honest readout, not a locked one.
  • It does not see changes made outside ABTestly. A pricing change on your site, a new banner shipped by another team, or a tag manager edit will move your numbers and ABTestly has no way to know. Those still need a note in your own log.
  • It does not correct the statistics. The numbers are not reweighted or adjusted to account for the change. The notice tells you the result mixes two setups so you can judge it. It cannot unmix them.
  • It does not apply to a draft. Nothing is recorded until the experiment has collected data.
Last modified on September 8, 2026