The projection
estimateTimeToDecision in worker/src/lib/time-to-decision.ts.
requiredSampleSize is the formula on the
minimum detectable effect
page. The inputs the results route supplies are all read off the
control arm of the primary goal: baselineRate is the
control’s conversion rate, currentNPerVariant is the control’s
visitor count, and mde is the experiment’s stored mde_bps divided by
10000, falling back to 0.10 when the field was never set.
Accrual comes from recentDailyAccrual, which is the mean of the last
up to 7 daily control visitor counts. When there is no daily series to
average, it falls back to total control visitors divided by days
running, floored at one day.
The four statuses
A separate
rough flag is set whenever the control is under 100
visitors (ROUGH_BELOW). It does not suppress the estimate, it appends
a qualifier to it.
The no_conversions line is additionally suppressed by the dashboard
whenever the primary goal has any conversions at all, since the status
keys off the control arm alone and a treatment arm can be converting
while the control is not.
The copy thresholds
Fromdashboard/src/lib/time-to-decision-copy.ts:
- Over 8 weeks: “More than 8 weeks to a confident result at this pace, consider a bolder change or more traffic.”
- Under 14 days: the day count, for example “about 6 days to a confident result”.
- Otherwise:
ceil(days / 7)weeks. - With
roughset, “(rough, still early)” is appended.
Worked example
A 4 % control conversion rate on the primary goal, a 10 % MDE, 12,400 control visitors so far, and daily control visitor counts over the last seven days averaging 1,850.ceil(15 / 7). The control is well
past 100 visitors, so no rough qualifier.
Four things the estimator does not know
It does not round to whole weeks. Behaviour varies across the week, and a run that stops mid-week gives one variation an extra Saturday. The fix is to round your own end date up to a whole number of weeks so every day of the cycle appears the same number of times in both arms. Our projection returns 15 days; the schedule you should commit to is 21. It does not hold a business-cycle floor. If traffic is heavy enough that the sample arrives in two days, the projection will say two days. Two days is one narrow slice of your audience and one mood of the market. Hold a floor of at least one full cycle, and preferably two, so you can see the effect survive a second week. If your purchase cycle is longer than a week, stretch the floor to match it. It does not model novelty or primacy. Returning visitors react to a change because it is new, some engaging more than they will once it is ordinary and some thrown by the unfamiliar layout. Both fade. Neither is visible to a projection that only counts visitors. It does not know your confidence level or correction.required is
always computed at 95 % confidence and 80 % power. An experiment set to
99 %, or running several variations under a Bonferroni or Sidak
correction, needs more than the projection says. Treat the number as a
floor.
The interaction with early stopping
alreadySignificant short-circuits the projection to reached. The
moment the leading variation clears 95 % on the gated p-value, the
runway line stops showing a date and starts saying you have enough. On
day 4 of a planned three-week run, that is exactly the reading a
fixed-horizon test cannot support. The status describes the state of the
evidence right now. It is not permission to stop, and we
measured what stopping there costs.
If you want a runway that stays honest under continuous reading, run the
test on the sequential engine, where reading early is the design rather
than the leak.
The health banner above it
A cron computes the same projection independently and raises a warning when it exceeds 8 weeks (UNDERPOWERED_MAX_WEEKS in
worker/src/lib/collection-health.ts). The banner prints the MDE as a
percentage and links back to the field, because raising the MDE, adding
traffic or widening targeting are the only three levers that move an
8-week projection. It uses a 7-day site baseline for accrual and does not
evaluate an experiment until it has been running 48 hours.
That banner is about feasibility, not validity. A test can be perfectly
well formed and still take four months.
Estimating before you launch
The projection above needs a live control rate, so it cannot help you before the first visitor. The public A/B test calculator runs the samerequiredSampleSize function from your assumed baseline and MDE,
which is the number to take into the planning conversation.
Sources
The whole-week rule and the business cycle floor are established practice, not house preference.- Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press. Covers duration, weekly seasonality, and why running to a pre-committed sample size beats stopping when a number looks good.
Related
Minimum detectable effect
The input that sets
required, and the grid of what it costs.The peeking problem
Measured false-positive rates for four reading schedules.
Goal firing health
The other reason a test never finishes: the goal is not firing.
Time to decision
The product-side description of the line on your results page.
Already testing somewhere else
These pages assume you are deciding how to run a test. If you are already running them in another tool, the quickest way to judge this one is to rebuild a single live experiment here instead of starting from an empty account.Send us one live experiment
If you already run experiments in Convert, VWO, Optimizely, AB Tasty or PostHog, send us one that is live today and we rebuild it in ABTestly with you, free. Within two business days you get back three lists: what carries across as it is, what has to be re authored, and what we cannot reproduce. We never ask for a login to your current tool.