Getting started
Welcome to ABTestly
What ABTestly is, what makes it different, who it is for.
Quickstart
Signup to first live test in about fifteen minutes.
How ABTestly works
The whole system on one page. Dashboard, snippet, ingest, ledger.
Install and verify
Install the snippet
One tag in
<head>, first before other scripts, no async or defer.Verify install
Three quick DevTools checks to confirm the snippet is firing.
SPA support
Route detection, apply/cleanup, waitFor.
Consent
always_on vs deferred, the consentKey, fail-closed behavior.
Visitor opt-out
The URL parameter that removes a visitor from every test.
Build a test
Create an experiment
The editor’s seven sections, defaults, and the publish gate.
Variations
The Monaco editor, weights, JS/CSS, lints, editor freeze on running tests.
Dev libraries
Curated Splide, GLightbox, and friends from a picker.
Redirect tests
Split-URL tests where a variant navigates to a different page.
Clone
Fresh draft, new salt, new key, goals become shared.
Schedule
Start date, end date, max run days, auto-stop lands in Paused.
Autosave and drafts
Recovery, save-in-flight guard, publish outbox.
Target the right visitors
Overview
Locations, audiences, activation triggers, and how they combine.
URL rules
Operators, per-rule case-insensitive toggle, negations.
Audiences
Device, browser, OS, geo, history, source, custom conditions.
Geo
Country, region, city. Fails open on lookup miss.
Saved lists
Reusable Locations and Audiences.
Activation triggers
On load, on click, when visible, manual.
Mutual exclusion groups
Overview
Why groups exist, when to use them, when not to.
How allocation works
The 10,000-slot math, the salt, the generation counter.
Long-term holdouts
Measure the cumulative effect of every test you have shipped.
Managing membership
Add, retire, reorder. What generation bumps do to running experiments.
Measure
Goals overview
What a goal is and how ABTestly counts it.
Goal types
Click, page visit, form submit, scroll, engaged session, revenue, custom.
System defaults
Increase Engagement and Decrease Bounce Rate, auto-attached.
Primary vs guardrail
One decides the winner. The rest catch collateral damage.
Goal firing health
Silent, one-sided, insufficient. What each warning means.
Read results
Reading a result
A tour of the results page, verdict copy, when to call a test.
The three engines
Frequentist, sequential, Bayesian. When to pick which.
Revenue metrics
CR, RPV, AOV, OPV. Each with a proper CI.
Segments
Slice the whole report by device, source, geo, and more.
Baseline
Change the reference variant on the results page.
Time to decision
An honest estimate of how much longer this test may need.
Business impact
Project a winning lift into currency at current traffic.
Speed guardrail
Per-device LCP so a winner does not tank your page speed.
Exact ledger
Every event, kept. Where your final numbers come from.
Sample Ratio Mismatch
Chi-square on every experiment. Warns before a bad split misleads.
Exports
CSV of the results table, or async raw event export.
Preview and QA
Preview modal
URL, QR, incognito. Preview drafts. Never records data.
Forced variation
Pin yourself into a variant for a full session.
QA overlay
On-page debug panel showing everything the snippet does.
Dev debugger
window.__abtestly.debug() prints a snapshot to the console.Site settings
General
Site name, domain, timezone, currency, default stats engine.
Snippet panel
The install snippet, weight, last-published time.
Analytics adapters
GA4, Segment, RudderStack, Amplitude, Mixpanel.
SPA support panel
Route detection, apply/cleanup/waitFor templates.
Global JS
One script on every page, before every variant.
Blocked IPs
Drop events from your office, or from a bot range.
Danger zone
Delete a site. Removes everything. No undo.
Team and access
Members and roles
Admin, Dev, Viewer. Per-site scope. Twenty permission keys.
Permissions catalog
Every key, what it unlocks, which role it requires.
Activity log
Every meaningful change, grouped by day, filter-shareable.
Assets
Upload
Images and PDFs, dedup by content hash.
Serving
usercontent.abtestly.com, immutable URL, one-year cache.
Recycle bin
Soft delete vs permanent delete. What actually frees storage.
Integrations
GA4 dataLayer
experience_impression push, exp_variant_string.
GA4 export
Slice conversions by variant using your existing GA4 pipeline.
GA4 BigQuery
Join exposures to conversions in BigQuery.
Product analytics
Segment, RudderStack, Amplitude, Mixpanel.
Google Tag Manager
How the snippet coexists with GTM.
Billing
Plans
Tiers, MTU caps, seat limits, config-size budget, asset budget.
Monthly Tracked Users
What counts, what does not, per-org counting, monthly reset.
Refunds and cancellation
Thirty-day money-back on first paid month. Cancel any time.
Trust and compliance
Trust and compliance
Security posture, subprocessors, retention, GDPR rights,
uptime, incident handling.
Developer reference
window.abtestly API
Every method and event on the runtime object.
REST API
Endpoints under api.abtestly.com. Auth, rate limits, request shape.
Beacon protocol
Wire format for /e, /e/validate, /err, /m, /g, /preview-config.
Bucketing
Hash, two-step allocation, salt, reproducible in Node.
Runtime bindings
ABTESTLY_EXP_KEY and friends inside variant code.
SPA lifecycle
Route detection, ApplyContext, routeRevision, cleanup order.
Errors and telemetry
What lands in /err, first-occurrence dedup, where to find it.
Rate limits
100 req/min per user on /api. What is not rate-limited.
Troubleshooting
Snippet not firing
On the page, nothing happens. A five-step ladder.
Numbers look off
Conversions greater than visitors, split imbalance, flip-flopping.
Consent stuck
Deferred mode never fires. Common CMP bridging patterns.
Blocked IP
Your own visits show up as blocked. Fix and workarounds.