Signals evidence contract

Contract version 2.0, in force from the PXL v2 release · Last updated 2026-09-04 This page is the reference for how Signals decides how much weight a recommendation carries. It pulls together the grading rules that appear across Signals overview, Read a scan, Connect Google Analytics, and Build a test from an idea into one place, so you can see the whole model at once. Signals grades on two things: what evidence was credited on an idea, which becomes its evidence tier, and whether the idea’s PXL score clears the gate set for that tier. Everything below follows from those two ideas. Credited is not the same as connected: a source can reach the scan, appear as Used on its receipt, and be credited on no idea at all.

​

The evidence an idea can rest on

Every recommendation rests on the page structure, plus whichever of three further sources actually speaks to that idea. Structure is always there. The other three are there only when you connect or supply them, and each is scored separately below. When this page says a second independent signal agreed with the structure, it means GA4 funnel data or customer voice. The heatmap is not a second signal on its own: it strengthens GA4 rather than standing in for it. Structure tells you what the page is. Behavior tells you what is happening on it. Signals never invents a statistic when the data to support it is not there. When the number exists, you see it. When it does not, the idea says so plainly.

​

Who decides what

The AI evaluates the evidence. ABTestly computes the score. The model never hands back a score, a tier, a badge, a place on the board, or the next step an idea names. It answers eight yes or no questions about one idea, each with a short rationale, and ABTestly derives everything else from those answers and from what the scan actually had. An answer the evidence cannot support is overridden, and the rationale written for it is discarded rather than stored. If the model answers Q8 yes with a paragraph about funnel figures on a scan where GA4 never contributed, the point is not credited and the paragraph is not kept. The saved idea records the system’s reason in its place, so a stored idea can never argue against itself.

​

The four evidence tiers

The evidence tier says what was credited on one idea. It is not a measure of anyone’s confidence in that idea, and it is not a property of the scan: it is set per hypothesis, from the evidence the scan could credit on it. Every hypothesis carries exactly one tier, derived from the credited answers above, not chosen by you. From strongest to weakest: Because the tier belongs to the idea and not to the scan, two ideas on the same GA4 connected scan can land in different tiers: one the funnel data speaks to, one it does not. A source that contributed to the scan makes its question scorable for every idea on that scan; a connected source that returned nothing makes it scorable for none. It does not award the answer. The heatmap counts only alongside GA4. It corroborates funnel data, it never replaces it, so a heatmap on a scan with no credited GA4 answer lifts nothing. Used on the receipt is not the same as credited, and the two come apart by design. The receipt records what reached the run; the tier records what the scan could credit on one idea. A source can appear as Used on a scan’s receipt and be credited on no idea at all: GA4 can contribute and the scan still find it has nothing to say about a given idea, leaving that idea Structural, and a heatmap can be read and shown as Used while it is credited nowhere because there was no GA4 beside it.

​

How an idea becomes Test ready

An evidence tier says what was credited on the idea, not whether it is ready to run. Test ready takes two things, and both are required. First, a second independent signal has to agree with what the page structure showed. Second, the idea’s PXL score has to clear the threshold set by its tier:

​

The score is read against that scan’s maximum

The PXL score counts the points the idea earned. The number it is read against is the maximum that scan could reach, which is not a fixed scale: it is built from the evidence that was available on the run, so it is lower when a source is absent. So a card reads its score out of that scan’s own maximum. The same 7 points read as 7 of 12 on a scan with qualifying traffic and all three sources contributing, and as 7 of 8 on a scan that had qualifying traffic and customer voice but no GA4. Both are honest, because the denominator says what was genuinely reachable on that run.

​

The board badges

The badge on the idea board reflects the outcome:

​

Ideas scored under the previous model

Ideas scored before this contract took effect carry no version, and Signals will not vouch for their readiness. They read Unverified, captioned “scored under the previous model, readiness unverified”:

• their readiness badge is suppressed, so a Test ready stored by the old model is never shown as Test ready
• they are left out of the Test ready count in the scan header
• their score is shown as it was stored, against the old 14 point scale, and it is not comparable with a score from this contract
• they are never recomputed. The previous model wrote its reasoning as prose, with no structured answers underneath it, so there is nothing for the current arithmetic to run over. Scan the page again to get ideas scored under this contract.

An idea you added by hand is not affected: it reads as your own idea, never as one scored under the previous model.

​

Which experiments feed back

Signals learns from experiments, but not from every test you ship. The feedback rule is narrow and specific. An experiment feeds back into future scans only when both of these are true: When both hold, that outcome feeds back into future scans of the same site as ground truth, so the next scan already knows what worked on your pages and what did not. This is a data feedback loop that carries past results forward as priors in the next scan, not model training. See Build a test from an idea for the handoff.

​

Fields a scan reads

A scan assembles evidence from several categories before it grades anything: The thirty four structural signals the DOM read captures are: page type and framework flags (React, Vue, Angular, single page app, hydration); h1 count and text, heading hierarchy issues, page title, meta description, and viewport meta; form field count, autocomplete presence, and mobile keyboard hints; CTA button count, primary CTA text, add to cart candidates, hidden add to cart count, and whether add to cart is gated by a variant; product schema, aggregate rating, visible review count, rating value, visible star rating, review count source, and trust badge count; image count and image alt coverage; hero copy, value propositions, navigation labels, and footer copy; HTTPS, bot wall detection and vendor, and a tech stack summary (platform, checkout providers, review widget, experiment tools, and script count). Each stored hypothesis is itself a structured record of twenty eight fields, and the split between them is the split described above. The model writes eighteen: title, pillar, observation, proposedChange, mechanism, mechanismTags, expectedLift, effort (Low, Medium, or High), researchCitation, dataSource, statisticalNote, qualitativeEvidence, brandRespects, opportunitySize (a point estimate with a low and high range and its assumptions), trackingSpec, recommendedRunTimeWeeks, interactionSpec, and pxlAnswers, its answers to the eight questions. ABTestly computes the other ten from those answers and from what the scan had: id (the rank), pxlScore, maxPossible, pxlBreakdown, confidenceTier, gateStatus, readinessReason, promotionPath, pxlVersion, and pxlDisplaySuppressed. The pxlAnswers that is stored is the normalised one, so an overridden answer carries the system’s reason in place of the model’s discarded rationale. Every generated idea is put through the adversarial pass, and one that comes back with an argument against it carries a falsificationCheck (failure mode, confounder, alternative hypothesis, discriminating test, and brand risk). The pass never blocks a scan: if it cannot complete, the ideas still reach your board without that block rather than the scan failing. An idea you added by hand never has one.

​

Model and data dates

• Model provider. Signals runs on the Anthropic commercial API, whose terms do not use your data to train models.
• Model version. Reasoning and hypothesis generation run on claude-sonnet-5. The screenshot reading pass runs on the vision model claude-haiku-4-5-20251001. Both are set in the worker configuration and can be overridden per environment.
• GA4 data window. A scan reads the last 90 days of GA4 behavioral data, from 90 days ago through today, for every report in the pull.

​

Related pages