Variants & A/B testing
How multi-variant sequence steps work, how the platform rotates sends across variants, the exact thresholds the AI agent uses to judge a winner, and when to disable a losing variant.
A variant is an alternative subject + body pair within a single step. When you add variants to a step, the platform rotates sends across them, tracks performance per variant, and (if you let it) retires the losers. This is how you A/B test cold email at scale without manually running two campaigns.
How a step with multiple variants works
Each step in a sequence can have one or more variants. When a step's turn comes for a given lead:
- The platform picks one variant from the active set for that step.
- The send is attributed to that specific variant — all replies, opens, and bounces associated with this lead on this step are counted against that variant.
- The lead moves on to the next step on its scheduled timing.
Variant selection is rotation-based — roughly even distribution across active variants for that step — so you end up with balanced data across variants fairly quickly.
The four metrics per variant
Variant performance is rolled up in the Analytics tab under the step-by-step breakdown. Each variant row shows these columns:
- Sent — how many times this variant has gone out.
- Replied — count of replies received on this variant.
- Positive Responses — replies the AI SDR classified as positive.
- Booked Calls — meetings booked as a result of this variant.
Derived from these: Positive Reply Rate (PRR) = Positive Responses ÷ Sent. PRR is what the AI agent judges variant performance on, not raw reply rate.
How the AI agent judges a variant
The AI agent uses volume and performance thresholds from the Options tab:
| Threshold | Default | What it means |
|---|---|---|
| Minimum Threshold | 100 emails sent per variant | Don't judge a variant until it's sent at least this many emails. |
| Base Threshold | 250 emails sent per variant | The volume at which the agent becomes confident in a verdict. |
| Winner PRR | 2 (= 2%) | A variant is treated as a winner if its PRR is at or above this automation threshold. |
| Healthy PRR | 1 (= 1%) | A variant is treated as healthy if its PRR is at least this automation threshold. Below this, it's a candidate for review. |
These are campaign-level controls on the Options tab, not universal cold-email benchmarks. Direct and AI Business Audit campaigns can be healthy below 1%, so compare against the method benchmarks before disabling variants. See Thresholds for the plain-English version of each field.
What the agent does with a loser
If a variant is sending below the Healthy PRR threshold and has passed the Minimum Threshold in volume, the AI agent adds a DISABLE_VARIANTS action to the campaign's Pending Actions on the AI Agent tab.
What happens next depends on the campaign's AI campaign management setup:
- Optimize email variants set to Run automatically: the agent can disable weak variants when the campaign qualifies. The variant moves to Actions Successfully Taken with a timestamp. Rotation continues across the remaining variants.
- Optimize email variants set to Ask first: the agent creates an Agent Approval before disabling anything.
- Optimize email variants set to Off: the agent will not disable variants for you. You handle it manually on the Sequences tab.
When the agent waits instead of acting
If a variant doesn't have enough volume yet to judge confidently (under the Minimum Threshold), the agent logs a WAIT_FOR_VOLUME action instead of flagging it as a loser. This prevents premature disables on variants that might improve with more data.
Manually disabling a variant
You don't have to wait for the agent. On the Sequences tab:
- Open the step.
- Find the variant you want to retire.
- Toggle it off (or delete it).
The sends already in flight still complete, but no new sends are attributed to that variant.
How many variants per step?
Practical guidance, not a hard rule:
- Single variant — fine for campaigns where you're confident in the copy and just want to run at scale. You still get AI agent monitoring.
- 2–3 variants per step — the sweet spot. Enough to produce a winner quickly, few enough to hit the Base Threshold on each in reasonable time.
- More than 3 — only if your send volume is high. Each additional variant slows the time-to-confidence proportionally.
A campaign sending 500 leads/week with 4 variants per step takes much longer to produce a statistically meaningful winner than the same volume with 2 variants.
AI-generated campaign drafts can create more variants than this practical recommendation. That's okay as a starting point. Before launching, review the Sequences tab and disable or remove any variants you don't want in the active rotation.
Common pitfalls
- Adding a new variant mid-campaign — the new variant has no history, so it shows low volume and wide confidence intervals. The agent applies WAIT_FOR_VOLUME to it; don't panic that it "looks worse".
- Too-tight Healthy PRR threshold on a cold vertical — if you set Healthy PRR to 3 and you're targeting a hard-to-reach audience where 0.5-1% is realistic, the agent will flag every variant. Tune the threshold for the method and audience.
- Disabling too aggressively — do not turn off variants before they have enough volume. Once a winner is clear, retiring weak variants is fine; just make sure each step still has at least one active variant so sending can continue.
Related
- Sequences tab — edit variants
- Analytics tab — per-variant metrics
- AI Agent tab — agent decisions about variants
- Options tab — per-campaign thresholds and AI campaign management
- Thresholds — plain-English guide to speed presets and custom thresholds