A/B testing campaigns

An A/B test on a campaign sends two or more variants of the same message to slices of your audience, measures which one performs best on a metric you pick, and then optionally ships the winner to everyone else.

What it solves

Push notifications are a one-shot medium — you don't get to iterate after the user has seen it. A/B testing lets you de-risk a big send by validating copy, imagery, or CTAs on a small fraction of the audience first.

Typical questions A/B tests answer:

Does emoji in the title lift CTR?
Does showing a product image beat a brand logo?
Does urgency ("ends tonight") beat a friendly tone?

How variants are assigned

When a campaign with A/B testing enabled starts sending, ReachBell hashes each subscriber's ID and uses the hash to bucket them into a variant deterministically. The same subscriber always sees the same variant across re-evaluations and re-sends — they will not be flip-flopped between variants on subsequent touches.

Deterministic assignment matters. If a user clicks variant A on Tuesday and re-renders variant B on Wednesday, your metrics are noise. ReachBell guarantees they see the same variant.

Schema

A campaign with A/B testing has an abTest block:

{
  "abTest": {
    "enabled": true,
    "variants": [
      {
        "variantId":  "a",
        "percentage": 50,
        "content": {
          "title": "Spring sale is live",
          "body":  "30% off everything until Sunday."
        }
      },
      {
        "variantId":  "b",
        "percentage": 50,
        "content": {
          "title": "Save 30% this week",
          "body":  "Spring sale ends Sunday."
        }
      }
    ],
    "winnerCriteria":       "ctr",
    "evaluateAfterHours":   24,
    "winnerId":             null
  }
}

Field	Meaning
`enabled`	Toggle. When `false`, the campaign sends with the top-level `content`.
`variants`	Array of two or more variants. `percentage` values must sum to 100.
`winnerCriteria`	One of `"ctr"`, `"clicks"`, `"dismissRate"`. The metric used to pick a winner.
`evaluateAfterHours`	Hours after the initial send before the system picks a winner.
`winnerId`	Populated by the system after evaluation runs. The `variantId` of the winning variant.

Two-variant tests

The simplest setup: 50/50 split between two variants. This is what the dashboard's A/B toggle defaults to.

Multi-variant tests

You can add up to four variants. The percentages must still sum to 100 — 25/25/25/25, 40/30/30, etc.

Three or more variants means each variant gets a smaller slice of audience, which means more sends per variant are needed for the result to be meaningful. Stick to two unless your audience is large.

Winner criteria

Criterion	Meaning
`ctr`	Click-through rate (clicks / delivered). Best default.
`clicks`	Total click count. Use when variant audiences are equal and you want raw winners.
`dismissRate`	Dismiss / delivered. Lower is better — use when you're trying to reduce notification fatigue.

Evaluation cron

A scheduled job runs every hour. For each active A/B campaign whose sentAt + evaluateAfterHours has passed, the cron:

Pulls per-variant counts from the campaign stats.
Picks the variant with the best score on winnerCriteria.
Sets abTest.winnerId on the campaign.
If the campaign has "auto-send winner to remaining audience" enabled, it dispatches the winning variant to subscribers who matched the segment but weren't in the test pool.

You can also force evaluation manually from the campaign detail page in the dashboard.

Sending the winner to the rest

In the campaign builder, the A/B step has a checkbox: Send winner to remaining audience after evaluation. When checked, the segment is split:

Test pool: a fraction (default 20%) of the segment receives the variants.
Hold-out: 80% receives nothing until the winner is declared, then receives the winner.

This pattern protects you from a bad variant going to your full list while still letting you reach everyone.

Dashboard UI

The campaign builder has an A/B test toggle in Step 3 (Content). When enabled:

The content editor switches to tabs — one tab per variant.
A traffic split slider lets you adjust percentages.
Below the editor, a "Winner criteria" selector exposes the three criteria above.

After sending, the campaign detail page shows a side-by-side stats table with sent / clicked / CTR per variant, plus a "Winner declared" badge once the cron runs.

Best practices

Test one variable at a time. If you change the title and the image and the CTA, you won't know which one moved the needle. Run sequential tests, each isolating one variable.

Watch the sample size. Don't trust a winner with fewer than 500 sends per variant. Below that you're reading random noise. The dashboard adds a "low confidence" badge under 500/variant, but it's on you to honor it.

Pick CTR over raw clicks. Raw clicks are confounded by audience size if your variants aren't perfectly equal. CTR normalizes for that.

Give it 24 hours minimum. Push notifications are time-of-day sensitive — a 2-hour evaluation window can be fooled by people checking their phones during a lunch break. 24 hours is a safe default.

Don't run A/B tests for the sake of it. Every test costs you a slower campaign and a fraction of your audience receiving the loser. Test things you genuinely don't know the answer to.

A worked example

You're launching a spring sale and you're unsure whether to lead with the discount or the urgency. You set up a two-variant test:

Variant A: "Spring sale — save 30%" / "Until Sunday, sitewide."
Variant B: "Ends Sunday: 30% off" / "Spring sale ends in 4 days."

You split 50/50 across a 20% test pool of your US-engaged-30d segment. Winner criteria: ctr. Evaluate after: 24 hours.

24 hours later the cron runs and finds:

{
  "variants": [
    { "variantId": "a", "sent": 1812, "clicked": 124, "ctr": 0.0684 },
    { "variantId": "b", "sent": 1798, "clicked": 178, "ctr": 0.0990 }
  ],
  "winnerId": "b"
}

Variant B (urgency-first) wins by 45% relative CTR. Because you ticked "send winner to remaining audience," the remaining 80% of the segment receives variant B over the next hour. You've now broadcast the better-performing message to your full list without ever shipping the loser to most of it.

What's next?

Read Campaigns API for the full programmatic interface.
Build a segment so your test pool is the right audience.
Attribute the winner's clicks with UTM tracking.

Error codes

Automation