Skip to content
PrimoCRMPrimoCRM Docs

A/B Testing

A/B testing is a Pro feature that compares two subject lines for a campaign on a sample of your audience, then sends the better performer to everyone else. It takes the guesswork out of what to put in the subject line.

How it works

Section titled “How it works”
  1. Create a campaign and enable A/B testing.
  2. Set subject A (the campaign’s subject) and subject B (the alternate to test).
  3. Choose the split: what percentage of the audience receives variant A and what percentage receives variant B. The remaining contacts are held back as the winner pool.
  4. Choose the winning metric: open rate or click rate.
  5. Set the wait period (in hours) before a winner is decided.
  6. Choose the winner mode: automatic (PrimoCRM sends the winner to the held-back pool for you) or manual (you review the results and pick the winner yourself).

PrimoCRM sends variant A and variant B to their slices of the audience, waits for the evaluation window, then either auto-sends the winning subject to the held-back pool or waits for you to choose, depending on the winner mode.

A worked example

Section titled “A worked example”
  • Audience: 10,000 contacts.
  • Split: 10 percent to A and 10 percent to B, so 1,000 contacts each. The other 8,000 are the winner pool.
  • Metric: open rate. Wait: 4 hours. Winner mode: automatic.

After 4 hours, PrimoCRM compares the open rates of the two subjects, then sends the winning subject to the remaining 8,000 contacts. You get the stronger subject in front of most of your list without deciding by hand.

Tips

Section titled “Tips”
  • Change only the subject line between A and B so you know the subject is what moved the numbers.
  • Keep each sample large enough to be meaningful. Very small slices produce noisy results.
  • Give the wait period enough time, since opens trickle in for hours after a send.
  • Use open rate as the metric for subject-line tests; it is the most direct measure of what a subject line affects.
  • Feed what you learn back into your next campaign and your templates.

Winner evaluation runs on the same background engine that paces sending, so it happens reliably without you keeping a tab open. See Reports to review results afterward.