You are here:
A/B Test Overview
Use standard A/B Testing to determine which version of your Marketing Cloud Engagement email receives the highest click-through rate or highest unique open rate. After reviewing, send the best performing version to your remaining subscribers.
Test Distribution
Before the system sends your emails, it must generate the A/B split. When the A/B test starts, the system chooses the subscribers, assigns Condition A or Condition B to those subscribers, and then sends the emails. For large lists and at times when several send jobs are in process, splitting the A/B test can take longer.
Note: If you have a large list, we recommend accounting for the split process in your send.
To prepare, schedule deployment before you want it to send. You can define your test audience size by entering a percent or value in the Test Distribution section or adjusting the slider. The A/B Split slider displays the number of subscribers the system sends to for each test condition. These numbers dynamically change as you select recipients, suppression lists, and exclusion lists.
This number is approximately the number of subscribers you’re sending to. The exact number can be more or less depending on these factors:
- The number of unsubscribed subscribers in your lists
- The number of subscribers on a suppression list that are also on a list you selected in the Recipients area
- The number of subscribers on an exclusion list that are also on a list you selected in the Recipients area
- The number of subscribers that subscribe or you add between when you set up the A/B test and the time of send
These numbers don’t reconcile until the time of send. For example, if you select 500,000 recipients, and you select an exclusion list containing 5,000 of those 500,000 subscribers, that number may not reflect in the Test Distribution field immediately. If the total is an odd number, the system attributes the extra subscriber to Condition B. For example, if your recipients equal 300,001 and you split your A/B Test by 10%, Condition B receives 15,001 while Condition A receives 15,000. That has little to no effect on your test results. Tracking results can differ from the test you set up from this page. The number suppressed and excluded are processed at send time and aren’t reflected in tracking results.
Ties
If your A/B Test results in a tie, the system declares Condition A the winner. You can edit the winner criteria to see if one condition performed better than the other using a different winner criteria.
If you ran your test by the highest open rate and your test results in a tie, you can edit the winner criteria to use highest click-through rate.
If you ran your test by the highest open rate and Condition B had a higher click-through rate than Condition A had an open rate, you can edit the winner criteria to send Condition B. You’re able to make this change only if you scheduled the remainder to send later.
Test Status
A/B test statuses are:
| Status | Definition |
|---|---|
| Unscheduled | Your A/B test is created but not scheduled to begin. |
| Scheduled | Your A/B test is created and set to begin at a particular time in the future. |
| Initializing | The system is splitting recipients to send Condition A, Condition B, and remainder audiences. |
| Running | The system is sending to at least one or both test audiences, depending on criteria set-up, or waiting on the results of your A/B test. |
| Overriding Winner | A user has manually chosen a winner. |
| Winner Declared | The system declared a winner of your A/B test or overriding winner has completed but hasn’t sent to the remaining subscribers. |
| Sending to Remainder | The system is sending to the remaining subscribers. |
| Completed | The system finished sending your test to the conditions and to the remaining subscribers |
| Canceled | A user canceled the A/B Test. |
| Errored | An error occurred at some point in the testing process and the test hasn’t completed. |
Track and Declare Winner
After you perform an A/B test, you can access the tracking overview section of your test to view the results. The tracking pages always display the current results, but the system honors the winning condition that performed the best from the time you specified to declare a winner.
In rare cases, the tracking information can change after a winner has been declared in that the winner is no longer performing the best. For example, you set the A/B test to declare a winner an hour after the test begins. After an hour, Condition A is declared the winner, but after five hours, the winner is Condition B because more subscribers interacted with your message.
Save the Winning Subject Line
When running a subject line A/B test, you can have the winning subject line save back to the original email. You configure this in the Winner stage of the test creation process.
Optional: Select Yes to save the winning subject line to the email in the My Emails folder under the Content tab.
