Pardot Knowledge Base

Using A/B Testing

Last Updated: Aug 30, 2016 | Print this Article
 

Pardot's A/B Testing for emails allows you to change elements in your email, segment your recipient list into a test audience and winning audience, and then use the email's engagement data to automatically determine and distribute the winning email.

Email A/B testing is included in Pro and Ultimate editions. A/B testing is not available for Standard editions.

Usage Notes

  • If an A/B test includes multiple prospects with the same email address, only the prospect with the most recent Last Activity field value will receive an email. They will receive version A, version B, or the winner, but not more than one email.
  • You cannot pause an A/B test to edit the email before the winner is sent. However, you can pause the test, copy the email, and edit the new copy.
  • If you enable A/B Testing on an email, then change your mind, you can turn it off before sending the email. See How do I turn off A/B Testing for complete instructions.
  • Prospects added to recipient lists after testing has begun will recieve the winning version of the email.
  • If you schedule an email, any changes you make to the prospect list before the scheduled send time will be reflected on the email send.

Best Practices

  • Change One Element: Try only changing one element between version A and version B (for example: subject line, an image in the email body, or greeting). If too many elements are different you won't know what it was about the winning version that actually 'made' it win.
  • Confirm Send Date/Time: Bear in mind the date and time your winning email is scheduled to be sent. If you choose to start your test at 11:00 AM and you set the test period to be 9 hours then your winning email will go out at 8:00 PM. We've highlighted this time to ensure you are ok with it before you click 'send' or 'schedule'.
  • Balance Test Audience Size and Test Period Length: Test duration is really up to you as the sender. It's impossible to give precise instructions as test results will vary by industry, client base and topic. It's best to base your initial AB tests off of what has worked well in the past or any data you may have for your particular company. In general, test duration should be long enough that there is a reasonable expectation that an email recipient has had time to interact (open/click through) the email. In the case of a 24-hour sale, you might want to send the email 2 days out, with a 1-day test duration. For a case study, or industry news type of email, you might want to test opens over the course of a week. Test duration will not affect the validity of your A/B test, as long as all emails are sent at the same time, and data collection ends at the specified time. To ensure that the performance of the winning email stays the same between the test period and the full send, it is important to make sure that the conditions of the sending time are similar. For example, if you send the test on a Monday, you will also want your remaining emails to send on a weekday. You don't want to test until Friday and then send all your emails over the weekend.
  • Test Every Email: More testing can only lead to more knowledge. Knowledge can only lead to better content. We believe in you!

Instructions

To start an Email A/B Test:
  1. Navigate to Marketing > Emails > Drafts.
  2. Click +Send New List Email, or select an existing email draft to launch Email Flow.
  3. In the Building step, click Basic Info, then mark Enable A/B Testing to turn on A/B Testing for that email. This will copy your email, and creates an A version and B version of the same email.
  4. Next, click the A/B tab at the top of the editor to toggle bewtween versions. As a best practice, we recommend you change only one element in the email. Too many differing elements makes it difficult to pinpoint what made the winning version the winner.
  5. Once you're ready to send, click the Sending tab.
  6. Add your recipient and suppression lists. You must have at least 4 prospects on your recipient list to send the A/B email.
  7. Pardot will automatically designate 25% of the recipient list as the test audience. This is randomized. To adjust the percentage, either move the slider bar or enter a new percentage in the "What percent of your audience do you want to use for testing? box.
  8. View both versions of your email (A and B) in the right sidebar. Mouseover the desired version to view the email at full size.
  9. Choose to schedule your email or send immediately.
    • To Schedule the Email:
      1. Click the Schedule button. Select a future date and time and click the Schedule button again to confirm. The email is now scheduled to send to your test audience on the chosen date, and will send to your winning audience after the test period has ended.
        Schedule Email Send 2 AB Testing
    • To Send the Email:
      1. Click the Send button. The email will send immediately to your test audience, and will send to your winning audience after the test period has ended.
This video walks through enabling A/B Testing on a list email draft. Click to expand the video.
 

Note: If you enable A/B Testing on an email, then change your mind, you can turn it off before sending the email. See How do I turn off A/B Testing for complete instructions.

Determining and Sending the Winner

Determining the Winner

Your test audience recipients, determined by your given percentage, will receive one of the two versions of your email. Half the test audience will receive Version A, and half the test audience will receive Version B. Pardot will determine the winner based on your chosen criteria. In the below example, the criteria is link clicks. So the email version with the most email link clicks in 1 day will be the winning email. Please note, link clicks or email opens that take place after the test has finished running will not be counted towards the winning email totals.

Note: In the unlikely event that emails A and B receive the exact same email click percentage or email open percentage during the test, Pardot will send email A.

AB Test Section

Sending the Winning Email

The winning email is determined by the test audience's interaction with your email during the chosen test run time. If you chose the "Send Now" button when executing your test, the winning version will send immediately at the conclusion of the test.

Sending AB Email

Scenario: On Monday at 2:00 pm, you create your email and set up an A/B test. You choose to base the test on link clicks and provide a test period of 2 days. You click the "Send Now" button. Your Test Audience will immediately receive either Version A or Version B. The link clicks will be tallied and the winning email will be sent to the Winning Audience at 2:00 pm on Wednesday.

Sending the Winning Email if Scheduled

The winning email is determined by the test audience's interaction with your email during the chosen test run time. If you chose the "Schedule" button when executing your test, the test version will send once the scheduled date and time occurs. The winning version will send immediately at the conclusion of the test.

scheduling ab tests

Scenario: On Monday at 2:00 pm, you create your email and set up an A/B test. You choose to base the test on link clicks and provide a test period of 2 days. You click the "Schedule" button and choose to send your email on Wednesday at 3:30 pm. Your Test Audience will receive either Version A or Version B on Wednesday at 3:30 pm. The link clicks will be tallied and the winning email will be sent to the Winning Audience at 3:30 pm on Friday, allowing for the test period.


Not what you're looking for? Check out these other articles:
A/B Test Reporting
4 Steps to Successful A/B Email Testing

Need more? Start a conversation with other Pardot users in our Success Community