Email A/B Testing: What It Is, and Why Should You Do It

Diving into the world of online marketing can be confusing. Those who claim to be experts on the subject often throw around a laundry list of terms like “A/B tests,” “split testing,” or something about “bouncing.” (Bouncing is less fun than the term might imply.) These terms all have important meanings, but they’re useless to everyone who’s unsure of their importance when creating marketing campaigns.

For email marketers, A/B testing is a valuable and commonly-used tactic. Whether you’re a new entrepreneur with a lot of enamel pins and a laptop, or a seasoned professional in Silicon Valley with a man bun and two phones (one for tweeting, one for calling your boss about the tweets.) Let’s dive into the basics of email A/B testing, including what it is, and how you can harness its power to supercharge your email marketing strategy.

What is email A/B Testing?

Email A/B testing is the process of testing specific parts of an email to determine which version ultimately will convert the best.

For example, you probably have a go-to email you know will get solid results every time you send it. But maybe you’ve been sending the same subject line over and over and you’re curious if some different versions will help or hurt your conversions. With A/B testing, you can test out that change (and others!) to see how it performs.

How do you A/B test an email?

Before you start setting up the actual A/B test, there are a few things you ought to plan out to make the test accurate and meaningful.

1. Your Objective

First, make sure you’ve identified the purpose of the test. Are you trying to increase open rates? Trying to get more subscribers to purchase? Building a new product waitlist? Having a clearly defined objective will help you test the correct elements of your email and provide useful metrics to measure the test’s success.

2. Your Sample Size

The second thing is to make sure your email list is large enough to get meaningful data. While there’s no minimum sample size, many companies find lists under 1,000 subscribers may not yield statistically significant results to draw a conclusion.

3. The Control Email

For this strategy to work correctly, you’ll need to have two versions of your email – a “control” and a “test”. The control email is the standard or default email you typically send your email list. It might be a weekly sale, a new subscriber coupon, or any offer of some sort. The control email doesn’t change and remains the same throughout the A/B test.

4. The Test Email

The test email (also referred to as the treatment) is the email where you’ll make your change. We’ll talk more in a minute about some of the components you may want to vary in your A/B emails. But for now, it’s important to know that you should test only one change at a time in your test email and to keep the control email unchanged, otherwise, you won’t know exactly what led to the change in results.

Once you’ve set your objectives and established your control email, follow these fours steps to set up and create your email A/B test:

1. Identify the variable (subject line, email length, offer, etc.) you want to test against the control (your default email with no changes or optimizations) and make the change.
2. Send both the control email and the version with the change to your email list.
3. Measure the performance. Did your conversions increase with your test email? Was there any statistical significance or difference?
4. Once you’ve identified what’s working and what isn’t, continue to optimize other parts until you find a winning email.

What components can be tested in an A/B test?

Marketers looking to optimize their email campaigns will A/B test multiple parts of an email over time. Here are some of the most common elements marketers A/B test.

Send Time

Your audience may respond better to emails depending on the time of day or day of the week. If you test sending emails on the weekend vs on Tuesday mornings, you’ll know the best time for you to send future campaigns.

Visual Design

Does a basic email template raise conversions, or does it work better to include hi-res images, gifs, or other creative layout options? While the email content is important, the email design plays a huge role in performance too.

Email Subject Lines

Do creative, catchy subject lines perform better than straightforward CTAs? What about using emojis? Test which style improves the conversion rate with your audience. The Barack Obama for America campaign tested hundreds of subject lines until it found the winner. The result was an additional 2.2 million in donations raised from sending the best draft vs 1.2 million raised from sending the default draft.

Sender Name (From Name)

Virtually all email providers include personalization options. You can send from your company name or choose to send from a person’s name. An A/B test will tell you which your audience is more likely to open and engage with.

Preheader Text

Most email platforms allow you to include a brief sentence as a “preheader” or a preview text. You can test whether or not these preheaders work and which types of content increase the open rates of your email marketing campaigns.

Call To Action

Testing the call to action is key to understanding what message your audience responds to the most. For example, if your emails are all transactional and include a “buy now” button, try a different CTA like “save now” or “learn more about pricing” to determine what raises engagement. You may also try the same CTA, but two different styles—such as a hyperlink in version A, and a button in version B.

How long should I run my A/B email test?

The length of your test depends on how long it takes to get statistically significant results, and this will vary based on the test objective, sample size, the frequency of your normal email schedule, and your average open rate.

Some people recommend 24 hours, while others say wait a week. The bottom line is that you’ll need to run your test until you have a high level of confidence that you’ve garnered all the data you’re going to be able to pick up from your audience, based on your past data and experience.

Here are a couple example timeframes to demonstrate:

Example 1

Objective: The objective is to test open rates on two different email subject lines to a list of 10,000 that has a normal average open rate of 22% in the first 48 hours after send.

Timeframe: In this scenario, you would want to wait at least 48 hours and make sure your control email’s open rate has reached approximately 22% before closing the test window in order to accurately compare subject line version B.

Example 2

Objective: The objective is to test conversion rates on two different coupon offers to that same list, which normally has a click-through rate of 1.5% and conversion rate of 0.7% after 3 days.

Timeframe: In this scenario, your total test timeframe would need to be at least 3 days for you to compare conversion rates. If your click-through rate is lower than average, you may need to run another test in order to get a more statistically significant

How do I measure my A/B testing results?

Ultimately, success is measured by improvement in KPIs (Key Performance Indicators). If changing the CTA in your email raised revenue consistently by 3%, that would likely be significant enough to make the change permanent.

To keep your conclusions accurate, make sure to only test a single variable at a time. When multiple changes are tested simultaneously, it’s hard to determine which variable made the biggest impact.

As you test and refine your email strategy, you should see improvements like higher open rates and increased click-through rates for your email marketing campaigns.

Benefits of A/B Testing Your Emails

The digital ecosystem is a confusing one that changes fast, requiring you to be adaptable and creative. Split testing makes this process easy and continuous; you can run A/B tests for the entire lifetime of your site and business, making widespread changes where appropriate.

A/B testing is all about finding the best email for your audience. If you’re an eCommerce store, you can sell more products. If you’re a non-profit, you can find which stories and times of the month raise the most money. But the benefits don’t stop there. In fact, A/B testing data can be used for a wide range of other marketing outlets and inform your overall marketing strategy. Here are a few additional A/B testing benefits.

Low-cost Market Research

A good A/B test is one of the cheapest forms of audience research available. Rather than spending valuable money, time, and resources asking people what they would like to see, you can simply test which aspect of your site or business they like more. Your audience will tell you through their engagement what works best for your email marketing on a continuous basis, basically for free.

Learn More About Your Customers

A/B testing doesn’t just inform you about changes you should make to your site or funnels. Split testing can also be craftily applied to help you learn more about other aspects of your business that your customers value, like new products or offers. You can split-test different products in your marketing, feature different products on your landing pages, or offer different deals to determine what your customers value the most. If they go head-over-heels over a featured pair of heels, then you can safely guess they want to see more shoes on your site.

Omnichannel Optimization

By testing different messages in your emails, you can take your data and apply it to other advertising outlets. If videos get more engagement than pictures, then your audience may be more interested in watching a video advertisement on your site than reading your latest blog.

Email A/B Testing Made Easy With Kartra

You might be thinking that a good A/B test will require you to manually divide your audience in two, open up a bunch of spreadsheets, and make dozens of different email versions. Fortunately, we got just as tired of traditional manual split testing as everyone else did, so we made it easier!

Kartra’s marketing platform includes all the features you need to automate your A/B tests with options to test just about everything under the digital marketing sun, whether it’s email, a landing page, imagery, or anything else on your page or funnel that could use a little experimental dynamism. For your email marketing, the data gleaned from your A/B test is automatically analyzed, letting you see right away what the results of the test were and a recommendation on which approach to choose for better results.

Start automating and refining your email sequences today!