How to Use Artificial Intelligence to Split Test Email Campaigns

Marketing is a percentages game.

A couple percentage points may not seem like a lot, but, when you have an email list approaching 1 million followers, 2% means an extra 20,000 leads opening your email, interacting with your brand, and ultimately generating revenue.

The problem marketers face when optimizing emails is determining which factors create that 2% increase. Is it the placement of the CTA? The color? Does it matter who sent the email? With so many variables, it’s easy to turn email optimization into a guessing game and form opinions based on perspective rather than statistical evidence.

There’s a better way, of course—email split testing. When done right, you’ll be able to know exactly why one email performs 2% better than another.

Related Read: Check out this case study where Virgin Holidays made millions from a 2% increase in open rates.

But split testing is tricky. Only carefully designed tests end up producing results. Just ask split test experts HubSpot and Phrasee: they say that “Split testing the wrong way is almost as in-effective as not split testing at all.” This comes from their recent study, The Science of Split Testing.

There are two common problems that marketers run into when running split test campaigns: first, gathering enough relevant data, and second, analyzing that data to make better marketing decisions.

Consequently, our research and experience have shown us a solution to these problems: create clean data by taking the correct scientific approach, then use AI technology to gather more relevant information, analyze it, and automatically offer solutions.

With all this in mind, we’ll take a look at how HubSpot and Phrasee recommend you create some killer split tests.

Let’s go back to 5th grade science.

Remember when you had to run an experiment for the science fair? Well, it turns out that mentality is actually really valuable in marketing. The elements of the scientific method are the exact elements of an effective split test:

• Hypothesis: What you expect to happen in the experiment. For example, “I think that email subject line A, B, C, or D will have a higher open rate.”
• Independent Variable: This is what you change. In this example, the independent variables are the different subject lines you’re testing.
• Dependent Variable: The effect caused by changing the independent variable. In this case, the dependent variable is the open rate. It goes higher or lower depending on your subject lines.

Pretty simple, right? Here’s where your split test could go wrong:

• Controls: There’s a lot of different variables in your emails. Make sure you only change the independent variable, and that the rest remains as similar as possible. If you’re testing subject lines, for example, make sure you send your tests to the same audience segment at the same time.
• Statistical Significance: Your test group needs to be a certain size in order for you to determine that your changes did, in fact, make a difference. To spare you from the math involved, use this email split test calculator to determine that size.

More Data Is Better: Moving Past A/B Testing with AI

Marketers commonly refer to split tests as “A/B” tests, where they test only two independent variables (two different subject lines, for example). But why stop there? Why not test 10 different subject lines?

Sure, it takes a little bit more work, but the data that larger split tests produce is super valuable. Pair that data with AI, and you start to see some real incredible results.

That’s exactly what Phrasee does, actually. This AI tool captures data from each of your split tests, then uses machine learning and natural language processing to automatically generate the optimal subject line copy that will get people to click.

For more valuable information, be sure to check out The Science of Split Testing. You’ll be running split tests like a pro in no time!