What is email testing? What is its purpose?

A/B email testing is a great way to figure out what content resonates with your activists and what (despite your best efforts) does not. It’s the ‘scientific’ way of finding out what kind of content your activists will engage with because it prioritizes data over your instincts (your instincts are probably very good, but they aren’t as accurate as the data). Your activists may react unexpectedly to your content, and you’ll never know for sure until you test and measure your results.

Looking for automated testing down the list? Check it out here.

A/B email testing allows you to send different variations of an email to a subset of recipients in order to determine which variation gives you the best results. Then, you choose your highest performing ‘winning’ email and send it to the rest of your email list. You can test email subject lines, ‘from’ lines, and email bodies on Action Network to find which content your list likes best. Here’s how:

Step 1: Determine what metric you’re going to test.

Your first step to A/B testing is to decide what metric you’re trying to improve. There are three main metrics that A/B testing is used for:

• Open rate — the percentage of recipients who opened your email
• The best way to improve your open rate is to test your subject lines and from lines
• Click/open ratio — the ratio of email openers who clicked on a link in your email
• The best way to improve your click ratio is to test your email body
• Why should you use click/open ratio instead of click rate? Your click rate will increase as more people open the email. Ratios stay constant and allow you to make faster decisions than the click rate.
• Action/click ratio — the ratio of email clickers who took action
• The best way to improve your action ratio is to test your action landing pages via links in the email body
• Why should you use action/click ratio instead of action rate? Your action rate will increase as more people open the email. Ratios stay constant and allow you to make faster decisions than the action rate.

Step 2: Determine what variable you’re going to test.

Once you’ve decided what metric you’re going to test, you can decide what variable you’re testing in your email. There are three variables you can test on Action Network: subject line, ‘from’ line, and email body. 

• Subject line
• Email subject lines are the first message an activist receives from you in an email. It needs to interest your activists enough to motivate them to open your email.
• ‘From’ line
• The ‘From’ line can be your name, your organization, or a combination of both. It allows you to build a relationship with your activists, so it’s important to decide who that relationship should be with. Whose name at your organization should your activists recognize? After you’ve determined which from line performs highest, it’s important to stay consistent to build rapport with your activists. 
• Body
• This is the content that your activist sees once they open your email, and it should encourage them to take action. The email body lets you test a multitude of different variables, including button color, text copy, images, links, and more. 

A note on multi-variable testing: You can test multiple variables on Action Network. We will display your test results for every combination of variables you create in order for you to select the best email. For example, if you’re testing two subject lines and two email bodies, we will send four variations of the email. You will be able to view the results of each variation.

Step 3: Make a hypothesis

Next, make your hypothesis (think back to 10th grade science class). This is what you think will happen when you test your email. Some examples of hypotheses include:

• Emojis in my subject line will increase open rates.
• Emails from my executive director will increase open rates.
• An orange call-to-action button will increase click rates.
• An image in my email will increase click rates.
• An action landing page in my organization’s colors will increase action rates.

You should already know your baseline result (what your metrics currently are), so you can gauge what to expect when you send your test email. You want to test option A and B, but you also need to know if neither option A nor B are performing as well as you’d like -- it’s possible that neither option A nor option B will perform better than your baseline.

Step 4: Create variations of that email

Now that you’ve decided which metrics and variables you’re going to test and made a hypothesis, it’s time to test that hypothesis! Here’s how to test subject lines, ‘from’ lines, and email bodies within Action Network.

Testing Subject Lines

To set up a subject test, use the alternate subject field for a second subject, and click the green plus icon to add as many additional alternate subjects as you want.

If you want to remove a subject line, delete the content and leave the alternate subject field blank.

Note: Don't make your subject too long! Long subjects tend to perform worse, and subjects over the byte limit will be truncated and will break open and click statistic reporting. Less than 200 characters should be your goal. 

 

Testing From Lines

You can enter more than one ‘from’ line if you'd like to set up an automatic ‘from’ line test. If you enter more than one ‘from’ line, we will randomly distribute each ‘from’ line you enter among your email's targeted universe and allow you to track the statistics of each ‘from’ line individually to see which one performs best.

If you want to remove a ‘from’ line, delete the content and leave the alternate ‘from’ field blank.

To set up a ‘from’ line test, use the alternate ‘from’ field for a second ‘from’ line, and click the green plus icon to add as many additional alternate ‘from’ lines as you want.

 

Testing Email Bodies

You can test multiple email bodies, which lets you test your creative content like email text, buttons, images, and more. Underneath where you write your original email body, click the green plus button labeled ‘add alternate body’. This will copy your original body over to the alternate, where you can fully edit the body. Give your alternate body a name, which will appear on the ‘test results’ tab of your email manage page so you can track how your email is doing.

On the visual editor/HTML, if you want to remove an alternate body, delete the content and leave the alternate body blank. On the drag & drop email builder, click the ‘delete body’ button in the lower right.

 

Step 5: Determine who to test your email to

Now comes the important part: deciding who will receive your email tests.The answer is different depending on the size of your email list. An important thing to remember in email testing is that this is a ~ scientific ~ process, so an ample sample size is essential to getting useful results. The larger the sample size is, the more accurate your results will be. To determine a good sample size, we recommend using a tool like Optimizely or EvanMiller.org. 

So the first step to figuring this out is to calculate the number targeted in your email. For example, if you’re targeting people in Texas who are donors, calculate the number of Texan donors you have. If you’re sending to your full list, you can go by the number of subscribers on your group manage page. Whatever you’re targeting by, you can pull a report to do this OR use the email targeting feature and let it calculate.

Now that you know how many people are targeted, you can decide which method you want to test:

 

Method 1: For large lists (>10,000 targeted), send to a test batch

If you have a larger list (more than 10,000 activists who meet your targeting criteria), then you can send a test batch of your email. This will let you evaluate which email variation is performing better, then send that email to the remainder of your list. For example, if you have 100,000 activists subscribed to your list, you may test your email by sending to 10,000 activists. Whichever email does best, you would send to the remaining 90,000 activists.

To determine a good sample size, we recommend using a tool like Optimizely or EvanMiller.org. 

Whatever sample size you choose, scroll down to the bottom of your email targeting page. Input the amount you calculated into the ‘random limit’ box. This will take the total number of activists targeted with your ‘includes’ and ‘excludes’ and cut it down to the number you entered by choosing random activists in your targeted universe.

In some cases, you may want to do more long term testing and tracking with a randomized subset of people, you can apply a random limit to a report. You'll be able to target different emails and contact to this audience by targeting that specific report or applying a tag to it. Click here to learn more. 

Method 2: For small lists (0-10,000 targeted), send a 50/50 split

If you have a smaller number of activists you’re sending this email to (less than 10,000), your sample size would not be large enough to send a test batch. Instead, you should send the email to your full list and employ long-term list testing. The data you gather on the best performing email should inform the strategies you employ in the future. What type of subject line works best, what type of ‘from’ line works best, and what type of content works best? Over time, you’ll see these answers start to emerge. 

 

Step 6: Let the results come in

The hard part is done! On your email manage page, navigate to the ‘test results’ tab, and watch your results come in! We know you’re excited to watch those email statistics count up when you refresh the page.

Step 7: Choose the winner and duplicate

Wait an hour or two to see which test does best — is your hypothesis correct? Remember to make sure whatever metric you’re measuring is above your baseline. 

Once you’ve chosen your winner, click the “duplicate with exclude” button to the right of your highest performing email. You’ll be taken to an email writing page that will automatically duplicate that email version you chose. Then, click the button to save and go to the next step, and you’ll see everyone automatically excluded who received any version of the email you previously sent. From there, send your winning email to the rest of your email list.

And you’re done! Remember that your email list changes over time, so be sure to test your emails frequently.