Contents

What is testing down the list?
How does it work?
What is auto-win? How does it work?
Step by step instructions to testing down the list
FAQs

What is testing down the list?

Automated testing allows you to test how your email will perform among activists who were active on your list during different time periods, and send a "winning" email just to activists in those cohorts.

This kind of test is great for maintaining deliverability, among other uses -- in addition to unsubscribing inactive activists from your list or only targeting email to more active segments, you can test to a small random sample in each cohort and send the "winning" email just to cohorts you know are performing for this email. For more tips on email deliverability, check out our deliverability guide.

Note: Automated testing down the list only works well on larger lists, with more than 60,000 activists on your email list  (and at least 10,000 activists in each cohort, more on that later).

How does it work?

Step by step directions are below, but here's an overview: when you automate the test, the system will calculate the number of activists in each cohort who match your targeting includes and excludes. There are six cohorts — activists who have been active (open, clicked, taken action, or subscribed) in the last 30, 60, 180, 365, and >365 days. 

Once the cohorts have calculated, you'll be able to set a random limit. This will take the total number of activists targeted in each cohort and cut it down to the number you entered by choosing random activists within that cohort. In other words, your random limit will be assigned to each cohort. We recommend setting the limit to at least 1000, so you have 1000 activists per cohort. Your random limit may need to be higher depending on the size of your list and the statistical significance you prefer. (Note: Though we consulted with statisticians on this tool, we recommend doing your own math too if you want a certain level of statistical significance! 

When you send your email, you'll be able to see how each of your cohorts performed in the 'test results' tab. You'll then get to pick out the well-performing cohorts to include in the winning email and create the winning email, which will exclude activists who already got a test, but include all remaining activists in the specific cohorts you choose.

What is auto-win? How does it work?

Auto-win is an optional setting that allows you to automate the testing down the list process. If you select auto-win, you'll be able to set a number of hours, metric (opens, clicks, actions, or donations) and a threshold percentage. Any cohort that meets that threshold percentage will have a duplicated email triggered after the set number of hours. The duplicated email keeps your same targeting with includes/excludes, removes the random limit, and excludes people who received the test cohort emails. 

Step-by-step instructions to setting up your automated email test

Step 1: Write and target your email without subject/from/body testing

Step 2: Click 'Automate Testing' on the preview page. The page will reload, and calculate the number of activists in each of your cohorts (30 days, 60 days, 90 days, 180 days, 365 days, >365 days).

Step 3: Check your cohort sizes.

1. If each cohort size is greater than 10,000 activists, proceed.
2. If each cohort size is less than 10,000 activists, unfortunately your list isn't big enough for a good statistical test. Try testing down the list the manual way if you're trying to improve your deliverability.

Step 4: Set your random limit. The random limit will apply to each cohort (so if you input 5000, the email will be sent to 5000 people from the 30-day cohort, 5000 people from the 60 day cohort, and so on). For the purposes of ~ a good statistical test ~ you should set the random limit to at least 1000.

Step 5: Decide if you're going to use auto-win.

1. If you are, check off auto-win and select a number of hours, metric (opens, clicks, actions, or donations), and threshold percentage. Skip to step 8.
2. If you aren't, proceed. You can always change it to auto-win later (at the bottom of the 'test results' tab of your email manage page).

Step 6. Confirm and send. Wait for results to come in!

Step 7. After a period of time, pick your winners on the 'test results' tab. If you're not using auto-win, check off the cohorts you want to include in the winning email. Then scroll down to the bottom of the page, and click 'Create Winner'. This will take you to a preview page of the duplicated email. The targeting will be copied over as well, excluding activists who received the cohort tests. Then confirm and send your winning email!

Step 8. Bask in the glory of your beautiful email campaign. If you selected auto-win in step 4, after the cohort sends are finished, we'll wait the number of hours you chose. Then the winning email will automatically be created and sent to any cohort that meets or beats the threshold (copying your email targeting and excluding activists who received the cohort tests).

 

FAQs

I need to edit my targeting, do I need to click 'automate test' again?

Yep! Once you've made your targeting edits, save and preview. Then click 'automate test' again and follow the steps above to get set up.

 

I decided I don't want to automate the test (I haven't set the email yet). What do I do?

Just click the cancel button on the right. Easy peezy!

 

I sent my cohort test, but I want to turn on auto-win. How can I do that?

At the bottom of the 'test results' tab of your email manage page, there's an auto-win section. Check off that box, and  select a number of hours, metric (opens, clicks, actions, or donations), and threshold percentage (note that the number of hours refers to the number of hours since the cohort email was sent). Then click 'update'.

 

I'm creating my "winning" email but I want to change the targeting. Can I do that?

No, you can't change the targeting. It's copied over from the cohort test, and excludes anyone who received the cohort test.

 

I want to automate testing, but the button is greyed out. Why?

You probably either are using subject/from/body testing, or are trying to automate the testing of a “winning” email from a previous test.

If you want to test your subject/from/body first, we recommend sending a test using the normal random limit to your most active activists. More here. Then exclude that test email in your cohort test targeting.

 

Who can use automated testing?

People with 10,000 people in each cohort. See step 3.

 

What factors decide what cohort someone is in?

Good question! People are assigned to cohorts based on their engagement — whether they have opened an email, clicked an email, taken action, or subscribed to your email list in the last X number of days.