Split-test head-to-head. The only way to get accurate results is in a head-to-head (A/B) test within the same segments. Don’t test one segment against another, one region against another, or test in two separate mailings. Too many other variables can affect the outcome, and you could roll out something that won’t yield the same results you saw in the test.
Test thoroughly. Be sure your test sample size is large enough. I always split-test several segments, and ensure that split cells don’t fall below a minimum number of names needed to yield at least 100 orders from the segment.
Double check results. If a full-scale mailing is risky (i.e., it requires an up-front investment or has a potentially large opportunity cost,) then it’s a good idea to re-test in a subsequent mailing. Make sure everyone is confident with the results and that your lift is high enough in the test to suffer a drop-off in the rollout and still yield positive results.
Back test during the full-scale mailing. Hold out some names in the initial rollout, and back test the old control to be sure you’re still getting the lift you expected.
Now that we’ve established a good foundation of testing habits, let’s review several testing ideas. If you can’t implement these ideas exactly, maybe you can alter them to fit your company.
1. Cover concepts. Your cover is vital for getting prospects to “step inside” and see what you have to offer. If you’re doing a fair amount of prospecting, I recommend testing several cover concepts to make sure you’re selecting the most effective one. Test a product cover against a lifestyle one, several products against an individual product, selective focus or an intriguing perspective against something more straight forward.
It’s especially important when testing covers to understand the concept you’re testing and make sure it can be executed in different ways. Otherwise you’ll be testing only a particular cover execution and won’t be able to roll it out in consecutive versions. You might even want to test a couple of different executions of the same concept at the same time just to make sure that it is indeed the new concept that is out-pulling the old one, not just a specific cover.