Strategy: The Art and Science of the Test
Use the Same Catalog
A group receiving something different from the control — for instance, offering a promotional incentive to a group of customers or prospects — is the test group. The same catalog should be mailed to all test panels on the same mail date. If you’re promoting the offer on the front cover for one test panel, do the same for any others. Promoting your offer on the cover for one group and on the inside order form for another doesn’t constitute a valid test, unless you’re testing where best to promote an offer.
I always feel it takes a minimum of 100 orders from any one group to accurately read the results. Sometimes it’s less, depending on the leap you’re willing to make when rolling out.
A minimum of 100 orders per panel at a 1 percent response rate to prospects would mean your panels need to be 10,000 each. If you’re testing two promo offers against a control, print 10,000 copies for the A group (the control group) and 10,000 each for the B and C test groups. Then assign key codes to all three panels. This will enable you to track the results. Select a representative sample across all ZIP codes.
Also note that for reliable results, other variables must be identical for each group. All three groups need to be selected from the same list universe. They all need to be mailed on the same day. Then once the mailing is at least 50 percent complete, you can predict the winner with confidence.
Knowing how to test is critical to your success — it’s the only way you can determine what worked and didn’t work. Knowing what to test is critical, too. Most things can be tested. Don’t assume you know what will or won't work. Test. Test. Test! The results might even surprise you!