Catalog Testing: Whys and Hows
Easy Offer Tests
If you're just testing offers, which can often (but not always) be done in a small amount of space, check with your printer for the cost difference at your print quantities between plate changes, dot-whacks, bind-in cards and dry-release cards to see what's most economical for you. For example, if you can fit your entire offer on a dot-whack ("Save 20% through 3/31/14 by entering code SAVE20"), it could be your best way to test.
Don't Roll Out Without Statistically Significant Results
Any test needs to deliver a minimum of 50 orders per test segment (100 is better). If not, you're just looking at random results that aren't predictive. True story: A cataloger made a massive list test mailing to (despite many cautions) very small quantities of many lists. Since the mailings to each list were small, so too were the resulting order counts. Ignoring the statistical insignificance of the results, the "winners" were rolled out. The roll-out results bore no relation to test results, ending in big losses.
Test Twice Before Rolling Out
In real life, rolling out to large quantities doesn't always deliver results that track with test quantities. The safer option is to retest to confirm the first test. If you're fairly confident in the results, you can retest to bigger quantities than your first test.
Don't Bet the Farm on a Test
It's not necessary to do true A/B splits, where half gets the test and half gets the control. This is particularly true if management is worried about the test. It's fine to test say 10 percent and send the control to 90 percent as long as the test panel is big enough to generate statistically significant results.
Susan J. McIntyre is Founder and Chief Strategist of McIntyre Direct, a catalog agency and consultancy in Portland, Oregon offering complete creative, strategic, circulation and production services since 1991. Susan's broad experience with cataloging in multi-channel environments, plus her common-sense, bottom-line approach, have won clients from Vermont Country Store to Nautilus to C.C. Filson. A three-time ECHO award winner, McIntyre has addressed marketers in Europe, Australia and New Zealand, has written and been quoted in publications worldwide, and is a regular columnist for Retail Online Integration magazine and ACMA. She can be reached at 503-286-1400 or firstname.lastname@example.org.