Catalog Testing: Whys and Hows
Patient: "Doc, testing in a catalog is a pain and it takes a long time to get the results. Do I need to test, or can I just go with my best marketing instincts or common industry wisdom?"
Catalog Doctor: "It's very tempting to skip testing in a catalog. However, I've seen too many ‘great marketing instinct’ and ‘common wisdom’ ideas crash and burn. It's much better to test first."
Trim Size Test Delivered Surprising Results
Common wisdom is that a full-size catalog (around 8 inches x 10.5 inches) will always beat a slim-jim (around 6 inches x 10.5 inches). Not always so. One test (and it was a well-constructed test) showed absolutely no difference in results between the two trim sizes (but a big difference in cost in favor of the slim-jim). Don't always believe common wisdom; test to know for sure what works for your catalog and your customers.
What Else to Test
Tests I've seen that paid off big include covers, offers, page count, order form, photos, product density, creative and, of course, lists. Creative and product density are the most expensive to test because you need to create two entirely separate designs and pay for plate changes on every page. A page count test can be done by adding an insert (e.g., four more pages) that offers marginal products that didn't make it into the main book.
It's easy and affordable to test covers — it's a single plate change on one side of the sheet. And it can result in a big sales increase. One cover test had a beautiful image with messages vs. a cover with the same image without messages. The cover with messages won by 40 percent. Retested. Same results. That's not an unusual result. Other types of cover tests have yielded similar results.
Easy Offer Tests
If you're just testing offers, which can often (but not always) be done in a small amount of space, check with your printer for the cost difference at your print quantities between plate changes, dot-whacks, bind-in cards and dry-release cards to see what's most economical for you. For example, if you can fit your entire offer on a dot-whack ("Save 20% through 3/31/14 by entering code SAVE20"), it could be your best way to test.
Don't Roll Out Without Statistically Significant Results
Any test needs to deliver a minimum of 50 orders per test segment (100 is better). If not, you're just looking at random results that aren't predictive. True story: A cataloger made a massive list test mailing to (despite many cautions) very small quantities of many lists. Since the mailings to each list were small, so too were the resulting order counts. Ignoring the statistical insignificance of the results, the "winners" were rolled out. The roll-out results bore no relation to test results, ending in big losses.
Test Twice Before Rolling Out
In real life, rolling out to large quantities doesn't always deliver results that track with test quantities. The safer option is to retest to confirm the first test. If you're fairly confident in the results, you can retest to bigger quantities than your first test.
Don't Bet the Farm on a Test
It's not necessary to do true A/B splits, where half gets the test and half gets the control. This is particularly true if management is worried about the test. It's fine to test say 10 percent and send the control to 90 percent as long as the test panel is big enough to generate statistically significant results.
Susan J. McIntyre is Founder and Chief Strategist of McIntyre Direct, a catalog agency and consultancy in Portland, Oregon offering complete creative, strategic, circulation and production services since 1991. Susan's broad experience with cataloging in multi-channel environments, plus her common-sense, bottom-line approach, have won clients from Vermont Country Store to Nautilus to C.C. Filson. A three-time ECHO award winner, McIntyre has addressed marketers in Europe, Australia and New Zealand, has written and been quoted in publications worldwide, and is a regular columnist for Retail Online Integration magazine and ACMA. She can be reached at 503-286-1400 or email@example.com.