Getting the Most From Your Catalog Tests

PATIENT: “Doc, we do tests, but I don't feel that we learn much from them. What are we doing wrong? What should we change?”

CATALOG DOCTOR: "Try building your tests differently to get clearer, more actionable results. Here are some tips for what to do, and what not to do.”

Optimizing an Offer Test
The goal: determine which type of offer delivers the best sales and return on investment in both the short and long run. Here are three keys to getting useful results:

• Assign a unique promotion code for the test offer, and have that code appear only in the test catalog.
• Look at test results both early and late — they often change over time. Early results may make “A” appear to be the winner, but six-month or 12-month results may show that “B” is the actual winner.
• Track multiple result indicators, including response rate, average order, sales per catalog, and ROI. You may decide that response is more important than ROI (e.g., if your offer is to prospects and your goal is adding the most new names to the file), or you may decide that ROI is most important (e.g., if your goal is maximizing profitability).

DON'T DO THIS: splash the test offer code all over your website, where folks not in the test group can see and use it. That may be easier operationally and could garner more sales overall, but it's not a test at all. In the end you'll learn nothing.

Optimizing a Holdback Test
Also known as “holdout,” this test answers the question, "Do we really need to mail a catalog since 90 percent of our sales come online?” There are two keys to getting clear results.

• Identify names to “hold back” — i.e., names that won't get mailed a catalog. Flag those names and keep them held back for the entire duration of the test. And keep them held back even if they order during the test period. Don't keep adding and subtracting which names get held back and which get mailed while you're still in the test period.
• Let the holdback test run for long enough to really tell if sending a catalog is making a difference or not. For example, if you normally mail six catalogs during a four-month period, then hold back all six catalogs from your holdback names during that period. Whatever your catalog timing, holding back a sequence of four catalogs to eight catalogs should be enough to give a clear reading. Don't just skip one catalog mailing instead of a longer-term mailing sequence. Skipping only one catalog won't tell you if you do or don't need to mail catalogs at all, it will just show that some names may do as well with fewer (but not zero) catalogs.

Optimizing a Creative Test
Protect your control — i.e., don't redesign your control at the same time as you test a new design. This applies to copy, design, photography, rebranding or a combination of all of these things. When testing, you need to (A) protect the sales that your control has already been delivering, and (B) see if new, different creative can deliver even more sales.

DON'T DO THIS: True story. An in-house design team was building the control catalog. An outside design firm had been hired to build a new test design. The outside firm had to submit all test pages to the creative director of the in-house team. Whenever that in-house creative director saw something in the test catalog that he liked, he incorporated that new idea into his control catalog. When mailed, the test and control delivered approximately the same results (because they ended up looking about the same as each other). Result: nothing learned.