Are Your Catalog Test Results Worthless?

Testing is an excellent way to learn how to maximize sales and minimize costs. However, it's critical to construct your tests to deliver results that are both accurate and actionable. A poorly designed test can deliver unreadable results and money down the drain. Here are tips to get clear, actionable results every time you test:

1. Focus the test on one element. A cataloger with "selling" covers (i.e., products sold directly from the cover) decided to test a "lifestyle" cover to see if it could lift response. An attractive room setting was photographed for the lifestyle test, then the test and control catalogs were mailed. The results? The control "selling" cover appeared to win roundly.

Wait, there's more.

The "selling" control cover also had a great offer, but the "lifestyle" cover did not. Why not? Because "the offer would have hurt the graphic integrity of the beautiful photo!"

So look again at those results. Did selling products on the cover win? Or did the great offer win? Or did selling from the cover plus the great offer win? No one will ever know. The test results were worthless.

2. Make sure your systems can execute the test. New management at a catalog company questioned if the offers were really necessary. They decided to test catalogs without offers versus the normal control offers. Creative was the same on both catalog versions except that the control had the offer and the test didn't. Mail quantities were selected to deliver statistically significant order quantities. In other words, this was a well-constructed test.

Results? Just about even. Conclusion? "Our customers don't need offers; they'll buy just as much without one." A puzzling conclusion. What was up?

A closer look revealed that anyone who ordered via the web (whether they had received the offer catalog or the no-offer catalog) saw - and used - the offer.

How did this mistake happen? Some of the blame goes to poor internal communication. Some goes to the fact that it was hard to program hiding the offer from consumers getting the test, but it was easy to program showing the offer to everyone, so IT took the easy route. Test results were worthless due to poor channel communication and internal oversight.

A Frequency Test That Worked
"Do customers really need to get all these catalogs, or can we mail less often?" was the question the new owners asked. A test was set up to mail some customers all catalogs on the "regular" schedule, and other customers were mailed only half as often. The catalog team made sure that the test group didn't miss out on any sales or special offers. This was a well-constructed test.

The catalog team also realized that to make this test work, the name selection and data processing had to be done differently than normal. That is, the exact same names needed to be flagged and then kept in either the "regular" or "half-as-many" group throughout the entire six months of the test, even if individual names switched their RFM segments. The team worked with IT to make sure that happened.

Results? The return on investment was better with the full catalog mailing schedule. The test group did reduce costs, but reduced sales and profits even more.

Trustworthy results? Yes, because the catalog team had thought through all the issues and worked with the channels and systems teams to be sure no unexpected glitches would occur. The test quantity was high enough for statistical significance, but low enough to minimize risk of sales loss during the test period.