Are Your Catalog Test Results Worthless?

Testing is an excellent way to learn how to maximize sales and minimize costs. However, it's critical to construct your tests to deliver results that are both accurate and actionable. A poorly designed test can deliver unreadable results and money down the drain. Here are tips to get clear, actionable results every time you test:

1. Focus the test on one element. A cataloger with "selling" covers (i.e., products sold directly from the cover) decided to test a "lifestyle" cover to see if it could lift response. An attractive room setting was photographed for the lifestyle test, then the test and control catalogs were mailed. The results? The control "selling" cover appeared to win roundly.

Wait, there's more.

The "selling" control cover also had a great offer, but the "lifestyle" cover did not. Why not? Because "the offer would have hurt the graphic integrity of the beautiful photo!"

So look again at those results. Did selling products on the cover win? Or did the great offer win? Or did selling from the cover plus the great offer win? No one will ever know. The test results were worthless.

2. Make sure your systems can execute the test. New management at a catalog company questioned if the offers were really necessary. They decided to test catalogs without offers versus the normal control offers. Creative was the same on both catalog versions except that the control had the offer and the test didn't. Mail quantities were selected to deliver statistically significant order quantities. In other words, this was a well-constructed test.

Results? Just about even. Conclusion? "Our customers don't need offers; they'll buy just as much without one." A puzzling conclusion. What was up?

A closer look revealed that anyone who ordered via the web (whether they had received the offer catalog or the no-offer catalog) saw - and used - the offer.