Are Your Catalog Test Results Worthless?

How did this mistake happen? Some of the blame goes to poor internal communication. Some goes to the fact that it was hard to program hiding the offer from consumers getting the test, but it was easy to program showing the offer to everyone, so IT took the easy route. Test results were worthless due to poor channel communication and internal oversight.

A Frequency Test That Worked
"Do customers really need to get all these catalogs, or can we mail less often?" was the question the new owners asked. A test was set up to mail some customers all catalogs on the "regular" schedule, and other customers were mailed only half as often. The catalog team made sure that the test group didn't miss out on any sales or special offers. This was a well-constructed test.

The catalog team also realized that to make this test work, the name selection and data processing had to be done differently than normal. That is, the exact same names needed to be flagged and then kept in either the "regular" or "half-as-many" group throughout the entire six months of the test, even if individual names switched their RFM segments. The team worked with IT to make sure that happened.

Results? The return on investment was better with the full catalog mailing schedule. The test group did reduce costs, but reduced sales and profits even more.

Trustworthy results? Yes, because the catalog team had thought through all the issues and worked with the channels and systems teams to be sure no unexpected glitches would occur. The test quantity was high enough for statistical significance, but low enough to minimize risk of sales loss during the test period.