In Bad Taste
A national food cataloger wanted to test whether mailing a free product sample to customers would boost sales enough to cover the significant cost of the sample mailing. It divided test and control segments properly, coded them separately, captured key codes at order time, and even hired an outside fulfillment house to execute the test. (Its in-house fulfillment center couldn’t handle stuffing samples into mailers.)
The test spanned several months, and early results heavily favored sampling. But later results showed a puzzling drop-off in response. Finally, someone at headquarters inserted himself as a seed in the next sample mailing, got the sample package by mail, and realized that the fulfillment company wasn’t rotating its sample stock. The samples were stale and tasted terrible.
No matter how simple your test, something can go wrong. The actual source of the problem for the cataloger above was its decision to hire an outside fulfillment house to implement the sample mailing. It had never used an outside fulfillment vendor before, which meant that not only was it running a sample-mailing test, it also was learning how to manage an outside fulfillment house.
In any test, try to limit the changes to the specific thing being tested. By keeping everything else the same, you maximize your chance of getting a clean result.
A cataloger of luxury goods tested a very large number of prospecting lists one year. The following year it based its entire circulation plan on the prior year’s test results by rolling out to lists that worked and dropping lists that didn’t.
There was just one problem: To test the largest number of lists during the test year, the cataloger mailed fairly small quantities to each list. So the number of orders from each list looked like this: 14, 35, 40, 19.