Measuring Campaign Results: Short-Term or Long-Term Measurement?
In 1998, Eddie Bauer was a very profitable multichannel marketing brand. As its director of circulation at the time, I was responsible for maximizing ROI in catalog marketing.
We spent a lot of money on catalog marketing. We also spent a lot of money offering various discounts. Our favorite was “20% off your order of $100 or more.” When customers last purchased seven or more months ago, we began offering them this promotion. And it worked.
We routinely observed response increases of 10 percent to 20 percent. My team crunched through numerous P&L spreadsheets, illustrating the incremental profit this promotional strategy generated.
Life was good … that is, until we executed a test.
Our test had two panels. The first panel of customers were offered the same promotions in the same cadence as usual. The second customer panel wasn’t offered any promotions. Both panels were fixed for a period of six months.
At the end of six months, we analyzed the results. Customers in the first panel, the one eligible for promotions, spent $50 apiece on average. The second panel, not eligible for any promotions, spent the same amount per customer.
In other words, there was no difference in sales performance between the two test panels. All we did was move demand to the time periods when customers received promotions. We plotted demand on a graph and noticed spikes every time we offered a promotion. But the test panel that didn't receive promotions had steady sales on a consistent basis.
In terms of profit, the second panel was much more profitable because we didn't give away any margin dollars via promotions.
For the 1999 fiscal year at Eddie Bauer, we removed promotions from almost all active customers in our database. To date at that time, fiscal year ’99 was the most profitable year in the history of the catalog/online channel at Eddie Bauer.
Fast-forward to 2009, where we live in an analytics-obsessed marketing culture. We measure everything, and we measure everything in real time. Marketers tailor their analytics toward campaigns because they're responsible for proving their campaigns work.
When you step back and measure marketing effectiveness over time instead of at a campaign level, you obtain different results. When you choose to use a different set of metrics, you obtain a different ROI.
We're training an entire generation of analysts to give a disproportionate amount of focus to short-term results. By doing so, we optimize our world around short-term results. And by optimizing our world around short-term results, we demand that our customers convert now. We'll do anything we have to do to get a customer to convert now.
When measuring ROI over time, you sometimes observe a different result. You can learn that your metrics aren't telling you the truth.