Strategy: The Art and Science of the Test
The percent lift is calculated from the control in the chart above. To the housefile, for example, free shipping with no order minimum increased the RPC 33.58 percent compared to no offer. Free shipping with a $99 order minimum increased the RPC 21 percent over no offer. Also, notice how much better the no-minimum results were compared with the $99-minimum segment.
With regard to prospects, I often see an increase in both the response rate and average order size. When there’s a minimum, customers try to get to the minimum. Once they do, they stop buying. With no minimum, customers shop and spend more.
Simple Testing Rules
Test an offer with no minimum against your normal minimum order size. Or do a three-way test: the offer with no minimum vs. the offer with a minimum vs. no offer at all. This should be tested to both the housefile and to prospects. Execute the test properly so you can read the results. Follow these best practices I’ve found over the years that will ensure an accurate and measurable result and a sound conclusion:
1. Clearly define the purpose of the test; define the objective.
2. Prepare a pro-forma; do your financial analysis.
3. Always test against a control.
4. Only test one variable at a time; keep everything else the same.
5. Run tests during your off-season whenever possible.
6. Always retest against a control or another offer.
7. Make sure your sample size will yield statistically valid results.
8. Source code the control group and test groups properly.
9. Read the results and act on what you see!
Keep all elements, such as creative, the same except for the variable you’re testing. Always test against a control group — i.e., a group that receives exactly the same treatment you’ve been doing. If you haven’t been making any offer, this is the control. If you always offer free shipping, that becomes your control.