How and What to Test Effectively

Testing is the key to direct marketing success. The key, however, is knowing what, how and when to test.

While testing is important, it’s not always cost-effective. This month, I’ll discuss how to structure tests, how to read the results and when it makes economic sense to do so.

When creating test panels, you’ll need to weigh a number of factors. The method that’s mathematically most accurate may not be cost-effective. Conversely, the most cost-effective method may produce skewed results. And in some cases, it may not even make economic sense to test.

The point is to determine the economic benefit from the test compared with its cost. You also want to be sure that in the end you’ll be able to read the results accurately and come to a conclusion.

It’s important to think through the test and ask yourself what you’re going to learn in the end. The goal of testing is to increase revenue and/or reduce costs. You need to know you can measure against these objectives.

Basic Rules

When setting up your test, follow these basic rules:

- Only test one variable at a time.

- Always test against a control group.

- Test panels must be selected from the same list universe.

- Mail dates must be the same for all test panels.

Only test one thing at a time. You must be able to read the results accurately. Every variable you introduce can influence the outcome. If you’re testing two different offers against each other and against a control group, ensure that all variables and conditions are the same for each test group.

For example, the same catalog should be mailed to all test panels on the same date. Moreover, the offer should be promoted in the exact same way: If you’re promoting the offer on the front cover for one test panel, do the same for the other test panel.