CASE STUDY: MEC Optimizes Web Sales 
Via A/B Testing

PROBLEM: Mountain Equipment Co-Op (MEC), a cross-channel retailer of outdoor gear and apparel, wanted to increase its web sales.

SOLUTION: Hired a third-party provider of testing and personalization products for online retailers.

RESULTS: MEC's conversion rates and web revenues have increased since it began conducting optimization testing on its site nearly three years ago.

For MEC (think the REI of Canada), the inability to simultaneously test pages or functionalities on its website led to a lot of head-scratching. Testing was a part of the cross-channel retailer's culture, but tests were run at different points in time. This often left MEC wondering whether the results of its tests were affected by outside variables (e.g., a holiday, free shipping promotion) unrelated to what it was trying to measure in the first place and, more importantly, if it was leaving money on the table.

In August 2008, MEC partnered with Amadesa, a provider of testing solutions for online retailers, to help it optimize website conversions via simultaneous A/B and multivariate testing. Specifically, MEC is using a tool called InPlace, which allows it to see real-time test results from a dashboard right on the web page where a test is being run. Amadesa's tool has enabled MEC to leverage insights gleaned from tests to implement revenue-generating changes to its website.

"To be able to measure different versions of tests side-by-side at the exact same moment in time to see what everybody on the site thinks is the best took a lot of opinion out of the mix," says Bob Hermanutz, e-commerce development manager at MEC. "A/B testing allows us to take individual ideas, put them into a pot and see how our website visitors react by reviewing the data."

Testing Yields 
Conversion Gains
MEC's average website test runs nearly three weeks, with individual tests dependent on the web page in question. Pages that naturally see more traffic (e.g., homepage, product page) require a shorter period of time to reach statistical significance compared to other pages (e.g., shopping cart page). Deciding what to test is a shared responsibility. MEC's web team suggests tests to Amadesa, and Amadesa's web optimization consultants propose tests to MEC. Tagging of the actual web pages to be tested is done by MEC's in-house IT team.