Crocs, Aritzia Use A/B Testing to Optimize E-Commerce Businesses

A Testing Strategy
Crocs runs lots of tests, and those tests seek to answer the following question: How far from optimal are we — i.e., how far will the results of this test move the needle? Nemann provided some guidelines on how Crocs decides what's worthy of being tested. First the retailer looks at the ease of the execution of the test, as well as the risks associated with it (e.g., lost sales). It then examines the potential impact the results of the test could have on customer reach and lift. It weighs those factors and prioritizes which tests should be conducted.

Don't spend time testing the minutia, Nemann advised. Look at your purchase funnel, identify where the drop-offs are occurring, then test to see how you can optimize, she added.

Crocs recently tested the impact moving from a five-page checkout to a single-page checkout would have on its business. While not as simple a test to execute as say changing the size or color of a "Add to Cart" button, the effort validated what Crocs already believed — the easier the checkout process, the more conversions the merchant will get. Those who were given a single-page checkout started the purchase process 6 percent more than those with a five-page checkout.

Takeaway Tips
Keller and Nemann wrapped up their presentations with some tips for the other retailers in the audience on how they can best leverage A/B testing:

• Front-end testing (e.g., placement of a button) typically requires less development effort than testing business rules (e.g., product recommendation rules).
• Look at industry benchmark data to get insight before testing.
• Run your tests for a minimum of two weeks to four weeks to get statistically valid results.
• Failed tests shouldn't be considered failures.
• Design tests with a purpose, but optimize for multiple paths.