How to Measure the ROI of Machine Learning and Artificial Intelligence

Applications for machine learning (ML) and artificial intelligence (AI) are growing exponentially with the data that supports them. In 2019, more and more retailers will be adapting these solutions as another means of differentiation, gaining more insights into their customers, their journeys and overall experiences.

The ability to both develop the right solution and measure its effectiveness are key to any retailer’s success when it comes to ML and AI solutions. Doing so will not only cover the cost of developing and deploying the solutions, but will drive revenue growth as well.

Here are the top use cases we’ll see in 2019, as well as how to measure the impact these solutions can have to the bottom line.

The Best Ways to Use ML and AI Solutions

Two of the most common ML and AI use cases are image recognition and chatbots. Image recognition is predominately used in medical and healthcare industries, and chatbots have become the norm in financial services, but are becoming more widespread.

Aside from these, retailers have several opportunities to use ML and AI to have a positive impact on their business, including:

• Next Best Action: This is the logical extension of “next best product” or “next best offer” and optimizes the customer experience through selecting the right product as well as the right creative, offer and channel for that customer journey. Some call this personalization, but it’s really more than simply adding key pieces of unique information into a customer communication.
• Natural Language Processing: The most valuable use case for this technique is in measuring customer sentiment. While this is challenging, the consequences of negative sentiment going viral and not being addressed as soon as possible are impactful and need to be accurate and continuously improving. For example, retailers need to be able to differentiate this REAL social media post: “I like Brand A. I'm 58 and I don't work in an office so I want good, fun clothes. Brand B, I love you but you've gotten way too office like for me.” In evaluation of various techniques, this message was flagged by one algorithm as a positive sentiment for Brand B and another as negative. While this post is rather innocuous, there's a lost opportunity when the sentiment is interpreted incorrect, which can potentially have brand image ramifications.
• Segmentation and Personalization: This use case supports classification and segmentation based on data that's available in real time and supports optimization of the customer journey based on specific customer journeys. Segmentation allows retailers to also support the next best action in the customer journey, and algorithms can be used to align offline and online treatments.
• Marketing Attribution: Understanding those marketing activities that are driving customer conversion are key to retailers in an omnichannel world. The days of overcounting are over when you apply ML algorithms to define the actions that are most impactful to converting customers to engage with the brand, purchase products or services, and ultimately become brand advocates. An example of marketing attribution would be to look at two customer journeys that are exactly the same — perhaps two emails and two digital retargeting ads — and recognizing which drove the customer to action. Based on unique attributes of the different customers, you might see that the digital ads weighted higher on customer A’s behaviors, but the emails drove customer B to convert. This is done through both understanding the customer segments as well as reviewing customer journeys by segments on a daily basis, as models identify the drivers for conversion.

3 Keys to Successful ROI Measurement

After defining the specific use case, the next step is to determine how to measure the impact of the solution and to define what the potential return on investment of the ML and AI initiatives will be. This is done through the creation of a pro forma. Most service providers will offer proforma development as a service included with ML and AI development. This pro forma will have key assumptions that are validated and updated as the solutions are rolled out or tested.

1. Defining Key Performance Indicators (KPIs): Oftentimes, this is mandatory in ML training since there's typically a metric that you're trying to optimize. However, in the case of segmentation for example, you need to know what the segments are going to be used for and what success criteria will be defined. The metrics defined may differ by segment as well (e.g., increasing average spend for the best customer segments and improving retention rates for average customer segments).
2. Having a Benchmark of Comparison: Once performance metrics are defined, maintaining a valid control group or holdout groups (e.g., customers who are not assigned next best action based on that ML algorithm) are key to understanding the true impact that the ML and AI models are contributing. Marketers have been measuring the success of customer treatments in this way for decades, and the principles still hold true today. The segments and treatments are now at a personalized one-to-one level.
3. Monitoring Overtime: Many algorithms are updated on a regular basis as more training data is made available. This can be a blessing and a curse when it comes to implementing a ML model, as the models can change significantly over time and need to be monitored to ensure the targets are being met and the metrics for success are sustaining data changes. Be sure to have data audits in place. If the core data is changing, your algorithms will change and potentially degrade. This is especially true after you've rolled out the ML models at full scale and no longer have control groups against which to monitor the models. Have a rigorous process in place to monitor models, and re-train and measure impact frequently. The two steps forward, one step back approach is valid here in ensuring that your KPIs are moving in the right direction.

The process for defining and measuring ML and AI initiatives is a journey, not a destination. Once you've gained momentum through your first use case, the opportunities to affect change on the business are truly limitless.

Julie Schmidt is the chief analytics officer at Allant Group, a “model driven” marketing service provider that uses data to generate insights (e.g., customer likes/interests/behavior) to execute personalized campaigns that acquire, retain or win-back customers.