A recommendation engine is a software system that personalizes experiences by identifying the right offer, product, or content for the people on your website or mobile app, or for those who interact with you on any digital channel.
Recommendation engines allow companies like Amazon and Netflix to offer targeted, personalized experiences to consumers by highlighting products or content types that they know are most likely to appeal to customers based on their traits and behavior.
The best recommendation approaches and tools move beyond the most-purchased products or most-visited content to identify the most uniquely relevant content for individual customers.
Recommendation engines can work in two different ways — with content-based recommendation or collaborative filtering recommendation — or with hybrid approaches that combine these methods.
Companies can run into problems when they don’t tie to the right underlying success metric.
Rob Hornick is a product manager for Adobe Target, responsible for the machine learning personalization functionality within that product, including the recommendation system Adobe Target Recommendations.
Q: Why are recommendation engines necessary for today’s retail climate?
A: The recommendations problem is one that has been with us for a long time. If you look at the history of retail businesses, you've historically had a salesperson who has a knowledge of customers and their behavior. And that salesperson was mentally deciding what products or offerings to recommend to customers. It was a manual process.
In a digital world, there are a few differences. We have a vastly greater number of customers that are interacting with us. They don't necessarily have that human-to-human interaction, yet we still want to make personalized recommendations for each customer, which may be potentially millions of customers across billions of interactions. This is too much to do manually.
The other challenge that we have is a much wider variety of products or content that we might recommend. Before the internet, a large retail store might have stocked 10,000 SKUs. Now, a retailer may act as a marketplace for millions of different products. And in the content space, we have millions of different options available. When we think about movies, books, or music, we just have a vastly greater number of items available to us. So recommendations engines are a great way to use data science to help make sense of those millions of customer interactions and millions of potential offerings and find the right ones for each individual customer.
Q: What is a content-based recommender system?
A: It’s a recommendation system that looks at data describing the content or characteristics of an item in order to determine which items should be recommended to users.
For example, if we’re trying to recommend a book, we may have brief descriptions of each book. When we analyze the description of the book War and Peace and we see that it mentions Russia, and we look at the description of the book The Brothers Karamazov and we see that it also mentions Russia, we may conclude that these are similar items. We can use this knowledge to infer the items are related, so if you enjoy one book, you’ll probably enjoy the other.
Or we may have explicitly modeled attributes describing an item. For example, we know that a given shirt is a men’s shirt, it’s black, and it’s intended for golfers. If we also have access to information on user preferences — like if a user prefers men’s clothing, white clothing, and likes to golf — then we can use this data to determine the closest matched items for the user.
For example, we may know that a given item is a men’s golf shirt and want to use this information to recommend the item to relevant users. Notably, we need to know the user’s favorite color and their preferred sport to be able to use this information to make a recommendation. If we have enough distinct traits about the user, we can combine different traits to come up with a more individualized list of recommendations.
Q: What is a collaborative filtering recommendation system?
A: A collaborative filtering recommendation system taps into user behavior in relation to items to determine which items should be recommended to users. One of the potential problems with a content-based approach is that you need to have a deep knowledge of your users’ preferences and you need a lot of metadata describing items. If customers are visiting your site for the first time, you won’t have information about them, making it a challenge to have deep knowledge of their preferences. If your items are user-generated or turn over frequently, it can be a challenge to have good descriptive metadata.
In these cases, a collaborative filtering recommendation approach is often preferable. This wisdom-of-the-crowds technique is basically going to say that people who are interacting with items in similar ways will be likely to interact with items in similar ways in the future. There’s this idea that if two people liked the movie The Bourne Identity and also both liked the movie Mission: Impossible, then if one user also liked the movie Casino Royale, we can infer the second user will also probably enjoy this movie.
There's the notion that if you have similar preferences in the past, then your future preferences are also likely to be similar. Collaborative filtering approaches are basically going to extend this analogy and use information on one person’s likes and dislikes to make assumptions about another similar user's likes and dislikes.
Q: What is a hybrid recommendation system?
A: A hybrid system is going to combine multiple recommendation approaches. This could include a collaborative filtering approach, a content-based recommendation approach, or other approaches like applying marketing- or merchandising-driven rules to determine the best items.
Hybrid recommendation systems may also use A/B testing, multivariate testing, or reinforcement learning on top of these other approaches. You identify multiple options to recommend and send some users to one option and some users to another option. Then you measure which option performed better and use that outcome to send future users to the better option.
Q: How do companies determine a customer’s preferences?
A: There are two ways that a company could know a customer’s preferences — or their likes and dislikes. One way is through explicit feedback — such as when a customer assigns a rating to a movie, like a one-star or five-star rating.
The other way that a customer might signal interest or disinterest in something is through implicit feedback. With implicit feedback, a customer is not specifically giving a rating, but by choosing to interact or not interact with something — whether or not to watch a movie, for example — they are indicating a preference. Or, if a customer is shown five products and clicks on one of those products, they’ve given implicit feedback on which of the five products they’re interested in.
Q: What are the challenges?
A: One of the challenges of a recommendation engine is collecting and analyzing all the information on a massive scale. Being able to deal with literally billions of user interaction events across a catalog that may have millions of SKUs. And being able to determine and serve up the right recommendations in real time. It's relatively easy to do speedy requests if you're just doing a few customers an hour. It's also relatively easy to come up with millions of results if you have all day to do it. It's hard to do both at the same time. The engineering challenge is really in addressing speed and scale simultaneously.
Also, a lot of the computational techniques that are available to make sense of these millions of data points grow exponentially more expensive when you add the million and first data point. For example, if you’re building a matrix that's one million by one million rows, then every incremental product that you add is adding a million rows to that matrix.
Another challenge is the cold-start problem. If I have a new user, you don't know very much about their preferences. If I have a new item, I don't know how that item relates to other items.
Companies can also run into problems if they don’t analyze the data they receive. The most-purchased item may not be the best item to recommend. There's definitely a lot of work that goes into identifying the best models that work reliably — that aren't led astray by different customer behavior that ends up creating a poor recommendation because it’s simply reflective of overall popularity. That type of recommendation isn’t adding a lot of value because customers likely would have bought the popular items anyway.
Another potential problem that we could run into is just irrelevant recommendations — things that we're showing that customers aren’t interested in. That could have an adverse impact on the customer experience.
A final interesting problem is when you optimize your recommendations around the wrong metric. If you haven't thought about what metric you want to optimize around, you could end up showing recommendations that actually work against our business interests. For example, if you consistently recommend lower-profitability products, you could actually dilute the overall profitability of your business.
Q: How can companies address or overcome challenges?
A: There's no single approach or silver bullet that will solve all recommendation engine challenges. It's best to have a tool set that allows you to use multiple approaches.
Part of your recommendation strategy, and part of proving the ROI of a recommendation engine, is having the ability to A/B test your recommendations. The ability to test whether the recommendations you’re making as a whole are improving that metric that you're optimizing for.
Q: How can companies effectively implement a recommendation strategy?
A: The number one thing that businesses need to think about for recommendation strategy is their objective. What is the business metric that they ultimately are interested in driving? Are they interested in driving revenue? Are they interested in driving profit? Are they interested in driving conversions?
The second part of a recommendation strategy is thinking about what you actually want to recommend. So if you're a retailer, it's probably pretty straightforward — you want to recommend the products that you're selling. For other businesses, it's not necessarily quite so straightforward. Your recommendation strategy might be to maximize the number of marketing qualified leads that you have in your B2B sales business. So you might want to recommend content that introduces your products or services — like videos, blog posts, and white papers — to get prospective customers interested in your product lines.
A good sales strategy feels like good customer service. When you're having that personalized conversation with your customer, when you’re recommending something that's really relevant to them, really personal to them, that's the kind of thing that's going to surprise and delight your customer and provide them with a really great customer experience. It’s going to make them feel good about your brand. And that's also going to drive them to make a purchase in that particular session.
Customers really expect now that your interaction with them is going to be personalized and that it's going to be relevant. In a world where there are alternatives around every corner, it's really critical that you identify something relevant to your customer when you have the chance to interact with them.
Q: How does machine learning impact recommendation approaches?
A: Machine learning is increasingly an important part of leading recommendation approaches. Broadly, you have to be using some machine learning to make sense of all the data and uncover hidden associations that you may not have been aware of.
A lot of foundational work has been done to create a variety of relatively mature approaches. But what we're seeing is that there are new machine learning techniques that are being developed to allow for the refinement of these approaches. And our businesses are continuously generating new sources of data, which could be used to further improve recommendations. The development of new machine learning techniques and the development of new data sources are continuously driving improvement in recommendations.