Recommendation Engine

Quick definition: A recommendation engine is a software system that personalizes experiences by identifying the right offer, product, or content for the people on your website or mobile app, or for those who interact with you on any digital channel.

Key takeaways:

• Recommendation engines allow companies like Amazon and Netflix to offer targeted, personalized experiences to consumers by highlighting products or content types that they know are most likely to appeal to customers based on their traits and behavior.
• The best recommendation approaches and tools move beyond the most-purchased products or most-visited content to identify the most uniquely relevant content for individual customers.
• Recommendation engines can work in two different ways — with a content-based recommendation or collaborative filtering recommendation — or with hybrid approaches that combine these methods.
• Companies can run into problems when they don’t tie to the right underlying success metric.

The following information was provided during an interview with Rob Hornick, product manager for Adobe Target.

What is a recommendation engine?
Why are recommendation engines necessary for today’s retail climate?
What is a content-based recommendation engine?
What is a collaborative filtering recommendation engine?
What is a hybrid recommendation engine?
How do brands determine a customer’s preferences?
What are the challenges when using a recommendation engine?
How can brands address or overcome those challenges?
How can brands effectively implement a recommendation strategy?
How does machine learning impact recommendation approaches?

What is a recommendation engine?

A recommendation engine is a software system that filters through data to recommend products or services a website user is likely to be interested in. Recommendations may be suggested based on a customer’s activity or actions taken by other shoppers.

Many websites and apps use recommendations:

• Spotify recommends songs listeners might like
• Amazon links to product recommendations based on past purchases
• Ads pop up on various websites based on a user’s past internet usage

Why are recommendation engines necessary for today’s retail climate?

The recommendations problem has been with us for a long time. If you look at the history of retail businesses, you've historically had a salesperson who knows customers and their behavior. And that salesperson was mentally deciding what products or offerings to recommend to customers. It was a manual process.

In a digital world, there are a few differences. We have a vastly greater number of customers that are interacting with us. They don't necessarily have that human-to-human interaction, yet we still want to make personalized recommendations for each customer, which may be potentially millions of customers across billions of interactions. This is too much to do manually.

The other challenge that we have is a much wider variety of products or content that we might recommend. Before the internet, a large retail store might have stocked 10,000 SKUs. Now, a retailer may act as a marketplace for millions of different products. And in the content space, we have millions of different options available. When we think about movies, books, or music, we just have a vastly greater number of items available to us. So recommendation engines are a great way to use data science to help make sense of those millions of customer interactions and millions of potential offerings and find the right ones for each customer.

What is a content-based recommendation engine?

A content-based recommendation engine is a system that looks at data describing the content or characteristics of an item to determine which items should be recommended to users.

For example, if we’re trying to recommend a book, we may have brief descriptions of each book. When we analyze the description of the book “War and Peace” and we see that it mentions Russia, and we look at the description of the book “The Brothers Karamazov” and we see that it also mentions Russia, we may conclude that these are similar items. We can use this knowledge to infer the items are related, so if you enjoy one book, you’ll probably enjoy the other.

Or we may have explicitly modeled attributes describing an item. For example, we know that a given shirt is a men’s shirt, it’s black, and it’s intended for golfers. If we also have access to information on user preferences — like if a user prefers men’s clothing, white clothing, and likes to golf — then we can use this data to determine the closest matched items for the user.

For example, we may know that a given item is a men’s golf shirt and want to use this information to recommend the item to relevant users. Notably, we need to know the user’s favorite color and preferred sport to be able to use this information to make a recommendation. If we have enough distinct traits about the user, we can combine different traits to come up with a more individualized list of recommendations.

What is a collaborative filtering recommendation engine?

A collaborative filtering recommendation system taps into user behavior about items to determine which items should be recommended to users. One of the potential problems with a content-based approach is that you need to have a deep knowledge of your users’ preferences and you need a lot of metadata describing items. If customers are visiting your site for the first time, you won’t have information about them, making it a challenge to have deep knowledge of their preferences. If your items are user-generated or turn over frequently, it can be a challenge to have good descriptive metadata.

In these cases, a collaborative filtering recommendation approach is often preferable. This wisdom-of-the-crowds technique is basically going to say that people who are interacting with items in similar ways will be likely to interact with items in similar ways in the future. There’s this idea that if two people liked the movie “The Bourne Identity” and also both liked the movie “Mission: Impossible,” then if one user also liked the movie “Casino Royale,” we can infer the second user will also probably enjoy this movie.

There's the notion that if you have similar preferences in the past, then your future preferences are also likely to be similar. Collaborative filtering approaches are basically going to extend this analogy and use information on one person’s likes and dislikes to make assumptions about another similar user's likes and dislikes.

What is a hybrid recommendation engine?

A hybrid system is going to combine multiple recommendation approaches. This could include a collaborative filtering approach, a content-based recommendation approach, or other approaches like applying marketing- or merchandising-driven rules to determine the best items.

Hybrid recommendation systems may also use A/B testing, multivariate testing, or reinforcement learning on top of these other approaches. You identify multiple options to recommend and send some users to one option and some users to another option. Then you measure which option performed better and use that outcome to send future users to the better option.

How do brands determine a customer’s preferences?

There are two ways that a company could know a customer’s preferences — or their likes and dislikes. One way is through explicit feedback — such as when a customer assigns a rating to a movie, like a one-star or five-star rating.

The other way that a customer might signal interest or disinterest in something is through implicit feedback. With implicit feedback, a customer is not specifically giving a rating, but by choosing to interact or not interact with something — whether or not to watch a movie, for example — they are indicating a preference. Or, if a customer is shown five products and clicks on one of those products, they’ve given implicit feedback on which of the five products they’re interested in.

Companies can also use cookies to track preferences. Within a single session, a company can track customer interactions and then use that information to inform the algorithm for their recommendation engine. If a cookie is used, then the company can track those interactions and preferences across multiple sessions — your interactions with the 10 items that you viewed yesterday and the 10 items that you viewed today. With this history, they have information about your interactions with 20 different items, instead of just 10.

What are the challenges when using a recommendation engine?

One of the challenges of a recommendation engine is collecting and analyzing all the information on a massive scale. Being able to deal with literally billions of user interaction events across a catalog that may have millions of SKUs. And being able to determine and serve up the right recommendations in real-time. It's relatively easy to do speedy requests if you're just doing a few customers an hour. It's also relatively easy to come up with millions of results if you have all day to do it. It's hard to do both at the same time. The engineering challenge is really in addressing speed and scale simultaneously.

Also, a lot of the computational techniques that are available to make sense of these millions of data points grow exponentially more expensive when you add the million and first data points. For example, if you’re building a matrix that's one million by one million rows, then every incremental product that you add is adding a million rows to that matrix.

Another challenge is the cold-start problem. If I have a new user, you don't know very much about their preferences. If I have a new item, I don't know how that item relates to other items.

Companies can also run into problems if they don’t analyze the data they receive. The most-purchased item may not be the best item to recommend. There's definitely a lot of work that goes into identifying the best models that work reliably — that aren't led astray by different customer behavior that ends up creating a poor recommendation because it’s simply reflective of overall popularity. That type of recommendation isn’t adding a lot of value because customers likely would have bought the popular items anyway.

Another potential problem that we could run into is just irrelevant recommendations — things that we're showing that customers aren’t interested in. That could have an adverse impact on the customer experience.

A final interesting problem is when you optimize your recommendations around the wrong metric. If you haven't thought about what metric you want to optimize around, you could end up showing recommendations that actually work against our business interests. For example, if you consistently recommend lower-profitability products, you could actually dilute the overall profitability of your business.

How can a brand address or overcome those challenges?

There's no single approach or silver bullet that will solve all recommendation engine challenges. It's best to have a toolset that allows you to use multiple approaches.

Part of your recommendation strategy, and part of proving the ROI of a recommendation engine, is having the ability to A/B test your recommendations. The ability to test whether the recommendations you’re making as a whole are improving that metric that you're optimizing for.

How can you effectively implement a recommendation strategy?

The number one thing that businesses need to think about for recommendation strategy is your objective. The objective can be determined by asking the following questions:

• What is the business metric that you ultimately are interested in driving?
• Are you interested in driving revenue?
• Are you interested in driving profit?
• Are you interested in driving conversions?

The second part of a recommendation strategy is thinking about what you actually want to recommend. So if you're a retailer, it's probably pretty straightforward — you want to recommend the products that you're selling. For other businesses, it's not necessarily quite so straightforward. Your recommendation strategy might be to maximize the number of marketing qualified leads that you have in your B2B sales business. So you might want to recommend content that introduces your products or services — like videos, blog posts, and white papers — to get prospective customers interested in your product lines.

A good sales strategy feels like good customer service. When you're having that personalized conversation with your customer, when you’re recommending something relevant to them, really personal to them, that's the kind of thing that's going to surprise and delight your customer and provide them with a great customer experience. It’s going to make them feel good about your brand. And that's also going to drive them to purchase that particular session.

Customers expect now that your interaction with them is going to be personalized and that it's going to be relevant. In a world where there are alternatives around every corner, it's critical that you identify something relevant to your customer when you have the chance to interact with them.

How does machine learning impact recommendation approaches?

Machine learning is increasingly an important part of leading recommendation approaches. Broadly, you have to be using some machine learning to make sense of all the data and uncover hidden associations that you may not have been aware of.

A lot of foundational work has been done to create a variety of relatively mature approaches. But what we're seeing is that there are new machine learning techniques that are being developed to allow for the refinement of these approaches. And our businesses are continuously generating new sources of data, which could be used to further improve recommendations. The development of new machine learning techniques and the development of new data sources are continuously driving improvement in recommendations.

People also view