Welcome to RecoBase

RecoBase is a collection of python code, notebooks and other essential resources to know and build recommender systems.

What is a Recommender System?

At their core, recommendation systems are nothing but similarity hunters. Depending on how you define "similarity" between two "items", you can a range of applications for these systems. Given any reasonable similarity function between items, a recommender system presents relevant items to a user. This similarity function between 2 items could be common users that have interacted with those items in the past, similarity in their attributes, similarity in context in which they are used, and so on.

Recommendation engines are, at their core, information filtering tools that use algorithms and data to recommend the most relevant items to a particular user in a given context.

Recommendation systems are built to predict what users might like, especially when there are lots of choices available. They can explicitly offer those recommendations to users (e.g., Amazon or Netflix, the classic examples), or they might work behind the scenes to choose which content to surface without giving the user a choice.

Either way, the “why” is clear: they’re critical for certain types of businesses because they can expose a user to content they may not have otherwise found or keep a user engaged for longer than they otherwise would have been. While building a simple recommendation system can be quite straightforward, the real challenge is to actually build one that works and where the business sees real uplift and value from its output.

Recommendation systems can be built using a variety of techniques, from simple (e.g., based only on other rated items from the same user) to extremely complex. Complex recommendation systems leverage a variety of different data sources (one challenge is using unstructured data, especially images, as the input) and machine learning (including deep learning) techniques. Thus, they are well suited for the world of artificial intelligence and more specifically unsupervised learning; as users continue to consume content and provide more data, these systems can be built to provide better and better recommendations.

Recommendation systems can be an effective way to expose users to content they may not have otherwise found, which in turn can forward larger business goals like increasing sales, advertising revenues, or user engagement. But there are a few key points to find success with recommendation systems. Namely, recommendation systems should be, above all, necessary. Building a complex system that requires experienced staff and ongoing maintenance when a simpler solution will do is a waste of data team resources that could be spent elsewhere for more impact. The challenge lies in building a system that will actually have a business impact; building the system in and of itself shouldn’t be the end goal. Recommendation systems should also be agile. That is, adaptable and able to evolve as users do. Putting a recommendation system into production isn’t the final step in the process; rather, it’s an ongoing evolution, looking at what works, what doesn’t, thinking about additional data sources that might help make better recommendations, etc

A recommender system calculates and provides relevant content to the user based on knowledge of the user, content, and interactions between the user and the item.

A recommendation is a particular form of information filtering, that exploits past behaviors and user similarities to generate a list of information items that is personally tailored to end-users preferences. In general, the recommendation lists are generated based on user preferences, item features, user-item past interactions, and some other additional information such as temporal and spatial data.

In the last decade, the Internet has evolved into a platform for large-scale online services, which profoundly changed the way we communicate, read the news, buy products, and watch movies. In the meanwhile, the unprecedented number of items (we use the term item to refer to movies, news, books, and products.) offered online requires a system that can help us discover items that we preferred. Recommender systems are therefore powerful information filtering tools that can facilitate personalized services and provide tailored experiences to individual users. In short, recommender systems play a pivotal role in utilizing the wealth of data available to make choices manageable. ~d2lAI

The benefits of employing recommender systems are two-folds: On the one hand, it can largely reduce users’ effort in finding items and alleviate the issue of information overload. On the other hand, it can add business value to online service providers and is an important source of revenue.

Users rely on recommender systems to filter out the numerous uninformative messages and facilitate decision making. An efficient recommender system should accurately capture users’ preferences and suggest items that the users are potentially interested in, which can enhance users’ satisfactory towards platform and improve user retention.

They’re critical for certain types of businesses because they can expose a user to content they may not have otherwise found or keep a user engaged for longer than they otherwise would have been.

Recommender systems are mainly used for auto-suggesting the right content or product to the right users in a personalized manner to enhance the overall experience. Recommender systems are really powerful in terms of using huge amounts of data and learning to understand the preferences of specific users. Recommendations help users to easily navigate through millions of products or tons of content (articles/videos/movies) and show them the right item/information that they might like or buy. So, in simple terms, RS help discover information on behalf of the users. Now, it depends on the users to decide if RS did a good job at recommendations or not, and they can choose to either select the product/content or discard and move on. Each of the decisions of users (Positive or Negative) helps to retrain the RS on the latest data to be able to give even better recommendations.

The “What to Recommend” part totally depends on the context in which RS are used and can help the business to increase revenues by providing the most likely items that users can buy or increasing the engagement by showcasing relevant content at the right time. RS take care of the critical aspect that the product or content that is being recommended should either be something which users might like but would not have discovered on their own. Along with that, RS also need an element of varied recommendations to keep it interesting enough. A few examples of heavy usage of RS by businesses today such as Amazon products, Facebook’s friend suggestions, LinkedIn’s “People you may know,” Netflix’s movie, YouTube’s videos, Spotify’s music, and Coursera’s courses. Some of the immediate benefits that RS offer in retail settings are: 1. Increased Revenue 2. Positive Reviews and Ratings by Users 3. Increased Engagement.

Recommendations break down into two different categories:

1. Exploitation: This is a sensible suggestion. For instance, if you listen to a lot of classical music, perhaps you should listen to Bach.
2. Exploration: this is the irrational suggestion—chaos. Sometimes you don’t want to listen to Bach; sometimes you want to listen to Bachman Turner Overdrive.

Now, why do we even make recommendations in the first place? And this is an essential thing to realize, because recommendations are taking the haystack and finding the needle within it. Sometimes when going to a restaurant, the menu can overwhelm. That’s when the server would make a recommendation. What we want to do is automate that. Because if you are on YouTube or Spotify, you might have millions of items on the menu.

Walk past any store in the world and you’ll see the same tactics in effect: window displays showing the brand’s most attractive, best-selling products. Entering the store will see smaller displays like mannequins, collections, or recommended setups. These are traditional methods of recommending products to shoppers – of showing potential customers what is currently in fashion to form a connection and suggesting what they might want to buy. While eCommerce doesn’t have glass windows, well-executed product recommendations can offer even more to digital brands.

The Rise of Recommendation Engines

Historically, marketing recommendations were human-built. Based on qualitative and quantitative marketing studies, marketing moguls would set up rules that statically defined the impression (in the sense of advertising views) sent to a customer with given characteristics. This technique gave rise to the marketing data mining urban legend that a grocery chain discovered that men who bought diapers on Thursdays and Saturdays were more likely to buy beer as well and hence placing the two next to each other will increase beer sales.

Overall, recommendation engines created manually presented numerous bottlenecks that resulted in a significant amount of wasted money: it was hard to build rules based on many different customer features because the rule creation process was manual, it was hard to set up experiments to test many different kinds of impressions, and it was hard to update the rules when the behavior of the customers changed.

The Role of Machine Learning

The rise of ML has brought a new paradigm to recommendation engines, allowing for the elimination of rules based on human insight. A new class of algorithm called collaborative filtering dominates the field. This algorithm is able to analyze customer and purchase data with millions of customers and tens of thousands of products to perform recommendations without any prior marketing knowledge. By identifying efficiently what customers that look like the current customer bought, marketers can rely on automatic strategies that outperform traditional ones both in cost and efficiency.

Because the process of building strategies is automatic, it is possible to update them regularly and to compare them using A/B testing or shadow scoring schemes (including the way to choose the impression among all possibilities). Note that these algorithms may be combined with more classical business rules for various reasons—e.g., avoiding the filtering bubble, not selling a product in a given geographical area, or preventing the use of a specific association that is statistically meaningful but unethical to use (like proposing alcohol to a recovering alcoholic), to name a few.

Push or Pull?

When implementing a recommendation engine, it is important to keep in mind that its structure will depend on whether the recommendations are pushed or pulled. Push channels are the easiest to handle; for example, they can consist of sending emails or making outbound calls.

The recommendation engine can be run on a regular basis in batch mode (typically between once a day and once a month), and it is easy to split the customer dataset into several parts to perform analysis within a sound experimental design. The regularity of the process allows for regular review and optimization of the strategy.

Pull channels are often more effective because they provide information to customers when they need it—for example, when doing an online search or when calling a customer service line. Specific information from the session can be used (e.g., what the user has searched for) to precisely tailor the recommendation. Music streaming platforms, for instance, provide pull-channel recommendations for playlists.

Recommendations can be prebaked, as illustrated in the in-depth example in this chapter, or made in real time. In the latter case, a special architecture has to be set up to compute recommendations on the fly.

Comparing strategies in a pull context is more challenging. First, the customers who call in on a given channel are likely to differ from the average customer. In simple cases, it is possible to randomly choose the strategy to use for each recommendation, but it also happens that the strategy needs to be used consistently over a given period for a given customer. This usually results in an unbalanced proportion of recommendations made with each strategy, which makes the statistical treatment to decide which one is the best more complex. However, once a good framework is set, this allows a very quick improvement cycle, as many strategies can be tested in real time.

Almost everything we buy or consume today is influenced by some form of recommendation; whether that's from friends, family, external reviews, and, more recently, from the sources selling you the product. When you log on to Netflix or Amazon Prime, for example, you will see a list of movies and television shows the service thinks you will like based on your past watching (and rating) history. Facebook suggests people it thinks you may know and would probably like to add. It also curates a News Feed for you based on the posts you've liked, the people you've be-friended, and the pages you've followed. Amazon recommends items to you as you browse for a particular product. It shows you similar products from a competing source and suggests auxiliary items frequently bought together with the product.

Why do online platforms need personalization capabilities?

Personalization helps customers discover products and content that can meet their needs and interests. It motivates engagement on digital properties, as customers see more of what they want and like. This growth in engagement can be observed through measurable improvements in metrics such as Click-Through-Rates (CTR), watch durations on videos, dwell times on articles and bounce rates. On top of all this, personalization is proven to improve conversions, resulting in growth in revenues or other desired outcomes.

The massive business potential observed in personalization motivated foresighted companies such as Amazon to invest and capitalize on the technology very early on. Amazon uses personalization intensively across their suite of platforms ranging from the Amazon.com website to Amazon Prime Video, Amazon Kindle, Amazon Music and more. Today, the algorithms have become significantly advanced, seen in widgets that recommend based on browsing history or from categories that are of high interest to the customer.

Personalization Challenges

A common approach to starting personalization is with a rule based system that is designed with pre-existing ideas or trends in mind, for eg. a retailer recommending boots to women who visit their website from New York at the start of winter. This system makes sense, however, it misses out on capturing the diversity of needs across people as user numbers and catalog sizes grow. Hence using machine learning should prove a better solution, producing a comprehensive algorithm more catered to individual level recommendations. Working with ML however, comes along with some challenges:

• Recommendation systems should respond to the actions and intentions of a user in real time and they should be able to effectively handle new users and new items in the catalog – a common challenge for ML systems.
• Recommendations should not have biases for popular goods, pushing the most relevant product or content based on the needs of the customer.
• Algorithms are not one size fits all, requiring customizations for different use cases.
• Building good personalization models is very hard, requiring a high level of ML expertise.

These complexities push people away from using ML, choosing to go with rule based systems or no personalization at all. Using a rule based system may come with lesser complications, however they present poor performance, poor scalability and a high cost and effort of maintenance. Those that choose to tackle machine learning tools for better results can also find them hard to build and manage, in addition to their limitations in matching recommendations with customer intent and managing real-time personalization for new customers.

Personalization engines suggest relevant content with the objective of maximizing a specific metric. For example: a news website might want to increase the number of clicks in a session; on the other hand, for an e-commerce app it is very important to identify visitors that are more likely to buy a product in order to target them with special offers.

Personalization is a very broad concept. In this case, personalization equates to leveraging engagement data to build an interest profile for a user. Many of the apps you use every day leverage machine learning to personalize your feeds. Here are 4 examples:

1. Instagram created an “Explore” section that shows you pictures you’re likely to be interested in, based on your past clicks, likes and engagement.
2. Quora uses your engagement data, like the posts that you search for and comment on, to personalize your feed experience and emails.
3. Facebook uses advanced machine learning to prioritize posts from the friends you engage with most, after starting out with just a basic edge rank algorithm.
4. Etsy analyzes your search and shopping behavior to optimize your newsfeed, email and e-commerce conversion.

Scalability

It’s important to create a recommendation system that will scale with the amount of data you have. If it’s built for a limited dataset and that dataset grows, computation costs grow exponentially, and the system will be unable to handle the amount of data. To avoid having to rebuild your recommendation system later on, you must ensure from the beginning it is built to scale to expected data volumes.

Scalability is critical to the usefulness of recommendation systems in industry settings. To that end, recommendation models can definitely be more efficient by exploring these following problems:

• Incremental learning for non-stationary and streaming data (large volume of incoming users and items)
• Computational efficiency for high-dimensional tensors and multimedia data sources
• Balancing the complexity and scalability as the model’s parameters increase exponentially

A promising research area is to use compression techniques to compact the embedding space of high-dimensional input data, which can reduce the computation time during model learning. Another promising approach is to distill knowledge to learn compact models for inference in recommendation systems. The key concept is to train a small student model that absorbs knowledge from a large teacher model.

Agility

People’s tastes don’t stay static over time, and if a recommendation system isn’t built to consider this fact, it may never be as accurate as it could be. Similarly, there is a risk of building a recommendation system that doesn’t get better over time. As users continue to consume content and more data is available, your recommendation system should learn more about users and adapt to their tastes. A recommendation system not agile enough to continue to adapt can quickly become obsolete and won’t serve its purpose.

It’s important to create a recommendation system that will scale with the amount of data you have. If it’s built for a limited dataset and that dataset grows, computation costs grow exponentially, and the system will be unable to handle the amount of data. To avoid having to rebuild your recommendation system later on, you must ensure from the beginning it is built to scale to expected data volumes.

It’s also possible that after spending time, energy, and resources on building a recommendation system (and even after having enough data and good initial results) that the recommendation system only makes very obvious recommendations. The crux of avoiding this pitfall really harkens back to the first of the seven steps: understand the business need. If there isn’t enough of a content long-tail or no need for the system, perhaps you need to reconsider the need to build a recommendation system in the first place.

Finally, people’s tastes don’t stay static over time, and if a recommendation system isn’t built to consider this fact, it may never be as accurate as it could be. Similarly, there is a risk of building a recommendation system that doesn’t get better over time. As users continue to consume content and more data is available, your recommendation system should learn more about users and adapt to their tastes. A recommendation system not agile enough to continue to adapt can quickly become obsolete and won’t serve its purpose.

The moment that recommendations have the power to influence decisions, they become a target for spammers, scammers, and other people with less-than-noble motives for influencing our decisions. ~Daniel Tunkelang

Why deep learning for recommendations?

• Deep learning can model the non-linear interactions in the data with non-linear activations such as ReLU, Sigmoid, Tanh… This property makes it possible to capture the complex and intricate user-item interaction patterns. Conventional methods such as matrix factorization and factorization machines are essentially linear models. This linear assumption, acting as the basis of many traditional recommenders, is oversimplified and will greatly limit their modeling expressiveness. It is well-established that neural networks are able to approximate any continuous function with arbitrary precision by varying the activation choices and combinations. This property makes it possible to deal with complex interaction patterns and precisely reflect the user’s preference.
• Deep learning can efficiently learn the underlying explanatory factors and useful representations from input data. In general, a large amount of descriptive information about items and users is available in real-world applications. Making use of this information provides a way to advance our understanding of items and users, thus, resulting in a better recommender. As such, it is a natural choice to apply deep neural networks to representation learning in recommendation models. The advantages of using deep neural networks to assist representation learning are in two-folds: (1) it reduces the efforts in hand-craft feature design; and (2) it enables recommendation models to include heterogeneous content information such as text, images, audio, and even video.
• Deep learning is powerful for sequential modeling tasks. In tasks such as machine translation, natural language understanding, speech recognition, etc., RNNs and CNNs play critical roles. They are widely applicable and flexible in mining sequential structure in data. Modeling sequential signals is an important topic for mining the temporal dynamics of user behavior and item evolution. For example, next-item/basket prediction and session-based recommendations are typical applications. As such, deep neural networks become a perfect fit for this sequential pattern mining task.
• Deep learning possesses high flexibility. There are many popular deep learning frameworks nowadays, including TensorFlow, Keras, Caffe, MXnet, DeepLearning4j, PyTorch, Theano… These tools are developed in a modular way and have active community/professional support. The good modularization makes development and engineering a lot more efficient. For example, it is easy to combine different neural structures to formulate powerful hybrid models or replace one module with others. Thus, we could easily build hybrid and composite recommendation models to simultaneously capture different characteristics and factors.

How recommendation engines benefit sites?

Peoples’ brains are designed in such a way that they always separate the main points from the background. When you visit a website and your eyes catch a piece of information that correlates with your own interests, you’ll likely continue reading. Finding irrelevant information on a website creates the opposite effect. Therefore, media & e-commerce business owners, in collaboration with marketing specialists, began to research the personalization issue. The more relevant information the target audience gets, the more loyal it becomes.

You don’t need to carry out thorough market research to understand that a customer will be more willing to buy from a shop that gives them maximum help in finding what they’re looking for as fast and easy as possible. They’re also much more likely to return to that shop in the future. While there’s a lot more to creating this kind of experience than recommendations, implementing a sophisticated solution is the cornerstone of a relevant and personalized customer journey. Curious about the business value of recommender systems? Netflix estimated that their recommendation engine is worth a yearly $1 billion for the company. Another telling example is when back in 2011 eBay acquired Hunch, a 2-year-old recommendation engine startup for $80 million. Amazon founder Jeff Bezos understood the value of recommender systems as early as 1998: "In the online world, businesses have the opportunity to develop very deep relationships with customers, both through accepting preferences of customers and then observing their purchase behavior over time, so that you can get that individualized knowledge of the customer and use that individualized knowledge of the customer to accelerate their discovery process. If we can do that, then the customers are going to feel a deep loyalty to us, because we know them so well". According to Amazon’s much-cited statistic, 35% of their total revenues are coming from purchases of products that customers found through recommendations. A study from Accenture revealed that 91% of consumers are more likely to shop with brands who recognize, remember, and provide them with relevant offers and recommendations.

Recommendations echo the items suggestions made by in-store sales representatives that some customers desire or require. This is taken care of on-site with the product recommendation engine, automatically generating suggestions based on a combination of rules. It provides the e-shoppers with the quickest and easiest way to view all items relevant to them. This can further motivate ‘just browsing’ visitors to make a purchase, help ‘lost’ customers find that ideal product, and encourage ‘big’ shoppers to add an item or two to their carts. It is crucial to understand the philosophy of the right things happening the right way at the right time. Using product recommendation efficiently will optimize your product discovery process, enhance customer experience, increase impulse purchases, higher engagement, conversions, average order value, loyalty, and retention. What else can a marketer need?