International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 11 Issue: 05 | May 2024
p-ISSN: 2395-0072
www.irjet.net
Building Recommendation Systems Using Lambda Architecture Vipul Bharat Marlecha1 , Aqsa Fulara2, Sreyashi Das3 1Senior Data Engineer, Netflix USA 2Product Manager, Los Angeles, USA 3Senior Data Engineer, Netflix USA
---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - Recommendation systems are indispensable
Recommendation systems are essential tools for enhancing user experience and engagement across various online platforms. Understanding the intricate components and workflow involved in building recommendation systems is vital for designing effective and efficient solutions. This paper provides a comprehensive overview of these components and workflow, elucidating their significance in the recommendation system's architecture.
tools in today's digital landscape, empowering big tech platforms to deliver personalized experiences to users. This paper delves into the intricate architecture of recommendation systems, exploring their components, data requirements, and the amalgamation of batch processing and streaming data pipelines within the Lambda Architecture framework. We examine various recommendation models, such as collaborative filtering and content-based systems, shedding light on their methodologies and applications. Additionally, we dissect the workflow of recommendation systems, from candidate selection to real-time model serving, elucidating the challenges and strategies encountered at each stage. Through a comprehensive analysis, this paper not only provides insights into the current state of recommendation systems but also anticipates future trends and developments in the field.
2. Components of a recommendation system Whether you are building batch or real-time recommendation systems, there are a few components key to any recommendation systems.
2.1 Input systems
1. Recommendation systems form the backbone of intelligent applications Recommendations systems have been around for more than a decade. However, with the advent of intelligent systems, personalized recommendations have become increasingly important to build any applications. Think of the last video content, post, or article you saw on your newsfeed, or the product you bought on Amazon, when you exactly didn’t know what to consume or buy, how did you land on your final choice? It is likely the application showed you relevant, similar choices that resonated with your nuanced preferences, knowing what you've previously consumed, together with what you’re currently exploring. Amazon’s ‘frequently bought together’, or ‘deals for you’ panels to TikTok’s ‘For you’ feed engaged users to buy their next product or follow the next creator in intuitive and seamless ways. However, building recommendation systems, especially with growing data volumes and ever evolving user preferences is no easy feat.
|
Impact Factor value: 8.226
for
recommendation
Input Source 1: Near Real Time Data Source. When users interact with specific items, their actions provide valuable insights to recommendation systems, enabling personalized recommendations. Various interactions such as browsing history, watched content, searched items, purchases, likes, dislikes, clicks, and time spent engaging with content all contribute to understanding how, what, and when users interact with different items. These interactions are captured and stored in user logs, which are ingested into asynchronous processing systems like Apache Kafka. This allows our near-real-time processing pipeline to further clean, process, and feed the data into an online machine learning model.
Key Words: recommendation systems, lambda architecture, user preferences, real time analytics
© 2024, IRJET
sources
Input Source 2: Offline Data Source. To achieve highly available, read-heavy data storage for offline data, Cassandra can be a suitable choice due to its distributed architecture and ability to handle large volumes of read requests efficiently. Here's how Cassandra can be utilized for storing the two types of data mentioned. Items or Content: Cassandra can store the entire inventory of products or content along with their metadata. Each item can be represented as a row in a Cassandra table, with columns representing different attributes such as title, description, categories, genres, themes, images, popularity scores, etc.
|
ISO 9001:2008 Certified Journal
|
Page 698