Scaling Personalization: How Schibsted Built a Machine Learning Suggestion Engine for News
Personalizing the news experience is no longer a “nice-to-have” – it’s essential for engaging your audience and fostering loyalty. Many organizations struggle with the complexities of building and deploying effective recommendation systems.This article details how one media company successfully navigated those challenges, offering valuable insights for anyone looking to do the same.
The Challenge: Building From Scratch
Initially, the team faced a common hurdle: a lack of readily available, off-the-shelf solutions that met their specific needs. Existing options didn’t quite fit the unique demands of delivering personalized news content at scale. Therefore, they embarked on a journey to build a custom solution, leveraging the wealth of data they already possessed.
Data-Driven Feature selection: Less is Frequently enough More
A key element of their success was a rigorous approach to feature selection.Instead of throwing everything at the model, they focused on identifying the most impactful data points.
They started with a broad set of 158 potential features.
Using a mathematical approach, they pinpointed the features that truly mattered.
Ultimately, they distilled this down to just a dozen core data points.
This demonstrates a crucial principle: quality over quantity.Focusing on relevant features reduces noise and improves model accuracy.
infrastructure: The Hidden Complexity
Developing the machine learning model itself proved relatively straightforward. However,building the underlying infrastructure to support it – especially with a small team - presented notable challenges. This is where many projects stumble.
The team quickly realized they needed to invest in robust systems to handle the demands of a live, scaling recommendation engine. They learned a valuable lesson: allocate sufficient development resources from the outset.
The Tech Stack: Powering Personalized Experiences
Their solution combines the strengths of static ranking algorithms with the predictive power of machine learning. Here’s a glimpse into the technology powering their system:
Custom feature Store: This central repository provides the features needed to predict user preferences.
Flyte: An orchestration tool that ensures seamless operation.
in-Memory Database: Provides rapid data access for real-time recommendations.
AWS DJL (Amazon Web Services Deep Java Library): Enables efficient model inferencing within their Java-based application.This combination allows for a flexible and performant system capable of delivering personalized content quickly and reliably.
scaling and Future growth: Expanding the Reach
Now, the organization is actively scaling this solution across multiple newsrooms. They’re also transitioning to Tecton, a managed feature store, to streamline operations and accelerate onboarding for different brands.
This move is strategic. By centralizing feature availability, they aim to:
Expedite the integration process for new newsrooms. Ensure a consistent user experience across all platforms.
Further enhance personalization and engagement.
Key takeaways for Your Team
Building a triumphant recommendation engine requires careful planning and execution. Here are some key lessons learned:
Prioritize Data Quality: Focus on identifying the most relevant features.
Invest in Infrastructure: Don’t underestimate the complexity of building a scalable system. Embrace Orchestration: Tools like Flyte are essential for managing complex workflows.
Consider Managed Solutions: Platforms like Tecton can simplify feature store management.
* Don’t Be Afraid to Iterate: Continuously add new features and refine your models.
ultimately, personalization is about understanding your audience and delivering content that resonates with their individual interests. By focusing on these principles, you can build a recommendation engine that drives engagement, fosters loyalty, and helps your news organization thrive.