LinkedIn Achieves “Considerable Quality Enhancement” in Job Recommendations with Multi-Teacher Distillation
LinkedIn has considerably improved the quality of its job recommendations through a novel submission of large language model (LLM) technology, dubbed “multi-teacher distillation.” According to LinkedIn’s Head of AI, Bharath berger, the improvements represent “substantial quality improvement of the likes we probably haven’t seen in years.”
Why multi-Teacher Distillation Was a ‘Breakthrough’ for LinkedIn
Berger and his team aimed to create an LLM capable of interpreting job queries, candidate profiles, and job descriptions in real-time, while adhering closely to LinkedIn’s product policies.
The team collaborated with product management to develop a detailed, 20-to-30-page document outlining scoring criteria for job description and profile pairings “across many dimensions.” This document underwent numerous iterations to refine its accuracy.
This product policy document was then used in conjunction with a “golden dataset” of thousands of query-profile pairs. The team leveraged ChatGPT to generate a larger synthetic dataset,training a 7-billion-parameter “teacher model.”
however, Berger emphasized that aligning with product policy alone wasn’t sufficient. “At the end of the day, it’s a recommender system, and we need to do some amount of click prediction and personalization,” he stated.
To address this, the team developed a second teacher model focused on click prediction, using the initial product policy-focused model as a foundation. They then distilled both teacher models into a 1.7-billion-parameter “student model,” optimizing it through numerous training runs to minimize quality loss.
This multi-teacher distillation technique enabled the team to achieve strong alignment with both product policy and click prediction accuracy. It also allowed for a more modular and componentized training process for the student model.
Berger illustrates this with an analogy to a chat agent trained by two teachers: one focusing on response accuracy, and the other on tone and communication style. “By now mixing them, you get better outcomes, but also iterate on them independently,” he explained. “That was a breakthrough for us.”
changing How Teams Work Together
Berger underscored the importance of a strong product policy and iterative evaluation process.
Creating a “really, really good product policy” requires translating product manager expertise into a unified document. Historically,product managers concentrated on strategy and user experience,leaving modeling iteration to machine learning (ML) engineers. Now,the two teams collaborate to refine and align the teacher model.
“How product managers work with machine learning engineers now is very different from anything we’ve done previously,” Berger said. “It’s now a blueprint for basically any AI products we do at LinkedIn.”
Key Takeaways
- Multi-Teacher Distillation: LinkedIn successfully used two teacher models – one for product policy alignment and another for click prediction – to train a more effective student model.
- Product Policy as Foundation: A detailed and iterative product policy document was crucial for guiding the LLM’s behavior.
- Cross-Functional Collaboration: Closer collaboration between product managers and ML engineers was key to success.
- Modular Training: The distillation process allowed for self-reliant iteration and improvement of different aspects of the model.
Watch the full podcast to hear more about LinkedIn’s approach.
You can also listen and subscribe to Beyond the Pilot on Spotify, Apple or wherever you get your podcasts.
Keep reading