Snacks Weekly on Data Science

In this episode, we talk about how Dropbox improved the search ranking performance of one of its key products. The team used a small amount of human judgment to calibrate an LLM, then used the LLM to generate training labels at roughly 100 times the human effort and train a more efficient production ranking model. With this approach, the LLM can scale human judgment, while the human reference set...

In this episode, we discuss how Pinterest moved from three independently evolving ads engagement models toward a unified modeling framework. The team started with a simple merged model, then progressively refined it by combining complementary modeling components, adding surface-specific calibration and multi-task learning, preserving surface-specific model exports, and optimizing both model comput...

In this episode, we talk about Lyft’s need to predict whether riders will convert across a wide range of highly specific situations, where the available data becomes increasingly sparse as the situations become more specific. Instead of relying entirely on a global model or trusting noisy local estimates, Lyft built a hierarchical Bayesian framework where specific models can borrow information fro...

In this episode, we discuss the budget allocation problem faced by Atlassian when trying to optimize its marketing investment decisions across many channels, products, and geographic markets. The company solved this with a two-stage system: a global payout model that predicts how business outcomes change with spend, followed by a constrained optimizer that finds the best allocation across the enti...

In this episode, we discuss how LinkedIn moves beyond keyword matching to find candidates based on actual qualification fit. The team first created a scalable relevance definition using LLM-based supervision, distilled that knowledge into customized embeddings, and then designed the retrieval, ranking, infrastructure, and evaluation system around that signal. The result was a search system that im...

In this episode, we discuss how Meta measures the business value of shared ML infrastructure when that value appears indirectly through downstream model launches. Instead of relying on adoption, usage, or traditional infrastructure metrics, the team built an observational causal framework around individual model launches. By comparing feasible adopters with comparable non-adopters and controlling...

In this episode, we discuss two LLM-assisted pipelines for turning unstructured text into a structured taxonomy with minimal manual annotation. The bottom-up approach lets semantic representations and clustering discover categories from the data, making it useful for open-ended exploration and relatively efficient at scale. The top-down approach starts with LLM-generated categories and recursively...

In this episode, we talk about how Instacart identified a limitation of a standard taxonomy: it could classify products, but it couldn't connect them the way real shoppers do. The team developed a solution around Semantic IDs, which compress product embeddings into hierarchical discrete codes, while contrastive learning and taxonomy-based supervision make those codes more semantically meaningf...

In this episode, we discuss how Faire addresses a fundamental bottleneck in machine learning development: there are more potential experiments than human scientists can realistically run. They developed Autoscience, giving an AI agent a structured environment where it can plan, execute, evaluate, remember, and continuously refine experiments. What makes the approach effective is not the agent alon...

In this episode, we discuss how DoorDash built a reliable metadata generation platform that could operate across millions of constantly changing menu items. The solution combined multimodal generation, automated evaluation through LLM juries, optimization loops for improving model behavior, and distributed infrastructure for cost-efficient scaling. By transforming unstructured menu information int...

In this episode, we explore how The New York Times built a real-time decision system that estimates the causal impact of different paywall actions, optimizes those decisions based on business objectives, and continuously measures performance using long-term business outcomes. The result is a more flexible and personalized subscription experience that balances immediate conversions with long-term g...

In this episode, we explore how Airbnb built a destination recommendation model to predict the places users are most likely to travel to next. The model predicts travel intent using sequences of user actions, carefully designed training examples for both active and dormant users, and multi-task learning to capture geographic relationships. Together, these design choices led to more relevant recomm...

In this episode, we discuss how Meta built an AI system with persistent memory, structured organization, secure access to enterprise tools, and reusable workflows. The result was an AI assistant that could participate in ongoing work rather than isolated chat sessions. By carefully organizing knowledge and retrieving only the context relevant to the current task, the system enabled the assistant t...

In this episode, we explore how Booking.com scaled experimentation quality across the organization. Instead of relying on strict enforcement, the team chose to optimize for flexibility while building the supporting processes to foster a strong experimentation culture, defining a clear framework for measuring experimentation quality, and integrating those practices directly into the experimentation...

In this episode, we discuss how Netflix built an LLM-based evaluation framework for creative writing quality without losing the nuance that makes expert review valuable. They achieved this by creating a golden dataset calibrated by expert writers, splitting evaluation into specialized judges for each quality criterion, and applying targeted techniques—tiered reasoning, consensus sampling, and mult...

In this episode, we explore how Glassdoor customized the Two-Tower model for its specific production environment. The gap between generic semantic embeddings and behavior-driven embeddings proved significant, and closing it required thoughtful decisions regarding training data, negative sampling, and auxiliary learning signals. For more details, you can refer to their published tech blog, linked h...

In this episode, we talk about why Lyft needed to understand which rider experiences drive long-term retention. We discuss how raw observational data can produce misleading conclusions because of confounders such as geography, and how Lyft's Rider Experience Score addresses this challenge using Augmented Inverse Propensity Weighting. This doubly robust causal inference method adjusts for those...

In this episode, we explore how Flipkart rethought ranking in its "Similar Products" widget. Instead of relying on fixed ad slots, the team built a unified ranking system that enables sponsored and organic products to compete on the same playing field. By combining a unified ranking model with a score normalization layer, the system ranks content based on contextual merit rather than pre...

In this episode, we explore how Pinterest rethought its ads serving infrastructure to support more expressive ranking models beyond the traditional two-tower architecture. Their solution was not simply to build a better model, but to redesign the serving stack so that ranking logic and model inference work more closely together, allowing lightweight ranking models to capture richer signals while o...

In this episode, we explore how Meta addressed a fundamental limitation of applying LLMs to enterprise analytics. While modern models are highly capable of generating code and SQL, they still fall short when it comes to organizational context and deep domain understanding — both of which are essential for reliable, real-world analytical work. Meta’s approach focuses on closing this gap through sha...

In this episode, we explore how CVS Health tackled a classic large-scale engineering operations problem: keeping application runbooks accurate, complete, and continuously compliant across hundreds of internal systems. To solve this, the team built an LLM-based automated auditing pipeline. The result is a lightweight but effective system that turns documentation compliance from a periodic manual ef...

In this episode, we explore how Uber tackled one of the most operationally challenging parts of its marketplace: airport pickups. Unlike normal city rides, airport demand is highly bursty, queue-driven, and heavily influenced by flight schedules, delays, and driver positioning decisions. To solve this, Uber built a coordinated forecasting system composed of three specialized models: Estimated Time...

In this episode, we discuss Expedia’s need to evaluate many ranking ideas quickly so that A/B testing would not become a bottleneck. We explore their three-stage experimentation funnel: backtesting to remove weak candidates, interleaving to rapidly compare promising ones, and A/B testing to validate final business impact. What made this design approach effective was using the right evaluation tool...

In this episode, we look at how companies deal with large volumes of unstructured text and why traditional clustering methods often fall short at scale. We explore two LLM-powered approaches shared by data scientists from Microsoft: a bottom-up pipeline that builds structure from data using embeddings and clustering, and a top-down pipeline that starts with LLM-generated categories and refines the...

In this episode, we discuss a classic scaling problem in fraud and risk operations: too much manual review, inconsistent judgments, and growing complexity. We explore the team’s solution, Bumblebee, a multi-agent AI architecture that separates planning, evidence gathering, and analysis into specialized roles, enabling a robust and scalable system to solve the problem. For more details, you can ref...