Snacks Weekly on Data Science

In this episode, we’ll explore the unique security challenges posed by agentic AI systems and why embedding trust and safety into these systems from the ground up is critical. We’ll review a few key ingredients for building a secure agentic AI future. For more details, you can refer to the blog, linked here for your reference: https://medium.com/intuit-engineering/owasp-dishes-out-key-ingredients-...

In this episode, we explore how Oda scaled its A/B testing practices alongside its business growth, focusing not only on building a technical platform but also on creating a culture that supports high-quality, reliable experimentation. For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/oda-product-tech/odas-online-experimentation-journe...

In this episode, we explore how Booking.com tackled the challenge of predicting reservation cancellations in an ever-changing travel landscape. By shifting from a traditional classification model to a survival modeling approach, the team developed more time-sensitive and flexible predictions that better support their business needs and decision-making. For more details, you can refer to their publ...

In this episode, we will explore how Workday tackle the challenge of measuring the cost of GenAI features. We looked at why LLM-powered features require a new approach to cost tracking, and how the team engineered a telemetry-driven system to make those costs visible, actionable, and fair. For more details, you can refer to their published tech blog, linked here for your reference: https://medium....

In this episode, we will discuss how Expedia’s recommendation system is designed to handle both standard destination searches and property-specific searches. While traditional ranking models optimize for broad search behavior, Expedia’s team refines their learning-to-rank approach by integrating property similarity, ensuring travelers get recommendations that align with their intent. For more deta...

In this episode, we will explore why evaluating LLM-based chatbots is critical for businesses, the limitations of traditional evaluation methods, and what could be a good robust evaluation framework covering both search performance and LLM-specific metrics.  For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/data-science-at-microsoft/ev...

In this episode, we will explore how The New York Times balances algorithmic recommendations with editorial judgment. We discuss their business challenge, examine their hybrid content recommendation system, and look at refinements designed to improve the reader experience. For more details, you can refer to their published tech blog, linked here for your reference: https://open.nytimes.com/how-the...

In this episode, we will explore how Airbnb upgraded its conversational AI system, leveraging LLMs in a controlled and predictable way. We will first examine their business needs, highlighting why traditional chatbot-based workflows were no longer sufficient. Then, we will break down their technical solution, which combines structured workflows with AI-powered reasoning, context management, and a...

In this episode, we will explore the importance of the Large Language Model (LLM) and the forces shaping the LLM economy: competition among AI giants, GPU scarcity, and tokens as the new currency. These dynamics drive innovation and challenge businesses to optimize resources and costs strategically. For more details, you can refer to their published tech blog, linked here for your reference: https...

In this episode, we will explore why Klaviyo developed its global holdout group feature and how its engineering team overcame the technical challenges. This feature helps Klaviyo’s customers run fair and unbiased experiments across multiple marketing channels, ultimately enhancing the accuracy of their marketing performance insights. For more details, you can refer to their published tech blog, li...

In this episode, we will explore why code reviews are critical for a fast-growing marketplace like Faire and the challenges that come with scaling them manually as the engineering team expands. We’ll dive into how Large Language Models (LLMs) offer a game-changing solution—automating code reviews by providing instant, context-aware feedback, enforcing coding best practices, and integrating seamles...

In this episode, we will explore how Instacart uses data science to optimize its incentive promotions. We will discuss the business challenge, introduce the concept of surrogate indices, and walk through the step-by-step process of building and applying one. For more details, you can refer to their published tech blog, linked here for your reference: https://tech.instacart.com/instacarts-economics...

In this episode, we will explore Meta’s AI scaling challenges and how the company leverages productivity and efficiency to optimize its computing power. We also discuss how analytics insights help identify active levers to improve AI development. For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/@AnalyticsAtMeta/innovation-demands-comp...

In this episode, we will explore how Coupang integrates Large Language Models (LLMs) to enhance its machine learning ecosystem. We'll break down Coupang’s business model, key machine learning categories, and the role of Foundation Models in improving efficiency and accuracy. Additionally, we'll walk through the LLM development lifecycle, discuss critical infrastructure decisions, and exami...

In this episode, we will explore how Expedia ranks lodging options to optimize both customer experience and business objectives. We will discuss the business problem—how ranking impacts Expedia’s success and the challenges of hotel recommendations. Then, we will break down the data science solution, covering the objective functions, features/signals, machine learning architectures, and the evaluat...

In this episode, we will explore Thumbtack’s business model and the importance of recommendations in helping professionals grow their businesses. We will share the team’s machine learning solution architecture, which involved building two sub-models and deploying them using offline inference to meet the business needs cost-efficiently. This quick and nimble approach to machine learning demonstrate...

In this episode, we will explore Uber’s Model Excellence Scores (MES) framework, a robust system designed to maintain and enhance the quality of machine learning models at scale. We will unpack its core components—indicators, objectives, and agreements—and explain how they work together to ensure model reliability and performance. This framework enables Uber’s ML ecosystem to operate seamlessly an...

In this episode, we’ll explore RazorPay, its business, and the critical role customer service plays in its success. We'll dive into how RazorPay revolutionized customer ticket categorization using generative AI. By replacing customer-selected categories with an AI-driven system, they enabled automatic interpretation of ticket details. This approach incorporated pre-processing, prompt engineeri...

In this episode, we will explore the foundational concepts of causal analysis, focusing on its two main pillars: causal discovery and causal inference. We will discuss the types of questions these pillars aim to answer and provide illustrations of related methodologies to better clarify their concepts. For more details, you can refer to their published tech blog, linked here for your reference:  h...

In this episode, we will explore how Noom developed a customized AI assistant solution using generative AI models. By incorporating key components such as prompt engineering, dynamic personalization, and integration with their knowledge base, the team created a smarter, more reliable assistant to support their customers. This approach offers valuable insights for anyone looking to build tailored g...

In this episode, we will explore Airbnb’s approach to personal data classification. We will begin by introducing the importance of protecting personal data. Next, we will examine the technical solution, where the three pillars—Catalog, Detection, and Reconciliation—form the backbone of their workflow. Finally, we will discuss how performance metrics are implemented to ensure the system remains rel...

In this New Year’s episode, I reflect on one of my earliest data science projects. My son will “interview” me about my experiences during that time. The key project discussed in this episode is also featured in a LinkedIn article I wrote a few years ago (⁠ https://www.linkedin.com/pulse/my-first-data-science-project-pan-wu/⁠⁠ ). I’d love for you to check it out, leave a 5-star review, and subscrib...

In this episode, we will explore Netflix’s approach to content recommendation using contextual bandits and reward engineering. We will also discuss the important role of proxy reward functions and how Netflix leverages offline machine learning models to predict delayed customer feedback, enabling them to continuously improve their recommendation engine and deliver a more personalized viewing exper...

In this episode, we will introduce the concept of paid marketing. We’ll explore how Instacart’s team developed an adaptive experimentation framework that continuously balances exploration and exploitation, ultimately maximizing marketing efficiency. For more details, you can refer to their published tech blog, linked here for your reference:  https://tech.instacart.com/bandits-for-marketing-optimi...

In this episode, we will introduce what CVS Health is and the importance of product recommendations for their business needs. We will delve into how their data science team leveraged advanced technologies, including Graph Neural Networks and generative AI models like GPT-4, to develop a prototype system to make product bundle recommendations. For more details, you can refer to their published tech...