AI Evals and Analytics Podcast

Why do most AI teams only ask "is this actually working for the business?" after it's too late? When should you start connecting evals to business impact and how do you actually do it? Using the same medical insurance chatbot from the last episode, we show how to bridge the gap between model metrics and the outcomes your leadership actually cares about. We introduce the Eval-to-Impact Stack: a thr...

What is Evaluation-driven development? When should you start building evals for your product? How to build it from scrach? Using a real-world example of a customer chatbot for a medical insurance company, we walk through the process of setting up evals from scratch: translating product requirements into quantifiable metrics, curating quality test datasets (hint: you need fewer examples than you th...

What are the skills required for AI evals? Why data scientists have a natural advantage in AI evals?  Evaluating AI isn’t just about "vibe coding" with an AI assistant. It actually requires a solid foundation in statistics for picking sample sizes and coding to build your own testing frameworks. Data scientists have a huge head start here because they are already pros at designing metrics and...