Inference & Intelligence Lab

EP4: Two Ways to Measure Demand, and When the Market Lens Matters "Demand" can mean more than one thing. In day-to-day product analytics, it’s the chart right in front of us: sessions, searches, transactions, and conversion rates. This is Funnel Demand —the activity on our own surface. But there is a second, critical lens worth holding alongside it: Market Demand . In this episode of Inf...

Across the tech industry, many of the most valuable features rely on voluntary adoption. A traveler chooses whether to join a loyalty program, or a marketplace seller decides whether to opt into a smart-pricing tool. Because you cannot force users to adopt a feature, standard A/B tests leave teams with a diluted, flat topline result. Genuinely great features get prematurely killed simply because t...

EP2: Build the Camera — Why Measurement Design Trumps Statistical Testing Running a statistical test is simply pressing the shutter. But designing the measurement system? That is building the camera . In this episode, we challenge the industry’s obsession with "which test to run" and shift the focus to what actually matters: whether your metric captures meaningful change in user behavior...

No Interference, No Ambiguity: The SUTVA Assumption Your randomized experiment is clean. The groups are balanced and comparable. The p-value is significant. But behind the scenes, the treatment is leaking. User A shared their referral link with User B in the control group, and suddenly your "independent" comparison is contaminated. Welcome to the most common—and most ignored—failure point in exper...

No Overlap, No Answer: The Positivity Assumption A causal effect can only be estimated where a comparison is actually possible. Imagine evaluating a loyalty program where every enterprise customer is already enrolled—leaving you with no unenrolled counterparts to compare against. This is a violation of Positivity . While exchangeability requires that groups are comparable, positivity requires that...

Comparing Apples to Apples: The Exchangeability Assumption Your dashboard flags a troubling trend: users who contacted customer support have a 40% higher churn rate than those who didn’t. The immediate takeaway seems obvious—support is failing. But is it? Or did those customers contact support because something had already gone wrong? In this episode, we tackle the heart of the "Bad Comparison" pr...

You can never see the data you need most to make a decision. 📉 It sounds counterintuitive, but the core of Causal Inference isn't just math—it's imagination. 🌌 Most Data Science focuses on predicting the future based on what happened in the past. But Causal Inference asks a much harder question: What would have happened if we had acted differently? In Part 4 of my series, "Causal Inference from...

Headline: Why your "Perfect" Models are failing the Boardroom. For months, the team worked on the model. The AUC was 0.92. The validation sets were clean. But when you shipped it to the real world? Nothing happened. The metrics didn't move. The business didn't grow. Well, you didn't have a "data" problem. You had a Causality problem. You just hit the Data Validity Cliff —the point where your high-...

The Infinite Data Trap: Why More Data Won't Save Your Causal Models You have petabytes of user data. Your model has 99% validation accuracy. But when you ask it, "What happens if we change our strategy?" , it gives you an answer that is confidently wrong. Welcome to the Infinite Data Trap. In this episode, we reveal why "big data" is useless for decision-making without a critical, often neglected...

The Ranking Trap: Why PLR Fails with Multiple Treatments You’re testing four different promotional strategies—a discount, free shipping, BOGO, and loyalty points. You run a Partially Linear Regression (PLR) , get a clean table of coefficients, and rank them to decide your next big investment. There’s just one problem: Your ranking is likely a lie. In this episode, we dive into a dangerous "trap" i...

The Recipe for Causal Truth: Estimand, Estimator, and Estimate You have a dataset, a model, and a final number. But can you explain—with precision—what that number actually represents? In the world of causal inference, precision is everything. We often use terms like "the result" or "the algorithm," but failing to distinguish between the What , the How , and the Result is where most analytical err...