The Experimentation Edge

Summary Gaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of ex...

Summary ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer. She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline l...

Summary Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not. The centerpiece is a test that looked like a win. At Expedia, an account sign-up...

Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test progr...

Summary What changes when you take a mature B2C experimentation practice and apply it to a B2B storefront? Anuradha Tempe, Lead Product Manager at Samsung Electronics America, joins host Ashley Stirrup to share what she learned managing both sides of Samsung.com. She explains why B2B is not a sidekick to B2C, how a single bulk order can push a test into a false positive unless you normalize the da...

Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Joe Yevoli, Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between weekly therapy and hospitalization. Joe explains how removing a page from Charlie Health's intake form produced a winning test that created a new bottleneck further down the funnel, why Teachers Pay Teachers cut ab...

Summary What do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational da...

Summary How do you keep 1,000 experiments a year pointed at one North Star? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Mi Tian, Head of Applied Science at Zalando, about running experimentation inside a central economics org that reports to the CFO. Mi shares how Zalando balances safe confirmatory tests with game-changing bets, how the team measured the discovery f...

Summary Rich Liebling, senior director of engineering at Learneo, joins host Ashley Stirrup to explain the practice that came out of growing Shop It To Me from 50,000 subscribers to one million in nine months: test your hypothesis, and test its opposite. Rich covers the page where cutting text lost and adding text won, why the inverse wins more often than teams expect, and why small focused tests...

Summary What separates a valid A/B test from an expensive guess? Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning — the bank-owned consortium that fights payment fraud and operates Zelle, which processed a trillion dollars last year — joins host Ashley Stirrup to share the experimentation playbook she built leading storefront analytics at Wayfair. She walks through the...

Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Shan Huang, data scientist on the central experimentation team at Supercell, the Helsinki mobile game company behind Clash of Clans, Clash Royale, Brawl Stars, Hay Day, and Boom Beach. Shan explains how a famously decentralized, creative-first company with 300 million monthly active users runs fewer than 100 A/B tests a...

Summary How do you run an experimentation program when classic A/B testing is off the table? Ben Schein, Director of Product Management at Clover, joins host Ashley Stirrup to explain how a platform serving 300,000+ merchants and processing billions of dollars daily proves every feature through pilots and ground-level testing before rollout, why uncertainty and downside — not feature visibility —...

Summary What happens when the most logical feature you've ever built has zero impact? In this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Edd Saunders, product experimentation manager at JobLeads, to unpack the pizza personalization experiment that cut an ordering flow from 22 clicks to 5 and changed nothing. Edd shares the problem mapping framework...

Summary What does a winning A/B test mean when your website serves 1,100 dentist owned offices? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arie Polycarpou, Senior Manager of Test and Learn at Aspen Dental, about building experimentation programs at Kohl's, Marriott, Total Wine, and now the largest company in The Aspen Group's healthcare retail portfolio. Arie expla...

Summary What if you could A/B test on customers who don't exist before spending a single live impression? Erika Dunn, assistant director of data science at Principal Financial Group, joins host Ashley Stirrup, CMO at GrowthBook, to share how she built synthetic digital audiences entirely in house: profiles shaped by real data that rank content by likelihood of engagement, whose first live A/B test...

Summary How does a major bank scale experimentation when even one percent of customers missing an experience is unacceptable? Vijay Lal, Lead Product Manager for Experimentation at US Bank, joins host Ashley Stirrup, CMO at GrowthBook, to share how his team made their experimentation platform self serve for non technical marketers, how a login widget experiment led to a two second fallback that ac...

Summary Luis Trindade, Principal Product Manager of Experimentation at Farfetch, joins host Ashley Stirrup to explain how one of the world's largest luxury marketplaces built its own experimentation platform and the culture around it. Luis covers the move from a hybrid setup with an external testing vendor to Fabs 2.0, the in house system where a feature toggle is the single entry point for every...

Summary In this episode of The Experimentation Edge, host Ashley Stirrup talks with Fabian Hans, founder and behavioral psychologist at Cogniteer, a consultancy that helps enterprises build in-house experimentation programs and raise both test velocity and win rate. Drawing on fifteen years in conversion rate optimization, Fabian explains why mass-producing the same A/B tests across clients quietl...

Summary On this episode of The Experimentation Edge, host Ashley Stirrup talks with Pedro Tabacof, Principal Machine Learning Scientist at Fin (formerly Intercom), about how one of the most advanced AI customer support agents in the world is built on relentless experimentation. Pedro explains why unit tests don't work on non-deterministic AI, how Fin runs up to two dozen concurrent A/B tests pulli...

Summary In this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with James Falzone, Director of Product Management at Kargo, to unpack how a high scale ad tech marketplace turns failure into its biggest advantage. James explains how Kargo connects advertisers to publishers through real time auctions that resolve in milliseconds across up to 10 billion ad requ...

Summary In this episode of The Experimentation Edge, host Ashley Stirrup talks with Danielle Oleen, Director of E-commerce at Box, about what it really takes to build a culture of experimentation inside a B2B company. Drawing on 15 years across B2C and B2B at Wayfair, Drizly, Zoom, and now Box, Danielle explains why experimentation belongs to every product team and not just e-commerce, walks throu...

Summary Dan Layfield, Director of Product Management at Diligent, joins host Ashley Stirrup on The Experimentation Edge to trace what fifteen years of A/B testing across Codecademy, Uber Eats, and the Fortune 1000 boardroom actually taught him. He breaks down the Codecademy trial-model rebuild that took four months and several rounds to deliver a 35% conversion lift, why moving on from a losing ex...

Summary In this episode of The Experimentation Edge, GrowthBook CMO Ashley Stirrup sits down with Nick Beyler, data science manager at Stitch Fix, where he leads the decision and insights team and owns the company's internal experimentation platform. Nick shares why the metric he most wants is the one he can't measure yet, a North Star that predicts a client's long-term value from their earliest b...

Summary Amir Moghaddam, Director of Software Engineering at Expedia Group, joins host Ashley Stirrup on The Experimentation Edge to make the case that measurement is not a reporting step but a gate: what you cannot measure, you cannot ship. Drawing on nearly four years at DoorDash and his current work leading Expedia's air booking platform, Amir explains why he refuses to label experiments winners...

Summary In this episode of The Experimentation Edge, host Ashley Stirrup sits down with Raunak Kumar, senior manager of GTM analytics at Fin (formerly Intercom), to unpack how experimentation actually works when the data is messy and the traffic is thin. Drawing on nearly 12 years in marketing analytics across Atlassian, Stripe, and Fin, Raunak explains how AI tools like Claude Code have collapsed...