The Experimentation Edge

Summary Vamsi Krishna Kunapuli, Director of Analytics at New York Red Bulls, leads a small team that supports marketing, ticket sales and partnerships for one of the original MLS clubs and its 25,000 seat stadium. He shares how an experiment across five to six seasons changed the club's North Star from average ticket price to gross margin. Host Ashley Stirrup and Vamsi also discuss why a losing te...

Summary Shiva Manjunath, Lead Web Strategist at Motive and host of the From A to B podcast, joins Ashley Stirrup to explain why CRO works best as a mindset every team carries rather than a role one team owns. He shares how building bridges to sales and customer service turns everyday complaints into test ideas. The conversation covers why customers who ask for price are often asking for value, how...

Summary Mary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely.     Mary...

Summary Arv Ramchandani, Senior Product Manager for Experimentation & Analytics at Walmart Connect, runs ad experiments on pages owned by other teams, from search to the conversational AI shopping agents, each with its own primary metric. He explains how his team negotiates a degradation threshold with every page owner, why relevance and latency sit next to revenue as guardrails, and how month...

Summary Sapna Kashyap, CRO Lead at VidaXL, runs the experimentation program for an e-commerce company operating in more than 30 countries, where product, development and business teams all validate their decisions with an A/B test first. She explains the biggest winner of her career: B2B buyers were abandoning a registration page, a survey showed they could not see what the subscription included,...

Summary Michael Mathews, Director of Data and AI Product at FanDuel, runs the teams behind the auto researcher, an agent that takes a data set, an objective function and the production model, then proposes, writes and scores about 300 strategies in an hour for roughly $300 in tokens, handing back the winner as readable code. Host Ashley Stirrup gets the three things that surprised FanDuel as it sc...

Summary Grubhub does not run much classic A/B testing. Michal Lenik, Associate Director of Product Design at Grubhub, explains what the company does instead to make sure its big product bets win before they reach millions of users. She joins Ashley Stirrup to walk through Grubhub's playbook: small cohort rollouts that protect core order metrics, merchant round tables, AI built clickable prototypes...

Summary Dorothy Crepin runs the A/B testing program at MilliporeSigma, the life sciences business of Merck KGaA, where one global catalog has to serve three very different visitors: scientists hunting for a specific instrument, procurement buyers working off a list someone else wrote, and students doing research with no purchasing power at all. Add a regulated industry where what you can sell chan...

Summary Gaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of ex...

Summary ServiceNow runs its own platform on itself. Ashraf Karim, Senior Vice President, Connected Customer Experience, owns everything that happens after a customer buys, and her team is Customer Zero, testing every product and capability internally before it ever reaches a customer. She joins Ashley Stirrup to unpack what that vantage point teaches you about experimenting with AI. The headline l...

Summary Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not. The centerpiece is a test that looked like a win. At Expedia, an account sign-up...

Will Guyeskey is Director of Digital Product at GoPro, where his team owns the e-commerce side of gopro.com. Before GoPro he ran personalization at Gap and cut his teeth at Brooks Bell, testing for brands like Barnes & Noble, Under Armour and Ralph Lauren. In this episode of The Experimentation Edge, he tells Ashley Stirrup why knowing your customer is the through line of every good test progr...

Summary What changes when you take a mature B2C experimentation practice and apply it to a B2B storefront? Anuradha Tempe, Lead Product Manager at Samsung Electronics America, joins host Ashley Stirrup to share what she learned managing both sides of Samsung.com. She explains why B2B is not a sidekick to B2C, how a single bulk order can push a test into a false positive unless you normalize the da...

Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Joe Yevoli, Director of Growth at Charlie Health, a virtual intensive outpatient program that sits between weekly therapy and hospitalization. Joe explains how removing a page from Charlie Health's intake form produced a winning test that created a new bottleneck further down the funnel, why Teachers Pay Teachers cut ab...

Summary What do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational da...

Summary How do you keep 1,000 experiments a year pointed at one North Star? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Mi Tian, Head of Applied Science at Zalando, about running experimentation inside a central economics org that reports to the CFO. Mi shares how Zalando balances safe confirmatory tests with game-changing bets, how the team measured the discovery f...

Summary Rich Liebling, senior director of engineering at Learneo, joins host Ashley Stirrup to explain the practice that came out of growing Shop It To Me from 50,000 subscribers to one million in nine months: test your hypothesis, and test its opposite. Rich covers the page where cutting text lost and adding text won, why the inverse wins more often than teams expect, and why small focused tests...

Summary What separates a valid A/B test from an expensive guess? Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning — the bank-owned consortium that fights payment fraud and operates Zelle, which processed a trillion dollars last year — joins host Ashley Stirrup to share the experimentation playbook she built leading storefront analytics at Wayfair. She walks through the...

Summary On this episode of The Experimentation Edge, Ashley Stirrup talks with Shan Huang, data scientist on the central experimentation team at Supercell, the Helsinki mobile game company behind Clash of Clans, Clash Royale, Brawl Stars, Hay Day, and Boom Beach. Shan explains how a famously decentralized, creative-first company with 300 million monthly active users runs fewer than 100 A/B tests a...

Summary How do you run an experimentation program when classic A/B testing is off the table? Ben Schein, Director of Product Management at Clover, joins host Ashley Stirrup to explain how a platform serving 300,000+ merchants and processing billions of dollars daily proves every feature through pilots and ground-level testing before rollout, why uncertainty and downside — not feature visibility —...

Summary What happens when the most logical feature you've ever built has zero impact? In this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Edd Saunders, product experimentation manager at JobLeads, to unpack the pizza personalization experiment that cut an ordering flow from 22 clicks to 5 and changed nothing. Edd shares the problem mapping framework...

Summary What does a winning A/B test mean when your website serves 1,100 dentist owned offices? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arie Polycarpou, Senior Manager of Test and Learn at Aspen Dental, about building experimentation programs at Kohl's, Marriott, Total Wine, and now the largest company in The Aspen Group's healthcare retail portfolio. Arie expla...

Summary What if you could A/B test on customers who don't exist before spending a single live impression? Erika Dunn, assistant director of data science at Principal Financial Group, joins host Ashley Stirrup, CMO at GrowthBook, to share how she built synthetic digital audiences entirely in house: profiles shaped by real data that rank content by likelihood of engagement, whose first live A/B test...

Summary How does a major bank scale experimentation when even one percent of customers missing an experience is unacceptable? Vijay Lal, Lead Product Manager for Experimentation at US Bank, joins host Ashley Stirrup, CMO at GrowthBook, to share how his team made their experimentation platform self serve for non technical marketers, how a login widget experiment led to a two second fallback that ac...

Summary Luis Trindade, Principal Product Manager of Experimentation at Farfetch, joins host Ashley Stirrup to explain how one of the world's largest luxury marketplaces built its own experimentation platform and the culture around it. Luis covers the move from a hybrid setup with an external testing vendor to Fabs 2.0, the in house system where a feature toggle is the single entry point for every...