The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer]( astronomer.io ), joins Kenten Danas to walk through what's new in the release and where each f...

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan]( otpp.com ), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution. Key...

Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Pytho...

JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline...

Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses...

In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III , Data Engineer at Cargill , joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill'...

In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path. Shrividya Hegde ,  a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should m...

We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike. Filip Kunčar , Platform Director at ShipMonk Product Development , discusses migrating from a closed source tool to Airflow, orchestrating dbt with both Cosmos and the BashOperator and using Airflow to power custo...

Buğra Öztürk , Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94. Key Takeaways: 00:00 Introduction. 03:10 Buğra has contributed to Airflow since 2022, from docs changes up to Committer and PMC member — a path he hopes inspires...

In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next. Kaxil Naik , Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu , Lead Data Engineer at Experian and Apache Airflow PMC member, join us to walk through the provider's first release and the technical dec...

Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas , Software Engineer at Jeppesen ForeFlight , to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where AI is starting to fit into that picture. Key Takeaways: 00:00 Introduction. 04:05 Airflow orchestrat...

We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines. Astronomer ’s Head of Customer Education, Marc Lamberti , and Senior Manager of Developer Relations, Kenten Danas , break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows....

Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent. In this episode, Benjamin Rogojan , Owner and Data Consultant at Seattle Data Guy , joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is stil...

Data quality is not optional when you manage credit data at scale. In this episode, Ashir Alam , Senior Data Engineer at Credit Karma , joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale safely across thousands of DAGs. We explore how governance, PII protection and orchestration come tog...

Modern Airflow isn’t just orchestration. It's a contribution.  In this episode, we explore how open source investment drives real performance gains and deeper observability. We’re joined by Christos Bisias , Open Source Software Engineer , Apache Airflow at G-Research , to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughp...

In this episode, Karan Alang , Principal Software Engineer at Versa Networks , joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the shortage of SOC analysts. Karan also outlines a novel architecture that combines Airflow, XDR, graph datab...

In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow. Egor Tarasenko , Data and AI Engineer at Ponder Labs , joins us to share how Ponder Labs customizes Airflow for education organizations using...

Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge. In this episode, we’re joined by Ethan Shalev , Data Engineer at Wix , to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development. Key...

Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability. In this episode, Carlos Daniel Puerto Niño , Senior Analytics Engineer and Data Analyst at Addi , joins us to share how Addi uses Airflow to support batch orchestration, manage organizatio...

Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules. In this episode, Andrea Bombino , Co-Founder and Head of Analytics Engineering at Astrafy , joins us to discuss how event-driven scheduling in Airflow is evolvi...

A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business. In this episode, Jaime Oliveira , Lead Data Engineer at Uphold , joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares how the team uses Airflow and dbt to support analytics, reporting and data activation while evolving...

Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of how orchestration platforms evolve over time. In this episode, Airflow best practices shaped by real-world implementation are examined. Bhavani Ravi , Independent Software Consultant and Apache Airflow Champion, shares lessons on pipeline des...

Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems. In this episode, Han Zhang joins us to explain how Conviva uses Apache Airflow as the orchestration backbone for its batch workloads, h...

Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work together. In this episode, Arun Karthik , Director, Data Solutions Engineering at Condé Nast Technology Lab , joins us to share how data engineering evolves from relational databases and ETL into distributed processing, modern orchestration...

The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards. Max Calehuff , Lead Data Engineer at Vivian Health , joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data o...