Shows / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage…
Episodes
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating cloud data migrations at HCL Tech
Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding self-service Dags with GitHub Actions at Lyft
At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating cost-efficient data pipelines at Insignis
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIScaling to 1,000+ Airflow Dags at Descartes / Idelic
Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating local LLMs for sensitive data with Airflow
Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIImproving security in Airflow 3
Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating data across 30 companies at itti
Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding self-healing Airflow pipelines at ATC Drivetrain
Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating strictly sequential ETL pipelines at Synechron
Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIManaging financial datasets with Airflow at Wise
Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIManaging travel platform data with Airflow at Headout
Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIHow Airflow orchestration decisions impact Spark performance
Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIUsing Airflow for diverse client projects at Accion Labs
When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIConsolidating legacy job schedulers with Airflow at Trading Technologies
Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating AI video intelligence and evaluation pipelines at Firework
How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating Retail Data Pipelines at Saks Global
Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIWhat's New in Apache Airflow® 3.3
Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIRunning Airflow 3 in a regulated environment at OTPP
Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIManaging a Customer Analytics Platform with Airflow at Skimlinks
Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding a custom Tableau provider for Airflow at JLR
JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIEnhancing DAGs for Data Processing with William Orgertrice III at Cargill
In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIGetting Into Data Engineering with Shrividya Hegde, Data and AI Engineer
In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOrchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development
We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding Airflow CTL with Buğra Öztürk at Mollie
Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIIntroducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next. Kaxil Naik…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIIntroducing Airflow 3.2
We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines. Astronomer’s Head of Customer Education, Marc Lamberti, and Senior…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIReflections on a Decade of Data Engineering at Seattle Data Guy
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent. In this episode…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIManaging Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
Data quality is not optional when you manage credit data at scale. In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOpen Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
Modern Airflow isn’t just orchestration. It's a contribution. In this episode, we explore how open source investment drives real performance gains and…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIAutomating Threat Intelligence Using Airflow with Karan Alang
In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIUsing Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko
In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIScaling Airflow at Wix for Analytics and AI with Ethan Shalev
Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIUsing Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño
Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino
Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIUphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira
A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business. In this episode, Jaime…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIModern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi
Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIInside Conviva’s Decision To Power Its Data Platform With Airflow with Han Zhang
Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIWhy Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik
Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIThe Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff
The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIScaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers
The evolution of Intercom’s data infrastructure reveals how a well-built orchestration system can scale to serve global needs. With thousands of DAGs powering…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIHow Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie
Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability. In…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin
The use of Apache Airflow in financial services demands a balance between innovation and compliance. Agile Engine’s approach to orchestration showcases how…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIHow Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar
The life sciences industry relies on data accuracy, regulatory insight and quality intelligence. Building a unified system that keeps these elements aligned is…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIHow Airflow and AI Power Investigative Journalism at the Financial Times with Zdravko Hvarlingov
The Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data. In this episode, Zdravko Hvarlingov, Senior…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIInside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo
The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines. In this episode, we feature Oscar Ligthart…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AITransforming Data Pipelines at XENA Intelligence with Naseem Shah
The shift from simple cron jobs to orchestrated AI-powered workflows is reshaping how startups scale. For a small team, these transitions come with unique…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIScaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith
Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIScaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya
PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding a Unified Data Platform at Pattern with William Graham
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIHow Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty
The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIOvercoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov
The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah
StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIBuilding the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak
The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIScaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille
Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable. In this episode, Sébastien Crocquevieille…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIHow Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye
Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency. In this episode, we are joined by…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIInside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler
The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIInside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero
Managing modern data platforms means navigating a web of complex infrastructure, competing team needs and evolving security standards. For data teams to truly…
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AIThe Future of Airflow Telemetry with Bolke de Bruin
Telemetry has the potential to guide the future of Airflow, but only if it’s implemented transparently and with community trust. In this episode, we’re joined…