Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.