Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille
Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.
In this episode, Sébastien Crocquevieille, Data Engineer at Numberly, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.
Key Takeaways:
00:00 Introduction.
02:13 Overview of the company’s operations and global presence.
04:00 The tech stack and structure of the data engineering team.
04:24 Running nearly 2,000 DAGs in production using…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.