Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Scaling to 1,000+ Airflow Dags at Descartes / Idelic
Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI.
Key Takeaways:
00:00 Introduction.
01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.