Latest / Data Engineering Concepts
Apache Airflow: Core Concepts and Building Blocks
Hi, my name is Dmitrii Polovinkin, I'm a data engineer with ~7 years of experience (check out my LinkedIn) and welcome to the first episode of Data Engineering Concepts, which is a new educational podcast about data engineering, designed to be useful when you're learning on the go.
In this episode, I explain what Apache Airflow is, what problems it solves, and when it makes sense to use it in data engineering.
We go through the main Airflow concepts, including DAGs and DAG runs, tasks and task instances, operators, sensors, hooks, connections, variables, scheduling, data intervals, XComs…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.