OpenLineage, AI Agents, and the Future of Data Engineering - with Harel Shein, Datadog
Everyone says they need data lineage. Few can say why. Harel Shein, Senior Engineering Manager at Datadog and an OpenLineage steering committee member, makes the case that AI turns lineage from a nice-to-have graph into infrastructure. We cover what OpenLineage actually is, why agents querying your warehouse need it to understand what data means, the maintainer tax of vibe-coded PRs, and Harel's wishlist: push the standard into every engine like Spark, and extend it to ML. One spec to rule them all.
⏱ CHAPTERS 0:00 Intro — What we're getting into 1:05 Rapid-fire questions 2:18 What…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.