Latest / The Data Splash / OpenLineage, AI Agents, and the Future of Data Engineering - with Harel Shein, Datadog
Transcript
- Ido Bronstein: Welcome everyone to the Data Splash podcast brought to you by Upriver If you're a data engineer, a data leader, or just someone trying to make sense of what the hell is going on with AI and data right now, you are in the right place. Here with me today is Harel Shein, senior engineering manager at Datadog. And we are going to talk about OpenLineage, of course. Harel is part of Datadog's product engineering team. and works on data observability products. He previously led data engineering and R &D teams at WeWork Astronomer, and he also in the technical steering committee member of OpenLineage and also committed himself a lot of code. Now, say hi to Harel ⁓ Harel: Hi ⁓ Hi Ido. ⁓ I wouldn't say commit a lot of code, some code. Ido Bronstein: Some code, some code. So before we are diving in, we start with our transition, the data splash. 30 seconds, no time to think, just instincts. So let's in. most underrated skill in data. Harel: of getting people aligned on definitions. Ido Bronstein: Dream vacation destination? Harel: Cook Islands. Ido Bronstein: What is going to break most of the data pipelines in your environment tomorrow? Harel: anything you can think of. Ido Bronstein: One tool you cannot live without. Harel: last few months cloud code for sure Ido Bronstein: Cloud code. So he's committing a lot of code, cloud codes. You are doing nothing, cloud codes commit all the things to OpenLinage. Harel: Exactly, I do zero work. Ido Bronstein: zero work, exactly. So for the part of the audience, the very small part of the audience that does not know anything about OpenLineage and basically about like your work there, can you give us a bit overview about the open source, about the project? Harel: Yeah, yeah, of course. So OpenLineage is basically an open standard ⁓ for emitting lineage metadata from ⁓ pipelines where work is actually happening. ⁓ So ⁓ it's basically collection of a few things. So one is a and that's really the gist of it. It's like a JSON specification that talks about ⁓ how you ⁓ reason about job, how you reason about a job run. and the inputs and outputs of that job. And as well, it includes a bunch of clients in various languages, like Python, Go, Java, for emitting open-lineage events, as well as a bunch of out-of-the-box integrations for platforms like Spark, like DBT. For Airflow, for example, the implementation is actually built into Apache Airflow itself, and that's the case for a few. other projects who basically decided to build an OpenLineage ⁓ into their open source project. ⁓ And it is... ⁓ Ido Bronstein: Interesting who is the director in Astronomer decided on this architecture choice Harel: So it was, listen, it was an Apache Airflow decision. There was an Airflow improvement plan ⁓ spec that came out and actually it was not a trivial decision to make to be perfectly honest. Like it needed community consensus, is what I like about open source work. Kind of like it pushes you to do things just the right way ⁓ because you need everyone to align on that. thing. yeah, so just to wrap up on like OpenLineage itself, it's a, we basically incorporated under the Linux Foundation data and AI subdivision or project that you don't remember. ⁓ And, and why it matters is no one wants to write boilerplate integration code, even if it's cheap to do today with Claude as we've just established. ⁓ It's not really cheap, right? Like you still need to maintain it, it still needs to be evolved. Still you need to write point to point solutions for every single thing. Isn't it nicer to just have a spec that does this and everyone contributes to? So that's really what OpenLineage is about. Ido Bronstein: And before we are jumping into OpenLineage and how it's changing the way we work in data engineering, I'd love to hear how AI affect managing an open source and someone that's really deep into that. What did you see that really happened in this space during the last year? Harel: So I'll say some open source projects have it definitely harder than others. I know of maintainers that are really just like, there's ⁓ a huge stream of just people vibe coding PRs directly into the projects without knowing much about the project or what they're trying to do. Just trying to get like an open source contribution on their checklist. And that's really hard because it creates pressure. Ido Bronstein: Mm-hmm. Harel: on the maintainer of the open source project, right? Like, so anyone can go in, drive by, post a PR, and now it's on you to like, okay, well, what does this person trying to do? What do they care about? And it's not like, you know, open source maintainers are these like omnipotent people who can, you know, we're smarter than anyone else. We're the same, same humans as anyone else. Now, and we don't necessarily get paid for it, right? So like, now we need to figure out, okay. Ido Bronstein: Yeah. Yeah. You Yes. Harel: should I, how much time do I spend on this and how much effort do I put into unpack all of this? And that's really hard. We've seen some of this in OpenLineage too. And I think what we're trying to do there is A, we've started like one thing we pushed a couple months ago is basically like an agents markdown file to the OpenLineage repo to at least try to guide the agents that people are using on OpenLineage. ⁓ Because like, hey, Ido Bronstein: ⁓ this is cool. Harel: do not ever make a change to the spec unless you know what you're doing, right? ⁓ We don't want that to happen. If you're making an evolution to a thing, make sure that maybe there's no other consumers of it. Maybe the thing you're trying to change is already represented in the spec. So kind of trying to guide the agents in doing that work, ⁓ as well as how they draft their pull request. So that's one thing. ⁓ Hard for me to say if it's working well or not yet, but. Ido Bronstein: Excellent. Rule number one. It's a idea. Harel: Yeah, I hope so. I don't know. It was one of those very few PRs that I contribute. But I'd say one case where it's been useful, at least for us on the maintainer side, we've been trying to push for the last few weeks, we've been trying to push a change for more explicit table-to-table lineage in the spec. Ido Bronstein: Yes. Harel: getting a bunch of test cases, which was much easier because you can like, OK, you have all the spec, point cloud audit. And actually, a conversation I had with ⁓ some friends at OpenAI last week was like, coding agents are really good at data models for open source projects. They're really good at understanding that because that's what they were trained on. ⁓ Ido Bronstein: Yeah. Yes. Yes. Yes. Harel: So that plays in our benefit because coding agents do really understand open source way better than closed source. Ido Bronstein: Cool. ⁓ Well, I did not think about that, but it totally makes sense. So moving back to specifically OpenLineage, we love, I'm sure that the first thing that our audience think about is where I can use OpenLineage. Like what is the main real world use case set that it solved today? Harel: So. Ido Bronstein: And of course, lineage is a use case, like when companies use that, ⁓ where do you see like more need for that different industries, different data platform usage? Harel: Yeah. Yeah, yeah. And it's such a term that people just drop in. Like, I need lineage. It's like, why do you need lineage? You just really like looking at graphs with your data pipelines. And you know, sometimes the answer is yes. Sometimes people just want to see that you understand and capture well the semantics of their pipelines. Sometimes just like it instills trust in people. Ido Bronstein: Exactly. Harel: to know that their mental model exists in your product. And that's actually something that, ⁓ this is, I'm gonna wear a non-data observability and OpenLineage hat, like when I was working on the APM product at that was also part of the thing customers want understand, that you are capturing ⁓ their services understanding the architecture that they have and what are the and how they talk to each other. Which is that just kind of gives them trust that you understand what's going on. So What are the use cases? mean, ⁓ really care about operational use cases ⁓ For OpenLineage and working data dog. I think is ⁓ probably a reflection of that ⁓ Because ⁓ when something wrong ⁓ understanding very how it broke ⁓ why it ⁓ what is downstream impact of the thing to help you quickly triage and Ido Bronstein: Yeah. Harel: and understand the severity of it. For like propagating data quality issues, I found it really important and useful, right? So you can use lineage to say, ⁓ hey, I need to know if there's anything wrong upstream of this specific dashboard or this specific reporting table that's very critical from our organization. ⁓ The use cases that it powers that are probably... more meaningful to a bunch of other folks are like compliance and PII tracking across organizations, ⁓ cost retribution, ⁓ and even just being in like dev loop workflows where someone's about to push a thing and break everything for other people and kind of like, hey, don't do this. You have downstream consumers. Ido Bronstein: It's interesting. he said, ⁓ first of all, like operations of the data platform. And secondly, regulation and governance of the platform. These are the main things. Harel: Yeah, yeah. I would say those are like the top categories for sure. Ido Bronstein: Amazing. So how does AI change these use cases? And I know I have my own answer, but we love to hear how you see that. Harel: So one, I'm going to hold you accountable to give your own answer on this. ⁓ But for me, I think it's the velocity of how much of this is happening. the thing that AI does is it amplifies people, right? ⁓ For better or for worse. If you knew what you were doing and you're using AI and you're using AI well, then you can do a lot more. If you didn't know what you were doing and... Ido Bronstein: I know, I know. Harel: If you're not using AI well, then you're just going to do a lot more bad things. ⁓ And so I think it just really, really increases the signal on any interaction that a human does. So if you're just kind of stumbling around a data pipeline or a data warehouse and trying to ask a bunch of questions and you don't know what you're asking for and you're not giving good guidance, you're going to get a bunch of answers and... those might not be the right answers or very likely are not the right answers. And I think the impact here is having good observability of that layer and having good observability of your, and I know this is like a very overloaded term that everyone likes to say context now, but having really good context on all of these things, does help you because from that you can give a sort of like, analytics observability kind of ⁓ point of view where like, hey, by the way, in your organization, you have like a lot of people asking these questions and getting this data. You may want to either, you know, tune how they interact with agents and like be there in the loop and then help guide them. Or maybe you want to create a materialized view for this, or maybe you want to create like, you know, like a centralized data model. to answer this question and inject that context into how people interact with your data warehouse. So I think having lineage information helps you, like lineage and kind of like through observing all these different tools and how they're being used, ⁓ it helps you understand what's going on in your environment. So like for me, it's just, it's an intensity thing, but I'm curious to how you're seeing it. Ido Bronstein: Yeah, so I think from the usage perspective, I agree with you, like when agents wants to use the data that you have in your platform, lineage is a key part of their ability to do that. Because lineage represents the flow of information in the data platform. And the agents, when they are using the data, they need to know what this data means. So they can use some metadata description that someone's created for them, or they can look on the code and the lineage how this data created. And when people using this data, they pretty much know, like they have in their head how the data created, but agents don't have a head. Maybe they have memories that someone will update, but in order to really understand what the data means, they need to look back on how it created and which calculation he passed. So lineage become a foundation fundamentally important in order to give agents ability to understand what the data means and use this data from analytics. for any automation that you want to do. So I believe that lineage is going to be increasingly more important when organization will really try to adopt agent use cases. And we see this in our customer, we see this with conversation in the industry. So in the same way, it's exactly the same. There will be more interactions that... lineage will be required. And I believe it will be mostly in every agentic interactions that agent wants to understand what he has in the data. So this is how I see that. And yeah, it's really amazing. So do you see that it's already affecting the open source activity? Harel: For sure. For sure. Yeah. ⁓ You mean like usage of AI? Ido Bronstein: Yes, like from both sides, maybe it's great like more demand from vendors or like people in the community to create more plugins and more ways to like create a lineage and maybe some changes in the format that you need to create and the ways that people consume the lineage in the other side. Harel: So, one of the things I'm hearing and seeing ⁓ in other companies and people who I talk to is... It becomes a lot easier for people to, as I said earlier, think, like, you know, dip their toes and like use an MCP server and interact with a warehouse or interact with ⁓ any tools that is being offered and just kind of like self-serve themselves on the data. So something that maybe they wouldn't have done before, but now they have more capabilities. They feel like they have more power or the perception of they have more power to do this on their own. And... The impact of that is like sometimes they can do it right and you know, there's a chance that they did everything correct. But in many cases, they're not going to do it correct. like you see sometimes like a VP coming to a meeting with a clotted up analysis that they did without talking to anyone because for whatever reason, right, they maybe they didn't want to bother someone or, and they just, you know, they come in with their own analysis and you sit there and you know, maybe there's executives in the room like the VP of data is looking at like where did you get these numbers because this does not look right to me right but like imagine if they came with that analysis and they had like an appendix ⁓ and you know and and the thing they share like this is data from this table which is calculated from this table from this and that at least what will give the the audience a sense of like security of like Ido Bronstein: Ha ha! Harel: Okay, this is probably fine. I can see that they used like, you know, qualified tables here. ⁓ Of course, you know, there's a product to be built around this. ⁓ But I think it really, really, really makes the need for that kind of tooling for people ⁓ much more important than it was before, just because it's easier to do, quote unquote, easier, right? It's easier to do, not necessarily do right. Ido Bronstein: Amazing. So what I'm hearing from you two things. First of all, like a way to consume lineage with agent like MCP integration and etc. And secondly, a way to like people is starting to use more analytics because it's easier, everybody in the organization, but it's need to be much more easier to trust the analytics. And this way the OpenLineage need to create a way to consume it easily with AI, without AI. So when someone present a number, he can base this number upon the flow of data. Harel: Thank Ido Bronstein: And another thing interesting that I think is happening with AI is our ability to access unstructured data. And I know that we love to think about lineage as gathering of tables, columns, numbers, but organization has like bunch of documents, folders, that today we are not looking on them as part of the data flow. Like, how do you see, like, do you think lineage should? also include that. yeah. Harel: I mean... So first off, know, very, very leading question. Of course it should include, ⁓ right, like... But, but, Ido Bronstein: Yes, no, but how, how, how, how, how, and like it's a huge technology tech challenge to do that. Harel: 100 % because I think in some ways ⁓ because of the stochastic nature of language models, ⁓ breaks ⁓ an intrinsic part of the contract ⁓ ⁓ or like or of how we reason about data lineage we say, okay, lineage is for compliance ⁓ in some And if it's for compliance, it has to be deterministic. We can't ⁓ two different results for like scanning Unstructured document and or or doing like this, know scanning through a code base and trying to infer what that code base is doing And if sometimes you're getting not exactly the same result that might be an issue ⁓ So I think we need to have ways to reason about it for unstructured data that is slightly different from You know assuming we do kind of like LLM based ⁓ heuristics of figuring out ⁓ what this thing is. But maybe there are other ways. I don't know. But like this... I think there's a lot of work to be had here. I haven't put enough effort myself individually to figure it out. Maybe you did. Ido Bronstein: I did not, but for that we have the community. So anybody here that wants a super interesting and important problem to work about, here it is. And structure data to OpenLineage. It's quite a challenge. Harel: I will definitely review that PR. Ido Bronstein: Yes. So I think it's leading me to my next question. If you had an army of engineers to push code to OpenLineage, but good code, not like ⁓ code, ⁓ will you build next? And let's put the unstructured AI side. Harel: Yeah, yeah, because that was going to be my easy go to answer, right? ⁓ So I'd say a few things. ⁓ So one is, and this might be, might sound like a very boring problem, but I would actually deploy them not to work on the OpenLineage ⁓ project, but would want them to work on each individual ⁓ data technology out there. Ido Bronstein: Yes. Harel: and embed OpenLineage into it. So like I'd want ⁓ from Spark ⁓ add the OpenLineage integration directly into Spark. I'd want, you know, I'm not gonna through every single data ⁓ technology, but you the point. I think ⁓ that for me would like the top priority. ⁓ And then ⁓ really the ultimate goal the repository being ⁓ having just the spec ⁓ and client libraries and none the integration code. ⁓ That would by the way help with ⁓ the maintainer burden that we have today. So yeah, trying to get... ⁓ Ido Bronstein: Yes I want army of data engineers that will write code that I don't need to Harel: Yes, exactly. And I think a second thing, is more like, it's something we haven't thought of enough in OpenLineage, is going deeper on machine learning or model training, I should say, for AI and kind of like, does ⁓ spec ⁓ for ⁓ space? How do you know? ⁓ applied scientists, machine learning engineers use Open Miniatur for their work because it's pretty similar, right? Like ⁓ I know ⁓ the Ray community worked a little bit on capturing Open Miniatur and using that in Ray jobs. So I do think there's a need there and I would love this army of ⁓ engineers, or at least a battalion to work on this. Ido Bronstein: Yes. So last question, where do you see data engineering going in the next few years? Harel: I don't know where engineering is going in the next few years, let alone data engineering, but I'd say probably... The ownership of the business itself, I think, is going to become more and more ⁓ important and having more understanding of what it is that you're trying to do. ⁓ think... I think focusing more on contracts and meaning ⁓ and delegating a lot of the execution and like the nitty gritty of the work to AI and to agents and focusing on what really matters, which is your company's business, the problems you're trying to solve, ⁓ and then stepping in when you need to make an executive decision, right? Like still, as we've established, the AI doesn't think, it can... provide a bunch of things, but then you need to make the decision. Ido Bronstein: Yeah, so focusing on the business and let AI do the technical work. Harel: Some of it, not all of it, right? I wouldn't, yeah, like I would still very heavily review a spec proposal from ⁓ Claude on changing an OpenMini discriminative. Ido Bronstein: So, it's all a self-sufficiency. Yeah, for sure. Especially in the open source community, it's important because everybody can see the code. exactly. Amazing. So thank you very much, for your time. It was super interesting conversation. Couple of takeaways that I took personally from this conversation. First of all, ⁓ lineage is critical when you... ⁓ Harel: Exactly. Ido Bronstein: to use agents in organization. They ⁓ to use the lineage in order to understand what the data means and how to use that. Secondly, that we need ⁓ push ⁓ the data tools ⁓ adapt OpenLineage. It's critical. ⁓ we'll ⁓ one to look on ⁓ our data flow with one format ⁓ and we will be able to really create a map of the flowing of data. And lastly, ⁓ if you search an open and hard problem to solve, OpenLineage is a great and important open source ⁓ you can ⁓ try to tackle any of the problems that we talked about it in this episode. So, Harel: That second point was a bit like Lord of the Rings. One spec to rule them all and in the darkness bind them. Ido Bronstein: I think it needs to be on the description of the project for sure Alright, that is a wrap, huge thanks to Arel for bringing the insights and keeping it real If you enjoyed this Flow the Data Splash podcast we have got a lot more coming your way Until next time, keep your data flowing and we will see you in the next stream Thank you very much Harel Harel: Thanks, Edo. Thanks for having me.