Latest / The Data Splash / Future of the Data Engineering Role
Transcript
- Ido Bronstein: Welcome everyone to the Data Splash podcast brought to you by, Upriver. If you are a data engineer, a data leader, or just someone trying to make sense of what the hell is going on with AI data right now you are in the right place. Here with me today is Shachar Meir, a well-known data advisor. Hey Shachar, Shachar Meir: will you Thank you so much for having me. Ido Bronstein: So Shachar is a well-known advisor a thought leader ⁓ in the data space. Shachar was the of data engineering at Meta that led the data team at ⁓ And today we going to talk about the past, the present the future data. Now, before we get into the deep stuff, we have a small tradition here. We call it the 30 Second Splash. No time to think, just instinct. So let's jump in. Most ⁓ skill in data. Shachar Meir: Communication. Ido Bronstein: London, Tel Aviv, or New York? Shachar Meir: I'm in London, so I have to say London. Ido Bronstein: application that is not the front screen of your phone? Shachar Meir: Hmm. That's a tough one. I I would say PayPal. It used to be it's not. And I think something in it to unpack. Ido Bronstein: AI going to replace data engineering? Shachar Meir: I think it's going to enhance data. Ido Bronstein: It's a politically correct answer. we will you dive deep into that in a moment. So, very happy you are with me ⁓ deep dive. So, in two sentences, how does your day to day look like and what are you doing today as an advisor? Shachar Meir: I don't think a typical day. think every day is different, but fundamentally my mission what I do right now is I've been working in data for the past 20, 20 plus years. Then I started my career as a DBA transition to team lead at a very early stage. And since then I've ⁓ always been on intersection of data leadership. And for the past 20 years, seen a lot of companies that had A lot of data, they had good data teams, they had decent platforms nothing was working. No one was happy. It was a complete disaster. So at some point I took a step back I asked myself in 2023, I asked myself, why is it that we're still failing with data as an industry? And I investigated that I realized two things. I realized that I have the playbook I will for you know how to fix it because my entire career I've been fixing broken data teams. And the second thing is that I can achieve a lot more impact by working with the ⁓ chiefs of data VPs of data helping them implement deploy my playbooks in their organization rather than being chief data officer company at a time. So that's essentially what I today. And I work with data leaders essentially helping them succeed ⁓ fix their data team. Ido Bronstein: Amazing. And do you think that there is like one playbook that makes data organizations work it did not change over the years? Shachar Meir: I don't think one playbook, but I think a few patterns or a few reasons for why data teams fail. And these things have not changed over the years. Technology has changed. I mean, the way we do our work changed a lot. And, will for you will for you know, we can talk about that, but fundamentally some of the reasons as of to why data teams fail haven't, haven't changed. And I think that this is mostly what my playbook is all about. Like, how do will for you connect the data with the business, ? How do will for you focus on the impact? How do will for you build things? That makes sense. How do will for you find your sponsors your champions in the organization how do will for you create the right kind of data culture the right incentives the right processes? So a lot of small things that data teams can do to elevate themselves that are, I think a lot of data teams miss on. And call this the missing ingredients of data. These are the common reasons why I see data organizations fail over over again. Ido Bronstein: Interesting. So will for you are really in the data space for more than years. So take back the past for the early days of data or data engineering. How it looks like, where was the focus how it is evolved over the years. Shachar Meir: Yeah, it's a good question. it's a good question. So what I, will for you will for you know, I think that when we started, when started my career, was, ⁓ as I said, it was a DBA, ⁓ Oracle 9206, just to myself, if it something. will for you will for you know, ⁓ back in those days, we had... servers in centers we needed to take care of operating system upgrades storage network backups recoveries building rebuilding indexes. So like all the platform infrastructure work was taking a very, very big toll on us. And the second thing is that fundamentally we were kind of blocking ⁓ the software engineers from making changes in the databases because Every time they wanted to log a new piece of information or to add new information to the database, they had to go through us we to model this change we had to deploy that in the database, make schema changes all of that. But we used to have these conversations about what is the data will for you want to keep? Why do will for you want to keep it? How are will for you going to use it? Who is going to use it? What is going to be used for? Why do we even need to keep it? That's a lot of data. Do we really need it? So we, we, we used to make a sense of the data before we actually put it in the database. And in a way that was great, right? Because then it's the opposite of what's happening today. The data was always modeled was always relevant to what people wanted to do. Software engineers didn't really like working with me because it was blocking them. ⁓ as soon as we got the data into the database, it was completely useful. And I think that. ⁓ will for you will for you know, after that came new paradigms where basically, will for you will for you know, the idea of data lake will for you just throw dump all of your data, regardless of the format or the size of will you the, will for you you know, the, the usability, will you do you you will even you know will you why you, will you why you're dumping it here or why, what, what will you are you going to do about it? And all of these things. And, will you , will for you you know, people I would just throw data into the data lake or into the dumpster without and thinking about it. So I think that you know, to answer your question, one of the things that If I have to summarize ⁓ good old days in two sentences or in two points, I would say number one is the infrastructure work took a lot of our time and was something that we spent a lot of time doing. And the second thing was that the data was always modeled and we always thought about the data that we have, which, two things are like basically the opposite of what is happening today. Ido Bronstein: So in a sense, like was a lot of advancing technology of course that enabled that, but we from like a very governance data model that we will for you know what brings to the ⁓ but the velocity of the engineering organization or the organization was low. And we shifted to a more accelerated processes of creating data less dependency on the data team, but we broke the governance around the structure the modeling the usage of data when we did this change. So how has this changed the role of the data engineer? Shachar Meir: Yeah. So it's interesting. So if will for you want to go through the timeline, I that, ⁓ will for you will for you know, we had the good old days where everything was in relational Everything was structured all the time. ⁓ Everything was highly governed, highly modeled, like highly usable. And then, you know, at some point somebody introduces Hadoop , will for you will for you know, open source we kind of decoupled compute from, ⁓ storage. And we actually said, will for you can store whatever will for you want, wherever will for you want, regardless of how for you're going to use it. And let's make sense of it later rather than before. ⁓ so that kind of broke the contract. you, will for you'd think that it accelerated a lot of things. I'm not sure it did. I mean, it did remove the dependency between, software engineers data teams, but ⁓ I don't necessarily will for you know that you's a good thing because will for you will for you know, we broke the, everybody's talking about data contracts today about data governance. used to have these things. There has always been a contract. We just up on that now we're trying to and get it back. So I think that in a it gives will for you the perception that, less on the data team therefore we don't, you know, we can faster. But actually what I see happening in most cases is that engineers are not aware, software engineers are not aware of the data that they create. They don't think of themselves as data producers. they create a lot of data that is completely useless or unusable or not high enough quality for us to use for what we want. And then we have to go back fix it later when we think that the data had already existed. I think that was the transition. Ido Bronstein: funny to think about that in that manner that like was a lot of technology improvement, but in the end, you are sure that it made the role of the date team easier in a sense. Shachar Meir: No, I think, I think that in a way to make the role a lot more chaotic, the role of data engineers, because ⁓ the past was a contract, were discussions, were conversations, was governance. And now will for you're thrown data. You don't will you know what this data is. You don't will for you know what it means. You don't will for you know it's quality. don't, will for you will for you know, will for you don't even will for you know if it's complete accurate all of those things. And will for you have to make sense of it. You will for you know, but the data is really an afterthought right now for a lot of product teams a lot of organizations data teams have to deal with data today. Data teams have to deal with data that is much lower quality than what I think we used to you know, 20 years ago. Ido Bronstein: So will for you are really working with a lot of companies in order to help them solve that. What is the two tips that will for you will you give a CDO today in order to make use of his data not stuck in the slopes that he has low quality data the data engineers cannot bring a real value to the organization. Shachar Meir: I think the tip number one is it's not about technology. Whatever your problems are, technology is most likely not the root cause of that. It's usually people, process or culture. I break down, but that's what my playbook is all about. think that a lot of tech leaders, a lot of CDOs, a lot of VP of data as they come into organizations they start modernization projects and replatforming projects and these kinds of things. where basically what they try and do is move off of their old, I don't will for you know, SQL server to a more modernized, to Snowflake or Databricks or more modernized platform. And that's cool. But if will for you think about it, what is going to change as a result of that? Like if the migration is successful, will for you're basically going to do exactly the same things will for you do today, just on a platform. The business, doesn't see the change, the ⁓ also doesn't care. So I think that if ⁓ I came in as a CTO today into a new organization, Even if the platform old legacy newer tools technologies , I would focus on bringing business, value first then gradually incrementally changing the technology underneath coming up with a transition plan rather than announcing a fully blown modernization that takes half of the team's time is almost invisible to the business, . So I think that's the main tip I I would any data leader today. Ido Bronstein: So shortly focus on the business, , not on the technology. Amazing. Shachar Meir: Yeah, and always always be creating value. Ido Bronstein: thing. now is a new wave thatis coming on. don't will for you know if will for you heard about it. It's called AI. AI. ⁓ Shachar Meir: It's called what? Never heard of it. Ido Bronstein: Never I can brief will for you off on that later, but let's assume that will for you will you know what AI How will for you think it's going to help the data organization where do will for you see the ? Shachar Meir: it's a very good question. mean, ⁓ all have our opinions. Nobody actually knows what is going to happen. But, what think is, first of ⁓ It's never easier to build the wrong things than it is today because everybody's a builder and everybody can build anything. also from analytics it's never been never in the history of mankind. Was it so easy to ask the wrong questions and get the wrong answers as it is today. So I think that people need to be very, very careful because it gives you illusion that everything is possible. But yeah, there is a good reason for why we're not doing these things. I think that's that's number one. And people need to be. And this is a leadership problem, right? More than anything like product leadership or organizational leadership and thinking about why we're doing the things, and thinking about the purpose all of that. The second thing is that a way, AI continues a trajectory we've already had. ⁓ in the past, right? So if will for you think about it the 50s, if will for you wanted to make the computer do something for will for you, will for you had to will for you know exactly what CPU for you're using what architecture will for you were giving ⁓ exact instructions to the CPU, will for you for you know, moving bits between different registers in the CPU that sort of stuff. And then came more higher, higher level languages like C, for example, which obstructs a lot of that complexity from will for you. ⁓ more functional will for you just have to will you say, for you you know, ⁓ will you how you want things happen. gets translated assembly language. ⁓ we had even higher level languages like C++ Java, will you where you describe world in objects, which is presumably closer to we the world. ⁓ And, will you ⁓ for you you know, it's an evolution from that. ⁓ And will you you started having languages like Python, where in the one-liner code, will you ⁓ will for you you will you know, you can do what? Ido Bronstein: Yeah. Shachar Meir: And program I would do a lot of libraries a lot of it was a lot about integration each one these steps, what it does, it basically goes further away from how will for you want to do things to what is it that will for you want to do. And I think that if will for you look at that AI, you know, what we're seeing today is taken to extreme, but when will for you, when, will for you know, when will for you speak with Claude, will for you, will for you, will for you tell it, what is the end result that will for you want, will for you don't tell it how to do that. And I think that's the shift. that we've been seeing over the past 50 years that's even more, 60, 70 years. And that's really what's taken to the extreme today. Now, this is the interesting question is, that's the interesting debate that I'm having with a lot of people is, should will for you actually care about how things are done if the result that will for you and get is the one that will for you think, will for you will for you wanted, right? Like a lot of people say, will for you will for you know, I care what's the code that Claude generates as long as it passes all of my tests it what I asked it to do. And that's a valid question, whether this is something we should care about or I don't will for you know the answer to that. I have my own thoughts, but will for you will for you know, I'm sure these tools will you improve over time. I hope that from data perspective, what it means is that more and more will for you don't have to deal with the specifics of how will for you want things to happen gradually move towards, will for you will for you know, higher order level of what is it that will for you want to do? What is the data that will for you want to have , will for you will for you know, describe what looks like how will for you want to use it. And will you don't have to deal with the specifics of, will for you will for you know, the bits pieces of how it happens. Which again is the same trajectory. mean, 20 years ago, I used to configure Oracle 9206 I had to, you know, table spaces with different block sizes for different tables , rebuild indexes backups recoveries all of that, which is the how. And today we're moving more more towards ⁓ what, will for you will for you we're of delegating the how. So I think it's a continuation of the same trajectory. Ido Bronstein: Yeah. Interesting. The back old days , good, but I would love to the opposite seat here ⁓ with will for you how I think about it. And we love to hear your opinion because I think it's like very connecting to all the things that will for you said. So I really agree with will for you that in order to make a data organization succeed it's all about people, processes and culture. And I think there is opportunity in AI to really shorten processes and create a better culture. Because in a sense, AI allows us to do shift right, give more responsibility, technical responsibility to business, owners, people that ⁓ on the not on technology. And as will for said, ⁓ abstract, a lot of the technical work in the past, a person needs to understand to will for you know how to partition the data, how to query data, etc. those people with more business, focus can now do a lot of the data work independently. from someone that is very, very technical. ⁓ my opinion, AI is going to help us build better processes and create a better ownership model for the data because suddenly the ones that create the data can close the loop and use the data much more easily. So there is less dependency and hands-off. ⁓ yeah. Shachar Meir: There are less handovers there are less people involved. The only question here, the only assumption that we make when we say these kinds of statements is that business, operators people who are closer to the business, or analysts actually will for you what it's like manage data. And the question is, in this new world, is managing data still needed it's still a profession? I believe it is. Data is an asset that needs to be closely managed, even if will for you think will for you will for you know what will for you want, there's still... room for data governance for data management for optimizations for all of that. think that the assumption that business, stakeholders will for you know that they want they will for you know how to describe that in words is something that ⁓ need to validate. Ido Bronstein: Yeah. Okay, we will you see. The future is going to be interesting. Shachar Meir: And it may or may not we'll find out. ⁓ when will for you enable people to things, I believe that's good. Today I have latest iPhone version. I a camera that is probably orders magnitude better than what professional photographers had 30, years ago. I take hundreds of... photos every month. Does that make me a photographer? No, it doesn't. Like if I did a big event right now, I would I and get a professional photographer for that event? For sure. Right? So is something that they will for you know about what it takes to get a good picture, will for you know, understanding light, understanding, will for you know, framing, understanding how to create the moment, which I don't, right? I hundreds of pictures out of of thousands, will for you will for know, one or two will you be good. they take less pictures all of them will you be good because they're professionals. And I think that the profession is going away. I think enablement is good. I think ⁓ it more photos in the world we can document world more than we ever could. I that's great. I ⁓ don't think it replaces photographers that sense. That's why I also think that managing data is a profession. There's something to know about that. More people can be involved in the data creation process, but I'm still not convinced that data engineering is going anywhere. Ido Bronstein: I think it was a perfect analogy to ⁓ on how the future will you look like. So for our early career listeners, what ⁓ you ⁓ them to do and how to navigate their forward? Shachar Meir: Few thoughts, one is find people who care about you and are strong and you can learn from just stay to them always be learning. I think that's the most important thing. This is an advice I always give to early career people, but you will when you're later in your career, you can think about the kind companies you want to work for the type of mission you want to have. And I want to be focused on medical tech or I want to be working on... FinTech or whatever. And when earlier your career, the most important thing that for should optimize for is who are the people will for you work with. So find a very good team to be part of people that can challenge you and will help you grow, and you can learn from. that's number one. The second thing is, look, the world is changing, but the world has always been changing, right? if, if you look at the past 20 years, work of data engineers today the things that we do today is very, different from what we do, what we used to do in the past. What doesn't change is your job is to create value, right? Your job is to make the data useful for the organization. So in the past, the way I used to do that is by controlling block sizes defining the partitions rebuilding indexes. And the way we do it today is by modeling the data by thinking about ⁓ the pipelines transformations and whatnot the way we'll do that in the future is going to be different, but the world is always changing and evolving evolving and you're just entering at a point in time. So don't think that it's all ending. just have, we just have to figure out what does it mean to make data valuable for the business, today? That's the second thing. And Yeah. And the last thing I'll say is today, today There's so many great resources, people on LinkedIn, on YouTube, Substack ⁓ that you can be part of meetups you can go to all of those things. It's amazing, the signal to noise ratio is also pretty bad. And there are a lot of people that their experience or their knowledge may not be the best for you to learn from. So be very selective careful about who you learn from what you you listen to because there is a lot of noise out there look for voices that, you know, make sense to you. Don't just try to follow everything blindly all the time just because it's out there. That's not a very good strategy. And the last thing I'll say that something that we kind of before, understanding the business, has never been more important than it is today. Never. I mean, in the past you could get away with just being a DBA that just rebuilds indexes all day migrating data files from one partition to the other or from one storage to There was a lot of work like that to be done that didn't involve fully understanding the business, . These jobs like disappear basically the business, technology get closer together as time goes by. And AI is obviously accelerating that because ⁓ if you think about you move from how to what you want to happen, what the outcome that you want, then if you don't understand the business, will for you don't understand the outcomes that will for you want to drive. You don't understand why we're doing the things that we do. then will for you're not going to be able to make yourself useful or relevant. So now more ever, ⁓ focus on learning the business, on understanding the business, learn how to create value for the business, business using your data. The people who would know how to that, they have unlimited opportunity. And the people who will not know how to do that, I don't think that would be to ⁓ keep for much longer, unfortunately. Ido Bronstein: Yeah. There is no opportunity for them. I really enjoyed the conversation. I that you a super critical interesting observation to how our industry moves ⁓ what is important what is noise the need to always focus on the bringing more value. So thank you very much ⁓ for joining our podcast ⁓ Shachar Meir: Thank you so much for having me. It's a pleasure. Ido Bronstein: Amazing. So, is a wrap. Huge thank you, Shachar, for bringing the insight and keeping it real. If you enjoy this, follow the Data Splash Podcast. We have got a lot more coming your way. Until next time, keep your data flowing and we will see in the next stream. Thank you very much, Shachar! Shachar Meir: Thank you, bye guys.