Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Reflections on a Decade of Data Engineering at Seattle Data Guy

Transcript

Reflections on a Decade of Data Engineering at Seattle Data Guy · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: If you have a few tasks, I don't think you need that much extra overhead.
• 0:03
  SPEAKER_01: And if you have a lot of, you know, tons of tasks, tons of teams, it can be very helpful to have some sort of tool similar to Airflow to just manage all of that and allow, you know, different teams to kind of work and operate.
• 0:18
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:31
  SPEAKER_02: Okay.
• 0:31
  SPEAKER_02: Hi everyone.
• 0:32
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:34
  SPEAKER_02: I'm your host, Kent and Danis, and today I'm joined by Ben Robajan, who is a data consultant and owner of Seattle Data Guy.
• 0:41
  SPEAKER_02: Ben, how are you?
• 0:43
  SPEAKER_01: Uh I am well.
• 0:44
  SPEAKER_01: How about yourself?
• 0:44
  SPEAKER_02: Yeah, doing great.
• 0:46
  SPEAKER_02: Uh, I think a lot of our listeners will probably better know you as the Seattle data guy, though um, as you state, you're no longer in Seattle, which makes me very sad as someone who still lives in Seattle.
• 0:56
  SPEAKER_01: Yes, yes.
• 0:57
  SPEAKER_01: Well, we can both we can both be happy that the the Seahawks won.
• 1:00
  SPEAKER_02: Um That's right, that's right.
• 1:01
  SPEAKER_02: When we're recording, we're actually recording this on the day of the victory parade, um, which is a big deal happening.
• 1:08
  SPEAKER_02: Of course, by the time this comes out, everybody will have moved on from that.
• 1:11
  SPEAKER_02: But yeah, it's an exciting week in Seattle while we're talking now.
• 1:16
  SPEAKER_02: Uh okay, well, let's go ahead and dive in.
• 1:18
  SPEAKER_02: So for today's podcast, we're going to discuss some lessons from the past decade in data engineering, a little more high-level than we often go on our podcast, but we'll still have plenty to talk about with Airflow since it's certainly relevant within that conversation.
• 1:32
  SPEAKER_02: Uh, but before we get into that, let's start with some background on you.
• 1:35
  SPEAKER_02: For those who aren't familiar with your work, what do you do and how did you get into data engineering?
• 1:41
  SPEAKER_01: Yeah, yeah.
• 1:42
  SPEAKER_01: I mean, currently, you know, you brought up I work as a data consultant.
• 1:46
  SPEAKER_01: Um, but prior to being uh independent, worked at Facebook as a data engineer, um, as well as a few other companies as well.
• 1:53
  SPEAKER_01: I originally started my career in data on the more data analytics side.
• 1:57
  SPEAKER_01: I think, like a lot of people, I started around 2015 reading uh the 2012 article about from Harvard about data science and being interested in that because I had just taken um some courses on statistics and epidemiology and programming.
• 2:11
  SPEAKER_01: And I was like, oh, wouldn't it be cool if this all was together?
• 2:13
  SPEAKER_01: And you know, that just kind of came at the right time.
• 2:16
  SPEAKER_01: So I think that was where I originally started pushing for, but um, eventually I found data engineering and I found that I kind of like that work um overall.
• 2:24
  SPEAKER_02: Yeah, that's great.
• 2:25
  SPEAKER_02: I can certainly relate to that.
• 2:27
  SPEAKER_02: Someone with a data science turned data engineering background myself.
• 2:31
  SPEAKER_02: Uh, and in your current consulting role, what sort of projects do you typically work on?
• 2:36
  SPEAKER_01: Yeah, yeah, there's a lot of um everything from honestly like more greenfield projects where people want to set up their data infrastructure, set up, you know, uh a data stack that's reliable to brownfield where it's like something's been set up and maybe they need help to change it or migrate solutions, or someone else built something and no one knows how it works.
• 2:54
  SPEAKER_01: So then I kind of come in and help them uh figure out how to kind of operate from there.
• 2:59
  SPEAKER_02: Awesome.
• 2:59
  SPEAKER_02: That's great.
• 3:00
  SPEAKER_02: And I imagine uh with data engineering consulting, hopefully at least some of the time your airflow is in the mix.
• 3:07
  SPEAKER_02: So I'm curious, when do you typically recommend that companies you work with choose Airflow as their orchestrator?
• 3:14
  SPEAKER_01: Yeah, yeah.
• 3:14
  SPEAKER_01: And and obviously there's there's lots of different kind of nuances and reasons.
• 3:18
  SPEAKER_01: But you know, I I do find that it's it's great when workflows get more complex, you know, and and if especially if you've got a team that can then build it out and and manage it and maintain it, you know, depending on on how you plan to to manage it and kind of have it running in the background.
• 3:33
  SPEAKER_01: Um, so yeah, that that's generally when when I'll recommend it.
• 3:37
  SPEAKER_01: You know, there's there's some earlier companies where it's like, you know, if you have a few tasks, I don't think you need that much extra overhead.
• 3:42
  SPEAKER_01: And if you have a lot of, you know, tons of tasks, tons of teams, it can be very helpful to have some sort of tool similar to Airflow to just manage all of that uh and allow you know different teams to kind of work and operate.
• 3:54
  SPEAKER_01: Um I know I think even I want to say like now v3 has a lot more in terms of like making it easier to operate as a team as well.
• 4:01
  SPEAKER_01: And so I think that the the more you can it's usually around like uh more pipelines, more people where I start calling out towards towards Airflow.
• 4:09
  SPEAKER_02: Yeah, that makes sense.
• 4:10
  SPEAKER_02: And yeah, definitely there's been a lot of um progress within the Airflow project over the years to make it easier for enterprise use, both from like a managed services perspective, if you don't want to deal with it at all, or just from the project itself, making it you know more secure, more reliable, all that good stuff.
• 4:27
  SPEAKER_02: So yeah, that's great to hear that you're recommending it in those more complex use cases.
• 4:32
  SPEAKER_02: Awesome.
• 4:33
  SPEAKER_02: Well, let's talk a bit about, as I stated earlier, the past decade in data engineering.
• 4:39
  SPEAKER_02: So I understand, as you said, that's about how long you've been in the industry.
• 4:43
  SPEAKER_02: Uh, it also happens to be about how long Airflow has been around.
• 4:47
  SPEAKER_02: Um was originally created in 2014 uh and then open sourced in 2016, 2015, 2016.
• 4:54
  SPEAKER_02: So about that same amount of time.
• 4:57
  SPEAKER_02: Just starting with the big picture, what are some of the biggest lessons that you've learned during this time?
• 5:02
  SPEAKER_01: Yeah, yeah.
• 5:03
  SPEAKER_01: And I appreciate that you you know you called out because that was that was, I think, a big part of I did a talk at the Airflow Connor or uh summit last year.
• 5:10
  SPEAKER_01: And I kind of brought that that point up where I was like, yeah, like the last decade, Airflow and I, or Airflow and my careers are both kind of in line there.
• 5:19
  SPEAKER_01: In terms of like the lessons that I've I've picked up over the last few years, one I like to say what what hasn't changed that much, which um I think things that haven't changed is like one, we're still trying to make data more accessible.
• 5:33
  SPEAKER_01: That's a a massive part of the role of data engineering.
• 5:36
  SPEAKER_01: You know, now we have AI as an aspect as well where we're trying to make data more accessible to you know systems that are again LLMs and things similar to that.
• 5:45
  SPEAKER_01: I also I also like say the dashboards are still slow.
• 5:48
  SPEAKER_01: I still run into a lot of companies where their dashboards are still slow.
• 5:51
  SPEAKER_01: Batch batch is still probably the most popular approach for most for most companies, is what I find.
• 5:57
  SPEAKER_01: Um, although real time is becoming, I think, more popular.
• 6:00
  SPEAKER_01: I think it's just kind of running alongside with with batch where it's just we're just building more pipelines.
• 6:04
  SPEAKER_01: So yes, there's more real time, but there's also more batch.
• 6:06
  SPEAKER_01: And I think the the biggest challenge we still face is just aligning with the business um overall, because it's like we're we're just always playing that that catch-up game.
• 6:16
  SPEAKER_01: And I think that that's something that hasn't changed too much overall.
• 6:21
  SPEAKER_01: I I'm curious if you if you'd feel like that's similar on on your do you see similar things on your side?
• 6:26
  SPEAKER_02: Yeah, absolutely.
• 6:27
  SPEAKER_02: I think in terms of alignment with the business, I this is a um question that we include every year in the Airflow survey that is, at least as far as we know, kind of the biggest data engineering survey that we gather.
• 6:40
  SPEAKER_02: And and it's more kind of beyond just Airflow, but this question of like, what are the biggest challenges that you face, you know, using Airflow or kind of with the work in your team?
• 6:50
  SPEAKER_02: Um and like just alignment with stakeholders is always one of the top answers, you know, along with things like being under-resourced and um and stuff like that.
• 6:59
  SPEAKER_02: So I I think that makes sense.
• 7:01
  SPEAKER_02: It's always, you know, one of the things that we call out when we talk about best practices as well, um, is that, you know, just doing that alignment up front before you go into building pipelines is a really important part of a project.
• 7:14
  SPEAKER_02: And I think, like you said, that that just hasn't changed.
• 7:17
  SPEAKER_02: Um, it's been true, certainly in my entire career in this space.
• 7:21
  SPEAKER_02: And yeah, I think it will continue to be really important going forward.
• 7:26
  SPEAKER_02: I also like that you called out that batch is still important, uh, because I I feel it maybe it's sort of cyclical or in phases, but sometimes it seems like the trendy thing is to move everything towards real time.
• 7:38
  SPEAKER_02: And people say, like, oh, batch is kind of of the past, and that's you know, not the modern way of doing things.
• 7:43
  SPEAKER_02: And I think it's just not true.
• 7:45
  SPEAKER_02: Like, batch the reality is that real time isn't needed for a lot of workflows, or even when you have real time, it's almost impossible, I think, to do that in isolation without some batch supporting it.
• 7:57
  SPEAKER_02: So certainly relevant for airflow, and I think there's work to, you know, make airflow not not a streaming tool or anything like that, but just more supportive of kind of low latency workflows.
• 8:10
  SPEAKER_02: Uh, so that's perfectly valid too.
• 8:12
  SPEAKER_02: But yeah, I don't think the batch workflow is going anywhere.
• 8:16
  SPEAKER_01: Yeah.
• 8:16
  SPEAKER_01: Yeah, I think um, you know, one, it it's just the realities of how some businesses still operate.
• 8:22
  SPEAKER_01: I actually had a call recently where, like, yeah, someone brought up like, oh, we want these dashboards in real time.
• 8:28
  SPEAKER_01: But I was like, you know, I've been working with them a while.
• 8:30
  SPEAKER_01: I'm like, but you only get your files that they are pulling in like once a week.
• 8:33
  SPEAKER_01: So do you when you say real time, I think what you mean is as soon as you get that file, you would like that loaded by the next day.
• 8:40
  SPEAKER_01: And that's because right now you're currently like you get the file, when you get time, you process it, and and that's that's not real time.
• 8:46
  SPEAKER_01: But for you, real time right now would be just hey, once we get the file, at least by the next day, if we can load it in, that would be you know a massive move and a massive shift to what we're currently doing.
• 8:56
  SPEAKER_01: And so that's what I've I've often found for some people.
• 9:00
  SPEAKER_02: Yeah, definitely a difference between event-driven, like you're describing, and actual real time, like that, you know, sub-couple second, subsecond latency, um, which is I think harder technically to implement.
• 9:14
  SPEAKER_01: Yeah, yeah, yeah.
• 9:15
  SPEAKER_01: And and even when it's easy to implement, I think it's it still comes down to the like but why uh sides of things.
• 9:20
  SPEAKER_02: Right, right.
• 9:21
  SPEAKER_02: Do you really need that?
• 9:23
  SPEAKER_01: Yeah, yeah.
• 9:23
  SPEAKER_01: And then like, you know, depending on where you load the data, like let's say you load it into Snowflake, you're gonna pay a lot more um if you load it real time versus if you load it in batches, generally speaking.
• 9:33
  SPEAKER_01: So there's like other trade-offs um besides even like, okay, maybe we've made the actual ingestion part of real time easy, but what is the trade-off on on other components or other parts of the whole process as well?
• 9:45
  SPEAKER_02: Yeah, yeah, absolutely.
• 9:47
  SPEAKER_02: Well, speaking of kind of batch uh orchestration and stuff like that, thinking about Airflow specifically, how do you think Airflow's role in the data ecosystem has changed during this time?
• 10:00
  SPEAKER_01: Yeah, yeah.
• 10:01
  SPEAKER_01: I mean, I think one, you know, when it first came out, I I think it was kind of like the it wasn't like the it product, but it was definitely like a shift, right?
• 10:09
  SPEAKER_01: Because up until this point, most people you you'd often build your own solution.
• 10:14
  SPEAKER_01: You'd build something that looked very similar to Airflow or had like similar components.
• 10:17
  SPEAKER_01: Maybe you didn't have a UI.
• 10:18
  SPEAKER_01: Like I've worked for several companies where you know, you you had to kind of build everything yourself and and you just run it off all you'd run it off cron or something, and that would that would be how it operated.
• 10:28
  SPEAKER_01: And so I think Airflow really gave you this ability to think like, oh, I can get this out of the box, right?
• 10:35
  SPEAKER_01: Um, I think the original versions uh you know left a lot of people wanting, especially after you've now been, you know, especially if you just came in the data world, maybe in like 2020, you're like, oh, we can do this better.
• 10:45
  SPEAKER_01: And I think, you know, we there's been a lot of improvements since then.
• 10:48
  SPEAKER_01: But you know, in 2015, thinking back to then, it was like I I remember coming in and being like, well, I've got SSIS, I've got like some custom-built solutions that I've had, and obviously there's other tools, but yeah, you you have Airflow and like, oh wow, dependency management.
• 11:01
  SPEAKER_01: Like that was, I think, the big thing that was a shift for me, or having something that did that for me, because up until this point, I was like developing a table somewhere that would be managing, like, hey, what's run, what hasn't run, don't run something if if if we haven't run, you know, step five in this process.
• 11:15
  SPEAKER_01: And so having something where you could put that in code rather than kind of what I was doing, I think it made a lot of sense.
• 11:22
  SPEAKER_01: And so I think it really defined a lot of of how we've built things.
• 11:25
  SPEAKER_01: And even now, I think a lot of the tooling that you see has, you know, the the newer solutions are either one, kind of either built a little bit on Airflow or two, you know, built trying to differentiate itself from Airflow.
• 11:36
  SPEAKER_01: So in many ways, Airflow has kind of spun off all of these other other solutions as well, where it's like, yeah, well, we saw Airflow, here's what we'd like to do differently.
• 11:44
  SPEAKER_01: Um, and so I think it really became like the defining way of of doing more of this like code-based orchestration.
• 11:50
  SPEAKER_01: Obviously, then you still have the UI-based ones, but like I think in terms of like code-based, it really is a lot of stuff is based off of it, either by again trying to differentiate away from it or you know, being built on top of it.
• 12:01
  SPEAKER_02: Yeah, for sure.
• 12:02
  SPEAKER_02: We always say it's the standard for data orchestration, which I think is exactly what what you're saying.
• 12:08
  SPEAKER_01: Um, yeah.
• 12:09
  SPEAKER_01: Yeah, no, I think that's a good way of thinking.
• 12:11
  SPEAKER_01: And I think the other difference is people start with it now, right?
• 12:14
  SPEAKER_01: Like in the past, again, you'd probably start learning about programming uh and Python and you build like your own, again, your own pipeline systems.
• 12:23
  SPEAKER_01: And now it's kind of like, yeah, maybe you learn how to pull an API like from an API and maybe a few other things, but you a lot of people jump pretty quickly um in into orchestrators um because again, they're just accessible.
• 12:34
  SPEAKER_01: Airflow being open source makes it super accessible.
• 12:37
  SPEAKER_01: And so a lot of people, I think, jump into it.
• 12:39
  SPEAKER_01: Because again, you don't have to sign a contract, uh, you can just spit it up and have it working on your local computer and kind of testing it out.
• 12:48
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:58
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 13:09
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:18
  SPEAKER_02: Yeah, it's really cool from my perspective to see how it's just ubiquitous in like training for data engineers now or in curriculum or you know, boot camps or whatever it is that Airflow is always one of the top things of like this is the basis of what you need to learn, and then you can kind of go from there.
• 13:38
  SPEAKER_02: To your point, certainly was not the case, like when I got started in data engineering.
• 13:42
  SPEAKER_02: I first used Airflow in 2019.
• 13:45
  SPEAKER_02: So it wasn't like super new, but it was still like not really widely known or widely used.
• 13:51
  SPEAKER_02: And like at that point, I came from a data science background and sort of to move into data engineering, just like learned really good SQL, you know.
• 13:59
  SPEAKER_02: And then, like, yeah, every place that I worked had a different like drag and drop-y and kind of ETL tool that they used, and then finally learned Airflow, kind of ripping some of those out because they weren't working very well.
• 14:10
  SPEAKER_02: But yeah, I remember at the time doing some debate on like the first consulting project I worked at where I used Airflow, or me and the head engineer like, this thing is pretty new.
• 14:21
  SPEAKER_02: Like, do we think we can use this?
• 14:24
  SPEAKER_02: Like, can we trust this?
• 14:26
  SPEAKER_02: There are some blog posts out there about it.
• 14:28
  SPEAKER_02: We think we can figure out how to set it up on Kubernetes and like let it go.
• 14:32
  SPEAKER_02: But yeah, now it's just this is like out the gate, like one of the first things that you learn.
• 14:38
  SPEAKER_02: Um, and we see it taught, you know, even in school a lot, which is pretty pretty great.
• 14:43
  SPEAKER_01: Yeah.
• 14:43
  SPEAKER_01: Yeah, yeah.
• 14:44
  SPEAKER_01: No, I think it again, it it's become, like you said, that that kind of baseline that people reference these days.
• 14:49
  SPEAKER_01: So yeah.
• 14:50
  SPEAKER_02: Yeah, yeah.
• 14:51
  SPEAKER_02: Uh well, taking a little bit of a fork, very popular topic.
• 14:55
  SPEAKER_02: So I feel like I need to touch on it.
• 14:57
  SPEAKER_02: What is your opinion on the impact of AI on data engineering?
• 15:01
  SPEAKER_01: Yeah, yeah.
• 15:02
  SPEAKER_01: I think, you know, one, it obviously changes a lot of our workflows for sure.
• 15:06
  SPEAKER_01: I think that that's one of the big, big shifts is hey, now I can get pretty decent SQL um sometimes out of AI, at least, especially if it's like off some standard data set, uh, you know, like from HubSpot or something like that, right?
• 15:19
  SPEAKER_01: Like it's it's that stuff all exists.
• 15:20
  SPEAKER_01: It's in the documentation.
• 15:22
  SPEAKER_01: It's not hard for it to like uh like if you use, like, for example, a data share or sharing uh data share with from HubSpot from to Snowflake, the data comes out a little wonky, but it's pretty easy to say, like, hey, can you just make me the query that's gonna do this?
• 15:35
  SPEAKER_01: And so it's it's hard to deny that like it hasn't changed that workflow and and changed like, hey, I can write a really easy API extract.
• 15:43
  SPEAKER_01: I do think like when you get like larger and in and needing like more complex systems, right?
• 15:49
  SPEAKER_01: Like when you have to deal with more complex APIs or have to pull in large data sets, you're gonna run into problems where you know you're gonna find somehow that data, like how you're pulling it in, is it more efficient?
• 15:59
  SPEAKER_01: Um, but it's it's really great at getting you started.
• 16:02
  SPEAKER_01: I I think the other thing is like there's a greater demand from data engineers to be more, even more organized or even like just better for LLMs to sit on top of the data we create, just creating more of a data set that they can rely on.
• 16:16
  SPEAKER_01: Um so I think that that's become more pertinent.
• 16:18
  SPEAKER_01: I think the the last space is, and I think this is gonna slowly change, is that we're going to just have more, we're gonna have a greater need for more data.
• 16:28
  SPEAKER_01: I think we we don't we tend to want more granularity.
• 16:31
  SPEAKER_01: I always think about like I imagine back in the 90s and in 80s, you know, there was a certain level of granularity where it was really just based off the transaction system, um, you know, your your Oracle instances, SAP, etc.
• 16:41
  SPEAKER_01: And then like as we've moved from that, it's like, well, we want like we don't just want to know like what people bought.
• 16:47
  SPEAKER_01: We want to know what they were looking at before they bought it.
• 16:49
  SPEAKER_01: Like, did they go to another website?
• 16:50
  SPEAKER_01: What website did it become from?
• 16:52
  SPEAKER_01: Like, where did they click on our website?
• 16:54
  SPEAKER_01: Uh and so we just start asking more and more questions.
• 16:57
  SPEAKER_01: And I can only imagine that happens more and more in the future.
• 16:59
  SPEAKER_01: And it's gonna become more from unstructured data sets as well.
• 17:02
  SPEAKER_01: And so I'm gonna bet that you know there's gonna be a greater need for for data engineering.
• 17:07
  SPEAKER_01: So maybe we have AI help some of those workflows, but it's just going to be like great, you know, you can build faster and build more data sets that we can use and then integrate better.
• 17:17
  SPEAKER_01: So I think that's that's hopefully where we'll we'll see it go in the next few years as well with AI.
• 17:21
  SPEAKER_02: Yeah, I think that's a great summary.
• 17:23
  SPEAKER_02: At astronomer, of course, with such a heavy focus on airflow.
• 17:26
  SPEAKER_02: Uh, my team has kind of been talking about it in this like dual aspect of like there's AI for airflow, like how do you use AI to like help you write DAGs more efficiently or better or faster or whatever.
• 17:39
  SPEAKER_02: And then there's airflow for AI.
• 17:42
  SPEAKER_02: So, like whatever orchestration you need to do to get your AI workflows into production, like you can probably, or we think you should be using Airflow for that in a lot of cases.
• 17:52
  SPEAKER_01: Yeah.
• 17:52
  SPEAKER_02: Um, so yeah, both both certainly very relevant.
• 17:56
  SPEAKER_02: But I think I like that you sort of touched on the fact that the knowledge of the data engineer is still really important.
• 18:04
  SPEAKER_02: Like you're not at a point where you can just let the LLMs like do their thing blindly.
• 18:09
  SPEAKER_02: Like somebody who kind of knows what the output is supposed to look like needs to still be involved there.
• 18:15
  SPEAKER_01: Yeah.
• 18:15
  SPEAKER_01: Yeah, I will say that you brought up like, you know, having airflow manage from the processes around LLMs and like ML and like productionizing ML models as well.
• 18:24
  SPEAKER_01: That's something that I think I found interesting because I had always assumed that there must be a different way people are doing it, right?
• 18:32
  SPEAKER_01: Like there must be, like in my brain, like a more complex way, right?
• 18:35
  SPEAKER_01: Like, I mean, one, there's a lot of ML flow and other tooling as well.
• 18:38
  SPEAKER_01: Um, but I find that a lot of companies, yeah, use Airflow to train their model, retrain their model, push it back out.
• 18:44
  SPEAKER_01: Um, I remember I think I did one with Riot.
• 18:46
  SPEAKER_01: I want to say it was Riot games, but you can check back on my have a sub stack somewhere.
• 18:50
  SPEAKER_01: And that's what they talked about.
• 18:50
  SPEAKER_01: They're like, yeah, we would like pull in the data and they would deploy the model.
• 18:53
  SPEAKER_01: I think they were only doing it like weekly or monthly or some some cadence.
• 18:57
  SPEAKER_01: And so they're like, yeah, we would train the data and then re-push it back out for your Airflow.
• 19:00
  SPEAKER_01: And I thought I was like, oh, I would I was assuming it was gonna be a different tool, but um, and actually thinking back even to Facebook, you know, we used we used Dataswarm, and I would often see a lot of uh ML engineers use that as well to deploy models.
• 19:14
  SPEAKER_01: But and data swarm is like kind of iteration one of Airflow for I I think it's a little more common knowledge now, but for those who maybe don't know.
• 19:23
  SPEAKER_02: Yeah, yeah, certainly Airflow, I think more so now we've seen it grow a lot in the last couple of years, but historically it was not always thought of as like an MLOps kind of tool.
• 19:33
  SPEAKER_02: Um, I think for the Gen AI workflows, maybe even it's a little more straightforward because a lot of what you're actually doing is just like an ETL thing with like an LLM tacked on to it.
• 19:43
  SPEAKER_02: But yeah, we we have lots of customers implementing those workflows and it seems to work pretty well for them.
• 19:49
  SPEAKER_01: Yeah, yeah, yeah.
• 19:50
  SPEAKER_01: And and that I think part of it's also like you use the tool you know.
• 19:53
  SPEAKER_01: So um that that is the benefit of airflow getting, you know, having so much staying power, you know, getting in the all those.
• 20:00
  SPEAKER_01: I I don't think they meant to, but like getting in all those training programs and um, you know, now it's it's the tool people know.
• 20:06
  SPEAKER_01: So they're gonna use what they they know and they know they can deliver um, you know, with.
• 20:11
  SPEAKER_02: Yeah, definitely.
• 20:12
  SPEAKER_02: I think it helps to have that thread of consistency when one of the things that, you know, I certainly felt myself in data engineering is is just that the the tool landscape changes really, really fast.
• 20:24
  SPEAKER_02: Um it's just all the time there's a lot of different stuff, and it can be, I think, really hard for anybody to keep up with.
• 20:31
  SPEAKER_02: But if you do have at least some of it that is consistent and kind of standardized, that can help a lot in the day-to-day of especially when that thing is designed to you know swap in and out with other tools or just you know connect to anything.
• 20:46
  SPEAKER_02: Uh yeah, yeah.
• 20:47
  SPEAKER_01: Yeah, bringing it back to like the data, like different, like what's happened in the last decade.
• 20:51
  SPEAKER_01: Like there's just been a lot of change even the last decade for data engineers.
• 20:55
  SPEAKER_01: You if you came in 2012-ish, 2015-ish, you you could have either gotten pulled into Oracle and SQL and SSIS, you could have gotten pulled into Hadoop.
• 21:05
  SPEAKER_01: Um, because that peaked around 2015.
• 21:07
  SPEAKER_01: You you know, again, Airflow kind of pop peaked or started getting uh more intrigue in that time period.
• 21:12
  SPEAKER_01: And then you suddenly see it switch in 2020, where it's like, okay, there's some talk about MDS and you know Snowflake, um, and we're kind of dropping Hadoop and or at least we're keeping some components and re you know, reworking them.
• 21:25
  SPEAKER_01: And so so much has changed, especially I think in our industry.
• 21:28
  SPEAKER_01: And of course, every every tech industry has that, but it I think it's been even faster more recently, where you know, maybe more in the past in the decades prior, data was like SQL, Oracle, SQL Server.
• 21:41
  SPEAKER_01: Some somewhere in like 2010, it's like, okay, let's rehash everything.
• 21:44
  SPEAKER_01: And I think it's been us trying to figure out what it's supposed to look like for the last decade or so.
• 21:48
  SPEAKER_01: Like, okay, what does the future of the data stack look like for different companies of different sizes?
• 21:54
  SPEAKER_02: Yeah, absolutely.
• 21:56
  SPEAKER_02: Well, I think that's a good, you know, kind of summary of the state of.
• 22:00
  SPEAKER_02: The last 10 years and also um just in the interest of time, I go lead into my standard final question, uh, which is looking forward, what would you most like to see from development of the Airflow project in the near future?
• 22:12
  SPEAKER_01: I think there's been a lot of changes that have been already great.
• 22:15
  SPEAKER_01: I know that you you put some effort into trying to track data sets.
• 22:19
  SPEAKER_01: I think that's one big thing that um, you know, for a long time people were probably frustrated with, or like passing data back and forth uh in between tasks uh was was more complex in the past.
• 22:29
  SPEAKER_01: And now it's like definitely easier.
• 22:31
  SPEAKER_01: So I think those are those are the big things that I remember wanting to change.
• 22:35
  SPEAKER_01: Um on the event side, I think being clear on on how much it can handle.
• 22:40
  SPEAKER_01: I remember I was talking to someone at the Airflow Summit and I asked them, like, because they they were like, oh, we worked on the event um component.
• 22:47
  SPEAKER_01: I'm like, oh, that's awesome.
• 22:48
  SPEAKER_01: Like how much?
• 22:49
  SPEAKER_01: Like how, how, how, like, how many events?
• 22:52
  SPEAKER_01: Uh, and it didn't feel like I got a clear answer there, so I don't think it's been t I mean, one, I think it hasn't been tested, right?
• 22:58
  SPEAKER_01: Like, I don't know how much I would recommend people try to do event, like, you know, actual, like like almost replace Kafka with airflow.
• 23:06
  SPEAKER_01: Uh, sorry, vice versa.
• 23:07
  SPEAKER_01: Uh replace airflow with Kafka.
• 23:09
  SPEAKER_01: I don't know.
• 23:09
  SPEAKER_01: I I don't think I'd recommend that, but I want to know like at what point is there like is it just too chaotic?
• 23:15
  SPEAKER_01: Because I feel like it's really nice for like batch flow or again, like some events, but not events that are like a thousand events a minute, but maybe you know there's there's somewhere in there.
• 23:24
  SPEAKER_01: So I'd like to kind of know what what the thought is there going forward.
• 23:28
  SPEAKER_01: Like, is it that you're just gonna do events and then but never not really real time?
• 23:32
  SPEAKER_01: Because that's kind of kind of a little more difficult, or is the eventual goal to do that?
• 23:36
  SPEAKER_01: So I think it's more just clarity around what's going on there.
• 23:39
  SPEAKER_02: Yeah, for sure.
• 23:40
  SPEAKER_02: I think those are both really good points.
• 23:42
  SPEAKER_02: I from the data sets perspective, or now in Airflow 3, they're called assets.
• 23:46
  SPEAKER_02: That's a feature that is I we see like the most widely used feature in Airflow, which is great.
• 23:51
  SPEAKER_02: And yeah, there's a lot of work being done to add additional functionality there.
• 23:55
  SPEAKER_02: I do think that's a great question on the event-driven scheduling.
• 23:59
  SPEAKER_02: I also don't know the answer of whether any benchmarking has been done.
• 24:03
  SPEAKER_02: But yeah, I would also, from my perspective, not recommend people use it as a K replacement, at least today.
• 24:10
  SPEAKER_02: I think the use case you described earlier of like, I want my dashboard to update as soon as the data comes in, is like a great one for event-driven scheduling, assuming that data isn't coming in like every 10 seconds.
• 24:22
  SPEAKER_02: But yeah, hey, definitely something for the project to think about going forward.
• 24:27
  SPEAKER_01: Yeah, yeah.
• 24:28
  SPEAKER_01: And you know, I I in my mind, I don't ever envision Airflow being more of a real-time can like tool, but you know, you never know, like the future things things change quickly.
• 24:37
  SPEAKER_02: And it's hard to predict sometimes.
• 24:39
  SPEAKER_01: So yeah, yeah, it's it is hard to predict.
• 24:41
  SPEAKER_01: I maybe I can see it uh well, you you can already do like some unstructured data.
• 24:45
  SPEAKER_01: I don't know.
• 24:46
  SPEAKER_01: I think uh I think for what I've needed it for, airflow's always been sufficient.
• 24:51
  SPEAKER_01: So I'm like uh especially once you had uh data assets and being able to pass data back and forth.
• 24:55
  SPEAKER_01: I was like, all right, this was like one of my big pet peeves, but once that was a little more solved, it's like okay, this is better.
• 25:01
  SPEAKER_02: So yeah, nice.
• 25:02
  SPEAKER_02: Well, always lots to look forward to for sure.
• 25:05
  SPEAKER_02: Um well, Ben, thank you so much for coming on.
• 25:07
  SPEAKER_02: It's been really interesting speaking with you.
• 25:10
  SPEAKER_02: Uh just to close us out, what is the best way for folks to get in touch with you?
• 25:14
  SPEAKER_01: Yeah, um, you know, you can find me across YouTube, uh, LinkedIn, and um what do you call it, and Substack as well.
• 25:21
  SPEAKER_01: Usually I you can try to LinkedIn me and send me a DM there.
• 25:24
  SPEAKER_01: I don't do my best there, um, but I will do my best to respond.
• 25:28
  SPEAKER_01: Uh, I think I've also hit the 30,000 follower or uh network limit on LinkedIn, so I can't accept anyone else.
• 25:34
  SPEAKER_01: I need to start cleaning out all of that.
• 25:38
  SPEAKER_02: That's a lot of connections for sure.
• 25:40
  SPEAKER_02: Yeah.
• 25:40
  SPEAKER_02: Um okay.
• 25:41
  SPEAKER_02: Well, we'll include links to all of your stuff in the show notes so folks can follow you if they're interested.
• 25:46
  SPEAKER_02: Awesome.
• 25:47
  SPEAKER_02: Okay.
• 25:47
  SPEAKER_02: Well, thanks again for coming on.
• 25:48
  SPEAKER_02: It's been great having you.
• 25:50
  SPEAKER_01: Yeah, thanks so much, Kenton.
• 25:54
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 26:00
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 26:04
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five-star reveal.