Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti
Transcript
- 0:00SPEAKER_03: I try to stay as close as possible to the community and identify what people are struggling with.
- 0:05SPEAKER_03: And based on that, I tend to create content.
- 0:07SPEAKER_03: I try to create the content to match with those struggles and give answers.
- 0:12SPEAKER_03: So that's why sometimes you can find the certification tricky, but it's because that's the kind of issues you will see, or that's the kind of problems that you will have when you will use Airflow.
- 0:22SPEAKER_03: And I want to make sure that you can identify those problems when you will get them and you will have the solution as well.
- 0:45SPEAKER_01: Hey Airflow community, Airflow Curry is here and we are taking it on the road.
- 0:49SPEAKER_01: Join the Astronomer Roadco for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
- 0:58SPEAKER_01: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
- 1:06SPEAKER_01: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
- 1:11SPEAKER_01: Register now.
- 1:12SPEAKER_01: The link is in the show notes.
- 1:15SPEAKER_02: Hi everyone.
- 1:16SPEAKER_02: Welcome to the Data Flowcast.
- 1:18SPEAKER_02: I'm your host, Kenton Danis, and today we have a very special episode where we are flipping things around a little bit.
- 1:25SPEAKER_02: Today, my guest is typically our other host of the podcast, Mark Lumberti, who is the education content lead at Astronomer.
- 1:33SPEAKER_02: Mark, how's it going?
- 1:35SPEAKER_03: Doing great.
- 1:36SPEAKER_03: It's a pleasure to be on the show now as a guest.
- 1:40SPEAKER_03: So that's different than it used to be, but I'm really glad to be here.
- 1:44SPEAKER_02: Yeah, fun to be on the other side of it for a change.
- 1:47SPEAKER_02: Yeah, a little different, but I like it.
- 1:50SPEAKER_02: It's great.
- 1:50SPEAKER_02: I lots of relevant stuff to cover for today.
- 1:54SPEAKER_02: We're gonna discuss what Mark does outside of hosting the podcast, which is a lot.
- 1:59SPEAKER_02: Specifically, we'll talk about kind of the best ways to learn Airflow and how to prepare for the new Airflow 3 certifications.
- 2:06SPEAKER_02: Very hot topic.
- 2:06SPEAKER_02: I think a lot of folks are very excited about this.
- 2:08SPEAKER_02: So I think this will be a helpful resource.
- 2:11SPEAKER_02: Maybe, Mark, since perhaps some listeners know you most from the podcast or just from your presence in the Airflow community, maybe we can start with a little bit of background.
- 2:22SPEAKER_02: You've of course had a huge impact on those using Airflow.
- 2:25SPEAKER_02: I think I saw recently a stat that you have now helped over a hundred thousand people learn Airflow, which is a truly mind-boggling number.
- 2:35SPEAKER_02: But yeah, let's start there.
- 2:36SPEAKER_02: How did you start using Airflow yourself and what prompted you to kind of start with this role of helping learn, helping people in the Airflow community learn how to use it better?
- 2:45SPEAKER_03: Yeah, thank you.
- 2:46SPEAKER_03: That's a great question.
- 2:47SPEAKER_03: I think I think it started in 2016, I believe, because I was working in a bank in France at that time.
- 2:54SPEAKER_03: And I was a data engineer.
- 2:55SPEAKER_03: My background is basically coding, and I was truly like a data engineer at that time, and I was working on a few things in the bank, and basically I was looking for a solution that would help me to avoid coming up at the office on Monday just to realize that we have some issues to fix.
- 3:15SPEAKER_03: And it was pretty usual to have issues every Monday, and I was like, okay, I'm sure there is a way to automate some stuff, to automate scripts and so on, because a lot of things were done manually at that time.
- 3:30SPEAKER_03: And basically, I was looking on the internet and I came across uh Airflow, and I think at that time it was the version 1.10, I believe, or maybe, yeah, or maybe it was in 2018 or 2016.
- 3:43SPEAKER_03: I don't exactly remember the dates.
- 3:45SPEAKER_03: But the point is, when I came across Airflow, I really quickly realized how helpful this tool could be for me and also for data engineers in general.
- 3:55SPEAKER_03: But unfortunately, as it was and it is still an open source project, when I wanted to use it in the bank, my managers were not really okay with that because it was it's an open source project, and obviously the first thing that my manager asked me was, okay, is there any support for that?
- 4:15SPEAKER_03: How do we know if the tool is gonna last long enough?
- 4:19SPEAKER_03: All of those kind of questions.
- 4:21SPEAKER_03: So I I got a bit of frustration, I will say.
- 4:24SPEAKER_03: And then when I look on the internet to find some documentation about Airflow, I didn't find that much at that time.
- 4:31SPEAKER_03: I actually I think there was only one course on Udemy about Airflow, and that course wasn't great.
- 4:38SPEAKER_03: I'm I'm not saying that because I did a few courses after that, but actually because the rating of the course wasn't that high.
- 4:46SPEAKER_03: So what I did basically is I looked at the course, I looked at the reviews, and then I just said to myself, okay, maybe I should create a course because I'm sure people would love to learn Airflow because it's a useful tool for me, and I'm sure it will be helpful for other data engineers.
- 5:02SPEAKER_03: So let's create a course.
- 5:04SPEAKER_03: I created my first course on Udemy on Airflow.
- 5:06SPEAKER_03: I think the course was the complete introduction to complete hands-on introduction to Airflow.
- 5:11SPEAKER_02: And then small aside, I actually took that course before I worked as astronomer.
- 5:17SPEAKER_02: So I think we probably yeah, I think we probably started working with Airflow around the same time.
- 5:22SPEAKER_02: But I had a similar story kind of in my first role where we decided to move to Airflow when I was working as a data engineer from like a legacy scheduler.
- 5:31SPEAKER_02: And one of the things I it was on version 1.9, I think it was back in 2019.
- 5:37SPEAKER_02: And anyways, I remember we were also a little unsure about it because it was it seemed really popular, and there were a lot of kind of blog posts about people choosing Matt versus Luigi or kind of other tools that are no longer really big players in the space.
- 5:51SPEAKER_02: And we were also a little nervous about choosing an open source tool like that, and then actually seem oh hey, there's this Udemy course on it that this guy Mark has done, and it's actually a pretty good course.
- 6:03SPEAKER_02: So maybe that makes it more legit.
- 6:05SPEAKER_02: So we all took it on our team.
- 6:08SPEAKER_03: That's awesome.
- 6:09SPEAKER_03: I didn't know that.
- 6:10SPEAKER_03: I didn't know that.
- 6:11SPEAKER_03: That's funny.
- 6:11SPEAKER_03: Yeah, look, I actually I've been like uh it blew up so quickly.
- 6:16SPEAKER_03: Honestly, I wasn't ready for it.
- 6:18SPEAKER_03: When I've created that course, I wasn't even expecting to leave from my UDME courses.
- 6:24SPEAKER_03: Honestly, I just created that course because I wanted to help as many people as I could about Airflow.
- 6:30SPEAKER_03: But then, yeah, I think like just after a few months, I basically quit my job because I was able to earn enough from my first course, and then I decided to create other courses and it all started like that.
- 6:44SPEAKER_03: And actually, I joined Astronomer thanks to one of my students.
- 6:49SPEAKER_03: Yeah, I think I think it was in 2020 maybe one of my students made a tweet and he tagged Rai, which he was one of the co-founders, and he said, Hey, you should hire this guy, and that's how that's how everything happened.
- 7:05SPEAKER_02: It worked out very well for astronomer, and maybe that's a good next part of the story.
- 7:10SPEAKER_02: So, since you've been here for the past five or so years, probably almost exactly five years, actually.
- 7:16SPEAKER_02: That's funny.
- 7:17SPEAKER_02: You've been working to build out astronomers' education program, including our academy, which has a ton of resources for learning airflow and the certifications.
- 7:26SPEAKER_02: Talk a little bit about your process of building out that work with the folks at Astronomer and kind of why it's important and why you think it's a good focus for us.
- 7:35SPEAKER_03: Yeah, I think I think the I think the academy honestly is like when I started creating the academy, for me, I truly I truly see education as a journey, not just as a one-time event.
- 7:49SPEAKER_03: And the way I and others, because I have I haven't been the only one working on the academy in five years, actually many people helped me along the way, but truly like the way I wanted to design the academy was more like as a journey.
- 8:05SPEAKER_03: So, for example, if you are looking for airflow, you want to learn airflow, then it's just not one course, but it's actually a set of modules or like a learning path.
- 8:14SPEAKER_03: So you actually have some guidance about what you should know to learn the basics of Airflow, for example.
- 8:22SPEAKER_03: Or if you are looking for the how can you author your data pipelines in the best way, then again what you should learn.
- 8:28SPEAKER_03: Because Airflow is not that hard to learn, uh, in my opinion.
- 8:32SPEAKER_03: I might be piazed, but but uh but but it's very easy, I would say, to get lost because you have many features, right?
- 8:41SPEAKER_03: So I think like the thing that you look for when you land on the Astronomer Academy, and what you will find, hopefully, is some guidance about okay, this is where you should start.
- 8:51SPEAKER_03: Uh just follow the path, and then at the end you will get everything you need.
- 8:55SPEAKER_03: So you will feel comfortable about starting with airflow.
- 8:59SPEAKER_03: And then if you want to move so to some advanced features, again, you will have everything that you need on the academy.
- 9:05SPEAKER_03: Did I answer the question?
- 9:06SPEAKER_02: Yeah, yeah, definitely.
- 9:08SPEAKER_02: And yeah, I think that's a really good way of putting it.
- 9:11SPEAKER_02: I often think of Airflow myself as in that category of tools that's like easy to learn, hard to get good at, because it is fairly approachable.
- 9:20SPEAKER_02: And the most basic level of what you're doing, as you said, is pretty easy to pick up.
- 9:25SPEAKER_02: But with Airflow's kind of purposely flexible, it's got a lot of different features.
- 9:30SPEAKER_02: It can be really hard to figure out what you should do and what aspects of it you might not even be aware of.
- 9:35SPEAKER_02: We still run into folks like that, where I was just talking to somebody at Snowflake Summit about thinking they couldn't do something in Airflow, and I brought up a specific feature that was not as well known.
- 9:46SPEAKER_02: And they said, Oh, I've just never heard of that.
- 9:49SPEAKER_02: That perfectly solves my problem.
- 9:50SPEAKER_02: So I think there's so much good stuff in it, but yeah, it can be hard to keep up with everything.
- 9:56SPEAKER_03: And to your point, that's exactly why I keep posting uh stuff about Airflow every day on LinkedIn, is because I feel like I believe as a data engineer, even if you are not a data engineer, but whenever you are working in the tech industry, you have some deadlines to meet.
- 10:11SPEAKER_03: And actually, one reason why I loved teaching, I love teaching things to others is because I have more time to actually learn myself about what I love.
- 10:24SPEAKER_03: And uh, you know, as a data engineer, you just want to know what you need to know as quickly as possible, and that's really the goal of the way I uh the goal of the academy is to uh bring everything you need as quickly as possible so you uh don't miss any features uh that would that could be super useful for you.
- 10:43SPEAKER_03: Right?
- 10:43SPEAKER_03: I think it's actually a very big challenge today, is like we have so many tools that we should know that we work with, but we don't have that much time to learn about those tools.
- 10:54SPEAKER_03: Yeah.
- 10:54SPEAKER_03: So maybe you are using something and you are like, damn, that doesn't work, or that doesn't exist, but actually it does, but you don't know about it.
- 11:02SPEAKER_03: So the question is, how do you make sure that people know about what your tool can do and cannot do?
- 11:08SPEAKER_03: And this is a very challenging question.
- 11:12SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
- 11:22SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
- 11:33SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
- 11:42SPEAKER_02: The whole industry moves so quickly.
- 11:44SPEAKER_02: It's yeah, even when it is your full-time job, like you said, you you often don't have a ton of time to actually do the learning that it would take to kind of get really proficient at a lot of these.
- 11:56SPEAKER_02: I think that's a good natural lead-in to I want to talk about the certifications.
- 12:00SPEAKER_02: I think personally, even though not everybody likes taking tests, but tests are a really good way of reinforcing your learning.
- 12:09SPEAKER_02: And so they're a really good tool in your toolbox, and which I think is part of why Astronomer has created the certifications for Airflow.
- 12:16SPEAKER_02: And we just launched the new version of the Airflow Fundamental certification specifically for Airflow 3, which came out a couple of months ago.
- 12:25SPEAKER_02: So maybe talk a little bit about that and your experience building the new certification and what you want people to take away from it.
- 12:35SPEAKER_03: First, I will say I don't like passing certifications too.
- 12:38SPEAKER_03: Let's be honest, it's a stressful event, it's a stressful event moment.
- 12:43SPEAKER_03: But I think like people should take the certifications as a way to identify their gaps and like what they need to learn.
- 12:51SPEAKER_03: Basically, that's how I will take the certifications.
- 12:54SPEAKER_03: So, for example, the Airflow Fundamentals one is very interesting because even if you are completely new to Airflow, you can still get it as long as you uh are prepared for it.
- 13:09SPEAKER_03: So, for example, if you uh go through the learning path, which is the Airflow 101, okay, you will learn everything you need about the basics of Airflow.
- 13:20SPEAKER_03: And once you are done with that, then you will be ready to pass the certification exam.
- 13:25SPEAKER_03: So this Airflow fundamentals certification truly is just to assess that the fundamentals of Airflow.
- 13:31SPEAKER_03: So when I say fundamentals, DAG scheduling, like how do you schedule a DAG?
- 13:36SPEAKER_03: Like what happens when your DAG runs, when your tasks run, like what is a variable, what is a connection, how do you create your first data pipeline?
- 13:45SPEAKER_03: So some really basic stuff that you should know, otherwise, there is no way for you to actually write use your first data pipelines.
- 13:54SPEAKER_03: Now, the way I design certifications, look, I I try to stay as close as possible to the community and identify what people are struggling with.
- 14:07SPEAKER_03: And based on that, I tend to create content.
- 14:10SPEAKER_03: I try to create the content to match with those struggles and give answers.
- 14:16SPEAKER_03: And so that's why like sometimes you can find the certification tricky, but it's because that's the kind of issues you will see, or that's the kind of problems that you will have when you will use Airflow.
- 14:28SPEAKER_03: And I want to make sure that you can identify those problems when you will get them and you will have the solution as well.
- 14:36SPEAKER_03: That's why that's how I try to design the certifications.
- 14:39SPEAKER_03: And again, I would take certificate uh certificate the certification as a way to identify gaps.
- 14:44SPEAKER_03: And I think the fundamentals one is a really nice way to start with.
- 14:48SPEAKER_02: Yeah, yeah, I agree.
- 14:49SPEAKER_02: Definitely a great way to reinforce that learning when you're especially just getting started with Airflow.
- 14:56SPEAKER_02: And what I'm hearing from this is your recommendation for folks looking to take the exam and pass is that the best way to prepare is to take the course and practice.
- 15:05SPEAKER_02: And right, I think all of the courses on the academy are done in such a way that there is a hands-on component.
- 15:11SPEAKER_02: So you can actually do the things that you're learning about alongside.
- 15:16SPEAKER_02: It's not just watching the videos, which I personally think is really helpful.
- 15:20SPEAKER_02: I can never learn anything until I've done it myself.
- 15:23SPEAKER_02: So I think yeah, that's a really valuable resource for folks.
- 15:27SPEAKER_03: Yeah, you have many quizzes and obusil, if you have any questions or anything, you can still ask.
- 15:33SPEAKER_03: But yeah, like when you land on the academy, you will find basically if you are new to Airflow or if you are new to Astronomer as well, if you are a newer, you will just pick the path that you need.
- 15:42SPEAKER_03: And in the context of Airflow, you will just pick a new to Airflow and you will land on the Airflow 101 learning path and you just follow the path.
- 15:49SPEAKER_02: Yeah, great.
- 15:50SPEAKER_02: That's perfect.
- 15:51SPEAKER_02: And then so looking forward a little bit, I understand there is another course and certification in the works for Airflow 3.
- 16:00SPEAKER_02: Do you want to talk a little bit about that and kind of what's coming for the new certification?
- 16:05SPEAKER_02: Yes.
- 16:06SPEAKER_03: Yeah, yeah.
- 16:07SPEAKER_03: For a while we had the Dagothring for Airflow 2, but basically we're gonna have Dagothring for Airflow 3.
- 16:15SPEAKER_03: And this one is definitely, I would say, a little bit harder than the Airflow Fundamentals one.
- 16:20SPEAKER_03: Just to give you some context, both certifications last for 60 minutes and you have 75 questions.
- 16:28SPEAKER_03: The time is, I would say it can be challenging, especially for Dagothring, maybe, because you have more code snippets, so you have more code to uh to check.
- 16:38SPEAKER_03: But ultimately, DagoThring will be way more focused on okay, do you know like some advanced features to write uh data pipelines in uh in an efficient way?
- 16:52SPEAKER_03: Okay, so for example, do you know the task free API?
- 16:55SPEAKER_03: Okay, do you know how to uh use decorators in your data pipelines to create tasks in an easier and faster way?
- 17:03SPEAKER_03: Do you know how to use assets in your data pipelines?
- 17:08SPEAKER_03: Okay, which is a new totally new way of thinking about your tasks basically, and now you're gonna think about assets instead of tasks.
- 17:14SPEAKER_03: I love that concept by the way.
- 17:17SPEAKER_03: DAG versioning is another is another one.
- 17:19SPEAKER_03: Like when, like what kind of change will create a new version of your DAG?
- 17:27SPEAKER_03: Right.
- 17:27SPEAKER_03: So I would say like everything related to that, how do you create your uh tasks?
- 17:33SPEAKER_03: Do you know task flow?
- 17:34SPEAKER_03: Do you know dynamic task mapping?
- 17:35SPEAKER_03: Right?
- 17:35SPEAKER_03: Are you able to generate tasks dynamically on an input that you don't know in advance?
- 17:40SPEAKER_03: How do you generate DAGs dynamically, not tasks, but DAGs?
- 17:44SPEAKER_03: So all of those advanced features that truly you will use in the real world.
- 17:48SPEAKER_03: Okay, the idea of the DAG authoring is not to challenge you on features that you will never use, it's truly to challenge you on features that you will actually use and will make a huge difference in your workflows.
- 18:02SPEAKER_02: For sure.
- 18:02SPEAKER_02: I think compared to, in my mind, compared to the Airflow fundamentals, the DAG authoring certification is a little narrower in scope, but deeper.
- 18:10SPEAKER_02: So to your point about it is features that you would use.
- 18:14SPEAKER_02: I just took it myself recently.
- 18:16SPEAKER_02: And it's definitely more challenging, I think, especially with identifying things in the code snippets of does this work or is there an issue with it, or is this implemented in a way that is itempotent or whatever it is?
- 18:31SPEAKER_02: But to your point, I think those are often once you've got the hang of just airflow in general on how it works, these are the things that most like data engineers or data scientists are going to be encountering in their day-to-day when they're actually writing pipelines.
- 18:44SPEAKER_02: So it's a great way to, as you said, test your knowledge of whether you're doing things in the most efficient and best practices forward way and identifying those, especially for the Airflow three, the new features that you probably want to be using.
- 18:58SPEAKER_03: Yeah, exactly.
- 18:59SPEAKER_03: And I think ultimately we were discussing about the fact that you may miss some features in the tools that you use.
- 19:06SPEAKER_03: I think the certification like DAGO throwing is also one way to discover features that you didn't know about.
- 19:12SPEAKER_03: Hopefully, you will not discover those features during the exam because you would have taken the prep course for that.
- 19:18SPEAKER_03: But still, like that might happen.
- 19:21SPEAKER_03: You might come across some questions and features that you didn't know about.
- 19:24SPEAKER_03: And again, it's a great way to be like, oh wow, I didn't know that this exists.
- 19:29SPEAKER_03: This is powerful.
- 19:30SPEAKER_03: I'm gonna learn about it.
- 19:31SPEAKER_02: Yeah, yeah, absolutely.
- 19:33SPEAKER_02: We're really excited about the new certification that will be live soon, maybe already by the time this is published.
- 19:41SPEAKER_02: So we will provide links in the show notes to the academy for anybody who wants to take any of the available certifications.
- 19:48SPEAKER_02: We hope that everybody is moving forward to Airflow 3 and learning Airflow 3 and taking those courses and certs.
- 19:54SPEAKER_02: But the Airflow 2 is of course available for anybody who's still working with it as well.
- 19:59SPEAKER_02: Okay.
- 20:00SPEAKER_02: Mark, thank you for going through that.
- 20:01SPEAKER_02: That was great.
- 20:02SPEAKER_02: I always end my episodes with a standard final question.
- 20:06SPEAKER_02: I'm curious to hear your answer to this.
- 20:08SPEAKER_02: And that is, what would you most like to see from the Airflow project in the near future?
- 20:13SPEAKER_03: Wow, I wasn't prepared for that question.
- 20:16SPEAKER_03: Sorry.
- 20:17SPEAKER_03: No, that's okay.
- 20:18SPEAKER_03: No, I think I think I think I guess I I would love to.
- 20:24SPEAKER_03: I would say it's more I would love to see a better integration with AI.
- 20:30SPEAKER_03: And that might sound boring because AI is everywhere today, but that's that that's the truth.
- 20:35SPEAKER_03: Like I I feel I feel that you are seeing all of those orchestrators today, like all of those AI orchestrators.
- 20:41SPEAKER_03: And I don't see why Airflow couldn't be one.
- 20:44SPEAKER_03: Actually, you can already orchestrate your AI agents and so on with Airflow, but I don't think for many people it's obvious today.
- 20:52SPEAKER_03: And I hope that at some point it will be obvious that you can actually orchestrate your AI agents, that you can actually orchestrate your AI workflows with with Airflow.
- 21:01SPEAKER_03: And yeah, that's what I hope in the future is that we will have a better integration for those AI use cases.
- 21:07SPEAKER_02: Yeah, that's a great answer.
- 21:09SPEAKER_02: And I think there are certainly some conversations going on in the dev list about ways to either improve existing features or add new features to Airflow for those types of use cases.
- 21:19SPEAKER_02: I think to your point, you can also very much already do that.
- 21:22SPEAKER_02: And so in that way, your request, I think, is more for our own team than it is for the Airflow project of just providing some guidance on how you might do that.
- 21:31SPEAKER_02: And that is certainly something that we have in the works.
- 21:34SPEAKER_02: Yeah, very topical answer.
- 21:35SPEAKER_02: I love that.
- 21:36SPEAKER_02: Okay.
- 21:37SPEAKER_02: Mark, thanks so much for coming on.
- 21:39SPEAKER_02: It's been great having you on as a guest on the other side of the microphone.
- 21:43SPEAKER_02: What is the best way for folks to get in touch with you?
- 21:46SPEAKER_03: LinkedIn.
- 21:47SPEAKER_03: You just go on LinkedIn, you type my name and connect with me.
- 21:50SPEAKER_03: I will be more than happy to discuss with you.
- 21:52SPEAKER_02: Okay.
- 21:53SPEAKER_03: Love it.
- 21:53SPEAKER_03: Thanks.
- 21:54SPEAKER_03: Okay.
- 21:54SPEAKER_02: Well, thanks again, Mark.
- 21:56SPEAKER_03: Thank you, Caitlin.
- 21:57SPEAKER_03: See you.
- 22:00SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
- 22:06SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
- 22:10SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.