Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino

Transcript

Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: We are helping the different clients to get either like migrate to the cloud provider or either implement their data journey with data.
• 0:08
  SPEAKER_01: And they have strong requirements about okay, we want to have the data as soon as possible ready.
• 0:13
  SPEAKER_01: And it's really even driven real-time approach.
• 0:17
  SPEAKER_01: Now it's starting to become more and more often as a requirement in the data pipeline.
• 0:22
  SPEAKER_01: And at the time was not even driven possible in Airflow, so I come up with the option to use Hub and Apasset.
• 0:34
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:46
  SPEAKER_03: Hi everyone.
• 0:47
  SPEAKER_03: Welcome to the Data Flowcast.
• 0:49
  SPEAKER_03: I'm your host, Kenton Danis, and today I'm joined by Andrea Bambino, who is a co-founder and data analytics team lead at AstraFi.
• 0:58
  SPEAKER_03: Andrea, how are you?
• 0:59
  SPEAKER_01: Really, really good.
• 1:00
  SPEAKER_01: What about you?
• 1:01
  SPEAKER_03: I'm good.
• 1:02
  SPEAKER_03: I'm good.
• 1:02
  SPEAKER_03: It's great to have you on.
• 1:03
  SPEAKER_03: I think you might be our first return guest for the podcast.
• 1:07
  SPEAKER_03: So this is very exciting.
• 1:09
  SPEAKER_03: It's so good.
• 1:10
  SPEAKER_03: And I think I would be remiss if I didn't at least ask in our intro here, how is your winter running going?
• 1:17
  SPEAKER_01: Now it's getting really cold, but still I'll keep on going.
• 1:20
  SPEAKER_01: I'm preparing now the way that I have in December.
• 1:23
  SPEAKER_01: So it's on going well.
• 1:25
  SPEAKER_03: Great.
• 1:26
  SPEAKER_03: Love to hear that.
• 1:27
  SPEAKER_03: I'm sure that will have happened by the time this recording comes out.
• 1:31
  SPEAKER_03: So hopefully someone in our comments can wish you well and check in on the results there.
• 1:36
  SPEAKER_03: I promise we will talk about Airflow.
• 1:38
  SPEAKER_03: We will not spend the whole podcast talking about trail running, though I think we could.
• 1:43
  SPEAKER_03: Yeah, exactly.
• 1:44
  SPEAKER_03: So with that, for our listeners for today's podcast, we're gonna discuss how Astrafy is using Airflow 3's event-driven scheduling feature and PubSub to orchestrate dynamic workflows.
• 1:55
  SPEAKER_03: Before we get into that, for anybody who might not have listened to the first episode, can you tell us a little bit about AstraFi?
• 2:02
  SPEAKER_03: What does your company do and what is your role there?
• 2:05
  SPEAKER_01: Perfect.
• 2:05
  SPEAKER_01: Thank you.
• 2:05
  SPEAKER_01: AstraFi basically is a simple consultancy company that helps clients to differentiate from different industries through their data journey.
• 2:14
  SPEAKER_01: So we help them in order to enrich data and explore data within the model data stack.
• 2:20
  SPEAKER_01: And we are differentiated from the other because we say that we are doers and not talkers.
• 2:26
  SPEAKER_01: And our main provider is Google Cloud, but we are can easily adapt to any cloud provider.
• 2:32
  SPEAKER_01: And we do, as I said, all the model data stack part.
• 2:36
  SPEAKER_01: So from data ingestion, data transformation, and data distribution.
• 2:40
  SPEAKER_01: And we have like pillars that are like infrastructure, data engineering, machine learning, MBI on top of that.
• 2:48
  SPEAKER_01: So read the whole pipeline of the data of nowadays.
• 2:52
  SPEAKER_03: Okay, great.
• 2:53
  SPEAKER_03: And I imagine Airflow features quite heavily in all of that.
• 2:56
  SPEAKER_01: Yes, we are using Airflow in all our clients.
• 3:00
  SPEAKER_03: Okay, love to hear that.
• 3:01
  SPEAKER_03: When you, as we mentioned, were on the podcast last year, you discussed using data sets and Google PubSub to do dynamic DAG triggering.
• 3:12
  SPEAKER_03: We're gonna talk a lot about the next evolution of that.
• 3:14
  SPEAKER_03: But before we get into how your implementation has changed, maybe can you remind our listeners why you need to use that type of scheduling for running your DAGs?
• 3:24
  SPEAKER_01: Yeah, basically, like I had the idea to have a talk because I was working for a specific client where there are the strong dependencies between different DAGs and the data were coming from the same airflow instances, the where the dependencies were needed to have the data as much as possible refreshed when the source DAG has been updating the data.
• 3:46
  SPEAKER_01: We could use, of course, the trigger mechanism of Airflow, but then I was getting to the documentation and I read about the data set concepts.
• 3:54
  SPEAKER_01: And then I just had the idea to integrate the dataset concept with PubSub in order to have this combination between the two in Google Cloud ecosystem in order to have like multiple dial data depending between each other in order to process and get the data in the downstream application as soon as the data are going to be ready.
• 4:14
  SPEAKER_01: And it worked quite well.
• 4:16
  SPEAKER_01: And not only the Tolbete I did with you guys, but also my client was really happy about the solution at the time.
• 4:21
  SPEAKER_03: Yeah, that makes sense.
• 4:22
  SPEAKER_03: And so is this a need that has come up for multiple clients before to have DAGs be more event-driven or more data-driven, as we say, or data aware as opposed to time-driven?
• 4:32
  SPEAKER_01: Yeah, because as I said at the beginning, we are helping different clients to get either like migrate to the cloud provider or either implement their data journey with data, and they are strong requirement about okay, we want to have the data as as soon as possible ready.
• 4:48
  SPEAKER_01: And it's really even driven real-time approach is now starting to become more and more often as a requirement in the data pipeline.
• 4:57
  SPEAKER_01: And at the time, it was not even driven possible in NetFlow.
• 5:00
  SPEAKER_01: So I come up with the option to use Hub and Apasset.
• 5:03
  SPEAKER_01: So definitely is still today an open point, but with the new approach, it's even better.
• 5:08
  SPEAKER_03: That makes sense.
• 5:09
  SPEAKER_03: And so for that original implementation, you relied on PubSub to publish data set events that trigger your DAGs.
• 5:18
  SPEAKER_03: Now, as you just said, there's new functionality available.
• 5:22
  SPEAKER_03: So I think you recently spoke at the Airflow Summit about how you've changed this implementation to make use of the new event-driven scheduling functionality.
• 5:30
  SPEAKER_03: Can you talk about what changed there and what the setup looks like now?
• 5:33
  SPEAKER_01: Yeah, first of all, I want to say that when I was preparing the talk for this year for Airbus Sammy, I was like, okay, what is the subject that I can talk about?
• 5:42
  SPEAKER_01: And I had different topics, and because I all the time I like to prepare myself a bit before and I was reading the documentation.
• 5:47
  SPEAKER_01: Then reading the documentation of Airflow 3.0, I just, okay, I know what to talk about it.
• 5:52
  SPEAKER_01: Because for me, when I just read that the even-driver mechanism was already included in Airflow, I was like, okay, my mind is blowing here because I can just use this new ecosystem inside Airflow to tell a story from where we were last year to where we are right now.
• 6:08
  SPEAKER_01: And so it was for me was like, okay, it's still a really good storytelling, and see how Text to Airflow community and improvement we can use this new feature.
• 6:19
  SPEAKER_03: I think the perfect evolution there.
• 6:21
  SPEAKER_01: Yeah, indeed, exactly.
• 6:22
  SPEAKER_01: It was just like out of the box, ready for me.
• 6:25
  SPEAKER_03: Yeah, that's great.
• 6:26
  SPEAKER_03: And so, how has the new event-driven scheduling feature helped with your setup?
• 6:32
  SPEAKER_01: So basically at the beginning, without having needed the data set needs, I was trying to use, for example, the pub self operator.
• 6:39
  SPEAKER_01: Okay, the pub sub operator, you could sense data with it, but all the time there was a problem that because okay, sensor like taking a lot of consumption of the workload, all this kind of stuff.
• 6:49
  SPEAKER_01: And as well, uh then when after the sense, you need to like retrigger the data.
• 6:54
  SPEAKER_01: So there was like not really this event-driven logic behind.
• 6:58
  SPEAKER_01: And for the three or the zero, there is this mechanism that is called message Q trigger that uses the new component, the Airfoot trigger, that's now is working in Airfoot 3.0, that you can just subscribe to it and then define your own uh trigger event uh operator, or you can just use them, the one that are ready for you.
• 7:18
  SPEAKER_01: At the time of the talk is a funny thing because uh when I try to prepare the PubSub event driven watcher, it was not ready.
• 7:26
  SPEAKER_01: So I implement my own with the idea that okay, I could collaborate when I contribute to the community with that doing a PR and just okay, maybe it can be like my first contribution to the community, but actually it was not because the same day where in which I present the talk to someone else the day before the day after, I think it did PR about the PubSub watcher.
• 7:46
  SPEAKER_01: So it was like okay, another time.
• 7:48
  SPEAKER_01: Uh but I also what was great was really the straightforward implementation of just reading documentation and use directly in my code, the inheritance of the class, and like seeing how easily I could implement this with PubSub and easily without doing any tricky aspect in my Apple instance.
• 8:07
  SPEAKER_03: Okay, interesting.
• 8:08
  SPEAKER_03: It's funny that you mentioned that because I was just checking today, and I think that PR was actually reverted, that there was some question amongst the Google team on how they wanted it to work with their provider.
• 8:21
  SPEAKER_03: So I hope that this has changed by the time this episode is coming out and that it is officially supported.
• 8:27
  SPEAKER_03: But I think that at least as of recording, it has not been merged yet officially.
• 8:33
  SPEAKER_01: Okay.
• 8:34
  SPEAKER_01: Yes, I went and I need to reach the developer because on the day after the guy came to me and said, Oh, Andrea, I'm so sorry, we already merged the PR, but then maybe something changed.
• 8:43
  SPEAKER_03: Yeah, it looks like it was reverted.
• 8:45
  SPEAKER_03: So unless I miss something, maybe they've re-merged.
• 8:48
  SPEAKER_03: But in any case, I know for anybody listening who's not aware, at least at the time we're recording this, I think the asset watchers that exist in open source right now are for SQS and Kafka.
• 9:01
  SPEAKER_02: Yes, okay.
• 9:02
  SPEAKER_03: But then of course, as you're saying, you have the asset watcher class exists and anybody can build off of it so you can implement your own.
• 9:09
  SPEAKER_03: And it sounds like for you, at least with PubSub, it was quite easy to do that.
• 9:13
  SPEAKER_01: Yeah, quite easy because there was already an operator that I could use of PubSub.
• 9:18
  SPEAKER_01: And Google is like a kind of big supporter for Airflow as a readable operator and library that you can just use in Python to implement and use the PubSub option in order to just subscribe to a specific topic in Google.
• 9:31
  SPEAKER_01: And so adjusting everything in the class and using the Google Cloud provider of Airflow was really easy to use it.
• 9:40
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 9:49
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:01
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 10:10
  SPEAKER_03: That's really cool that it's implemented in a way that you're able to build off of it like that, even for more custom solutions.
• 10:16
  SPEAKER_03: I know a lot of people who are implementing things in Airflow for clients sometimes stray away from doing too much customization because it gets hard to maintain over time, especially if it's then handed over to a client to maintain as opposed to yourself.
• 10:30
  SPEAKER_03: But it sounds like in this case it's relatively easy.
• 10:33
  SPEAKER_01: Yeah, it was relatively easy.
• 10:35
  SPEAKER_01: And then also like now if the merge request is going to be there and the queue is going to be available for everybody.
• 10:40
  SPEAKER_01: Anyone that's going to use it in Google Cloud ecosystem, you can use easily.
• 10:43
  SPEAKER_01: And as well, not only this one change, but as well the new concept of data set became as an asset.
• 10:49
  SPEAKER_01: And as well, the new asset in M43.0 is really well improved because we can see as well all the time, the update of data set it has been done.
• 10:57
  SPEAKER_01: We can update the data set manually, asset manually or from the UI.
• 11:01
  SPEAKER_01: There are a lot of improvement on this on this topic as well.
• 11:04
  SPEAKER_03: Yeah, definitely the new asset interface is really powerful.
• 11:08
  SPEAKER_03: And I'm curious with either that or the asset watchers for event-driven scheduling, now that you've implemented this in both ways, the Airflow 2 version and the Airflow 3 version.
• 11:18
  SPEAKER_03: What are the benefits of the Airflow 3 version given that the end state, I think, is probably basically the same as that you're able to trigger DAGs when these events come into PubSub.
• 11:30
  SPEAKER_03: But is it more efficient now?
• 11:32
  SPEAKER_01: Or the thing is like it's more efficient because before there was a delay or there was not really an event-driven approach.
• 11:39
  SPEAKER_01: It was like a kind of scheduler we need to pass to a change the asset, and there was not really applicable this event-driven concept.
• 11:47
  SPEAKER_01: But now, like whenever there is a subtopic that's pushed in the topy, then we can have our multiple DAG that are just subscribed for the specific asset that are going to just be triggered automatically.
• 11:58
  SPEAKER_01: And this is kind of powerful.
• 12:00
  SPEAKER_01: And and as well, like me in the way that I use it right now, is it was a bit more say elegant because I was having triggering all my DAs they were depending from a specific table in BQuery.
• 12:12
  SPEAKER_01: So basically, all the time there was a change in a big query table, then other desks could start to be triggered and to load their data.
• 12:19
  SPEAKER_01: Because this is the Ashwiz idea that you can have a transformation process that is doing from the ingestion, that data transformation, and then updating the final data table that later on needs to be used as a source for other DOS triple application.
• 12:31
  SPEAKER_01: You can just use this even-driven approach with the asset and PubSub, and it's working really perfectly.
• 12:39
  SPEAKER_03: Yeah, interesting.
• 12:40
  SPEAKER_03: So I was also gonna ask since you've implemented this with PubSub, but I know and we were talking about other asset watchers that are available in other systems, but it's pretty common for people to want to schedule based on a table being updated or something like that.
• 12:56
  SPEAKER_03: And I know there are open AIPs in Airflow right now discussing the best way to implement watchers in the context of not a message queue where you would have to maintain some sort of state within Airflow or watermarking or something like that.
• 13:11
  SPEAKER_03: And it's quite challenging, I think, in terms of the discussion on the best way to implement that, which is why it's still ongoing.
• 13:17
  SPEAKER_03: But the eventual goal being that there would be watchers available for, say, Snowflake tables or big query tables or something like that.
• 13:27
  SPEAKER_03: Based on your work with your clients currently, how are you managing that where PubSub might not be necessarily the natural thing that they're watching?
• 13:36
  SPEAKER_03: Are you just inserting that in the middle to make everything work?
• 13:40
  SPEAKER_01: Exactly.
• 13:40
  SPEAKER_01: It's going to be like kind of met in the middle.
• 13:50
  SPEAKER_01: And then the asset is going to either do like just simply an asset change.
• 13:55
  SPEAKER_01: So this one is going to trigger not the DAG as I was doing in 2.9.
• 13:59
  SPEAKER_01: Or either that using using this pub because basically the airflow instance is in another project that is depending from another team or whatever.
• 14:06
  SPEAKER_01: And this actually is what I'm really looking forward for one of the next features, but I will not spoil it.
• 14:12
  SPEAKER_01: I will try to see if it's going to be implemented someday.
• 14:15
  SPEAKER_03: Okay, interesting.
• 14:16
  SPEAKER_03: Yeah, that makes sense.
• 14:17
  SPEAKER_03: And I would imagine that it's relatively straightforward to implement something similar in other clouds or depending on what tools folks are using to have the message queue be that thing in the middle.
• 14:29
  SPEAKER_03: But yeah, definitely a lot of demand for watching something that's not a message queue, file system, database, something else like that.
• 14:37
  SPEAKER_03: Yeah, cool.
• 14:38
  SPEAKER_03: I think that's a nice tie-in, maybe related or maybe not to always my standard final question that I like to ask folks is what would they like to see from the Airflow project in the near future?
• 14:49
  SPEAKER_03: So maybe other than additional watcher capabilities, anything you're looking forward to?
• 14:54
  SPEAKER_01: Actually, yes, no requests, but improvement that I would like to have.
• 14:57
  SPEAKER_01: The first one is considering now this data governance is coming more and more like requests within a different team, and we would like to have different that in one airflow instance that are not going to be triggered by everybody or access from everybody.
• 15:11
  SPEAKER_01: So like something that can be like role-based access on Airflow with a specific rule that because today I find it like that the rules and the user in Airflow are a bit tricky to maintain and they are not really easily flexible, let's say.
• 15:25
  SPEAKER_01: And secondly, I would like to see if it's gonna be a possibility to have when you have different airflow instances that they don't know each other, to have maybe like an asset in the middle or whatever that can make the link between these two instances, because it actually is at the end is airflow.
• 15:43
  SPEAKER_01: And say if I update this asset that is refers as well in another instance of airflow, maybe then my other DAG can be triggered or something like that.
• 15:51
  SPEAKER_01: So I ought to see, for example, having a non-multi-dag in the same instance, but multi-dug in multiple different instances together.
• 15:59
  SPEAKER_03: Those are both good.
• 16:00
  SPEAKER_03: And I think even maybe a little bit related, it sounds like the first one you're talking about with DAG level RBAC, I think is a really common topic that comes up.
• 16:07
  SPEAKER_03: It's certainly something that we hear about from our customers at Astronomer a lot and have been thinking about the best way to implement, but it definitely impacts how you decide to set up your airflow infrastructure, whether or not you have different deployments.
• 16:22
  SPEAKER_03: And I think that's one of the really common push and pulls all the time is you don't want to have everybody have access to all of the DAGs, but there are dependencies between the DAGs.
• 16:31
  SPEAKER_03: So do you put them in the same environment or do you put them in different environments?
• 16:35
  SPEAKER_03: And then in either case, how do you manage that?
• 16:38
  SPEAKER_03: So definitely I think something where airflow could improve.
• 16:42
  SPEAKER_03: And the implementations that I've seen for cross-deployment DAG dependencies are, I mean, it's possible, but a little bit weird.
• 16:54
  SPEAKER_03: Yeah, a little tricky.
• 16:56
  SPEAKER_03: I think with assets in Airflow 3, it's better, at least with the API triggering or for adding an asset event, but it still requires some setup on the part of the user to figure out what system is gonna make that update.
• 17:11
  SPEAKER_03: Yeah, interesting.
• 17:12
  SPEAKER_03: Well, that's really cool.
• 17:13
  SPEAKER_03: Well, Andrea, it's been really great talking to you.
• 17:15
  SPEAKER_03: I think really love that it worked out for you to come and talk about the next evolution of this implementation.
• 17:22
  SPEAKER_03: What a cool use case for this new Airflow 3 functionality and a really good highlight of how it's helped in a particular case.
• 17:30
  SPEAKER_03: Love that you were able to talk about that at Airflow Summit and here on the podcast.
• 17:34
  SPEAKER_03: Just to close out, what is the best way for folks to get in touch with you?
• 17:37
  SPEAKER_01: LinkedIn, actually, and you can find me isn't in LinkedIn, or if you want to have to shoot any question, you can just send me an email to my personal account.
• 17:45
  SPEAKER_01: It's Andrea at astrophie.io.
• 17:47
  SPEAKER_01: And uh simple as it is.
• 17:49
  SPEAKER_03: Okay, great.
• 17:50
  SPEAKER_03: We'll throw those in the show notes.
• 17:51
  SPEAKER_03: And I imagine if anybody caught on to my running chat of the beginning, Strava.
• 17:57
  SPEAKER_02: Yeah, Strava, yeah, exactly.
• 17:59
  SPEAKER_02: Strava.
• 17:59
  SPEAKER_02: Strava, I really I more like updating Strava than my LinkedIn, but uh yeah.
• 18:04
  SPEAKER_03: Yes, I totally relate to that.
• 18:06
  SPEAKER_03: We can throw that in the show notes as well, too, in case anybody is interested.
• 18:10
  SPEAKER_03: If anybody's following along, they can check out how your marathon went.
• 18:14
  SPEAKER_01: Yeah, definitely.
• 18:15
  SPEAKER_03: Yeah, okay.
• 18:16
  SPEAKER_03: Well, thank you again for coming on and best of luck with the winter running and everything at Astrophy.
• 18:21
  SPEAKER_03: Thank you so much.
• 18:24
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 18:31
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 18:35
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star reveal.