Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko

Transcript

Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:05
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:17
  SPEAKER_02: Okay.
• 0:17
  SPEAKER_02: Hi everyone.
• 0:18
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:20
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Igor Teresenko, who is a data and AI engineer at Ponder Labs.
• 0:28
  SPEAKER_02: Igor, how are you?
• 0:30
  SPEAKER_01: Great.
• 0:31
  SPEAKER_01: How are you doing?
• 0:32
  SPEAKER_02: Yeah, I'm good.
• 0:33
  SPEAKER_02: I'm good.
• 0:33
  SPEAKER_02: Thanks.
• 0:34
  SPEAKER_02: It's great to be chatting with you today.
• 0:36
  SPEAKER_02: Yeah, it's wonderful to have you on you're somewhere in Europe.
• 0:39
  SPEAKER_02: Is that correct, if I remember right?
• 0:41
  SPEAKER_01: I'm an Israel kind of way.
• 0:42
  SPEAKER_01: Oh, an Israel.
• 0:43
  SPEAKER_01: Okay, sorry.
• 0:44
  SPEAKER_02: I got that completely wrong.
• 0:46
  SPEAKER_02: Um, it's great to speak with you in a very different time zone from myself.
• 0:52
  SPEAKER_02: Appreciate you making the time.
• 0:55
  SPEAKER_02: Uh we'll go ahead and dive right in then.
• 0:57
  SPEAKER_02: So for today's podcast, we're going to discuss how Ponder Labs customizes Airflow for their clients using plugins.
• 1:04
  SPEAKER_02: I'm super excited to talk about this, very timely with all of the work in the project that has gone on with plugins in the past, I'll call it six months or so, maybe even more with Airflow 3.
• 1:16
  SPEAKER_02: Uh, but before we get into all of that, let's back up a bit, start with some background.
• 1:21
  SPEAKER_02: Uh, can you tell our listeners what Ponder Labs does?
• 1:24
  SPEAKER_01: Yeah, sure.
• 1:25
  SPEAKER_01: So uh as you said, uh I'm data and AI engineer at Ponder.
• 1:28
  SPEAKER_01: And in simple terms, we we basically help education organizations make better use of their data.
• 1:35
  SPEAKER_01: Um, schools and education systems usually have data coming from various different systems.
• 1:42
  SPEAKER_01: And uh our job is basically to bring that data together so it becomes useful.
• 1:47
  SPEAKER_01: Uh, for example, teachers of uh our clients can look at assessment data from students and work to notice some gaps in learning uh or in teaching and to use that to help their teaching basically.
• 2:02
  SPEAKER_01: And as well, schools manager, school leaders can see trends across schools, across different areas, across different uh programs.
• 2:12
  SPEAKER_01: So, yeah, I think like that's more or less in a nutshell what we do.
• 2:16
  SPEAKER_01: And me as a data engineer, um, my uh main goal is to make sure that teams, uh school teams can rely on a consistent, trustworthy data from for for their decisions and for their work with students and in schools, right?
• 2:34
  SPEAKER_01: So um, yeah, so um we basically build many different data platforms, analytics tools.
• 2:41
  SPEAKER_01: We uh pretty fairly often contribute to open source.
• 2:45
  SPEAKER_01: Uh so probably we will talk about it um about open source plugins for Airflow for DBT, mainly like data engineering uh tools.
• 2:53
  SPEAKER_01: Uh and also recently, probably as the whole world, we try to do more stuff in AI port solutions.
• 3:00
  SPEAKER_01: Obviously, that's probably all the rich right now.
• 3:03
  SPEAKER_01: Yeah, so um yeah, so we so basically um that's pretty much it.
• 3:10
  SPEAKER_01: Again, on the data engineering side, we uh we mainly like I think the vast majority of our clients use Airflow because we build an operative data infrastructure for our partners, and uh, this basically includes like the whole pipeline from the ingestion and to the uh BI stuff or uh reverse ETL.
• 3:32
  SPEAKER_01: Uh so we sort of uh we have to build some sort of uh orchestrator that will you know handle this whole flow of the data from its ingestion and to the end, like to the final user.
• 3:46
  SPEAKER_01: So uh yeah, we use like often we use airflow, so it's uh yeah, it's a it's a pleasure to be here.
• 3:53
  SPEAKER_01: Yep.
• 3:53
  SPEAKER_01: That's that's pretty much it.
• 3:56
  SPEAKER_02: Yeah, that's great.
• 3:56
  SPEAKER_02: I have a couple of follow-up questions.
• 3:58
  SPEAKER_02: One, I'm just curious for you, you said you work with schools and the education systems.
• 4:03
  SPEAKER_02: Is this universities or primary school systems or both?
• 4:07
  SPEAKER_02: Is it everything?
• 4:09
  SPEAKER_01: So it's mainly chariot schools in on the east coast of the United States.
• 4:15
  SPEAKER_01: Mainly, so so it's basically, yeah, I think um that's the reason that they need us because they have many, many uh schools, and it's not like the management of one particular school.
• 4:27
  SPEAKER_01: They have many different schools across different states, and they need to they need some way to work efficiently with their data and basically handle all these schools together from from the management office, right?
• 4:41
  SPEAKER_02: Yeah.
• 4:41
  SPEAKER_02: Um very interesting.
• 4:43
  SPEAKER_02: And I think you touched on this a little bit, but tell me more about the use cases that Airflow is supporting for these schools, and that you mentioned a little bit is providing their data and analytics, but what does that look like in a little more detail?
• 4:57
  SPEAKER_01: Sure.
• 4:58
  SPEAKER_01: So um maybe you know that all the educational organizations have a lot of different source systems uh when they uh work with their data.
• 5:06
  SPEAKER_01: So in order to bring this to some data warehouse to Data Lake and work efficiently with this, uh they need some single place, first of all, to ingest this data.
• 5:17
  SPEAKER_01: So we use Airflow in order to orchestrate all these pipelines from the data ingestions to to transform and then to to promoting into production.
• 5:27
  SPEAKER_01: But basically, yes, uh, it starts from the data ingestion, and we use it like there are totally different uh ways to ingest the data from uh Trino to just basic Python models and uh and all the other stuff.
• 5:42
  SPEAKER_01: But like our idea and our main rule is to no matter what we use to ingest the data, everything must be triggered from the Airflow.
• 5:51
  SPEAKER_01: Because once we do it, we can handle the dependencies and we can see the whole picture in one place.
• 5:57
  SPEAKER_01: So even if we use tools for some particular systems that are not like their logic is not integrated in Airflow in any way, we still want uh Airflow to be a scheduler for this process.
• 6:09
  SPEAKER_01: So we still want to be Airflow to be triggered, and it's also true for transformation process.
• 6:16
  SPEAKER_01: I think in the vast majority of cases we use dbt and uh sometimes it's uh dbt core, in some cases it's dbt cloud, and like obviously in dbt core, we just need this whole infrastructure and operator, et cetera, et cetera.
• 6:28
  SPEAKER_01: And when we use dbt cloud, we just need to trigger the job in in dbt cloud, but we still want to do it in Airflow, even though uh dbt cloud has their all-window scheduling system, et cetera, et cetera.
• 6:38
  SPEAKER_01: But like we just want this one single ecosystem when we have uh an opportunity to define all the different dependencies between different tasks, jobs, and just have this, you know, yeah, this sort of DAG of DACs, right?
• 6:52
  SPEAKER_01: Uh, this big lineage of processes.
• 6:56
  SPEAKER_01: Yeah, and and finally we have uh also reverse details.
• 7:00
  SPEAKER_01: Uh so sometimes we have to return the data to the uh back to the operational systems, and after they the data was transformed, sometimes we need to promote the data to to some other places.
• 7:14
  SPEAKER_01: And yeah, so that's basically, I think I would say that yes, we have these three main groups of DACs, which is data ingestion, data transformation, and uh reverse ETLs.
• 7:25
  SPEAKER_01: And I think that yeah, uh there uh as usual, like there are many minor DACs for this and that purpose.
• 7:32
  SPEAKER_01: Um, but yeah, I think I would say that this this is pretty much the way we use it.
• 7:36
  SPEAKER_02: Great.
• 7:36
  SPEAKER_02: That's very helpful.
• 7:38
  SPEAKER_02: And I think if I understood you correctly, when you initially described your setup, you're providing this data platform that is built on Airflow and the analytics on top of it to each individual client.
• 7:51
  SPEAKER_02: So they are actually interacting with Airflow or the platform themselves.
• 7:55
  SPEAKER_02: Is that accurate?
• 7:57
  SPEAKER_01: Yeah, that's right.
• 7:58
  SPEAKER_02: Okay, that's very cool.
• 8:00
  SPEAKER_02: And I given that you have this setup where your clients are interacting with Airflow directly, what are some of the biggest challenges that you've faced using Airflow in this way?
• 8:09
  SPEAKER_01: So I think that as data platforms grow, both in complexity and uh in in in size and complexity, it basically is very logical that the the whole uh management process becomes much more complicated.
• 8:25
  SPEAKER_01: I think it's pretty much natural.
• 8:27
  SPEAKER_01: And we have like for for the same client, we can have many DACs created by different teams within the same organizations, and we would like to keep it one cluster per client.
• 8:39
  SPEAKER_01: I mean, like um not including uh the development and uh staging in other environments, but I mean like the one production cluster per one client, uh, because uh it's just more um, I think like it's more comfortable to manage all the pipelines in one in one cluster.
• 8:55
  SPEAKER_01: So yeah, uh so we just have in the same cluster, we have many different DAGs that were created by different teams, all of them running in the same cluster.
• 9:04
  SPEAKER_01: And basically at the scale, it becomes important to focus on structure, visibility, and consistency because otherwise it's just very hard to understand what's going on when you have like, I don't know, like 200 DAGs and uh a lot of different dependencies.
• 9:19
  SPEAKER_01: Like Airflow really helps by creating these dependency systems of uh data sets and triggers and and sensors, etc.
• 9:26
  SPEAKER_01: But it's still uh it's still pretty much because otherwise it just probably I think it would be impossible to manage this whole thing uh at all.
• 9:34
  SPEAKER_01: But now it it's possible with the dependencies, uh cross-deck dependencies, but uh it's still pretty complicated when you have different DAX and like some team, which is I think it's pretty natural to to treat different DAX as a black box, right?
• 9:47
  SPEAKER_01: Because we we cannot know what's going on in each one of them.
• 9:51
  SPEAKER_01: And it's I think it's a good thing that you just know that there is some DAG that your DAG depends on, and there is some DAG that depends on your DAG, and you have no idea what exactly they do, but you sort of know you treat it as an API.
• 10:03
  SPEAKER_01: Uh and I think it's a good thing about uh about DAGs, but um at the same case when there's an error happens somewhere there in the upstream uh DAG, you and it can be like five, six levels above, it can be pretty much complicated to understand what's going on and like what's what's the reason there.
• 10:20
  SPEAKER_01: So I think I think that's that's the the biggest issue that we have.
• 10:24
  SPEAKER_02: Yeah, of course.
• 10:25
  SPEAKER_02: I think that totally makes sense.
• 10:26
  SPEAKER_02: And it's uh one of the reasons on astronomer's side, I think we're very familiar with this.
• 10:30
  SPEAKER_02: And one of the reasons we've built our observability product uh into our platform for exactly that reason when you've got lots of teams using Airflow, and if you do run into an issue, debugging it can be really hard when it's happening very upstream or or something like that.
• 10:46
  SPEAKER_02: So yeah, I think that totally makes sense.
• 10:49
  SPEAKER_02: Uh I do want to dig into the plugins piece uh that I mentioned earlier, because I think this is really interesting what you've done there.
• 10:58
  SPEAKER_02: So I understand that you've built a couple of custom plugins to extend airflow to help your clients.
• 11:04
  SPEAKER_02: So tell me about what those are and what they do.
• 11:07
  SPEAKER_01: Yeah, sure.
• 11:08
  SPEAKER_01: So uh the first one, I think uh it's highly related to the same topic that we just talked about.
• 11:12
  SPEAKER_01: We call it schedule insights, and we had this problem of like before we moved to the trigger dependent DAGs, we had just a lot of different schedules, and obviously that becomes just impossible to manage as your complexity grows because you have to change schedules over there in the absolute DAG, and it means that you have to schedule, reschedule all the dependent DAX, uh, and obviously you forget something.
• 11:38
  SPEAKER_01: So uh we migrated to the event trigger DAX, but we suddenly faced other problems that it's fairly complicated thing to predict when a particular DAG is going to run, when it depends on five, six layers of other event-driven DAX.
• 11:57
  SPEAKER_01: So we created this schedule insights that basically it's basically a graph and you just a GAN chart.
• 12:04
  SPEAKER_01: You can you can take a look on a grant on a GAN chart and understand what is the next time the particular DAG is going to run, even if it's schedule, even if it's event trigger DAG.
• 12:14
  SPEAKER_01: And basically the way it works is just looks on the the looks at all the dependent schedule DAGs.
• 12:22
  SPEAKER_01: Search for average duration for each one of the layers below, and somehow uh it like it's not exact duration, not exactly start time, but it pretty much accurate in the possibility of cases.
• 12:36
  SPEAKER_01: Uh obviously it's much more complicated than it because you have regularly not one schedule DAX, you have like seven, and uh, I think uh starting from Airflow to nine, I think you can like define really complicated conditions for triggering a particular DAG.
• 12:52
  SPEAKER_01: You can like and or you can uh schedule your DAG on scheduling and on other event trigger DAC.
• 12:59
  SPEAKER_01: So yeah, it can be pretty much complicated to handle together.
• 13:03
  SPEAKER_01: Uh and basically, yeah, uh this this uh plugin just uh has this general image of uh we have all these DACs, half of them are scheduled this the next time that they're gonna run, and these DAGs are event triggered, and it's highly likely that they're gonna run in a couple of hours.
• 13:21
  SPEAKER_01: And then you can see in the gun chart all the parallel running DAGs, and sometimes it can you know answer some questions about uh how's it possible that like I run this DAG and once in a while the table is dropped to the same time or something.
• 13:34
  SPEAKER_01: So I think like uh the vast majority of us uh tried to solve this problem uh at least once in a while.
• 13:40
  SPEAKER_01: So I think, yeah, so I think uh yeah, it it it it always helped us.
• 13:44
  SPEAKER_01: And I think the second plugin is also somehow relevant to the same problem.
• 13:49
  SPEAKER_01: As I said, when your um complexity grows, uh it's it becomes real complicated to understand um who's responsible for what pace in your cluster and what's what's exactly going on.
• 14:02
  SPEAKER_01: Basically, especially I think people who would like to know this thing are product managers because they are sort of above all all the different teams and they would like to understand what's going on in the cluster.
• 14:13
  SPEAKER_01: And also at the same time, project managers aren't the ones who work with the code exactly and not ones who would like to join there and and spend hours probably in order to understand what's going on uh in in Python.
• 14:28
  SPEAKER_01: So um as I think like the the um answer, like I think usual answer in the in the last couple of years is AI.
• 14:37
  SPEAKER_01: It was pretty much answer for everything.
• 14:39
  SPEAKER_01: So uh here uh we actually tried to make it work.
• 14:42
  SPEAKER_01: And we created Airflow chat, which is basically UI chat that uses Airflow MCP.
• 14:49
  SPEAKER_01: Uh it's also something that we created.
• 14:51
  SPEAKER_01: Like I know that there are pretty much plenty of them right now.
• 14:55
  SPEAKER_01: Uh so we just used ours, but I believe like we can use anything and it will still uh work great.
• 15:01
  SPEAKER_01: And basically, just uh, you know, uh for us, it was important that project manager could uh go into just some tab in Airflow and ask how many DACs are created by this or that team, how many DACs are owned by some particular uh group of people, and and basically just understand cross-dependency between different DAGs, uh, what's what are the tasks uh in some DAG.
• 15:24
  SPEAKER_01: And it's also I think it's also become like people started trying to write the DAGs as simple as possible because they know that AI will take a look on it, and uh as a result, like the some some places are uh just much more no, it's it's it sort of became in some in some way it became like semantic layer of our pipeline.
• 15:46
  SPEAKER_01: So uh people just start describing what they're doing exactly, etc.
• 15:50
  SPEAKER_01: etc.
• 15:50
  SPEAKER_01: Because they know that this information will be actually used uh by someone.
• 15:54
  SPEAKER_01: Yeah, and um, I think that like less exciting things uh are plug-in.
• 16:00
  SPEAKER_01: We build some plugin in order to manage permissions for our data warehouse, uh, because yeah, yeah, it's a long story short, uh, just we had a couple of limitations from other products, so we could not use uh IM roles and users for a couple of our clients.
• 16:16
  SPEAKER_01: So we had to use just um good old um roles and users with passwords.
• 16:23
  SPEAKER_01: Uh so in order to make it work and not just you know go to the database and querying all the users and their roles and like uh have all this like just dozens of different queries uh that you have to run often.
• 16:35
  SPEAKER_01: We just create this beautiful UI where you can create user, attach some particular role to particular user, and just have this mapping on UI.
• 16:44
  SPEAKER_01: It's just fast and simple.
• 16:46
  SPEAKER_01: Yeah, and and I think like even something even more I would say like something that is specifically was created for our use case, is that we use um ECS for some of our clients, and ECS uh still has this another thing which is called task definition.
• 17:06
  SPEAKER_01: And sometimes you would like to play with different task definitions for different runs.
• 17:10
  SPEAKER_01: So we created this uh additional plugin where you can sort of choose different versions for different logics in order to execute them and check what what's going on.
• 17:20
  SPEAKER_01: Like it's usually um used in stage environment because like people just would like to merge something and then test how it was changed, like how particular logic was changed and how it changes particular data in some tables.
• 17:34
  SPEAKER_01: But yeah, it's still it's still something that um is being used by us.
• 17:37
  SPEAKER_01: So yeah, we uh I think that like my go-to rule is that once you have some problem you would like to integrate into Airflow, just create um custom plugin uh and it will solve the vast majority of your problems.
• 17:50
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 18:00
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 18:12
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 18:20
  SPEAKER_02: Yeah, that's great.
• 18:21
  SPEAKER_02: That's a lot of customization that you've done.
• 18:24
  SPEAKER_02: Uh, and those are all really interesting problems.
• 18:27
  SPEAKER_02: Maybe one brief follow-up question.
• 18:30
  SPEAKER_02: I I guess this probably applies mostly to the plugin that's predicting when DAGs are going to run, but could potentially apply to some of the other situations as well.
• 18:39
  SPEAKER_02: I'm curious what your experience has been with how I guess bulletproof they are in terms of anomalies coming up.
• 18:46
  SPEAKER_02: Like how often do you have a scenario that's sort of outside the norm that then throws off whatever the prediction is or even the model is that's you know, returning, answering questions, I should say, about what's going on in an Airflow instance.
• 19:02
  SPEAKER_01: Yeah, sure.
• 19:02
  SPEAKER_01: So I think each major change that somehow changes the duration of the DAG, somewhere there in the in the this whole chain of different DACs, obviously can affect uh the prediction because uh that's exactly how how it's working.
• 19:18
  SPEAKER_01: It basically just looks on the history of different runs.
• 19:20
  SPEAKER_01: And if like your current run is different from the medium time of medium duration uh of the same uh DAG in in the previous couple of weeks or like just previous couple of runs, then your um your current duration, like predicted duration uh will be also problematic.
• 19:40
  SPEAKER_01: And the second issue that we uh we actually faced with uh with the same uh plugin is that sometimes we'd like to build one DAG for different purposes.
• 19:52
  SPEAKER_01: So, for example, you would like to just handle a different task by some parameter.
• 19:58
  SPEAKER_01: For example, you have a DAG of data ingestion, and you have a parameter that handles whether it's incremental ingestion, or you just trunk it and insert all the data.
• 20:08
  SPEAKER_01: So obviously, the duration is significantly different from time to time.
• 20:13
  SPEAKER_01: And like someone can just execute manual run in order to just clean and re-ingest all the data from this from scratch, and it affects uh the duration of prediction.
• 20:25
  SPEAKER_01: And uh there, like we try to basically uh implement a few worker runs to handle this particular situation.
• 20:31
  SPEAKER_01: For example, take a look on more scheduled uh runs than manual runs and stuff like this.
• 20:37
  SPEAKER_01: But yes, basically each time there is some significant change that uh changes the duration, it can affect uh the prediction time, or it can basically just you know not affect that like the prediction time will be based on the previous runs and it's it for now it has nothing to do with the current run.
• 20:54
  SPEAKER_02: Yeah, yeah, interesting.
• 20:55
  SPEAKER_02: Yeah, well, that's very cool that you're still able to provide some insight in those cases.
• 21:00
  SPEAKER_02: I'm curious with all of these plugins that you've developed, what has been your experience generally working with the new plugin interface in Airflow 3?
• 21:08
  SPEAKER_01: So uh yeah, so uh actually it's it's it first of all, it was great.
• 21:12
  SPEAKER_01: For now, like all the plugins that I mentioned, we built them for, we started building them for Airflow 2.
• 21:18
  SPEAKER_01: And then right now we are continuing to migrate to Airflow 3.
• 21:23
  SPEAKER_01: So we started looking on how to migrate all of our plugins to Airflow 3.
• 21:29
  SPEAKER_01: And um, for some of them, it was significantly easier than uh I was thinking.
• 21:34
  SPEAKER_01: And uh obviously it's great at like this modern fast API-based interface.
• 21:39
  SPEAKER_01: First of all, uh, I think it's great because uh it's much more generic than it was, even though it was generic with uh uh with a flask.
• 21:46
  SPEAKER_01: But I think that fast API based interface is it's just the best solution that uh Airflow community could could take.
• 21:52
  SPEAKER_01: Uh, and also I think that APIs started being more, I would say, clearer and more predictable than than they were.
• 21:59
  SPEAKER_01: Um Uh and in general, it's just uh a much better developer experience.
• 22:03
  SPEAKER_01: So um we started like we started migrating some of our plugins to Airflow 3, and we suddenly realized that uh we couldn't, we didn't have at least like it was a couple of months ago, so maybe it's uh outdated information, but we didn't have these graph dependencies, dependencies of diff between different DACs in Airflow 3.
• 22:22
  SPEAKER_01: Um, so and we really needed it.
• 22:24
  SPEAKER_01: So we basically just I think in a couple of hours created our old plugin uh in order to see different DACs and whether they're triggered by SendAC or they depend on data sets and what is the condition, et cetera, et cetera.
• 22:37
  SPEAKER_01: And we also added a couple of capabilities that we uh really waited to have in Airflow 2.
• 22:43
  SPEAKER_01: For example, like when you have 90 or a couple of hundred DAX that depend on each other, sometimes you would just really like not only to search for particular DAGs, you would also like to filter it and see only like particular DACs and their upstream uh DAX or like the all of the children.
• 23:03
  SPEAKER_01: So basically just filter out all the all the unnecessary stuff that you would like to see in order just to understand what's going on.
• 23:09
  SPEAKER_01: So we added this thing, and yeah, this this development was surprisingly with surprisingly well and and fast.
• 23:16
  SPEAKER_02: That's great.
• 23:16
  SPEAKER_02: That's great.
• 23:17
  SPEAKER_02: I love to hear that.
• 23:18
  SPEAKER_02: Uh yeah, definitely a lot of changes to the interface in Airflow 3, but it seems like there's been a lot of great community feedback so far.
• 23:26
  SPEAKER_02: Uh, I think what you just mentioned, so adding creating your own view for this view that existed in Airflow 2 but was not in Airflow 3.0, I think you alluded to, and has been added back in in 3.1, I believe, or will be.
• 23:40
  SPEAKER_02: Maybe I got that wrong.
• 23:41
  SPEAKER_02: But in any case, um, I think it could lead into my standard final question uh for folks, which is what would you like to see most from the Airflow project in the near future?
• 23:51
  SPEAKER_01: Well, I think as someone who would like to delegate airflow work to the clients, I think it would be great to have.
• 23:59
  SPEAKER_01: I know that uh astronomer has this Astro Asterisk Decay, right?
• 24:04
  SPEAKER_01: Or uh something like this, which is basically helps with uh AI help build different uh different DAX.
• 24:11
  SPEAKER_01: So I think if like sometimes uh there is this like one of our key challenges is to combine this work of people who would like to work with the code, basically engineers, uh, and people who would like to work with UI and to work in some drag and drop UI tools.
• 24:27
  SPEAKER_01: It's pretty much a complicated thing because when you work with code, a lot of different people uh it becomes much more complicated to dig in and understand what's going on there.
• 24:35
  SPEAKER_01: Uh and when you work with uh only with the drag and drop in UI, uh it's a fairly complicated thing to track the history and changes in some VCS system, for example, Git.
• 24:46
  SPEAKER_01: So uh I think this challenge of being able to work on the same project for both of these groups of people, both for UI Fox and uh for people who would like to code, and being able to synchronize their their work.
• 25:01
  SPEAKER_01: So the code would be synchronized in UI and UI changes would be synchronized in code.
• 25:06
  SPEAKER_01: And you know, uh if Astro um CLI Astro thing would be on top of that, so you could basically talk with AI, ask for to define your couple of data sets, and then manually change something, for example, some connections between different decks, and then it would be reflected in card itself.
• 25:27
  SPEAKER_01: That would be just a degree, that would be great because we could just basically uh give it to clients and say, uh, you can work with it as you like.
• 25:34
  SPEAKER_01: Everything is saved in in VCS.
• 25:38
  SPEAKER_01: And yeah, that's basically I think I think that would be the great case.
• 25:42
  SPEAKER_01: Yeah.
• 25:43
  SPEAKER_02: Yeah, I can see where that would be very helpful for working with a lot of clients that have varying degrees of airflow and Python background.
• 25:50
  SPEAKER_02: I'm not aware of anything in the open source project for what you're describing, but as you alluded to, we do have within our Astro platform the IDE, which you can use, the AI assistant for helping write DAGs.
• 26:02
  SPEAKER_02: We're also adding in uh what's called Blueprint, which is basically a form that comes up.
• 26:08
  SPEAKER_02: So for like very standardized DAGs where you're doing, you know, maybe an ETL flow or something like that, and you just need somebody to come in and say, I want this form, but for this database or this table or whatever, then it's just a form that they fill out.
• 26:23
  SPEAKER_02: Um, so that is something that we're adding there, kind of like a dread, not actually drag and drop in terms of the GUI interface, but I think it was similar to what you're describing.
• 26:33
  SPEAKER_01: Yeah, that's very cool.
• 26:34
  SPEAKER_01: Is it is it clo a part of the cloud solution, right?
• 26:36
  SPEAKER_01: So it's like uh part of the club.
• 26:39
  SPEAKER_02: Yeah, yeah, it will be.
• 26:40
  SPEAKER_02: Yeah.
• 26:41
  SPEAKER_02: Awesome.
• 26:41
  SPEAKER_02: Uh well, Igor, thank you so much for taking the time.
• 26:44
  SPEAKER_02: It's been really interesting talking to you and learning more about the solutions that you've built out.
• 26:48
  SPEAKER_02: Uh, just to close us out, what is the best way for folks to get in touch with you?
• 26:52
  SPEAKER_01: So you can look for Punal Labs on LinkedIn or Substack.
• 26:56
  SPEAKER_01: Uh you can uh read our newsletters, and you can find me on LinkedIn as well, Igor Turosenko, and just Google it and I will be there.
• 27:03
  SPEAKER_02: Great, great.
• 27:04
  SPEAKER_02: And I we really appreciate you taking the time to write up also on your Substack and those other mediums, everything that you've done with Airflow.
• 27:11
  SPEAKER_02: I think it's so helpful for the community to be, you know, inspired by how companies are using it and customizing it in your case.
• 27:18
  SPEAKER_02: So great work there.
• 27:20
  SPEAKER_02: Thank you for remembering for having me.
• 27:21
  SPEAKER_02: Yeah, yeah.
• 27:22
  SPEAKER_02: Thanks again.
• 27:26
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering Airflow for data engineering and AI.
• 27:33
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 27:37
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.