Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino

Transcript

Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: Our only instance of variable, we know that with the M chart that we can deploy the easily in Kubernetes is what we like, but most of the time, like other clients are using Cloud Composer, and also when they are new feature retail, do we that the feature is available in the new version of Cloud Composer?
• 0:16
  SPEAKER_03: So this is more like tricky.
• 0:22
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:33
  SPEAKER_00: Airflow users, we have exciting news.
• 0:36
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:40
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow.
• 0:43
  SPEAKER_00: And as a thank you for filling it out, you'll get a free Airflow Fundamentals or DAG Authoring certification.
• 0:49
  SPEAKER_00: Those are worth$150 each.
• 0:52
  SPEAKER_00: Plus, you'll also be entered to win an exclusive workshop with Mark Lumberti.
• 0:56
  SPEAKER_00: So make sure you don't miss out.
• 0:58
  SPEAKER_00: The survey link is in the show notes and it closes December 20th.
• 1:03
  SPEAKER_00: Hi everyone.
• 1:04
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:06
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Andrea Bombino, who's co-founder and head of data analytics at Astrify.
• 1:15
  SPEAKER_00: Andrea, how are you?
• 1:18
  SPEAKER_00: Thanks for joining me.
• 1:20
  SPEAKER_00: Yeah, it's great to it's great to have you on.
• 1:23
  SPEAKER_00: I think last time we spoke, you were recovering from your hundred-ish mile running trip around Montblanc.
• 1:31
  SPEAKER_00: How are you feeling now?
• 1:32
  SPEAKER_00: Couple, couple more weeks removed.
• 1:34
  SPEAKER_03: Yeah, now I'm ready back to go running again.
• 1:37
  SPEAKER_03: And so I'm really happy.
• 1:38
  SPEAKER_03: It's too crazy how my body is recovering bus.
• 1:41
  SPEAKER_03: So now happy back to be back in the truck.
• 1:44
  SPEAKER_00: Yeah, that's great.
• 1:46
  SPEAKER_00: That's great to hear.
• 1:46
  SPEAKER_00: I for our listeners who are probably not familiar with anything that we're talking about.
• 1:51
  SPEAKER_00: This is a big trail race in Europe.
• 1:54
  SPEAKER_00: So Andrea and I are both trailrnness as we discovered at last year's Airflow Summit.
• 1:59
  SPEAKER_00: But before we go totally rogue and turn this into a running podcast, I guess we can also talk about airflow.
• 2:05
  SPEAKER_00: Um so let's go ahead and dive in.
• 2:09
  SPEAKER_00: For today's podcast, we're going to be discussing how AstraFi uses data sets and pub sub to orchestrate dynamic workflows.
• 2:17
  SPEAKER_00: Andrea, first tell me a little bit more about AstraFi.
• 2:21
  SPEAKER_00: What does your company do and what is your role there?
• 2:24
  SPEAKER_03: Yes.
• 2:25
  SPEAKER_03: So basically AstraFi is the company that's been found in Latin 2032.
• 2:29
  SPEAKER_03: We are four ex-colleagues that we as some client work together in different projects.
• 2:35
  SPEAKER_03: And then we understand that we share the same passion about data, but we have different backgrounds.
• 2:41
  SPEAKER_03: So somebody was like more concerning data scientists and machine learning, someone has all the DevOps and laptops infrastructure, and me and the shadow that is like the CEO, actually, we are more interested in data engineering.
• 2:53
  SPEAKER_03: So we say, okay, why don't we try to create our own company to see how we can help other people with data in general?
• 3:00
  SPEAKER_03: 2022, we start to create our own company in Geneva, where the idea was like to found something that was called the new data model stack and to help people and client can be like small startup or a big corporation in order to ingest data from all the data journey, from the ingestion, the transformation until a bit of like using machine learning, business intelligence, or dashboarding or whatever is behind the scene.
• 3:27
  SPEAKER_03: And with having a solid infrastructure and a solid architecture.
• 3:31
  SPEAKER_03: So like we see having all the our knowledge and all our expertise in all this data infrastructure, we could help a lot of people.
• 3:38
  SPEAKER_03: So we say, okay, we start our own business and now we are up with it.
• 3:43
  SPEAKER_03: And yes, so basically, me, I am end of data analytics, engineering, whatever is just a title.
• 3:50
  SPEAKER_03: And then basically, I more concentrate on what we do with the data in the size of data and transformation from roles system that can be like a legacy database or like a CSV file and whatever to in the data warehouse.
• 4:04
  SPEAKER_03: And Advive, we are Google partners.
• 4:07
  SPEAKER_03: So like our main favorite cloud provider is Google, but we are not like just Google's focus, but it's our main focus on Google.
• 4:15
  SPEAKER_03: But we are open to work with any kind of cloud provider in any way that's similar and they're not requinted all the time with just like different technology behind the scene.
• 4:24
  SPEAKER_03: So yet it's like a nutshell, what we do, and and all we are using in all the data transformation that I'm we are using different clients, we are using everywhere Airflow.
• 4:34
  SPEAKER_03: They can be like either instantiated with all instance of Kubernetes cluster and deployment of airflow in their own cluster, or using the management service per separ from Cloud Composer, and then we integrate that one with DBT for the transformation.
• 4:47
  SPEAKER_03: And that's how we do mainly.
• 4:50
  SPEAKER_00: Okay, cool.
• 4:51
  SPEAKER_00: And so are you helping your clients implement this solution and then they're running with it on their own, or do you take some form in like the ongoing management of their data?
• 5:02
  SPEAKER_03: We are doing like not only like City Dump, but we are can be there are different clients.
• 5:07
  SPEAKER_03: Like in some kind, we are just going there setting up air soap and like doing the maintenance for them, like when they want to release new airflow version or like they want to use new features, all this kind of stuff.
• 5:18
  SPEAKER_03: But at the same time, uh, for example, in different projects and working really daily basis on the client side when they want to do the data transformation within Airflow using any kind of SQL transformation, in this case, like Maily DBT, and I'm there to help them really to do the transformation with Airflow.
• 5:36
  SPEAKER_03: And so like I see all the pipeline that is going from taking data from the source system, transforming them, and then going to the transform application.
• 5:44
  SPEAKER_03: It can be AI or can be just other team, like with this data mesh concept that everybody nowadays is using.
• 5:50
  SPEAKER_03: So other teams that need to have the RB data data in producing for them.
• 5:54
  SPEAKER_03: And so there's sometimes to speak to the stakeholder, okay, this data needs to be ready at this time, so your own airflow can be scheduled at this time.
• 6:04
  SPEAKER_03: So it is for this reason, also when the topical data set is helpful because we wanted the data already as soon as possible.
• 6:12
  SPEAKER_03: So we have different pipelines that we wanted to trigger when whenever the data is are ready.
• 6:17
  SPEAKER_03: So this is a bit what I'm doing.
• 6:20
  SPEAKER_00: Okay, got it.
• 6:21
  SPEAKER_00: That's helpful background.
• 6:23
  SPEAKER_00: And so you recently gave a talk at the Airflow Summit back in September on using data sets and PubSub for dynamic DAG triggering.
• 6:33
  SPEAKER_00: This is a really interesting use case, and I think one that is very applicable.
• 6:37
  SPEAKER_00: We hear a lot about wanting more event-driven, dynamic workflows within Airflow.
• 6:44
  SPEAKER_00: So walk me through the backstory here and what originally prompted you to explore a solution like this.
• 6:50
  SPEAKER_03: Yes, thank you.
• 6:51
  SPEAKER_03: Basically, what I've seen in different kinds of use case, and then also for other internal internal projects, that you want to see like when a DAG is finished to transform the data, we want to have the data that are triggering the DAG, another DAG, for example, from different team or whatever, as soon as possible, as soon as they say before the data are ready.
• 7:12
  SPEAKER_03: And we could not just use the trigger DAG rules because we want to have like a different semantic, a different logic.
• 7:19
  SPEAKER_03: And for this kind of stuff, we want to have kind of a notion of real time.
• 7:22
  SPEAKER_03: So for this reason, we use PubStat.
• 7:24
  SPEAKER_03: And again, PubStat, because as I said at the beginning, we are more focused on cloud, on Google Cloud.
• 7:29
  SPEAKER_03: And then also as well, our clients are mostly in Google Cloud, so we adopt the solution that are in Google Cloud.
• 7:35
  SPEAKER_03: And then the technology that when I see that the data set was uh out as a feature of airflow, I was okay.
• 7:42
  SPEAKER_03: Now I need to try it because I'm solving the problem that I need.
• 7:45
  SPEAKER_03: Because basically, we have different workflows that are processing different data, and we want that as soon as different data, these different data are uh updating a data set, the data set are triggering different other workflows behind, I mean, let's say the different downstream application.
• 8:03
  SPEAKER_03: What we do today, I try different approach.
• 8:06
  SPEAKER_03: Um so what we do today is like we have a similar first DAG, let's say, that is like doing taking the data from the ingestion system, the let's say legacy or like uh raw system, taking the data ingest and transform them.
• 8:19
  SPEAKER_03: Uh and then when uh the this uh transformation is done, we have data data set.
• 8:24
  SPEAKER_03: Okay, when this data set is updated, uh then we enter another function in an airflow that is triggering a pass up topic.
• 8:32
  SPEAKER_03: And then this pass up topic is triggered whatever the downstream application it is.
• 8:36
  SPEAKER_03: And most likely what we do since we have different instances of airflow.
• 8:40
  SPEAKER_03: Since we cannot communicate like with different data sets are from the different instance of airflow, we use the PubSub topic to say to the other airflow, okay, you can trigger and run your pipeline.
• 8:51
  SPEAKER_03: So because we want that to say, like first the data, as soon as they are ready, we process them.
• 8:57
  SPEAKER_03: And the the downstream application, they don't just query without the data outdated, right?
• 9:02
  SPEAKER_03: Because what is the use case is sometimes uh before that the user data setting were we were implemented from my side, the downstream application were just okay, we ran running query on the table that I provided based on schedule time.
• 9:16
  SPEAKER_03: And this sometimes was like not working because okay, I run the query, this is costly, it's costly, and then second the data are not there.
• 9:22
  SPEAKER_03: So, how we can do that.
• 9:24
  SPEAKER_03: With that setup, we are we cannot resolve the problem.
• 9:27
  SPEAKER_03: And I adopted data centers is at the beginning, and now seeing how this is evolving, I'm really happy to see how the community and uh basic airflow is working on it because uh there are a lot of features that uh I'm really willing to use.
• 9:40
  SPEAKER_00: Yeah, that makes sense.
• 9:42
  SPEAKER_00: And so because you've got the airflow DAGs in multiple different airflow environments that need to be coordinated, that's where you use PubSub because as you said, the data sets as they're currently implemented are only listening for updates within their own Airflow environments.
• 9:57
  SPEAKER_00: They can't they can't go beyond their own metadata database.
• 10:00
  SPEAKER_00: Yeah, and so that's an interesting way of tying those together.
• 10:04
  SPEAKER_03: Exactly.
• 10:05
  SPEAKER_03: And then as I say that the final part of my talk, like maybe sometimes it was not clear for people that they're not really using that data set, but uh one thing that's for kind of more specific and like detailed, like needs what I'm doing right now, is like using like dataset and XCOM.
• 10:22
  SPEAKER_03: So basically, what when I need to know when a data set is changed and the time, since I don't want to query the database BNDC of Virtual because it's like costly and slowly, I just do an accommodation between data set and is comm to understand when the data set is updated.
• 10:39
  SPEAKER_03: Because I and all the time that update a data set, I save this value in an iscome uh variable, and then the other DAG with another DAG and read this value can say, okay, I think this data set is updated within today or based on the logic that I want to use, I then process the data behind.
• 10:55
  SPEAKER_03: I'm using all the kind of feature that are possible in Appflow in order to make it flexible and also like reusable in all different environments.
• 11:02
  SPEAKER_00: Okay.
• 11:03
  SPEAKER_00: So you have another DAG that is responsible for orchestrating the connection between your upstream DAGs and the pub sub that then sends the request to the students.
• 11:14
  SPEAKER_00: Okay.
• 11:15
  SPEAKER_03: And I was like, as well, I was without also reading the reading the documentation of data set.
• 11:20
  SPEAKER_03: I was like, when I was trying to use it, it's like, oh, it could be really cool if I if the data set could be used as a schedule uh the tag itself.
• 11:28
  SPEAKER_03: And when I then I read the documentation, I was like, okay, I can redo that.
• 11:32
  SPEAKER_03: I was really happy because the data tag should not like continue listening like with a sensor operator, whatever.
• 11:37
  SPEAKER_03: It would just be triggered when the data set is changed.
• 11:39
  SPEAKER_03: It changed my life actually.
• 11:41
  SPEAKER_00: Yeah, that's amazing to hear.
• 11:43
  SPEAKER_00: I think the data sets feature, like one of the big benefits I usually tell people who are newer to Airflow over sensors is that performance aspect of you don't have to have something that's just sitting running.
• 11:54
  SPEAKER_00: You don't need to know around when it should start so that you can minimize that waiting time as you just running as part of Airflow.
• 12:03
  SPEAKER_03: After like the new uh adoption of airflow, like with the version to the nine to 10, I see that there are a lot of like changes that are like the new feature of data set that they are really looking forward to try.
• 12:16
  SPEAKER_03: For example, right now, because could could be uh depends on the logic of the downstream application that you have.
• 12:21
  SPEAKER_03: But for example, right now, some customer they are they want to say, okay, even though the data are not there, or not all of the data are there, so the data set is not completely all the data set that are like the trigger from the airflow are not ready.
• 12:35
  SPEAKER_03: I want that my that my DAG is still running because I don't want to have delayed over my touchboard, whatever.
• 12:40
  SPEAKER_03: So, what I see now there's a combination you can combine between data set and also like the schedule time.
• 12:46
  SPEAKER_03: This is a feature that I think I'm going to use in order to provide like data availability like as soon as possible.
• 12:53
  SPEAKER_03: And even though like they're like I have a part of data, this one depends as well from the need of the client and the like the need of the downstream application.
• 13:00
  SPEAKER_03: Or, for example, like another thing that for the moment I don't see a real need in my personal use cases, but uh the possibility to put like a that uh data set in and or in order to trigger or do an action of the DAG.
• 13:14
  SPEAKER_03: This I think it could be really useful.
• 13:18
  SPEAKER_03: And then also the data set assets, yeah.
• 13:20
  SPEAKER_03: I mean to see also with the new the new UI or is like crazy.
• 13:25
  SPEAKER_03: Like sometimes and like this, I want to trigger the DAG and when a data set is uh updated, right?
• 13:31
  SPEAKER_03: And I could not do it manually because if I was triggering the DAG manually, the data set was not updated, it was not running at all.
• 13:37
  SPEAKER_03: But now with the new UI, it could just schedule it and update the dataset manually.
• 13:42
  SPEAKER_03: So this is something that it will is supposed to save some time when you want to just schedule the data set and just okay, you like just updated data set manually, and then the DAG is working as expected.
• 13:53
  SPEAKER_00: Yeah, there's definitely been a lot of great updates to the datasets feature, as you said, in the past couple of releases.
• 13:59
  SPEAKER_00: So your original implementation with the datasets and pub sub combination, I assume then is on a version older than two nine at least.
• 14:07
  SPEAKER_00: Is that right?
• 14:08
  SPEAKER_03: Yes, exactly.
• 14:09
  SPEAKER_03: Yes, exactly.
• 14:10
  SPEAKER_03: Yeah, because also you're not depending a lot of the from the client side when they want to uh update airflow, because when they want to update airflow, they don't want to have like down the system down like mini mini production, and uh some of them they have like only production airflow running and solid data and airflow depends.
• 14:30
  SPEAKER_03: If we are able to have our own instance of airflow, we know that with the end chart that we can deploy the easily in Kubernetes is what we like, but most of the time, like other clients are using like Cloud Composer, and they also when they have new feature read the way that the features available in the new version of Cloud Composer.
• 14:48
  SPEAKER_03: So this is more like tricky.
• 14:51
  SPEAKER_00: Yeah, upgrading in general is always a tricky topic.
• 14:55
  SPEAKER_00: I think it's something that a lot of companies struggle with, especially with, as you said, upgrading the providers that go along with the versions, or if you're on a managed service that is taking a little longer to upgrade and that adds an extra consideration.
• 15:09
  SPEAKER_00: Definitely some exciting stuff has happened with the sets in the recent features, but you still are at least can benefit from data sets have been around for quite a while now.
• 15:18
  SPEAKER_00: I still can run into a lot of people who think they're a newer Airflow feature, which I guess is true in the scheme of Airflow's lifetime, but they have been out for a couple of years now.
• 15:28
  SPEAKER_00: So I think most of the community that we've seen is running at least a feature where the or a version, excuse me, where the basic feature should be available.
• 15:36
  SPEAKER_03: Yeah, indeed.
• 15:37
  SPEAKER_03: Also, I was like, I was impressed because when I asked during my talk how many people know data set, there were not many that were doing the concept of data set or like really using it.
• 15:46
  SPEAKER_03: So at the end of the talk, I had a lot of questions, like and I was really happy to answer it because I s I think that a lot of people can just use the uh the feature that they have because like they can easily really easier life as a data engineer.
• 16:01
  SPEAKER_00: Yeah, it's definitely very flexible.
• 16:03
  SPEAKER_00: It's I think me and folks on my team are data set evangelists, especially given, as you said, the developments on the feature and future developments, which I do want to get to in a few minutes, but it's uh there are a lot of questions where I think there's just a very easy answer of oh, you can just use this feature.
• 16:22
  SPEAKER_03: Yes.
• 16:23
  SPEAKER_03: At the beginning, I was also like, okay, I'm not how can I use data set, but how can I just update a data set?
• 16:32
  SPEAKER_03: The first question was like, okay, I had the data set concept, but how can I update it?
• 16:36
  SPEAKER_03: Because actually, the data set is just a simple object that can be whatever can just be a string, whatever.
• 16:41
  SPEAKER_03: And how can I update that?
• 16:42
  SPEAKER_03: So how should I do that?
• 16:43
  SPEAKER_03: I didn't mean it was like, oh, I don't know how to do that, and then there were not a lot of questions, a lot of documentation how to use it, and then like after that, you uh you understand the logic behind is really easy to end using within your DAG pipeline.
• 16:57
  SPEAKER_03: And and as I said, you don't need to have a sensor or whatever, you can just use data set in order to do like your logic behind in order to trigger the DAG and like also define a specific logic to understand which part of your DAG flow needs to run.
• 17:11
  SPEAKER_03: So this is really good.
• 17:15
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 17:24
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 17:36
  SPEAKER_01: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 17:45
  SPEAKER_00: Speaking of updating data sets, one of the new additions to the feature in I think 2-9 was the ability to update them using the Airflow API, meaning that you can send that update from somewhere that's not another DAG within the environment.
• 18:02
  SPEAKER_00: It sounds like this is something that would be useful for the type of use case that you're describing for your clients.
• 18:08
  SPEAKER_00: Is that something you foresee yourself using when they do get around to upgrading to two nine or beyond?
• 18:14
  SPEAKER_03: And like for me, like will be RAIDs that we have this feature, not only like with API, but I don't know, like something within your code, within your an operator that can just tell you okay, I update the data set, and this will be like really nice.
• 18:29
  SPEAKER_03: And the other thing that for me today is a big uh pain, is like, for example, to retrieve the last dates of the last update of the data set.
• 18:39
  SPEAKER_03: Today you need to query the database.
• 18:41
  SPEAKER_03: You cannot, there is not a feature that you can just uh do it directly.
• 18:45
  SPEAKER_03: And this is sometimes for kind of some logic, I was I was needed, and today I was not able to do it.
• 18:52
  SPEAKER_03: For this reason, as well, I'd use the XCOM variable to in order to say, okay, this data set has been updated this time.
• 18:59
  SPEAKER_03: Or for example, as well, it could be great if we have two distant instance of F when one from one instance you could update the other data set to the other instance.
• 19:08
  SPEAKER_03: But for example, I think that if we have the F API, I could just call from API the data set of the other instance and it could do that.
• 19:17
  SPEAKER_03: But yeah, a lot of things that are like just describing it, I'm happy, but let's see where it goes.
• 19:23
  SPEAKER_00: Yeah, that's something that we recommend as the first option for implementing like a cross-deployment dependency.
• 19:30
  SPEAKER_00: Like that is now using Airflow API.
• 19:33
  SPEAKER_00: For what you're describing, getting the metadata associated with the data set event.
• 19:40
  SPEAKER_00: I wonder, I don't actually know, but one of the updates in 210 was that you can attach metadata to a data set.
• 19:46
  SPEAKER_00: So when it does get updated, you can pass information with that.
• 19:50
  SPEAKER_00: I'm curious.
• 19:50
  SPEAKER_00: I don't actually know whether that would work for what you're describing, but potentially, or maybe a future iteration of that might be helpful.
• 19:59
  SPEAKER_03: Yeah, it's true.
• 20:00
  SPEAKER_03: I should uh I read about it, and but I should try it because the thing that I did, I was like, okay, at the moment I don't think it is going to be helpful for me.
• 20:08
  SPEAKER_03: Uh, but yeah, definitely I want to try all the different stuff that turned around in order to see also okay, because right now the logic, sometimes the logic that I have is a bit like complex, less than complex, but what I want to do is like kind of complex because there are like different kinds of situations.
• 20:26
  SPEAKER_03: And so, okay, if I could get, for example, this metadata easily, it could be great.
• 20:32
  SPEAKER_03: But from the moment when I read it, I was not seeing something that was facilitated the way that I was processing the data set.
• 20:38
  SPEAKER_03: But for children, all the time there is an update, I believe all the time about it because also what is good as well is I load the UI, like when you can see, for example, because not only you see it's not only okay.
• 20:51
  SPEAKER_03: This done and is triggered based on the data set, and this present in the UI.
• 20:55
  SPEAKER_03: And when I saw it, I was like, oh yeah.
• 20:57
  SPEAKER_03: Sometimes to understand to give to the stakeholder how the flow burp is working, I would just go in the UI and send a picture and that's all because it's PSP at least and also now with the tests, is gonna be even more easy to understand.
• 21:11
  SPEAKER_03: Yeah, read all the time and reading all the time the update that is RPI and because uh I think it's they're going to be more and more useful feature bugs around the set.
• 21:20
  SPEAKER_00: Yeah, yeah, that's great.
• 21:21
  SPEAKER_00: Where you've implemented this solution with your clients, were they or maybe it varies, but were they using sensors before him?
• 21:31
  SPEAKER_00: Did they have another airflow solution implemented and it's now better that you've implemented it with data sets, or was it not in Airflow at all?
• 21:40
  SPEAKER_03: Or how's that work?
• 21:42
  SPEAKER_03: Because I do remember because I was I was updating as I was doing working only in EastCon variable, but this one was really a mess and test it that was triggered where what when it was not needed, and when I I started.
• 22:00
  SPEAKER_00: to use data set this is the reason the real reason why i see data set i was like okay this can solve my problem because sometimes i get like issue or alerts in my like dashboard or logins say okay this that has been triggered uh anyways like uh it should not it should not be triggered why it is triggered and then i see because some variables were not clean up from the day before uh this actually was the the the the the case where basically the uh it is called variable that was setting true or false uh but uh the one was set to true uh and i could not set it back to false because the was set there for the day before so we're a lot of different logic that i need to use and when i see data set and say okay this is the thing that i need to use right now and the things i try they were they work out out of the box and then the stakeholder happy also like my logic was became more easy and yeah so like everybody was happy and me as well so that's a great success story yes love to hear that especially with like I said we are data set evangelists over here at astronomer and I know there's a lot more development coming for them in airflow three which we're all really excited about it leads me to my next question which is I always like to end with the same final question for everybody and that's what would you most like to see from development of the Airflow project in the near future?
• 23:20
  SPEAKER_03: About data set or in general it can be anything what is a tough question some people give me lists of five things you can do that too if you want yeah as I said for data set I would like to be able to see not bigger change a data set in other instance easily with the API or even something more sophisticated or I don't know I don't know I don't have the answer otherwise I was proposing in a merge request or something.
• 23:47
  SPEAKER_03: But also like today what's we are facing we as a five we are facing the standoff kind of like a day is the role but not raw access with the user because today we need to do this within the UI and this sometimes is getting a bit blurry because we wanted for example some people they can run specific dance and their specific action within airflow.
• 24:12
  SPEAKER_03: And I think this part is it can be improved a bit because the role are like the definition of what the permission they can do of a user are not really linked together.
• 24:22
  SPEAKER_03: It's like the interface is like because the UI is not like really convertible to see because are two different spaces.
• 24:28
  SPEAKER_03: And so right now for example I'm building a like um a framework for myself myself where I am a list of all the users with all the rights and then I'm entering right now for example I will if I want to add a user or if I want to add a role link to the user and it this kind of stuff like I think about role and user if it can be cool from my point of view.
• 24:47
  SPEAKER_03: And then I don't know because already the from the last year the UI the improvement has been done already are really cool.
• 24:54
  SPEAKER_03: And I think with the info 300 we are going to see even more features yeah either the main thing as well.
• 25:02
  SPEAKER_00: Definitely more to come for the UI and 3.0 as well as the data sets feature I think some of which will probably make it easier to implement those things across deployment.
• 25:12
  SPEAKER_00: I'm not sure about the RBAC stuff whether there's more development there.
• 25:16
  SPEAKER_00: I think yes I actually think some of the folks at AWS have been working on an AIP about that.
• 25:21
  SPEAKER_00: I'd be wrong but those are all great requests I will because I forgot to do it at the beginning make a plug for by the time this episode airs the Airflow survey should be out this is something we do or the community does every year and is a great way for all of our listeners to make requests such as the ones you've just made for what should be coming in Airflow.
• 25:42
  SPEAKER_00: It's taken very seriously by the developers and we would love obviously for you Andrea to fill it out as well.
• 25:48
  SPEAKER_00: We'll send a link over but we'll put a link to it in the show notes for everybody who also has Airflow feedback that they would like to put out into the community.
• 25:59
  SPEAKER_03: I remember like last year in the Airflow Summit there was the session where what you want what's workful and there was there like a lot of people asking for the new UI stuff and I think it's what happened afterwards.
• 26:11
  SPEAKER_03: So I know that all the people behind when they requested for them makes sense they are working with the articles in place.
• 26:16
  SPEAKER_03: So yeah I know that if there's something that they said it could be cool for everybody, I even get it.
• 26:22
  SPEAKER_00: Yeah absolutely and a lot of the features that are coming in Airflow three are the direct result of past surveys and what folks have said at events like that at the Airflow Summit for what they wanted.
• 26:33
  SPEAKER_00: I think DAG versioning is the the very top of the list that everybody has said for forever this is what we want.
• 26:38
  SPEAKER_00: And so it it is being developed now.
• 26:43
  SPEAKER_03: All right thank you Andrea so much for coming on it's been great to have you what is the best way for folks to get in touch with you you can just like on LinkedIn you can just add me on LinkedIn and I'm also in the NFL Slack channel and yes also my email is needed and but I think there's LinkedIn and Airflow channel is the best.
• 27:04
  SPEAKER_02: Okay great and any uh random runners we might have happened to reach on this podcast I assume can follow you on Strava or something like that yeah Strava with my TB index whatever yeah if I when I found this around it I was like really excited oh wow another person should share my craziness passion.
• 27:22
  SPEAKER_00: Yes you never know when you're going to run into somebody who has the same hobbies as you so this is another plug for the airflow summit is you can meet people who use airflow for cool things and also who share entirely crazy hobbies.
• 27:45
  SPEAKER_03: So I I was like okay one day we're gonna talk about it this because it's going to be it could be interesting yeah.
• 27:51
  SPEAKER_00: I would love to have you on for another episode to talk about that.
• 27:55
  SPEAKER_00: But again I promised not to go entirely rogue into uh a podcast about running we'll go ahead and wrap it up there but it's been really great to talk to you again and yeah thanks again for coming on.
• 28:07
  SPEAKER_01: Thanks to you thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 28:17
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 28:21
  SPEAKER_01: If you're enjoying our podcast please subscribe so you never miss an episode and leave us a five star review