Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar

Transcript

Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: The main reason we're considering DBD cloud is because let's face it, code and installing all these things are a bit complex for people who've not worked a lot with it.
• 0:10
  SPEAKER_02: It's a bit much to ask for people whose primary focus is to generate reports or maybe make value out of the data that we've already ingested for them.
• 0:19
  SPEAKER_02: So it's probably a bit of extra overhead that they could avoid by just having a nice IDE on the cloud somewhere to play around with and get better representation of your models that have run and so on.
• 0:37
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:46
  SPEAKER_00: Hi everyone.
• 0:47
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:49
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Arjun Annan Kumar, who is a data engineer at Telia Denmark.
• 0:56
  SPEAKER_00: Arjun, how are you doing today?
• 0:58
  SPEAKER_02: Pretty good, Kenton.
• 0:59
  SPEAKER_02: Thanks for having me here.
• 1:01
  SPEAKER_00: Yeah, it's great to have you on.
• 1:03
  SPEAKER_00: So for our listeners for today's podcast, we're going to discuss how Telea uses Airflow to provide data for its analytics and AI teams.
• 1:12
  SPEAKER_00: To start, Arjun, for those who might not be familiar, can you tell us what Telia does?
• 1:18
  SPEAKER_02: Sure.
• 1:18
  SPEAKER_02: So Telia is a Swedish telecommunications company that has a presence in all of their Nordics and the Baltic states.
• 1:25
  SPEAKER_02: I work for Telia Denmark, which is now part of a bigger conglomerate called Norlus.
• 1:30
  SPEAKER_02: And Norlus is a Danish energy and now telecoms group.
• 1:36
  SPEAKER_02: So yeah, that's a brief bit about the company.
• 1:39
  SPEAKER_02: It's a telecoms company.
• 1:40
  SPEAKER_02: Yes.
• 1:41
  SPEAKER_00: Okay.
• 1:41
  SPEAKER_00: Great.
• 1:42
  SPEAKER_00: And what does your team do?
• 1:44
  SPEAKER_02: So I'm a data engineer and I'm in the data engineering team.
• 1:48
  SPEAKER_02: As data engineers, we are responsible for ingesting data from various sources and presenting them in a usable manner for our upstream consumers, like primarily the analysts and the air team.
• 2:02
  unknown: Yeah.
• 2:02
  SPEAKER_02: It's a bit of an interesting period for me because, as I mentioned, we've been acquired by Norleus.
• 2:08
  SPEAKER_02: And as a result of that, our team is growing much larger.
• 2:12
  SPEAKER_02: So we are moving into this structure where various data engineers are placed in their own realms of focus because there will be many different areas to focus on.
• 2:22
  SPEAKER_02: And I myself will be joining the data platform team.
• 2:25
  SPEAKER_02: So I'll have a bit more of focus on the platform side of things.
• 2:30
  SPEAKER_00: Okay, great.
• 2:31
  SPEAKER_00: That is a really interesting transition period.
• 2:33
  SPEAKER_00: I'm curious to hear more about where Airflow fits into that.
• 2:37
  SPEAKER_00: But let's talk first about how Airflow is used just in your current team, and then maybe we'll move on to where it might be used in the future post-acquisition.
• 2:45
  SPEAKER_02: Yeah.
• 2:46
  SPEAKER_02: I would describe Airflow as the silent star in our stack.
• 2:50
  SPEAKER_02: We are using the managed version right now on AWS called MWAA or Manage Workflows for Apache Airflow, as it's called.
• 2:59
  SPEAKER_02: It's just happily sitting there kicking off our tasks in a timely manner.
• 3:03
  SPEAKER_02: The primary use case for us is to run our DBT models.
• 3:08
  SPEAKER_02: And we do that using Cosmos and Astronomer, of course.
• 3:12
  SPEAKER_02: And I don't really know how it was before Cosmos because that's before my time at the company.
• 3:17
  SPEAKER_02: But I can imagine that once upon a time it was just with the bash operators and not really knowing what's going on under the hood and just hoping that it works, right?
• 3:25
  SPEAKER_02: But Cosmos is something that I started to explore even at my previous job.
• 3:30
  SPEAKER_02: And it was good to see it in action already here when I got it.
• 3:33
  SPEAKER_02: And I think it's quite brilliant.
• 3:34
  SPEAKER_02: It offers a very wonderful way to orchestrate DBT models from within Airflow.
• 3:40
  SPEAKER_02: And it's quite configurable too, from how much I've played around with it at least.
• 3:45
  SPEAKER_02: We don't really use the DBT DAG or task groups feature because of the way we've structured our DBT models into this medallion architecture.
• 3:55
  SPEAKER_02: Besides that, we also use Airflow for a bunch of our different tasks, which is to mostly surrounding ingestions like our Lambda functions on AWS or the containerized ECS test, as well as various snowpark, I think that's what it's called.
• 4:12
  SPEAKER_02: Snowpark functions on Snowflake for our AI team.
• 4:16
  SPEAKER_02: So they've just recently been onboarded onto it.
• 4:19
  SPEAKER_02: Actually, it's one of the first things I did when I started here.
• 4:22
  SPEAKER_02: I helped onboard them onto Airflow and helped them commit their first tags into production.
• 4:29
  SPEAKER_02: And it was uh quite a fun journey, actually.
• 4:32
  SPEAKER_00: Awesome.
• 4:33
  SPEAKER_00: That's great.
• 4:34
  SPEAKER_00: So you guys have that kind of classic Airflow is the central orchestrator managing all of the other tools in your data ecosystem and providing data for other teams.
• 4:45
  SPEAKER_00: And as you've said, you even have the other teams contributing their own DAGs to Airflow.
• 4:50
  SPEAKER_00: So your AI team.
• 4:51
  SPEAKER_00: That's really great.
• 4:52
  SPEAKER_00: Yeah, we love to hear that.
• 4:54
  SPEAKER_00: I'm curious to dig in a little bit more on your Cosmos usage since you brought that up.
• 4:59
  SPEAKER_00: So for anybody who's listening who is not aware, Cosmos is an open source project that was developed by Astronomer that's designed to make integrating Airflow and DBT much easier.
• 5:10
  SPEAKER_00: So it provides a lot of extra visibility into the DAGs that you're running.
• 5:15
  SPEAKER_00: I think prior to Cosmos, as you mentioned, it was a little bit of a black box.
• 5:20
  SPEAKER_00: So you often had to use like the bash operator or some folks use the Kubernetes pod operator to kind of send these DBT commands like off into the ether.
• 5:29
  SPEAKER_00: You just had to hope that everything was going okay.
• 5:32
  SPEAKER_00: And if it failed, it was a little bit hard to debug.
• 5:34
  SPEAKER_00: So Cosmos does provide you a lot more visibility there.
• 5:37
  SPEAKER_00: But I do think that one of the most common features of Cosmos that I've seen folks use is the kind of DBT task group, where it's really easy to implement that whole DBT model as a task group with just a little bit of code on the airflow side.
• 5:54
  SPEAKER_00: But you mentioned that you're actually not using that because you have a medallion architecture.
• 5:58
  SPEAKER_00: So tell me a little bit more about that and what that looks like.
• 6:02
  SPEAKER_02: So medallion architecture is just an organizational approach that's being popularized by Databricks where you divide your data into three layers: bronze, silver, and gold.
• 6:12
  SPEAKER_02: Where bronze is the raw data, silver is just your cleaned and refined.
• 6:17
  SPEAKER_02: I think that's how they name it.
• 6:18
  SPEAKER_02: And finally, gold, where you have the aggregated business ready data as in your facts and dimensions.
• 6:26
  SPEAKER_02: Normally it would have been let easy to let DBT do this heavy lifting for us with DBT, for example.
• 6:33
  SPEAKER_02: But many of our sources are not ready at various times.
• 6:36
  SPEAKER_02: And once it gets into the silver layer, it gets a bit tricky because we start intermixing a lot of different things.
• 6:42
  SPEAKER_02: And letting it run every time a source updates just means that you run a model maybe five times more than it really should.
• 6:51
  SPEAKER_02: So instead, what we do is we tell our silver tags, for example, that uh these are your data sets.
• 6:59
  SPEAKER_02: And then it just has to run when the final data set is ready.
• 7:03
  SPEAKER_02: That way we avoid running things uh more often than it should.
• 7:08
  SPEAKER_02: Um we considered or actually we did try using this freshness selector uh within the dbt runs, but it doesn't really solve the problem because uh once again, intermixed sources and one of them being fresh should be enough to kick start that uh workflow.
• 7:26
  SPEAKER_02: Yeah, we've been told by dbt that they are actually working on this adaptive selection or adaptive run uh feature, which actually solves this problem.
• 7:35
  SPEAKER_02: So we may consider moving to task groups or even a dbt deck at that point.
• 7:41
  SPEAKER_02: But that remains to be seen.
• 7:43
  SPEAKER_02: We've actually played around with task groups, and I actually have a branch lying around somewhere with everything converted to task groups, but we've had some performance problems and we had this question of whether dbt cloud is going to enter the picture.
• 7:57
  SPEAKER_02: So we've let it stay host for the moment.
• 8:01
  SPEAKER_00: Got it.
• 8:02
  SPEAKER_00: Interesting.
• 8:03
  SPEAKER_00: And so without the task group, so which what are you using Cosmos for today?
• 8:08
  SPEAKER_02: Uh we just use a combination of the dbt run local operator as well as the test operator.
• 8:16
  SPEAKER_02: And uh many of our DAGs are written in plain YAML, and we use this DAG factory-like approach to generate them.
• 8:23
  SPEAKER_02: So you pretty much just say where the model is located, and then you get a DAG generated with everything for you.
• 8:30
  SPEAKER_02: If there are tests, then there's a test group and so on.
• 8:34
  SPEAKER_02: Besides that, we also use it for a few seed operations, but nothing very fancy beyond that.
• 8:41
  SPEAKER_00: Got it.
• 8:42
  SPEAKER_00: Interesting.
• 8:43
  SPEAKER_00: And you mentioned so you're evaluating dbt cloud.
• 8:46
  SPEAKER_00: I that makes sense, as especially as you're growing to support other teams, I think.
• 8:51
  SPEAKER_00: Then dbt cloud also has a pretty, pretty solid integration story with Airflow, although it's obviously not Cosmos related at that point.
• 9:00
  SPEAKER_02: I've tried it out just to get a taste of how things would be.
• 9:03
  SPEAKER_02: The main reason we're considering DBT cloud is because uh let's face it, code and uh all installing all these things are a bit complex for people who've not worked a lot with it.
• 9:16
  SPEAKER_02: It's a bit much to ask for people whose primary focus is to generate reports or maybe make value out of the data that we've already ingested for them.
• 9:25
  SPEAKER_02: So it's probably a bit of extra overhead that they could avoid by just having a nice IDE on the cloud somewhere to play around with and get better representation of your models that have run and so on.
• 9:38
  SPEAKER_02: So that's one of the primary reasons for looking at DBT cloud.
• 9:41
  SPEAKER_02: And we already know that orchestration works well from Airflow even then.
• 9:46
  SPEAKER_02: So shouldn't be a problem, really.
• 9:49
  SPEAKER_00: Yeah, yeah, that's interesting.
• 9:51
  SPEAKER_00: Cool.
• 9:51
  SPEAKER_00: And then so you mentioned with this recent acquisition and kind of a broadening of the scope of what your team is going to do, and it sounds like what your role is.
• 10:01
  SPEAKER_00: You mentioned moving a little more into the platform management side of things.
• 10:06
  SPEAKER_00: And I'm curious for, especially in this period of kind of rapid scaling and managing a lot more data and probably supporting more teams.
• 10:16
  SPEAKER_00: What do you look at for that platform management side of things?
• 10:21
  SPEAKER_00: Are there best practices that you're trying to make sure that your team follows?
• 10:26
  SPEAKER_02: Already today, we do a lot of good things, as in we template most of our tags.
• 10:30
  SPEAKER_02: I would say at least 95% of them are templated.
• 10:33
  SPEAKER_02: So we work with this whole concept of YAML inputs, a little ginger template in the back where we use as the base and then generate an output tag.
• 10:42
  SPEAKER_02: A lot of tests are done within TPT, so that's great.
• 10:46
  SPEAKER_02: We have tests that way.
• 10:48
  SPEAKER_02: Then we have a few CICD tests, which checks if your data sets are badly referenced and things like that.
• 10:54
  SPEAKER_02: So I think one of the first things I want to do is introduce more robust tests, like maybe check across our DAG back, look for bad or excessive top-level imports, things like that.
• 11:05
  SPEAKER_02: Those things that you should never ever do.
• 11:07
  SPEAKER_02: So those are some of the first things that I want to do.
• 11:10
  SPEAKER_02: Of course, I have some other ideas as well, which are not fully fleshed out.
• 11:15
  SPEAKER_02: But another thing that I want to do is make this sort of feedback mechanism where we start to decide what is most important by collecting feedback from people who are actually using AirPlay and other things in our stack and then prioritizing them based on how they should be solved and what could be the biggest win, for example.
• 11:37
  SPEAKER_00: Yeah, I love that democratizing the airflow usage within your team.
• 11:41
  SPEAKER_00: We're currently I'm it's probably going to be over by the time this comes out.
• 11:46
  SPEAKER_00: But while we're recording, the Airflow survey is still live.
• 11:49
  SPEAKER_00: So this is the broader Airflow communities version of what you just said, asking the users what would be most helpful for them.
• 11:57
  SPEAKER_00: But yeah, that's really cool.
• 11:58
  SPEAKER_00: So you're already doing some automated testing for your DAGs, which is great.
• 12:05
  SPEAKER_00: I think we see that even today, like a lot of Airflow teams don't do that necessarily, or they find that's a difficult thing to implement.
• 12:14
  SPEAKER_00: And what you described in terms of implementing tests that sort of check for DAG writing best practices is I would say even less straightforward, especially checking for things like top-level code.
• 12:26
  SPEAKER_00: We can consistently see open source projects, as maybe even too strong of a word, repos pop up here and have kind of utilities to help with testing and things like this.
• 12:38
  SPEAKER_00: I'm curious to hear more about what your thoughts are on that and what you've found successful in that realm.
• 12:44
  SPEAKER_02: Um it's difficult.
• 12:45
  SPEAKER_02: Some of the things I've created are a mashup of many things that I've read and seen and tried.
• 12:51
  SPEAKER_02: So it's hard to say that one thing in particular really is what solved it for me or us in my previous chapter.
• 12:59
  SPEAKER_02: I use Airflow.
• 13:00
  SPEAKER_02: But you're right that there's a lot of people out there writing a lot of good code and a lot of good best practices.
• 13:06
  SPEAKER_02: And I follow a lot of content creators.
• 13:08
  SPEAKER_02: Mark, for example, is one of them.
• 13:10
  SPEAKER_02: And surely enough you cannot.
• 13:12
  SPEAKER_02: Yeah, exactly.
• 13:14
  SPEAKER_02: Yeah, and surely enough you cannot keep up to date with every single thing that goes on in Airflow.
• 13:18
  SPEAKER_02: So it's a really nice way to get these tidbit-sized thesis.
• 13:22
  SPEAKER_02: Oh, you should do this.
• 13:23
  SPEAKER_02: And then suddenly an idea pops up saying that, all right, okay, maybe I could do this in my own uh job, for example.
• 13:31
  SPEAKER_02: So that's some of the ways I go about it.
• 13:34
  SPEAKER_02: But technically speaking, many of the things come from trial and error.
• 13:40
  SPEAKER_02: So, for example, there's this problem on MWA, and um, it was one of my biggest pain points.
• 13:48
  SPEAKER_02: It's solved now, but uh AWS has their own logic about how things work under the hood, even though they really use salary.
• 13:55
  SPEAKER_02: But uh at some point it took each of our workers about three minutes or so to come up when you needed to scale, and that's an eternity in computing terms, right?
• 14:06
  SPEAKER_02: So yeah, played around a lot and we cut it down to under 40 seconds with one of the oldest tricks in the book, which is just packaging the Python wheels into our deployment, and then we also switched to UV instead of pip.
• 14:20
  SPEAKER_02: And this brought down another, this cut down another 10 seconds, I think.
• 14:26
  SPEAKER_02: So that's great.
• 14:27
  SPEAKER_02: But it was still quite frustrating.
• 14:29
  SPEAKER_02: But then again, I understand that it's a managed service and it's not as flexible as it is if you would deploy it yourself.
• 14:35
  SPEAKER_02: So I guess that's one of the drawbacks.
• 14:39
  SPEAKER_00: Yeah, for sure.
• 14:40
  SPEAKER_00: You're certainly not the first person I've heard have issues with scaling on MWAA or managed services like that.
• 14:45
  SPEAKER_00: We at Astronomer obviously on our own managed service have our own opinions on how we manage that for scale, but it's neat to see that at least even given those limitations, you've found a workaround that works for your team.
• 14:59
  SPEAKER_02: Yeah.
• 15:01
  SPEAKER_02: The basic thing is that you have to play around with it and tweak it a lot to get it to work how you would want it to.
• 15:08
  SPEAKER_00: Yeah, lots of trial and error, especially with best practices for sure.
• 15:12
  SPEAKER_02: Yes.
• 15:12
  SPEAKER_00: Yeah, I think a lot of the ones that you mentioned, taking into account the things that the folks in the community are saying, such as Mark is really good about uh posting things on LinkedIn and other places like that.
• 15:24
  SPEAKER_00: I manage the DevRel team at Astronomer, so we do a lot of community educational work and technical content to try and help with those things.
• 15:32
  SPEAKER_00: But a lot of the best practices that we are seeing are, you know, things that we've gathered from working with a lot of companies who run Airflow and run on our managed service and what has helped them.
• 15:42
  SPEAKER_00: So it's the same sort of thing that you're describing, just taking into account all of that experience and packaging it up so that hopefully other teams can benefit from it, maybe without having to go through all of the trials first, right?
• 15:55
  SPEAKER_02: I guess in a way it's a learning experience.
• 15:58
  SPEAKER_02: So it's also nice.
• 15:59
  SPEAKER_02: It can be frustrating, but then you get that win and it's really motivating, right?
• 16:04
  SPEAKER_02: But and I think that's also the beauty about Air Force.
• 16:06
  SPEAKER_02: A lot of people have already gone through that hill, which you are just trying to climb.
• 16:11
  SPEAKER_02: So you might already have your answer out there.
• 16:13
  SPEAKER_02: So that's pretty nice.
• 16:17
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:26
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:38
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:47
  SPEAKER_00: As you move forward into this new stage of your project where you might be working at a higher scale than you are at now to support these other teams, are there other kind of platform considerations that you're taking into account or other best practices that are on your radar?
• 17:02
  SPEAKER_02: This news is just about a week old, so we've only had time to formalize it in our heads.
• 17:08
  SPEAKER_02: But I know that one of the few things that will really come into play is infrastructure as code because we don't want to do the plastic old school clicking and creating things.
• 17:19
  SPEAKER_02: We want everything defining code and we want to template it in such a way that different domain teams can just make requests.
• 17:25
  SPEAKER_02: And we are once again going with the idea of YAML inputs.
• 17:28
  SPEAKER_02: I think that's just the standard now.
• 17:31
  SPEAKER_02: To try and package everything that way so that they can work within their own teams, make a pull request, have it approved between each other, and then just get your stuff created for you.
• 17:43
  SPEAKER_02: Uh airflow-wise, I don't really know exactly how it'll be scaled out, but I imagine that there will be a lot more AI workloads coming that way.
• 17:53
  SPEAKER_02: So it'll probably be more of a question of education for the AI teams rather than how do you say actual code-related work.
• 18:02
  SPEAKER_02: I don't know how else to praise it.
• 18:04
  SPEAKER_02: But yeah, so it remains to be seen how exactly it'll scale, but I am quite sure it's uh staying around in some capacity.
• 18:12
  SPEAKER_00: Yeah, that's great.
• 18:14
  SPEAKER_00: So I'm curious on the infrastructure as code, and you mentioned managing it using YAML.
• 18:20
  SPEAKER_00: You also mentioned a little bit earlier managing your DAG files using YAML that then gets converted to a DAG so that folks aren't having to write all of the Python themselves.
• 18:31
  SPEAKER_00: Are you using other tools to manage that?
• 18:34
  SPEAKER_00: Like on the DAG side, I know DAG factory is one that Astronomer now manages an open source project.
• 18:39
  SPEAKER_00: On the infrastructure side, there's like Terraform and other tools like that.
• 18:44
  SPEAKER_02: Do you roll your own or are you using so for infrastructure as code, we are definitely going with Terraform.
• 18:51
  SPEAKER_02: We already use it for our snowflake roles and other schema creations and things like that.
• 18:56
  SPEAKER_02: And now we just want to scale it out into a complete thing, right?
• 19:00
  SPEAKER_02: And as for the DAGs, we don't use Tag Factory.
• 19:03
  SPEAKER_02: I it's mainly because some templates already existed and they are purely in YAML, and then you have a Python pile that just parses the YAML, uses Ginger in the back to render the template, and then you get your desk.
• 19:18
  SPEAKER_02: So it's quite simple.
• 19:19
  SPEAKER_02: There's nothing much to it except Ginger.
• 19:22
  SPEAKER_00: Okay.
• 19:22
  SPEAKER_02: Really?
• 19:23
  SPEAKER_02: And the method really works for us right now.
• 19:26
  SPEAKER_02: And unless there is a reason to switch out, we probably won't.
• 19:30
  SPEAKER_02: But believe it or not, I have DAG Factory open as a tablet for reading.
• 19:34
  SPEAKER_02: It's something I'll look into for sure.
• 19:37
  SPEAKER_02: I know that uh astronomer maintains it now, right?
• 19:40
  SPEAKER_00: We do, yeah.
• 19:41
  SPEAKER_00: We took it over maybe six months ago.
• 19:44
  SPEAKER_00: Maybe it's been a little longer, I'm not sure.
• 19:46
  SPEAKER_00: But yeah, so yeah, we are working on developing it.
• 19:50
  SPEAKER_00: I know there's already been a little bit of work done to bring some of the newer airflow features into it, which is very exciting because I know, yeah, lots of teams want to make use of those, but I think the Terraform use for the infrastructure is also something we see super frequently.
• 20:06
  SPEAKER_00: We have a Terraform provider for our managed service now that we rolled out.
• 20:11
  SPEAKER_00: I've completely lost track of time.
• 20:13
  SPEAKER_00: Maybe it was this year.
• 20:14
  SPEAKER_00: So it was earlier this year, I think, because yeah, we have a lot of customers who want to do exactly the same thing that you're describing, and it's it's much easier for them to manage than to do it via the UI.
• 20:26
  SPEAKER_02: I guess it is easy to begin with the UI, but then it becomes a mess that uh no one really knows what's going on.
• 20:32
  SPEAKER_02: And then and then you start to think about moving it all away, and then it's an even bigger mess.
• 20:38
  SPEAKER_00: Yeah.
• 20:39
  SPEAKER_00: It's fine if you have one airflow deployment.
• 20:41
  SPEAKER_00: It's not so fine if you have multiple teams or many airflows, and yeah, definitely easier to manage it as code.
• 20:50
  SPEAKER_02: Definitely agree.
• 20:51
  SPEAKER_02: And I think right now for us it's a no-brainer because we are starting some new things from scratch.
• 20:57
  SPEAKER_02: So it makes a lot of sense to just do it the hard way first.
• 21:01
  SPEAKER_00: Yeah, definitely.
• 21:04
  SPEAKER_00: Right.
• 21:04
  SPEAKER_00: Probably both.
• 21:05
  SPEAKER_00: Sometimes those go those go together.
• 21:08
  SPEAKER_00: But as you said, it's an investment up front for hopefully like bigger payoff and easier time down the road.
• 21:14
  SPEAKER_02: Exactly.
• 21:15
  SPEAKER_00: Yeah.
• 21:15
  SPEAKER_00: But that sounds like an interesting opportunity, as you said, to build from not from scratch, but starting with a clean slate for some new workflows.
• 21:23
  SPEAKER_00: I'm curious to hear, after you've been at it for a while, how your AI teams have found working with Airflow.
• 21:30
  SPEAKER_00: That's something we see a lot more frequently now.
• 21:33
  SPEAKER_00: And my team has been pretty invested in the past year and a half on creating a lot of content for how you can use Airflow for those workloads.
• 21:40
  SPEAKER_00: So I'm curious to hear more about how that goes.
• 21:43
  SPEAKER_02: Yeah, I'll be closely following.
• 21:45
  SPEAKER_02: I'll be closely following that just to make sure that uh I'm not uh feeding up some old information.
• 21:52
  SPEAKER_00: Yeah, yeah, it's definitely helps to be working from the most recent versions for sure.
• 21:58
  SPEAKER_00: Jen, I know we're We're almost at time, so I want to close with my standard final question that I asked to all of my guests, which is what would you most like to see from development of the Airflow project in the near future?
• 22:12
  SPEAKER_02: That's a touch question because I'm quite happy with how it is.
• 22:16
  SPEAKER_02: But on a personal level, I feel like I'd like to see some of the more seasoned developers do intro sessions to allow other people to contribute to the project.
• 22:26
  SPEAKER_02: I have done a few of them in the past, but it always takes a lot of time because I'm just navigating blindly.
• 22:33
  SPEAKER_02: And um I was lucky enough to gu work with a guy who's part of the Apache Software Foundation and he has contributed to Airflow, so it was not as difficult as it could have been.
• 22:44
  SPEAKER_02: And I'm guessing it has come quite a long way since then.
• 22:48
  SPEAKER_02: But yeah, it will be one thing I would like to see.
• 22:52
  SPEAKER_02: And then another one is something that I can share from having heard it from many other sources, but can't really relate to myself.
• 22:59
  SPEAKER_02: It's that the local Airflow development experience is not so easy or that it's bad.
• 23:05
  SPEAKER_02: I can't really recall the reason why, but I'm just so used to it that it's not really a problem for me.
• 23:12
  SPEAKER_02: But maybe it's maybe it's a point to think about.
• 23:15
  SPEAKER_00: Yeah, it can be challenging to get started locally.
• 23:18
  SPEAKER_00: Obviously, I'm biased, but at Astronomer, we have our Astro CLI, which is open source, so anybody can use it locally.
• 23:25
  SPEAKER_00: It runs on Docker and it makes getting started.
• 23:28
  SPEAKER_00: You run two commands, and then you've got Airflow running on your local.
• 23:31
  SPEAKER_00: I remember when I so I had used Airflow at a previous role before I came to Astronomer and before, probably actually before the Astro CLI existed, or certainly I didn't know it existed at the time.
• 23:42
  SPEAKER_00: And I remember the first time I used the CLI, and I was like, oh my gosh, this would have saved me so much time previously, at least in that initial like figuring out how it works stage, as you once you've got it running, usually it's a little bit easier.
• 23:56
  SPEAKER_00: But that is our answer to that one.
• 23:58
  SPEAKER_00: Again, it's open source, so you don't have to be a customer to use it.
• 24:01
  SPEAKER_00: But I love your request for better contributing guidance.
• 24:06
  SPEAKER_00: That is something that I know a lot of the Airflow PMC are aware of.
• 24:10
  SPEAKER_00: It can be, I could say just super intimidating to start committing if you haven't already or contributing.
• 24:18
  SPEAKER_00: It's uh there's a lot to sift through, and it's not always super straightforward.
• 24:23
  SPEAKER_00: I have hit that myself and I work at astronomer, so I have a lot of help available to me, folks who are who are used to doing this, but that is something that has been brought up before, and I think there are still some improvements that could be made, definitely.
• 24:39
  SPEAKER_02: But I'm keeping an eye out, and I really want to get back into the game at some point.
• 24:44
  SPEAKER_00: Yeah, we would obviously love to have you contribute.
• 24:48
  SPEAKER_02: Yes.
• 24:48
  SPEAKER_02: Maybe maybe once Airflow 3 is out.
• 24:51
  SPEAKER_00: Yeah, yeah, definitely everything is very busy with Airflow 3 right now, but I think there's a lot of opportunity both in that and uh certainly afterwards for contributions.
• 25:01
  SPEAKER_00: So that's great to hear.
• 25:03
  SPEAKER_00: Cool, Arjun.
• 25:04
  SPEAKER_00: I know we're about at time.
• 25:06
  SPEAKER_00: It's been really great to have you on.
• 25:08
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 25:11
  SPEAKER_02: Uh it's probably my LinkedIn.
• 25:13
  SPEAKER_02: That's the one thing that's open.
• 25:14
  SPEAKER_02: It's just LinkedIn slash Arjun Anand1.
• 25:18
  SPEAKER_00: Okay.
• 25:18
  SPEAKER_00: Great.
• 25:19
  SPEAKER_02: Yep.
• 25:19
  SPEAKER_02: So it's just my name and half of my last name, plus the number one.
• 25:24
  SPEAKER_00: Okay.
• 25:24
  SPEAKER_00: We can link to your LinkedIn in the show notes for folks so they don't have to write that down from audio.
• 25:30
  SPEAKER_00: Awesome.
• 25:31
  SPEAKER_00: Thank you again for coming on.
• 25:32
  SPEAKER_00: It's been a pleasure talking to you.
• 25:34
  SPEAKER_02: Yeah, thank you, Kent.
• 25:35
  SPEAKER_02: Likewise, and have a great day.
• 25:37
  SPEAKER_00: Thanks.
• 25:38
  SPEAKER_00: You too.
• 25:40
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 25:47
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 25:51
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.