Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating Retail Data Pipelines at Saks Global

Transcript

Orchestrating Retail Data Pipelines at Saks Global · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: Around 8 million schools uh across our business.
• 0:04
  SPEAKER_00: So 8 million schools can get updated.
• 0:06
  SPEAKER_00: Sometimes the name is updated or the image is updated, or sometimes inventory is updated.
• 0:11
  SPEAKER_00: So all of this has to be real time.
• 0:13
  SPEAKER_00: And there is there is point of sale systems that currently work on, and uh sometimes people buy gift card-based transactions, the points are updated and all those things.
• 0:20
  SPEAKER_00: So all this all these things orchestrated using DAGs.
• 0:24
  SPEAKER_01: You're listening to the Data Flowcast, the podcast about Apache DR flow and the world of data and AI around it.
• 0:32
  SPEAKER_01: Let's get into it.
• 0:35
  SPEAKER_02: All right.
• 0:35
  SPEAKER_02: Hi everyone.
• 0:36
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:38
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Shilash Kadam, who is an architect at Sax Global.
• 0:44
  SPEAKER_02: Shylash, how are you?
• 0:46
  SPEAKER_00: I'm very well.
• 0:47
  SPEAKER_00: Thank you for asking.
• 0:48
  SPEAKER_00: How are you doing?
• 0:49
  SPEAKER_02: Yeah, I'm good as well.
• 0:50
  SPEAKER_02: I'm good as well.
• 0:51
  SPEAKER_02: It's great to have you on.
• 0:52
  SPEAKER_02: I'm excited for our conversation today.
• 0:55
  SPEAKER_02: Um, we're gonna talk about how Sax uses Airflow for ETL and Snowflake and things like where Agent AI and MCP fit into your roadmap.
• 1:04
  SPEAKER_02: So definitely big topics.
• 1:05
  SPEAKER_02: And I think um Sax will probably be a pretty familiar company to a lot of our listeners.
• 1:11
  SPEAKER_02: Um, which leads me to my first question, which is always I want to start with some background.
• 1:15
  SPEAKER_02: So um, in case Sax isn't familiar to our listeners, or I think once you describe it, they will be.
• 1:20
  SPEAKER_02: But tell me a little bit of background about what the company does.
• 1:23
  SPEAKER_00: Uh yes, absolutely.
• 1:24
  SPEAKER_00: So SAX is a major brand in the US and uh they are into the retail industry.
• 1:29
  SPEAKER_00: So in recent years, the business has gone through some significant transformation.
• 1:34
  SPEAKER_00: So we separated the e-commerce business uh from the uh Motar and uh Blick and Motar stores.
• 1:40
  SPEAKER_00: So most of the presence is online now, which is uh which because many people prefer uh doing things online uh rather than going to the stores anymore.
• 1:48
  SPEAKER_00: And then we took over Neiman Marcus just last year.
• 1:51
  SPEAKER_00: So it's the biggest retail brand now.
• 1:52
  SPEAKER_00: There is no competition to Sax anymore.
• 1:55
  SPEAKER_00: And uh since then we have rebuilt around a modern cloud native stack.
• 1:59
  SPEAKER_00: So most of the things are uh AWS, Snowflake, uh data warehousing, and then Airflow is the orchestration tool, uh, which is our Nova system, as you if we if we can say, to keep this business running.
• 2:11
  SPEAKER_00: So that's really has unlocked us a lot of things uh in on the online side of the business.
• 2:17
  SPEAKER_02: Yeah, great.
• 2:18
  SPEAKER_02: So it makes sense.
• 2:19
  SPEAKER_02: I mean, huge e-commerce business um in in retail.
• 2:23
  SPEAKER_02: So that's that's really cool to hear that Airflow is a big piece of that.
• 2:26
  SPEAKER_02: We'll get into that in more detail in just a minute.
• 2:29
  SPEAKER_02: Uh, but first, what does your team do within SACs?
• 2:34
  SPEAKER_00: Uh yes, uh, so our team is mostly on the online side of it.
• 2:37
  SPEAKER_00: We maintain the data that uh refreshes the page and shows all the products on the on the website, shows the promotions that their products are in, shows if they're in stock or not.
• 2:46
  SPEAKER_00: It actually maintains uh a link between the web page and the data that it resides uh behind it.
• 2:53
  SPEAKER_00: So practically speaking, we are responsible for getting the data from all our different operational systems, uh, point of sale, if you can see this e-commerce database uh which is maintained on some tools like uh catalog management, where we have a list of catalogs for this year, or if it is a season-based catalog or fall-based.
• 3:12
  SPEAKER_00: So we have a tool that keeps all those inventory in there.
• 3:15
  SPEAKER_00: So we maintain data flow between those.
• 3:18
  SPEAKER_00: And we also have a Snowflake, which uh the business analyzes on uh how is the uh sales going on or what are customer feedback about it, and uh also gift cards and how much how they're performing.
• 3:30
  SPEAKER_00: So all this analysis is done in Snowflake.
• 3:33
  SPEAKER_00: So we provide data to the analyst team so that they can look at all those numbers and then they can uh decide what's the next uh which customers can be targeted next and all those things.
• 3:43
  SPEAKER_00: And uh Airflow is a part of it, uh provides all this syncing between these different point of sale systems, e-commerce systems, and Snowflake so that the businesses can get real-time value as much as quicker as possible.
• 3:56
  SPEAKER_00: So uh we we want to make sure in case of uh there's a there's a sale, like a season sale, and it's only for one hour.
• 4:03
  SPEAKER_00: So we want to make sure the data flows through all the uh front-end systems and the back end is very quick as possible.
• 4:08
  SPEAKER_00: So the promotions are uh are updated quickly and efficiently.
• 4:12
  SPEAKER_00: So I'm in part of that team.
• 4:14
  SPEAKER_00: Uh we can say as analytics and engineering team, uh, which we if we sit on top of data to provide this data to our business uh users.
• 4:23
  SPEAKER_02: Okay, great.
• 4:24
  SPEAKER_02: So it sounds like uh your team and then Airflow, you mentioned, is powering both kind of your internal analytics that's used by the business as well as the platform itself uh for managing all the data coming through.
• 4:37
  unknown: Cool.
• 4:37
  SPEAKER_00: Uh yes, just to add to that, we also have a product catalog that gets updated and that's sent to social uh like Meta and Google.
• 4:45
  SPEAKER_00: They run ads for us.
• 4:48
  SPEAKER_00: So we also have to build that product catalog and send it, send it to them hourly sometimes because it's updated.
• 4:54
  SPEAKER_00: The stay invent is updated or the price is updated.
• 4:57
  SPEAKER_00: So we have to maintain uh people who are using apps on their phone, so they get the updated inventory all the time.
• 5:04
  SPEAKER_02: Okay.
• 5:05
  SPEAKER_02: Yeah, that's very cool.
• 5:06
  SPEAKER_02: And so you mentioned that Airflow is your primary orchestrator and that you're pulling data from sources like gift cards, frog data, your online product catalogs, all of that into Snowflake.
• 5:18
  SPEAKER_02: Can you tell me in a little bit more detail kind of what your pipelines look like in Airflow?
• 5:23
  SPEAKER_00: Oh yes, absolutely.
• 5:23
  SPEAKER_00: So as I say, it is a nervous system for us.
• 5:25
  SPEAKER_00: Uh without that, it's very difficult for us to know or I mean to coordinate everything across these different systems.
• 5:32
  SPEAKER_00: And uh we have around 8 million schools uh across our business.
• 5:38
  SPEAKER_00: So eight million schools can get updated.
• 5:40
  SPEAKER_00: Sometimes the name is updated or the image is updated, or sometimes inventory is updated.
• 5:45
  SPEAKER_00: So all of this has to be real time.
• 5:47
  SPEAKER_00: And there is there is point of sale systems that currently work on, and uh sometimes people buy gift card-based transactions, the points are updated and all those things.
• 5:54
  SPEAKER_00: So all this all these things orchestrated using DAGs.
• 5:57
  SPEAKER_00: So our airflows, uh the core of this entire airflow-based uh system is ingestion, data ingestion.
• 6:06
  SPEAKER_00: Like we have a huge amount of data that's coming in, so you ingest data from these different uh uh various uh file formats and different heterogeneous data systems, like Snowflake sometimes, and we orchestrate them in such a way that uh they are efficiently processed, and we get to see the status every every like every hour or every 30 minutes to see that all of this data is flowing through, and there is no error assessors.
• 6:29
  SPEAKER_00: And Airflow uses a holistic view of how this different data from different domains is flowing through.
• 6:35
  SPEAKER_00: So we use it for scheduling, we use it for coordinating the movement of data across uh different operational systems, and then we trigger we also trigger the downstream systems based on this using Airflow uh for Snowflake for data analysis and other things.
• 6:49
  SPEAKER_00: And in terms of specific pipelines, uh point of sale data is probably the most important for us.
• 6:54
  SPEAKER_00: So we have to be absolutely uh critical of looking at the SLAs for this data that's flowing through to the reporting systems, and it's all near real time.
• 7:04
  SPEAKER_00: And uh during sale, there are people who buy items worth one million or something more than that, which can be a fraud.
• 7:13
  SPEAKER_00: So we normally don't get such type of transactions, so we need to have uh there are third-party systems uh like FISERV which look at this kind of data, and we have to be very efficient at sending this data to third-party companies to understand whether the transaction is valid or not.
• 7:27
  SPEAKER_00: Uh, so this is very latency sensitive data that uh that has to be handled, and Airflow does a pretty much good job of running uh with all of this huge amount of data that flows through, uh uh, which is probably probably allowed us to make some changes because Airflow is Python-based.
• 7:43
  SPEAKER_00: So we also have made some changes in it, and it has helped in a way that no other tool can give us.
• 7:48
  SPEAKER_00: So it's it's basically the one of the most core of our entire orchestration, if we can say.
• 7:54
  SPEAKER_02: Wow, yeah, that's so cool.
• 7:56
  SPEAKER_02: And it sounds like you're running Airflow at a pretty high scale or working with it at a pretty high scale.
• 8:02
  SPEAKER_02: How are you running Airflow?
• 8:04
  SPEAKER_00: So, one of the things is we we have Astro, which is our astronomy, uh, which is a managed platform for Airflow.
• 8:09
  SPEAKER_00: We used to run it uh through Kubernetes sometimes, uh Kubernetes-based uh deployments.
• 8:14
  SPEAKER_00: And the issue with them is that we have to manage the platform more than looking at our jobs that are running.
• 8:19
  SPEAKER_00: So we were uh mostly occupied with uh handling errors in uh those uh Kubernetes uh kind based implementation.
• 8:26
  SPEAKER_00: Then we also have tried IBM's uh Tivoli kind of a scheduler, which was there.
• 8:30
  SPEAKER_00: We had Control M, which was another tool for scheduling and all those things.
• 8:35
  SPEAKER_00: And they had a lot of uh, I mean, we are not able to understand what the errors were sometimes.
• 8:39
  SPEAKER_00: Uh and we we need to have a separate team to look at what those errors are, specifically just to look at what those errors look like.
• 8:46
  SPEAKER_00: And uh we lost track of what exactly we were doing and we were we were delayed because of these things.
• 8:51
  SPEAKER_00: Uh, but uh Astronomer actually takes care of all this for us.
• 8:55
  SPEAKER_00: So Astro is uh we did we don't we didn't want to be in the business of managing Airflow infrastructure.
• 9:01
  SPEAKER_00: And Astro actually did everything for us, and uh uh it uh and it's uh proximity to the Airflow's project.
• 9:08
  SPEAKER_00: I mean, it uh Airflow is uh heavily open sourced and people are working heavily on it, and they they actually look at all these problems and improve the platform efficiently uh as if there are any issues or uh that comes across.
• 9:22
  SPEAKER_00: So that has helped us to uh put all the burden on the open source community to some extent.
• 9:30
  SPEAKER_02: Yeah, that's great to hear.
• 9:31
  SPEAKER_02: I have also been in the position myself of managing airflow on a Kubernetes cluster that we managed in-house before, before I worked for Astronomer, and it was not a good use of time at all.
• 9:43
  SPEAKER_02: Um so yeah, it's it's cool to hear that you guys have had that success with Astro.
• 9:48
  SPEAKER_02: Um, of course, as an astronomer, we always love to hear that.
• 9:51
  SPEAKER_02: Um very cool.
• 9:53
  SPEAKER_02: Well, on a slightly separate topic, I understand that you have been working recently on upgrading to Airflow three, and I'd love to hear more about what is driving that um that migration right now and uh kind of where you're at in that upgrading process.
• 10:09
  SPEAKER_00: Well, yeah, so some point on the Airflow three, and we are anticipating that there are architectural improvements that are coming in Airflow 3, and we are also anticipating there will be some sort of task isolation that will come across in Airflow 3, and it's going to help us, uh, is what our uh uh belief is.
• 10:28
  SPEAKER_00: That's what we are thinking.
• 10:30
  SPEAKER_00: Uh, that that will help us when we implement uh uh Airflow 3.
• 10:34
  SPEAKER_00: And so when we have a large DAG of inventory, we have like a 60, like like we have so many so much amount of data that's flowing through, we have so much of the DACs that are going through.
• 10:44
  SPEAKER_00: Uh we are looking at what other some of the things that can be that are causing errors, we are uh analyzing those kind of workflows and seeing which can be fixed uh with by moving through Airflow, uh Airflow 3.
• 10:58
  SPEAKER_00: So one of the things is basically I'm thinking Airflow 3 has event-based, uh asset-based uh uh data dependency.
• 11:08
  SPEAKER_00: So if there's a change in some of the underlying structure, then the DACs are going to adapt to that kind of changes, like event-based kind of uh execution of uh tasks.
• 11:18
  SPEAKER_00: So especially for making our uh data aware of changes and then react actively to those changes and then uh processes as and when it comes through.
• 11:27
  SPEAKER_00: So not as a fixed uh, if I'm making sense, not as a fixed clone-based job, not as a fixed set of DAG that runs every hour.
• 11:34
  SPEAKER_00: It's something like that adapts to an event and then it runs.
• 11:37
  SPEAKER_00: So suppose there is a change in one of the product catalog, it actually just triggers that product and makes a change, and then we don't have to run like six million and just run one or two of those and then uh transfer that changes downstream.
• 11:48
  SPEAKER_00: That's what we are looking at for Airflow 3.
• 11:51
  SPEAKER_00: Uh, we uh and we are going through a phased approach.
• 11:54
  SPEAKER_00: We are looking at some of the DACs that are very uh less critical.
• 11:58
  SPEAKER_00: We are going to move slowly uh through identifying those and then moving, changing those uh things.
• 12:05
  SPEAKER_00: And auto uh is the is the new AI-based uh uh uh tool that we're trying, uh, which can actually convert uh most of the some of these DACs into Airflow 3, and then we run parallelly and then we see how this actually are accommodating to our changes and um and and probably make things a little different than Airflow 2.
• 12:26
  SPEAKER_00: And definitely there will be more uh community involved in Airflow 3, I believe, to answer our queries uh as we move progress towards a newer version.
• 12:36
  SPEAKER_00: And I think there's one more thing I might like to mention.
• 12:38
  SPEAKER_00: Airflow 3 has version-based changes.
• 12:40
  SPEAKER_00: Um if I need to check that if there's a change in some of the DAGs, it actually keeps a version of it rather than just overriding it.
• 12:47
  SPEAKER_00: So maybe that is something we really want to uh keep a track of, what changed.
• 12:52
  SPEAKER_02: Yeah, yeah, that totally makes sense.
• 12:54
  SPEAKER_02: I think DAG version, you mentioned event-driven scheduling, those are certainly two of the most requested features are requested prior to Airflow three, and I can see where those would be very helpful for your team.
• 13:05
  SPEAKER_02: I love that you also mentioned Auto, which is our uh astronomer's data engineering agent uh that has some capabilities built in to help teams upgrade specifically from Airflow 2 to Airflow 3.
• 13:17
  SPEAKER_02: So I'm glad to hear that that's that's part of your process and that it's helping out.
• 13:22
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed Aircraft platform built for data environment teams.
• 13:31
  SPEAKER_01: From better pipelines to training models and wiring up data.
• 13:35
  SPEAKER_01: Astro handles the infrastructure upgrades and scaling so you can focus on building.io.
• 13:46
  SPEAKER_00: Yeah, that's a really cool way of doing it.
• 13:48
  SPEAKER_00: And it really helps.
• 13:50
  SPEAKER_00: I would I would say going forward, these tools are going to make life easier for a lot of people.
• 13:54
  SPEAKER_02: Yeah, absolutely.
• 13:56
  SPEAKER_02: Well, on that topic of AI, since this is a very uh you know hot topic right now within uh the Airflow community, I understand that you're also moving towards leveraging more agentic AI and MCPs.
• 14:10
  SPEAKER_02: Uh, where does uh what does that process look like and where does Airflow fit into that picture?
• 14:15
  SPEAKER_00: Yeah, so before I answer your question, we are very excited about it.
• 14:18
  SPEAKER_00: It's one of the things uh that we see that uh Airflow has taken that step towards creating an MCP for it and then making it more AI-aware, uh context-based.
• 14:32
  SPEAKER_00: So that way we don't have to uh think about some other tool which does that.
• 14:37
  SPEAKER_00: So it's a very right step.
• 14:38
  SPEAKER_00: It's done at the right time, and I think uh uh it will come as a winner because a lot of data they handle and they understand these patterns, and it's definitely going to help them to be a leader in um in the in the in this space.
• 14:51
  SPEAKER_00: And so one of the things that we're looking forward to is uh LLM-based uh uh outputs for our underlying data as well.
• 14:59
  SPEAKER_00: So we want to uh send data to an LLM and then the LLM answers the queries to us.
• 15:04
  SPEAKER_00: So and using Airflow on top of it, looking at how this uh uh interaction between the DACs work, it's going to help us to uh make sure that the LLM gets all the updated data and then uh it provides the same kind of output.
• 15:18
  SPEAKER_00: So one of the things is uh classification is what we are trying to look at, product classification based on our school.
• 15:24
  SPEAKER_00: So, and we want to enrich our pipelines as well based on how the products uh attributes are changing.
• 15:30
  SPEAKER_00: So that kind of uh things we want to run using these uh AI-based tools, and and Airflow is going to handle most of these things for us, so because it has its own MCP, so we're gonna give access to all of use all of those tools and then make use of this context-based reasoning, which Airflow can do it for us.
• 15:47
  SPEAKER_00: So we want to be a part of it and inside Airflow itself if possible, with including all the other features inside it.
• 15:54
  SPEAKER_00: Uh, and the agents are only good as how much data they get.
• 15:58
  SPEAKER_00: So uh so we are going to provide as much data that we have, and we have an ample amount of data, at least in the retail space.
• 16:05
  SPEAKER_00: We know each and every attribute of a particular product, and then we're going to make sure that it gets all of the data.
• 16:11
  SPEAKER_00: And uh if you look at uh Snowflake is going to provide other analysis analysis as well, like customer churn or customer um spending and everything.
• 16:20
  SPEAKER_00: So we need to make sure that uh Snowflake also gets all of this data that comes from uh this uh AI-based, uh LLM-based uh reasoning that goes to Snowflake.
• 16:30
  SPEAKER_00: So Airflow is going to uh make we're gonna make use of these MCPs uh provided by Airflow, especially S3 reading security-based guardrails or top of it.
• 16:39
  SPEAKER_00: So these things are very important before we even run the uh LLM uh based on this data.
• 16:47
  SPEAKER_02: Yeah, absolutely.
• 16:48
  SPEAKER_02: And so if I understood you correctly, you're using Airflow both to orchestrate the agentic workflows themselves and also to, of course, prepare and you know feed the data that's powering them.
• 16:59
  SPEAKER_00: Is that right?
• 17:00
  SPEAKER_00: Yes, absolutely.
• 17:00
  SPEAKER_00: Yes, absolutely.
• 17:01
  SPEAKER_00: We have a list of uh various lists of vendors, and we send them data like every hour or every day or maybe quarterly, and we don't know which vendors gets all this data, and then we need to understand the patterns, like what the vendor is doing on the data.
• 17:14
  SPEAKER_00: So maybe we can do it and then we can save some costs uh there as well.
• 17:18
  SPEAKER_02: Okay, yeah, cool.
• 17:20
  SPEAKER_02: Uh and I'm curious, have you tried the new common AI provider yet?
• 17:24
  SPEAKER_02: Is that part of your exploration within this airflow for AI workflows?
• 17:29
  SPEAKER_00: Uh we are still discussing on it.
• 17:32
  SPEAKER_00: We have a team reorg recently happened.
• 17:35
  SPEAKER_00: So the new organization, the new org is going to be totally heavily focused on uh the AI side of it.
• 17:41
  SPEAKER_00: Uh, as you can understand, we we everybody wants to be part of it now because it's getting matured.
• 17:47
  SPEAKER_00: And we also have uh tie up in Microsoft, which is going to give us some sort of this compute power uh which is coming with it, the co-pilot version of it.
• 17:57
  SPEAKER_00: And uh and we also going to use Amazon S3 to use bedrock on top of it to see if that can be used as well.
• 18:05
  SPEAKER_00: So uh yeah, so we are working on different options right now, and then we will be maybe mature.
• 18:12
  SPEAKER_00: I mean it's trying to get mature.
• 18:14
  SPEAKER_00: They are a little early in this phase where AI is going to help them.
• 18:16
  SPEAKER_00: I mean, the business is even the entire industry is looking at what new roles or what AI is going to impact them to such an extent.
• 18:23
  SPEAKER_00: So maybe by the end of the year, or maybe early next year, we know how much benefit it's going to bring us.
• 18:30
  SPEAKER_00: So that's what we are thinking.
• 18:32
  SPEAKER_02: Yeah, makes sense.
• 18:33
  SPEAKER_02: Certainly lots to explore there, both within Airflow and outside of it.
• 18:38
  SPEAKER_02: Uh you mentioned uh kind of managing the security and guardrails around these AI workflows.
• 18:45
  SPEAKER_02: And I know that's uh obviously a particularly important topic when it comes to AI, but also just more broadly with teams of your size that are using Airflow to manage so much data, a lot of it potentially quite sensitive.
• 18:59
  SPEAKER_02: Uh how does your team handle all of that?
• 19:02
  SPEAKER_00: Uh yeah, that's a good question, uh Kenton.
• 19:05
  SPEAKER_00: So most of the data from the customer is their addresses and their and their names and then spending and their maybe credit card details.
• 19:14
  SPEAKER_00: So those are all very P2 kind of data, uh personally identifiable.
• 19:19
  SPEAKER_00: And it has to be absolutely secure.
• 19:21
  SPEAKER_00: We have rules and regulations in place that is data, even I can't say it.
• 19:25
  SPEAKER_00: It's it's uh it's all uh uh heavily masked kind of data.
• 19:30
  SPEAKER_00: And we have third-party companies which process this data for us.
• 19:32
  SPEAKER_00: For example, Capital One is one of them, which gets all this data and then processes all this transaction.
• 19:37
  SPEAKER_00: So uh it at least in the retail space, uh this data flows through from different systems, and then uh we also don't know these files are sometimes uh uh flat files.
• 19:50
  SPEAKER_00: We don't even know which column belongs to which uh which field, and it's all heavily encrypted and we don't know what's happening.
• 19:58
  SPEAKER_00: So it's a very important feature, uh important uh concern for us that whatever credentials we pass through through DAG or to environment variables, uh it has to be heavily masked and it uh it has to be heavily secure uh for us to stay in the business, if if we can say.
• 20:15
  SPEAKER_00: And then we have uh basically we have secrets backend that we keep, which is a file, which is heavily it's it's password protected.
• 20:24
  SPEAKER_00: So everything has to be come from a secrets file.
• 20:27
  SPEAKER_00: We don't actually hardcode anything in our file, uh in our code as such.
• 20:31
  SPEAKER_00: So uh airflow secret management integrates uh with the clouds uh secret manager.
• 20:37
  SPEAKER_00: So that's what we're using right now, and then we also have Snowflake-based uh analysis that is done.
• 20:42
  SPEAKER_00: Snowflake also is pushing us towards using a key value pair kind of a thing where uh they are they encrypt this value in a config file somewhere, and we just reference the file and then we keep on rotating it.
• 20:52
  SPEAKER_00: And then a separate team that uh checks all of these rotations, so they keep on rotating these values maybe one week or maybe uh depends on how the uh secrets are the retention period is now for this.
• 21:06
  SPEAKER_00: And then on the Airflow side, we looked at the agents which are currently being built, the MCP side.
• 21:11
  SPEAKER_00: And uh, I'm happy to know that they basically don't send anything in the data uh in the DAG as such.
• 21:17
  SPEAKER_00: It's part of the configuration file where we can store that and then it uses it based on that configuration file.
• 21:22
  SPEAKER_00: So that's a good feature that uh that we're happy about that Airflow is providing uh for us to uh mask that value in the configuration file somewhere, not in the any of the jobs that run.
• 21:34
  SPEAKER_00: So so that is something that's uh an important uh thing for us to use Airflow.
• 21:41
  SPEAKER_02: Yeah, absolutely.
• 21:42
  SPEAKER_02: We'll love hearing about how Airflow is very effective there.
• 21:45
  SPEAKER_02: Uh Helish, this has been so interesting.
• 21:49
  SPEAKER_02: I uh we could keep talking about this, I think, for a long time, but just in the interest of time, I always like to ask my guests a standard final question, and that is what would you most like to see from the Airflow project in the near future?
• 22:00
  SPEAKER_00: So most of the things I'm thinking, Kentin, we need to see data lineage, like how the data is flowing.
• 22:06
  SPEAKER_00: It's top of our list.
• 22:07
  SPEAKER_00: So we have so many DACs running at the same time.
• 22:09
  SPEAKER_00: We don't know what data is flowing through where.
• 22:11
  SPEAKER_00: So my kind of like a graph kind of a thing, hey, you know, this DAG is running and then this is dependent on this one, and this is currently at 30%, is currently 40%.
• 22:18
  SPEAKER_00: So kind of a data lineage towards uh uh uh like uh understanding how the data is currently in which stage, like uh staging or processing or final layer.
• 22:29
  SPEAKER_00: So those kind of things, uh maybe like a very interactive kind of a graph.
• 22:33
  SPEAKER_00: Uh and uh what uh Airflow has done is basically used agent-based workflow, agent, agent uh based workflow, asset management kind of workflow.
• 22:42
  SPEAKER_00: So that's a very important step, which I think is already there in place, which is looking at like event-driven kind of uh thing.
• 22:49
  SPEAKER_00: And uh absolutely valuable is debugging and for debugging kind of a thing, where we can actually look into Airflow and say, hey, uh this particular DAG is at this stage and it's waiting for this one and it's maybe maybe not getting a proper value at here.
• 23:04
  SPEAKER_00: So like a UI-based uh debugging kind of interface, uh which uh which many like uh you know the compilers that they have.
• 23:13
  SPEAKER_00: So maybe something uh shows me that while running that this is waiting for this thing, and then have a have a little user-friendly way of uh showing some debug errors.
• 23:24
  SPEAKER_02: Okay.
• 23:24
  SPEAKER_02: Yeah, that makes sense.
• 23:26
  SPEAKER_02: I know at Astronomer we are working on, you mentioned auto, a little bit of an investigation capability within our products that is AI driven to help with that debugging uh before you ever have to go into like the task logs and kind of look at it yourself.
• 23:41
  SPEAKER_02: Uh, but I can see where what you're saying would also be very helpful in the open source project, just more visibility into what's happening in the DAGs.
• 23:49
  SPEAKER_00: Yeah.
• 23:49
  SPEAKER_00: I also feel that many of the users don't go because the debugging is something because there are so there are support teams which join in.
• 23:56
  SPEAKER_00: If a developer did something, the support team has to go and then on the weekend go and check what the issue is, and then well, maybe they they take some time, and we don't want to wait on the call for like uh for 30-50 minutes just to understand what the issue is.
• 24:08
  SPEAKER_00: Yeah, is what I feel.
• 24:10
  SPEAKER_02: Yeah, absolutely.
• 24:11
  SPEAKER_02: Those that's that's a great, great point.
• 24:14
  SPEAKER_02: Uh okay, Shailash.
• 24:15
  SPEAKER_02: Well, it's been so great speaking with you.
• 24:17
  SPEAKER_02: This has been a really interesting conversation.
• 24:19
  SPEAKER_02: I think it's always so great for everybody in the community to get to hear uh or kind of peek you know behind the curtain of like these huge companies using Airflow and how it all works.
• 24:28
  SPEAKER_02: So appreciate you sharing everything.
• 24:31
  SPEAKER_02: Uh just to close us out, what is the best way for folks to get in touch with you?
• 24:35
  SPEAKER_00: Uh so uh this is enough it's very good that people want to get in touch with me.
• 24:40
  SPEAKER_00: So if they want to, then uh I'm always on LinkedIn.
• 24:43
  SPEAKER_00: Uh they can always reach out to on LinkedIn.
• 24:45
  SPEAKER_00: I'll also be on the Apache Airflow Slack community.
• 24:48
  SPEAKER_00: I'll be part of it.
• 24:49
  SPEAKER_00: So people can come and just say hi there as well.
• 24:53
  SPEAKER_00: So that's how it is.
• 24:54
  SPEAKER_00: Yeah.
• 24:55
  SPEAKER_02: Great.
• 24:55
  SPEAKER_02: Well, we will put links to both of those things in the show notes.
• 24:59
  SPEAKER_02: Uh okay.
• 25:00
  SPEAKER_02: Well, Shilash, thanks again.
• 25:01
  SPEAKER_02: It's been really great speaking with you.
• 25:03
  SPEAKER_00: Yes, thank you so much.
• 25:04
  SPEAKER_00: It was great talking to you too.
• 25:05
  SPEAKER_00: Uh thank you.
• 25:06
  SPEAKER_01: Thank you for listening to the data flowcast.
• 25:09
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 25:13
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 25:18
  SPEAKER_01: And leave us a five-star review.