Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar

Transcript

How Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: In our industry, we work with a lot of regulatory documents, and these documents sometimes can be like more than 200 pages, 300 pages.
• 0:07
  SPEAKER_02: Summarizing them and getting the value out of it is very critical.
• 0:10
  SPEAKER_02: Building insights from there is also very critical.
• 0:12
  SPEAKER_02: Now you can understand like getting through this entire process if you want to do this on the fly becomes very difficult, right?
• 0:18
  SPEAKER_02: It can be very unpredictable in terms of behaviors and things like that.
• 0:22
  SPEAKER_02: And that's why we don't want to do that.
• 0:23
  SPEAKER_02: We want to make sure like we are doing this in a very predictable manner within SLA and then improving the customer experience in Channel.
• 0:35
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:47
  SPEAKER_01: Hey Airflow community, exciting news.
• 0:50
  SPEAKER_01: This fall, we're launching Beyond Analytics, a virtual data ops conference happening September 16th from 12 to 3 p.m.
• 0:57
  SPEAKER_01: Eastern.
• 0:58
  SPEAKER_01: You'll hear from data leaders on how modern data ops and orchestration are powering AI, machine learning, and production grade data products.
• 1:05
  SPEAKER_01: And here's the best part.
• 1:06
  SPEAKER_01: You can join Mark Lumberti for a fast track Air Force 3 certification prep session and get a discount code for a free certification, a$150 value.
• 1:16
  SPEAKER_01: The conference is virtual, it's free, and it's made just for you.
• 1:19
  SPEAKER_01: You can register now at astronomer.io slash beyond slash data flowcast.
• 1:27
  SPEAKER_01: Hi everyone.
• 1:28
  SPEAKER_01: Welcome to the Data Flowcast.
• 1:30
  SPEAKER_01: I'm your host, Kenton Danis, and today I'm joined by Shankar Mahindar, who is a senior data engineer at Redica.
• 1:36
  SPEAKER_01: Shankar, how are you?
• 1:37
  SPEAKER_02: I'm doing good, Kenton.
• 1:39
  SPEAKER_02: Thank you for having me here.
• 1:40
  SPEAKER_01: Yeah, it's great to have you on.
• 1:42
  SPEAKER_01: From our listeners for today's podcast, we're going to discuss how Redika built their data platform around airflow.
• 1:48
  SPEAKER_01: Great topic, but let's start with some background.
• 1:50
  SPEAKER_01: Can you tell me what Redika does?
• 1:52
  SPEAKER_02: Yeah, absolutely.
• 1:53
  SPEAKER_02: So Redica is actually our data analytics platform.
• 1:56
  SPEAKER_02: We help life sciences companies and customers in the pharma and medtech domain, specifically in terms of reducing their compliance risk, ultimately, like improves their overall product quality, be it medical devices, pharmaceutical drugs, biologics, in that space.
• 2:11
  SPEAKER_02: We have like a few domains in our business that includes the quality and regulatory intelligence platform, that is a core product that we have developed, with which we help the customers be inspection ready, give them insight as to how their vendor qualities are looking at, provide them with site profiles and data analytics around that, and basically make sure like they're up to date on the regulatory and the intelligence domain.
• 2:36
  SPEAKER_01: Okay, interesting.
• 2:37
  SPEAKER_01: Yeah, that sounds like a lot of analytics work that goes on in the background to make that happen.
• 2:43
  SPEAKER_02: Yep, absolutely.
• 2:44
  SPEAKER_02: We acquire a lot of data from multiple sources.
• 2:47
  SPEAKER_02: These are our external agencies, I would say, who are responsible for quality monitoring in terms of, for example, MHRA, FDA.
• 2:54
  SPEAKER_02: So these are like some of the global agencies there who kind of do a lot of inspections and they do in sportsmen's, typically making sure like the quality of drugs and the products that touch the human lives are always monitored and they are educated from time to time.
• 3:08
  SPEAKER_02: So what we do is we acquire this data from a lot of different sources.
• 3:12
  SPEAKER_02: We then combine them, make sure like we find out the unique ID from there, because there is none, then that's what we create out of that.
• 3:19
  SPEAKER_02: And then eventually kind of tying back the whole story as to how that product or how that particular device has been monitored all through its life cycle, and then making sure like any future companies or any even the same company, if they try to create a new maybe version of the same drug or maybe a version of the same medical device, they can actually look back and refer to this inspection data and then make sure like they can make the quality of that particular product can be improved from time to time.
• 3:46
  SPEAKER_01: That's so interesting and obviously very important work for an industry that's both highly regulated and where the testing data is so important, right?
• 3:54
  SPEAKER_01: And I think it's really cool that you do that because the success stories around, you know, when drugs and medical devices are able to, you know, be successful and get to market faster have such a big impact on people's lives.
• 4:05
  SPEAKER_01: So being a part of that process and managing all of that data sounds really cool.
• 4:09
  SPEAKER_02: Yeah, it's cool and it's exciting as well.
• 4:11
  SPEAKER_02: It's a good data challenge to solve, I would say.
• 4:14
  SPEAKER_01: Yeah, that's great.
• 4:15
  SPEAKER_01: And so I imagine that Airflow fits into how you manage all of this data.
• 4:20
  SPEAKER_01: And I understand that over the past couple of years, you've gone through a pretty significant journey in implementing a new data platform.
• 4:27
  SPEAKER_01: So tell me that story.
• 4:28
  SPEAKER_01: Where did you start a couple of years ago and why did you need a new data platform to manage all of this?
• 4:34
  SPEAKER_02: Yeah, absolutely.
• 4:35
  SPEAKER_02: A great point.
• 4:36
  SPEAKER_02: So this started, I would say, probably three and a half years back when our CTO, HSahaki, joined Redika, I think early 2022, and he built in the new data team to prevamp the overall data infrastructure that we had.
• 4:47
  SPEAKER_02: So previously, we were all building things out of a specific cloud platform.
• 4:52
  SPEAKER_02: They were all disparate data sources.
• 4:54
  SPEAKER_02: So we would acquire these different data sets from multiple sources, and they would all reside into multiple several databases, which we would call it like zones in different zones.
• 5:03
  SPEAKER_02: And what eventually we were trying to do is we were trying to tie back this data with each other.
• 5:07
  SPEAKER_02: Like, for example, an inspection data was getting tied with sites data, with the inspectors who have inspected that, and then what products they have inspected, and so on and so forth.
• 5:15
  SPEAKER_02: But they were not unified together.
• 5:17
  SPEAKER_02: What it meant was at the end of the day, when you are trying to triage or troubleshoot or basically build a story out of all this information, which are all interconnected, by the way, was difficult.
• 5:28
  SPEAKER_02: And that is the reason why we saw multiple, there was like a lack of ownership, there was a lot of missed SLAs.
• 5:35
  SPEAKER_02: Then the overall solving the problem was difficult.
• 5:38
  SPEAKER_02: Figuring out issues and then debugging them was a problem as well.
• 5:42
  SPEAKER_02: And then overall, one of the biggest problems that we had was entity resolution.
• 5:46
  SPEAKER_02: So we work with, like I said, since we work with multiple data streams, there could be no single source of truth.
• 5:52
  SPEAKER_02: So we may be getting an inspection data or a site data from maybe hundreds of sources, and then finding that out that is the unique record out of it is pretty difficult because there is no unique way to identify that site.
• 6:04
  SPEAKER_02: Or example, so for some of the sites that are maybe, let's say, US based, there might be an FEI, which is uh like a federal establishment identifier, which can help identify it uniquely.
• 6:13
  SPEAKER_02: But then for sites or locations which are outside the US, they might not have such an identifier.
• 6:19
  SPEAKER_02: So it becomes very difficult.
• 6:20
  SPEAKER_02: So at the end of the day, it becomes a very classic entity resolution problem that you have to go through, which is maybe use the name and the address, and then figuring out how to get past that.
• 6:29
  SPEAKER_02: And this overall problem was we were getting into more than millions of sites getting generated, and then building analytics out of such a discrete source of data becomes very difficult at the end of the day.
• 6:41
  SPEAKER_02: Because if you think very simply at the end of the day, in a SQL terms, you're trying to do a group by, and then getting that unique ID was difficult.
• 6:48
  SPEAKER_02: So those were some of the challenges.
• 6:49
  SPEAKER_02: And again, this is purely like I would say from a data standpoint, but if you take a bigger picture out of it, there were like more other challenges that were out there, and that's when like he DVM the entire architecture, and then we caught in a very classic data warehouse kind of a solution, which becomes our core of our platform.
• 7:05
  SPEAKER_02: And with it, definitely we wanted to also solve all these different problems, like the sites' entity resolution that had to be like a big data problem itself.
• 7:13
  SPEAKER_02: It needed to go through its own batched process of capturing all this information, ingesting this data, going through entity resolution of multiple stages, and eventually like figuring out one specific unique identity, which we call it as a golden record, and that is what would then get treated as the site.
• 7:30
  SPEAKER_02: And one of the outcomes at PickAchieve was like we were able to bring this down from like 13 million to a million initially, and then from there we have brought it down to 300,000 records, which is right now the unique ID.
• 7:43
  SPEAKER_02: There are still, I would say, some very close matching sites that could be very batch related, but then that improvement itself was a very big problem.
• 7:49
  SPEAKER_02: And then the way we did it was with the help of a core cloud data warehouse, making sure like we are using Airflow right from the start itself, and that's where Airflow kind of became a very cool picture for us because it helped us build up our batch jobs, it helped us build the processes through a very systematic way of ingestion layer, then followed up with some parsing layer, then intermediate layer, but we would add business logic and then eventually building the golden records or data marks out of it for analytics.
• 8:16
  SPEAKER_02: At the end of the day, we are solving analytics problems and we are trying to build trends out of it.
• 8:20
  SPEAKER_02: And for that, it requires a pattern of a series of records and a pipeline for right.
• 8:25
  SPEAKER_01: So you had to have that data governance and the data quality in place before you can really create the gold layer, as you said, that's used for that to be effective.
• 8:34
  SPEAKER_01: That's a huge change in the number of records.
• 8:37
  SPEAKER_01: Sounds like a really big impact, and I'm sure it's much easier to manage now.
• 8:41
  SPEAKER_01: Tell me a little bit more about how Airflow is set up within your data platform.
• 8:46
  SPEAKER_01: How are you running Airflow and how does that work for managing all of those data streams?
• 8:50
  SPEAKER_02: Yeah, absolutely.
• 8:51
  SPEAKER_02: So initially, when we were evaluating the different choices of how we can probably use Airflow or any other orchestration tool for that matter, Airflow became a very natural choice for us because the way the technology has evolved over the last maybe 10 years, the number of capabilities it offered and the ease of development from a developer standpoint, that became a very, I would say, natural choice of why this particular technology.
• 9:12
  SPEAKER_02: And then from that point onwards, why manage?
• 9:15
  SPEAKER_02: Because managed becomes then you do not have to spend too much of money in terms of building this thing when you're trying to overall do our architecture of your entire data platform.
• 9:24
  SPEAKER_02: And solving or creating that out of our own was not the primary goal.
• 9:28
  SPEAKER_02: Our goal was to make sure we can help build up this data that we are acquiring and create insights out of it.
• 9:33
  SPEAKER_02: So that was the, I would say, the main focus area, and that's why the managed instance.
• 9:38
  SPEAKER_02: We are right now using the salary mode of execution.
• 9:41
  SPEAKER_02: Pretty much we initially started off as a pure orchestration kind of tool where we would be able to build up tasks using the task mode of executions and then starting from source itself, like acquiring the data from the raw source itself, like maybe websites, APIs, flat files, whatever could be the source of data, even customer data.
• 10:01
  SPEAKER_02: And then that would go through this entire process of ingestion, then over to the next step of transformations, and then eventually building the golden objects with site entity resolutions and other steps in place.
• 10:13
  SPEAKER_02: The transformations itself are taking place with the help of DBT.
• 10:16
  SPEAKER_02: We're using DBT.
• 10:17
  SPEAKER_02: So it's basically like the classic modern data warehouse stack, which is build it on Snowflake, use DBT for the transformations as and where it's required, orchestrate everything through uh Airflow.
• 10:27
  SPEAKER_02: And at the back end of it, it's ultimately SQL with Ginger template or Python.
• 10:32
  SPEAKER_02: And the set of engineers that we had initially, and then the tech stack, and then getting the learning cover out of it, it was very easy.
• 10:40
  SPEAKER_02: Python, it ultimately at the end that it was solved with Python and SQL.
• 10:43
  SPEAKER_02: So it was very easy to scale up and then basically focus more on the solving the data problem than the technology side of things.
• 10:49
  SPEAKER_01: Makes sense.
• 10:49
  SPEAKER_01: That's great.
• 10:50
  SPEAKER_01: And so just to summarize some of that back to you, so you're using Astro, which is Astronomer's Managed Airflow service, to run Airflow because you didn't want to spend the time managing Airflow.
• 11:00
  SPEAKER_01: You wanted to spend the time on the analytics piece, which is so important to your business.
• 11:05
  SPEAKER_01: And then from there you've layered in DBT and everything to help you with the SQL queries and stuff like that.
• 11:10
  SPEAKER_01: Yeah, that sounds like a great setup.
• 11:12
  SPEAKER_01: Now that you're a couple of years into that journey, what are you looking towards next with this data platform?
• 11:19
  SPEAKER_01: Kind of what plans do you have in the near future?
• 11:22
  SPEAKER_02: Yeah, absolutely.
• 11:23
  SPEAKER_02: When we started off, we initially wanted to build up this central process so that we can have this clear-cut mission of different layers, different processes, and then business logic, we would be able to separate them out in this classic warehouse solution.
• 11:36
  SPEAKER_02: But now that we are into this stage and we have hacked most of the soft problems that we had initially, we are now looking towards to make the systems much more faster, much more efficient.
• 11:45
  SPEAKER_02: And then, of course, with how we can embed AI into our entire solutions when we are offering this to customers.
• 11:51
  SPEAKER_02: Basically making sure, like when they're asking to be inspection ready, when we're asking them to be inspection ready, how best we can offer the solution for them and how fast they can get these answers.
• 12:00
  SPEAKER_02: Because in our industry, we work with a lot of regulatory documents, and these documents sometimes can be like more than 200 pages, 300 pages.
• 12:08
  SPEAKER_02: Summarizing them and getting the value out of it is very critical.
• 12:11
  SPEAKER_02: Building insights from there is also very critical.
• 12:14
  SPEAKER_02: Now you can understand getting through this entire process, if you want to do this on the fly becomes very difficult, right?
• 12:19
  SPEAKER_02: It can be very unpredictable in terms of behaviors and things like that.
• 12:23
  SPEAKER_02: And that's why we don't want to do that.
• 12:25
  SPEAKER_02: We want to make sure like we are doing this in a very predictable manner within SLA and then improving the customer experience in Channel.
• 12:32
  SPEAKER_02: So, from that point of view, we are trying to make sure like whenever we are acquiring data, we are trying to shift away to a more of an event-driven system.
• 12:40
  SPEAKER_02: And then that is where we are trying to see instead of running it on a specific day, on a specific time of months, like if we run jobs which are monthly, which are weekly, which are hourly, how can we make this mode event-driven so that the source to destination availability becomes faster?
• 12:56
  SPEAKER_02: And then that basically improves the customer's experience.
• 13:02
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 13:11
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 13:23
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:31
  SPEAKER_01: Are you looking into event-driven scheduling with Airflow 3 and the new asset watchers, or how are you implementing event-driven today?
• 13:39
  SPEAKER_02: So right now it's a mix of the classic AWS solutions that we have in place.
• 13:44
  SPEAKER_02: So it's, I would say, distributed between some part of it is within AWS, which we're acquiring and ingesting.
• 13:50
  SPEAKER_02: And then the ingestion process and entire pipeline itself, and I call it data pipeline, master data pipeline, that still runs on Airflow and backed on Snowflake.
• 13:59
  SPEAKER_02: What we're trying to do is we are trying to bring these two together as much as possible and then have a unified logging system so that at the end of the day, when we're debugging issues and solving problems, that becomes much more faster at the end of the day.
• 14:10
  SPEAKER_02: And that's where we are looking to evaluate basically Airflow 3, because I see some of the improvements that Airflow 3 has done over from Airflow 2.02.1.0 is the asset-based way of scheduling.
• 14:22
  SPEAKER_02: So previously, the task-based scheduling method was pretty much pretty neat and clean in terms of you are treating it as a pure orchestration tool from an Airflow standpoint.
• 14:30
  SPEAKER_02: But with the data being the focus of attention and assets getting created out of it, I see for companies like us, we can definitely get benefit out of it where data becomes a very crucial aspect of our product itself, of the product offering that we are going to provide you to customers, be it, let's say, as data service or dashboards or analytics or even our application, AI-based solutions.
• 14:51
  SPEAKER_02: Everything is getting served through this data.
• 14:53
  SPEAKER_02: Creating the data asset as your final product and building SLS or alerts out of it becomes a very natural choice, I would say.
• 15:00
  SPEAKER_02: So that's where like one of the use cases I would say.
• 15:03
  SPEAKER_02: With it, some of the things that kind of also comes up is what I liked on AirPro3 in terms of my initial assessment is the fact that we can do versioning, DAG versioning.
• 15:13
  SPEAKER_02: So previously, I would say like there are chances, there are several times when a system is re-architected or redesigned, the classic brownfield projects, we go ahead and do all these things, which is to add tasks initially, and then over a system or a period of time, we improve that and we reduce those tasks or take them off.
• 15:30
  SPEAKER_02: But then from any new engineer who joins, or maybe any architect who looks at that system later on becomes difficult for that person to evaluate what went, how was it built, what were the logs, and all those things.
• 15:40
  SPEAKER_02: So with DAG version, I feel like that is a very good, like I would say, feature that has come in, which we can definitely leverage and specifically helps us tie back to what happened in the past, and then from there we can take those things forward.
• 15:53
  SPEAKER_02: So I think those are some of the things that have caught up.
• 15:55
  SPEAKER_02: And of course, like observe is other piece.
• 15:57
  SPEAKER_02: Like we are also evaluating different tools to see like how observability can be tied back because then at the end of the day, that helps us tie back the SLS that we want to tag to each and every data asset that we produce.
• 16:08
  SPEAKER_02: And it's not just about your pipeline has failed, and that's why we need some engineer to go back and fix it.
• 16:14
  SPEAKER_02: It's more about how much delayed are we, how fresh the data is, and those are the questions that we want to kind of answer and provide to our customers to gain their trust and then basically add value to their work.
• 16:25
  SPEAKER_01: Absolutely.
• 16:25
  SPEAKER_01: That was a great summary, and I think so many new features in Airflow 3 to be excited about.
• 16:29
  SPEAKER_01: I would love to double-click on the observability piece that you just brought up because I can definitely see where with your industry and the use cases that you're working with, that observability and that data quality piece is really key.
• 16:42
  SPEAKER_01: So tell me a little bit more about what does observability look like in your data platform?
• 16:47
  SPEAKER_01: You mentioned a little bit beyond just knowing that a task has failed via notification or the Airflow UI or something.
• 16:53
  SPEAKER_01: So, how does observability fit in and what does it impact in your day-to-day?
• 16:57
  SPEAKER_02: Yeah, absolutely.
• 16:57
  SPEAKER_02: I can probably give you a quick background as to how we are doing it right now.
• 17:00
  SPEAKER_02: So, from our standpoint, what we have done is we have built in a lot of custom, I would say, connectors and tools so that we can get alerts in Slack or maybe in email whenever a task or a specific DAG fails.
• 17:11
  SPEAKER_02: But that is something that we had to architect or design ourselves based on the different choices that were available from Astronomer itself.
• 17:18
  SPEAKER_02: But now with IC with Airflow 300, these are some of the out-of-the-box kind of solutions that are provided.
• 17:23
  SPEAKER_02: So that means that going forward, we can always do this work much more easily without kind of copying over code and then just trying to tweak things like that.
• 17:32
  SPEAKER_02: So that becomes much more system like easy for us, basically.
• 17:35
  SPEAKER_02: From a developer journey standpoint, I feel like that becomes much, much more easier.
• 17:39
  SPEAKER_02: So that is how we were getting alerted.
• 17:40
  SPEAKER_02: But then from an observability standpoint, like making sure like how many tags fail, or what is the SLA of that DAG, and then what is the most recurring failures, those were being done through Astro UI itself, where we would look at recurring failures and things like that.
• 17:53
  SPEAKER_02: But then that while it existed was still the precursor of what was coming next.
• 17:57
  SPEAKER_02: With observe, I see there is much more capabilities.
• 18:00
  SPEAKER_02: I see a much more granular way of looking at data.
• 18:03
  SPEAKER_02: For example, I see that there is this Snowflake connector which can actually tie back to the Snowflake cost itself, in which you can actually run and create tasks or maybe create tables out of it from there.
• 18:14
  SPEAKER_02: Though we are not using that, we are still using our solution as DBT-based transformations running on Snowflake.
• 18:19
  SPEAKER_02: But I still feel like those capabilities are some things that we can definitely try out and figure out, like whether some of those kind of tie with our roadmap and overall engineering re-architecture, if we want to kind of do the other piece which I feel is going to be useful from an observability standpoint, is tying back the SLAs to each and every data product that we are building.
• 18:38
  SPEAKER_02: So right now, whenever a failure happens, we do have some sort of SLA internally built in to make sure like we are always working very closely with our CSMs and product team to make sure any pipeline, any critical pipeline that fails, there is a leak time that is always alerted within which we can fix it.
• 18:54
  SPEAKER_02: And again, most of the solutions are frequently asked questions for us.
• 18:58
  SPEAKER_02: We have some standard procedures that we have written, which the engineers can look up and solve the problems.
• 19:03
  SPEAKER_02: But then that is still, I would say, pretty much something that we had to spend time and build it out of the box.
• 19:09
  SPEAKER_02: Having this system uh built in from the platform itself, which we're using, let's say, in this case, an astronomer, that would definitely be much more useful.
• 19:17
  SPEAKER_02: Looking at it, maintaining it, and then creating governance out of that particular process would be very crucial.
• 19:24
  SPEAKER_02: That helps any new engineers who are kind of getting joined or onboarding, for them, it becomes much more simpler.
• 19:30
  SPEAKER_02: And then they can also understand the story much more better and provide support as and when which is required.
• 19:36
  SPEAKER_02: So that is, I would say, in very early stages of it, but that's what we are trying to figure out like how we can do this.
• 19:41
  SPEAKER_02: And Observe become like a very natural choice of evaluation criteria because most of our data engineering pipelines are built out of Airflow, and then that's why like batch job-based workbook leaves is what we're trying to kind of evaluate through.
• 19:54
  SPEAKER_02: We're also evaluating some other tools as well, because, like I said, half of our workload is also on AWS and cloud-based solutions.
• 19:59
  SPEAKER_02: So we're trying to see like how best it fits the entire picture of things.
• 20:04
  SPEAKER_01: Okay, makes sense.
• 20:05
  SPEAKER_01: Thank you for going through that.
• 20:06
  SPEAKER_01: And so just to summarize that back to you.
• 20:08
  SPEAKER_01: So you're using Astra Observe, which is the observability piece within Astronomer's Managed Service within our data platform.
• 20:15
  SPEAKER_01: And it sounds like you're using mostly the data products and the SLA features today, and of course, the lineage and kind of the holistic overview that it gives you so that it's easier for everybody to understand what's going on.
• 20:26
  SPEAKER_01: And then maybe you're playing around with some of the additional functionality in the future.
• 20:30
  SPEAKER_01: So you don't have to create those things yourself.
• 20:33
  SPEAKER_02: That is what we're kind of right now evaluating.
• 20:35
  SPEAKER_02: And then basically, as and when we can get to the stage that we can tie back the value to the entire story, we can then start taking it more in a formal manner and then productionizing it eventually.
• 20:45
  SPEAKER_02: That's the ultimate goal.
• 20:46
  SPEAKER_01: Well, I would be very curious to speak with you again, however many months down the road, and see how that's going and how it has changed the work within your data platform.
• 20:55
  SPEAKER_01: Just in the interest of time, I always like to end these chats with a standard final question.
• 21:00
  SPEAKER_01: And that is what would you most like to see from the Airflow project in the near future?
• 21:04
  SPEAKER_02: I've been following Airflow for quite some time now.
• 21:07
  SPEAKER_02: And one of the couple of features that I wanted to end up have was having a partition-based asset kind of information or buildup.
• 21:14
  SPEAKER_02: So when we call it as data asset or data product, then creating asset-based partitions or refreshes is very critical.
• 21:22
  SPEAKER_02: And I feel like that is one of the features if we can have within Airflow itself, that will be really critical and useful, specifically for data companies who revolve around data as that kind of center kind of point of decision making.
• 21:34
  SPEAKER_02: And the other piece which I was always curious if could be possible was having an ID itself on Astro itself that can help developers build and debug.
• 21:44
  SPEAKER_02: Like the from a developer's journey, you do not have to have a separate ID to kind of author or DAG, create a DAG or something like that.
• 21:50
  SPEAKER_02: If it can be done within Airflow itself, that could have been really clear.
• 21:53
  SPEAKER_02: But I do see very recently, I think from some of the posts that have come up, is that from Airflow 3.1, that is right now Bosswell, like an ID is already there, which you can create and manage DAGs and stuff like that.
• 22:04
  SPEAKER_02: Those are some of the features that I feel like would be very useful from a developer's journey standpoint.
• 22:08
  SPEAKER_02: Again, I'm purely thinking from an engineer's perspective, like how I can make my work much more simpler, much more easier.
• 22:14
  SPEAKER_02: And then answering back the questions back to, let's say, leadership or executives, how easy is it for us to answer what's the average duration of this test?
• 22:24
  SPEAKER_02: So building those analytics out of it, which I think partially has already been done through the observe capability itself, are I think at the end of the day, some of the cool features that we would definitely love to see come out there from Astro.
• 22:35
  SPEAKER_01: That's a great answer.
• 22:36
  SPEAKER_01: I think for the open source Airflow partitioning for assets is certainly coming.
• 22:41
  SPEAKER_01: I think it's in the near future.
• 22:42
  SPEAKER_01: Folks spoke about it at the Airflow Summit.
• 22:44
  SPEAKER_01: And I have heard a lot of talk about that.
• 22:46
  SPEAKER_01: You're not the only one requesting that.
• 22:47
  SPEAKER_01: So I would imagine that will be part of open source.
• 22:50
  SPEAKER_01: Yeah, as far as the analytics, that's our team loves to hear that.
• 22:53
  SPEAKER_01: I think it's a little easier for managed services to implement their own analytics on top of Airflow than it is for the open source project.
• 22:59
  SPEAKER_01: But yeah, both great answers, either case.
• 23:02
  SPEAKER_01: Shankar, it's been so nice speaking with you.
• 23:04
  SPEAKER_01: Um, I'll just close out by asking what is the best way for folks to get in touch with you?
• 23:08
  SPEAKER_02: Well, yeah, you can reach out to us at www.readyca.com and you would see us, or you can reach out to our company and from there you can reach out to us.
• 23:17
  SPEAKER_01: Okay, great.
• 23:18
  SPEAKER_01: Love it.
• 23:18
  SPEAKER_01: Again, thank you so much.
• 23:20
  SPEAKER_01: It's been great speaking with you.
• 23:21
  SPEAKER_01: We really appreciate you coming on the podcast.
• 23:23
  SPEAKER_02: Thank you, Kenton.
• 23:24
  SPEAKER_02: I loved speaking with you here and thank you for the opportunity.
• 23:27
  SPEAKER_02: Thank you.
• 23:30
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:36
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 23:40
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.