Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Consolidating legacy job schedulers with Airflow at Trading Technologies

Transcript

Consolidating legacy job schedulers with Airflow at Trading Technologies · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: Need to sol uh solve your business problem.
• 0:03
  SPEAKER_01: So I'm trying to minimize our efforts around managing infrastructure towards solving real problems around data.
• 0:12
  SPEAKER_02: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:20
  SPEAKER_02: Let's get into it.
• 0:22
  SPEAKER_03: All right.
• 0:22
  SPEAKER_03: Hi everybody, welcome, welcome in this amazing episode.
• 0:25
  SPEAKER_03: I'm super excited for today.
• 0:28
  SPEAKER_03: I'm your host, Mark Romerti, and welcome to the Data Flowcast.
• 0:32
  SPEAKER_03: And today I'm super excited because I'm joined by Sanket Patel, Director of Engineering and Training Technologies.
• 0:39
  SPEAKER_03: And first thing first, how are you doing today, Sanket?
• 0:44
  SPEAKER_00: It's extremely busy day today, Mark.
• 0:48
  SPEAKER_00: I mean, thanks for having me and show.
• 0:52
  SPEAKER_00: Awesome.
• 0:53
  SPEAKER_03: So uh let's dive into what we're gonna cover today.
• 0:57
  SPEAKER_03: So we are talking about how trading technologies is migrating 20 years of legacy job orchestration to Airflow, an astronomer, and the reverse engineering uh strategy that they used to make sure nothing broke in production, which I think is a very common issue uh across the industry.
• 1:15
  SPEAKER_03: You know, it's very common to actually have some legacy systems, and whenever you want to migrate away from those systems, it's uh pretty painful, unfortunately.
• 1:24
  SPEAKER_03: So, can you introduce yourself and tell us what you do at at uh trading technologies today?
• 1:29
  SPEAKER_01: So, well, I'm a strategic engineering leader and director of engineering at trading technologies.
• 1:35
  SPEAKER_01: I have around 15 plus years of experience to try and data and software projects.
• 1:40
  SPEAKER_03: Okay.
• 1:41
  SPEAKER_03: And um as you are like a director of engineering, like um just to give you know a better idea to the audience, what what is the team size like typically?
• 1:52
  SPEAKER_03: Like, you know, do you interact with uh many data engineers or do you have a mix of data scientists and then data engineers?
• 1:59
  SPEAKER_01: And like, you know, tell us more about the context in which you are so when I joined training technologies, uh, I think I'm the first one who actually driving data initiatives, especially for data platform to elementary data silos.
• 2:12
  SPEAKER_02: Okay.
• 2:13
  SPEAKER_01: Um, my my role is definitely an engineering role here, uh, to oversee engineering efforts.
• 2:19
  SPEAKER_01: But Tom is finding more value if I focus on executing data platform-related projects.
• 2:27
  SPEAKER_01: So in last one year, we built almost 25 plus engineers team in telling technologies, um, including FT and the consultants.
• 2:40
  SPEAKER_03: Okay, so that's a lot.
• 2:42
  SPEAKER_03: That's quite a lot.
• 2:43
  SPEAKER_03: Okay, interesting.
• 2:45
  SPEAKER_03: But can you tell us more about like what used to run or what is still running uh for like the um orchestration side?
• 2:53
  SPEAKER_03: Because obviously, you know, when we talk about unifying your uh your data stack, most of the time you have like an orchestrator that actually talks to all of those systems.
• 3:03
  SPEAKER_03: Um can you tell us more about this?
• 3:06
  SPEAKER_03: Like what is the sure sure.
• 3:08
  SPEAKER_01: So so the still the large part of the organization are actually running on top of their operational data stores only, right?
• 3:17
  SPEAKER_01: So they're the reports being pulled out from their own operational data stores only.
• 3:22
  unknown: Okay.
• 3:22
  SPEAKER_01: And and their OLTP database are penetrated a lot to pull out those kinds of architecture and uh uh you know building an insight on top of it.
• 3:34
  SPEAKER_01: Um TD basically acquired one firm three years back, uh, which basically emerged as an data and an artics team in TD.
• 3:45
  SPEAKER_01: And that team has their own line of business software, okay, which has orchestration abilities.
• 3:54
  SPEAKER_01: And that particular software was built around 17, 18 years back.
• 3:59
  SPEAKER_01: Okay, almost uh almost uh a two decade before.
• 4:03
  SPEAKER_01: And primarily it's been written with the old school ASP.NET application that are certain stuff is running on on top of SQL Server, stuff like that.
• 4:14
  SPEAKER_00: Okay.
• 4:15
  SPEAKER_01: And the legacy infrastructure has limited resource who has SME level understanding of these systems, uh, which is not in line with the core technology stack been used by the other line of businesses of TT.
• 4:34
  SPEAKER_01: So it's more about we are solving two, three problems together.
• 4:37
  SPEAKER_01: One is like how to absorb a new acquired firms technology with the TT's landscape, and also started to build something which basically offers other line of businesses also to consolidate that data platform related stuff into one particular space.
• 4:56
  SPEAKER_01: So we choose Snowflake as a data platform as a choice.
• 5:00
  unknown: Okay.
• 5:00
  SPEAKER_01: So we are hosting our data on Snowflake as a target system.
• 5:04
  SPEAKER_01: Okay, but the legacy is more about spinning up those jobs over C.NET based application, which basically we have some logics in place to schedule jobs, orchestrate the data, building typical fact tables and dimension tables on top of it.
• 5:25
  SPEAKER_01: Now, with new data stacks, what we are trying to do is we choose Airflow as our orchestration tool.
• 5:31
  SPEAKER_03: Okay.
• 5:32
  SPEAKER_01: And the Snowflake is our cloud database.
• 5:37
  SPEAKER_01: And this whole mix of it, we would like to call it as a data platform.
• 5:43
  SPEAKER_01: With data platform, we are also using a lot of Snowflake native features like OpenFlow, Snowflake intelligence.
• 5:52
  SPEAKER_01: So those features allow us to enhance our ability to address data problems.
• 6:01
  SPEAKER_03: And so just to uh so like why did you actually decide to use Airflow?
• 6:10
  SPEAKER_03: Um because like there are there are many orchestrators, right?
• 6:13
  SPEAKER_03: I mean, I won't I won't, you know, I won't give names here, uh, but there are many orchestrators.
• 6:19
  SPEAKER_03: So why did you choose Airflow?
• 6:21
  SPEAKER_03: Did you uh already use Airflow before joining training technologies?
• 6:25
  SPEAKER_03: Did you uh like why?
• 6:29
  SPEAKER_01: Yeah, so see, uh Mark, the thing is that throughout my career I spent a lot of time to building or supporting an existing jobs driven infrastructures, batch-driven infrastructures.
• 6:46
  SPEAKER_01: Because large part portion of my career is about you know being a data architect or or database administrator, or sometimes being a principal architect of the whole project.
• 6:58
  SPEAKER_01: So most of those batch processing been running on tools like autosys or control am, which offers you a lot of resilience for sure, yeah, and um a lot of scalability as well sometimes.
• 7:15
  SPEAKER_01: Um there are there are a lot of flaws into it, what I outlined, right?
• 7:21
  SPEAKER_01: And those flaws are like you know, they don't offer you in a cloud native ability by nature, right?
• 7:32
  SPEAKER_01: Okay, so you don't have any direct connectivity or any plugins in a place for which you can directly talk to the cloud providers.
• 7:45
  SPEAKER_01: You probably need your own way to connect it, you probably need to rely on certain scripts, what you have written, homegrown scripts, what you have written to connect to systems.
• 7:56
  SPEAKER_01: It emerged that Airflow has all those abilities to which you can natively view all the dependencies, right?
• 8:06
  SPEAKER_01: Right, uh, in both structured and graphical way, right?
• 8:12
  SPEAKER_01: So and graphical representation is also easier to understand that what is going on, what is not going on, if you just segregate your milestones in a way that is performing as DAG and all your tasks are fitting to the DAG.
• 8:33
  SPEAKER_01: And use maybe a DAG generator through which you can orchestrate more so many stuffs within the same DAG, but you you at at the end goal you have one DAG which at least serves your one milestone.
• 8:48
  SPEAKER_01: So 2000, 3000 jobs, which appears you to monitor them very difficult, that can squeeze up to 15 penti decks.
• 8:58
  SPEAKER_03: Right.
• 8:59
  SPEAKER_03: Yeah.
• 9:00
  SPEAKER_01: Right.
• 9:01
  SPEAKER_01: And uh that basically exciting to watch for sure.
• 9:06
  SPEAKER_01: Uh right.
• 9:08
  SPEAKER_01: So that's how this whole thing emerged that okay, Airflow is something what I would like to work against.
• 9:15
  SPEAKER_01: But that doesn't mean that I only work with Airflow, which is also an open source in nature, which has again a good advantage over other propriety softwares.
• 9:26
  SPEAKER_01: Uh I also have been introduced with Argo CICD.
• 9:32
  SPEAKER_03: Okay.
• 9:33
  SPEAKER_01: And and Argo batch processing jobs, uh batch processing software, and which is also phenomenal software, which is also kind of open source and runs really well under Kubernetes infrastructure.
• 9:47
  SPEAKER_01: But when I compare Argo versus Airflow, these are the two major cloud native software I was evaluating when I was introducing a new orchestration tool in TT.
• 9:59
  SPEAKER_01: Um, I really think that okay, the kind of community support what we have around Airflow is phenomenal compared to Argo.
• 10:09
  SPEAKER_01: And in my search, I uh I also identified that okay, astronomer is this one airflow provider who basically solved a lot many other problems.
• 10:20
  SPEAKER_01: So what I faced in my past experiences with Airflow where while I was using it on top of MWAA or as a composer in GCP.
• 10:32
  SPEAKER_01: So the major problems what I face is okay, establishing in CI CD with standard way.
• 10:38
  SPEAKER_01: So CI C D standardization with that astronomer is already given.
• 10:48
  SPEAKER_01: Astronomer also simplify the obsoletity even more.
• 10:53
  SPEAKER_01: Uh, you can configure SLAs out there, you can configure pager duty as a native connections.
• 11:02
  SPEAKER_01: Um, and there are ways to generate alerts on your billing usages within one platform in in different setup.
• 11:13
  SPEAKER_01: You probably need to configure too many other services within the cloud provider to manage it.
• 11:20
  SPEAKER_02: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 11:29
  SPEAKER_02: From batch pipelines to training models and wiring up agents.
• 11:33
  SPEAKER_02: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 11:39
  SPEAKER_02: Check it out at astronomer.io.
• 11:44
  SPEAKER_03: It's great that you are talking about Astronomer because that that's exactly my next question to you.
• 11:50
  SPEAKER_03: Uh, because you mentioned that you wanted to have like a cloud native solution.
• 11:55
  SPEAKER_03: Um, and obviously you can run Airflow in the cloud, but also to your point, like Airflow is still uh and is an open source project.
• 12:03
  SPEAKER_03: And the I think, like you actually said, that the the observability part is super important for you, especially when you have like thousands of pipelines, jobs, and so on uh to monitor.
• 12:15
  SPEAKER_03: Airflow definitely has you know like some monitoring aspects, uh, but let's be let's be also uh clear.
• 12:22
  SPEAKER_03: There are a few things that you know are not part of Airflow.
• 12:27
  SPEAKER_03: And you said that you are actually uh using Astronomer today.
• 12:31
  SPEAKER_03: Uh so can you can you tell us more about like why why are you using Astronomer today?
• 12:37
  SPEAKER_03: Like um and maybe more on the observability part, if this is something that you know you find a lot of value.
• 12:43
  SPEAKER_03: Uh, because I I think you know, for some for some reason, like when you when you do some some, you know, when you talk about like how you can author your DAGs and so on, this is super popular, you know, how you can make your pipelines better.
• 12:55
  SPEAKER_03: But whenever you talk about observability and and you know how can you actually monitor your pipelines in the most efficient way, this is less popular but super critical.
• 13:05
  SPEAKER_01: Uh so so so stability is uh core of any data pipeline.
• 13:11
  SPEAKER_01: If you if you have written your data pipelines with any software or any tool, but if you could not able to provide resilience and stability, then that's a problem, right?
• 13:24
  SPEAKER_01: You you you basically reun not just your customer delivery, but also in long term, your employees also will be annoyed if you if you yeah, to support such instable system.
• 13:41
  SPEAKER_01: So that's a major reason I was you know really emphasizing to use astronomer to reduce overhead to manage airflow.
• 13:52
  SPEAKER_01: So for example, airflow is community-driven product, and if something goes in, right, you and whenever you see you know major releases being released, you want it to yourself upgrade it to that release, right?
• 14:14
  SPEAKER_01: And if you wanted to do it by your own, even though you are running through services, I have seen people are spending more than a week, sometimes months, to make sure that what we're usually running into previous version is not impacting in the newer version.
• 14:36
  SPEAKER_01: This is also one aspect, right?
• 14:38
  SPEAKER_01: When it comes to observability part, I would say observability required dashboards, right?
• 14:47
  SPEAKER_01: A lot of firms have their own SRA team.
• 14:51
  SPEAKER_01: We also have our SRA team, right?
• 14:52
  SPEAKER_01: To observe a lot many metrics in various industrial standard products.
• 14:59
  SPEAKER_01: Uh but in most of the cases, teams are doing their L0 work and basically wants to initiate uh a ticket through which you can engage your developer group.
• 15:15
  SPEAKER_01: And stuff like you know having the connectivity with PagerDuty, which basically absorb your current logs and given into the ticket.
• 15:25
  SPEAKER_01: Those are like really cool features for even SRETM to work with.
• 15:39
  SPEAKER_01: Copied, put it into some systems like ServiceNow or a patron duty, whatever.
• 15:46
  SPEAKER_01: Um all of those network, uh all of those native connectivity is already in place in astronomer.
• 15:54
  SPEAKER_01: So you don't need to solve bigger problems, you just need to uh solve your business problem.
• 16:00
  SPEAKER_01: So I'm trying to minimize our efforts around managing infrastructure towards to solving real problems around data.
• 16:10
  SPEAKER_00: Yeah.
• 16:11
  SPEAKER_01: And astronomers sort of helping me to accelerate that process while I choose astronomer.
• 16:20
  SPEAKER_03: Yeah, I mean, I think like I mean, you you I think you said it extremely well, right?
• 16:24
  SPEAKER_03: I mean, you don't want to spend a ton of time on month, like, you know, on the things that actually don't matter for your business.
• 16:30
  SPEAKER_03: Uh right.
• 16:31
  SPEAKER_03: And uh and if you go with Airflow open source, go ahead.
• 16:35
  SPEAKER_03: But just like you will have to manage a lot of things yourself that you actually really don't want.
• 16:40
  SPEAKER_01: Um and there is another aspect also.
• 16:43
  SPEAKER_01: Let's suppose if you have if you need to manage Airflow by yourself, you to offer a global setup, you at least need three, four engineers working in various regions.
• 16:54
  SPEAKER_00: Right, true.
• 16:56
  SPEAKER_01: Various regions, right?
• 16:57
  SPEAKER_01: Yeah.
• 16:58
  SPEAKER_01: So if you propagate the cost of those four resources, just manage Airflow, which cannot be used for other stuff as well, maybe.
• 17:09
  SPEAKER_01: That's an operational approach, but I'm thinking against that okay, why I hire four resources to just keep alive one system.
• 17:19
  SPEAKER_01: Uh maybe that person could be a cloud engineer also can do that.
• 17:25
  SPEAKER_01: But there's an argument around that.
• 17:27
  SPEAKER_01: So if you have engineers who do a general work, then you don't have a specialized and you cannot rely on them to do a specialized work.
• 17:38
  SPEAKER_01: So when it comes to upgrading Airflow, you cannot rely on cloud engineer to do upgrade.
• 17:43
  SPEAKER_03: That's right.
• 17:45
  SPEAKER_03: That's right.
• 17:45
  SPEAKER_01: So these are like trade-offs you need to take.
• 17:47
  SPEAKER_01: And I I had taken that trade-offs, okay.
• 17:49
  SPEAKER_01: Okay, instead of paying three, four engineers, I probably opt some service in a way through astronomer, which which is also a costless model.
• 17:59
  SPEAKER_01: And we able to lock a rook really good deal with them so that we got early engagement, one month free period to onboard us with basic minimal setup with their own engineers.
• 18:15
  SPEAKER_01: That basically expedite a lot of efforts, right?
• 18:17
  SPEAKER_01: And the kind of projects what I'm executing, Mark, as you understand that a lot of organizers used to take two, three years to hire like you know, 20, 30 bunch of engineers to work against, right?
• 18:31
  SPEAKER_01: We not just build those amounts of team, but we also started delivering mission critical projects for some uh with resilience.
• 18:41
  SPEAKER_01: So if you need to really work fast and if you really need to execute projects, then you need to take certain trade-offs which allow you to focus on business problems first than the infrastructure problems.
• 18:54
  SPEAKER_03: Yeah, I mean, I fully I fully agree with you.
• 18:56
  SPEAKER_03: Um fully agree with you.
• 18:58
  SPEAKER_03: Um and actually, like just on on that, uh I just wanted to come back a little bit uh on you know the legacy systems and so on, because I think it's it's gonna be important for for many people that uh listen to this episode.
• 19:11
  SPEAKER_03: Um you know, like migrating is painful for sure.
• 19:15
  SPEAKER_03: Like migrating from Airflow 2 to Airflow 3 is painful.
• 19:18
  SPEAKER_03: Um for actually for any tool, it's always painful to migrate uh from one major version to another one.
• 19:24
  SPEAKER_03: But it might be even more painful when you have to migrate from one tool to another one, you know, when people are actually used to use uh you know, Odosys and then they have to use Airflow, right?
• 19:36
  SPEAKER_03: Uh with your experience, can you tell us more about like what strategy should I take if I want to ensure that my team will follow me?
• 19:46
  SPEAKER_03: Uh, you know, will follow with the with that huge uh migration.
• 19:50
  SPEAKER_03: I mean it's uh it might be more like a human problem to solve than really a technical one, but I think like many people would love to have your uh your thoughts on that.
• 20:02
  SPEAKER_01: Yeah, yeah, you you absolutely nailed it when it comes to you know moving away from one tool to another tool.
• 20:10
  SPEAKER_01: It's always going to be challenged because you probably have a few people who love the previous tool, who is familiar with that tool.
• 20:18
  SPEAKER_01: Now all of a sudden they need to evolve with the new set of tools and whatnot.
• 20:24
  SPEAKER_01: But you know what?
• 20:26
  SPEAKER_01: I mean, I have this take, right?
• 20:28
  SPEAKER_01: And I I have seen this in the past, right?
• 20:30
  SPEAKER_01: That where I used to be working with some investment banks where it's very difficult to manage bureaucracy, uh, to be honest, uh, to move away from one to another and having enough energy to support you to execute such projects sometimes.
• 20:49
  SPEAKER_01: But in last couple of years, what I see as a trend that okay, after you the AI is evolving, um a lot of stubborn people also becoming uh flexible enough nowadays to add off new stuff.
• 21:09
  SPEAKER_01: So that's right.
• 21:10
  SPEAKER_01: So that that's the external factor, what I observe that is actually helping a lot.
• 21:16
  SPEAKER_01: And a lot of organization and uh leadership of the firms is passing this message again and again to their teams that okay, if we really need to be ahead in AI curve, then you need to eliminate your data silos.
• 21:36
  SPEAKER_01: You must need to have a strong data strategy before you start executing AI strategy.
• 21:44
  SPEAKER_01: Right?
• 21:44
  SPEAKER_01: So such push is really helping on the ground sometimes to organize people to at least accept and work against it.
• 21:56
  SPEAKER_01: And also they'll get an opportunity to, you know.
• 22:00
  SPEAKER_01: To work with the latest tool.
• 22:03
  SPEAKER_01: So it's more about spreading more awareness to the teams who is going to be move away from the legacy tool to the new tool.
• 22:14
  SPEAKER_01: And I think if you're able to address the why questions from all of them, um with providing them enough comfort, they will be accepted the way they are, and they will be trained over the new systems.
• 22:33
  SPEAKER_01: And that's the major aspect, what I would say.
• 22:37
  SPEAKER_01: Then people give you a lot of confidence to work against such migrations as well.
• 22:42
  SPEAKER_03: Okay.
• 22:43
  SPEAKER_03: So there is definitely like a psychologically uh psychological uh aspect for it.
• 22:49
  SPEAKER_03: Yeah, I agree.
• 22:50
  SPEAKER_03: And and to your point, I mean it's it's moving away from one tool to something that hopefully will deliver more value and will be easier to use than you know the one that you used to use.
• 23:01
  SPEAKER_03: Uh so you should be pretty uh happy about it.
• 23:05
  SPEAKER_03: Um but okay, that this is super interesting.
• 23:07
  SPEAKER_03: Uh, one last question to you, um, because you just mentioned before that you are running mission critical uh pipelines today.
• 23:16
  SPEAKER_03: Um and during our uh preparation call, you also talked about AI, which obviously like AI is everywhere today.
• 23:23
  SPEAKER_03: Uh so I have to ask you something about AI.
• 23:26
  SPEAKER_03: Um are you using AI today um with Airflow?
• 23:31
  SPEAKER_03: Um and uh if not, that's fine.
• 23:35
  SPEAKER_03: Just like what you know, what typically are you planning to use uh AI with Airflow or what what what is your uh current state with AI?
• 23:42
  SPEAKER_03: Because you know, we are all talking about AI, but it's always hard to know where you know where people are with this.
• 23:48
  SPEAKER_01: I think uh that's an interesting question.
• 23:50
  SPEAKER_01: I mean, um nobody's actually asking real questions sometimes when it comes to AI, right?
• 23:57
  SPEAKER_01: I mean a lot of people are actually in the journey of proving themselves that they are onboarded to AI, but are we are you really solving real problem with AI or not?
• 24:07
  SPEAKER_01: And the other yeah, so I'll say so.
• 24:13
  SPEAKER_01: There are two aspects when I'm talking about AI, right?
• 24:16
  SPEAKER_01: So so one is so one side of AI, what we are building is are we able to offer an opportunity to our customers that they can dialogue with their own data, right?
• 24:30
  SPEAKER_01: And they should not wait for client services in future that okay, I need this kind of reports, I need this kind of dashboards, I need this kind of insights, and they'll go and commit them.
• 24:42
  SPEAKER_01: It may take some time to deliver this report.
• 24:45
  SPEAKER_01: Um, it's been treated as an unchange request, and it would be in our backlog, and it would be in a path of so-and-so releases, and you'll get that.
• 24:57
  SPEAKER_01: Instead of that, if you're able to give them the ability to dialogue with their own data and insights in a way, so that they can pull out some of those custom reports by themselves.
• 25:09
  SPEAKER_01: So that's one part of AI, what I'm talking about when it comes to the data platform.
• 25:13
  SPEAKER_01: How we can solve customers' problems or how we can make our customers happy by using AI products.
• 25:20
  SPEAKER_01: So that's one aspect.
• 25:22
  SPEAKER_01: The other aspect, when what when you asked me that what we do with as astronomer or airflow, uh with when it comes to AI.
• 25:31
  SPEAKER_01: So see, Airflow is going to be definitely our primary choice of orchestration tool, which basically build our data strategy.
• 25:43
  SPEAKER_01: And as I said, if you have a strong data strategy, then you can run your AI strategy really appropriate and accurate.
• 25:53
  SPEAKER_01: Because if you don't have an accurate data, then you cannot provide them a response accurately.
• 26:00
  SPEAKER_01: So that's one aspect.
• 26:01
  SPEAKER_01: The other aspect, what we do around astronomer and airflow with AI.
• 26:07
  SPEAKER_01: So we have created certain skills, Mark, uh plot skills, for which we can um orchestr, uh we we can orchestrate uh a new decks based on certain requirements being in place.
• 26:24
  SPEAKER_01: So and all other objects as well.
• 26:30
  SPEAKER_01: So we we have created certain proof of concepts around that.
• 26:34
  SPEAKER_01: We are testing around it to to publish it in a way that okay, if we have an appropriate requirements in place, and if we orchestrate those requirements appropriately, then the boilerplate decks would be created, right?
• 26:52
  SPEAKER_01: Uh so that kind of skills we have created.
• 26:56
  SPEAKER_01: So some of it we borrowed uh from Cloud Marketplace, right?
• 27:06
  SPEAKER_01: And some of it we basically emerged uh based on our own custom requirements.
• 27:15
  SPEAKER_01: So as a beginning, what we have done is we we just gone through with certain skills and borrow it, borrow some of those foundational skills, and then we ended up creating the custom skills on top of it.
• 27:32
  SPEAKER_01: So that when we run when it runs it with co-op, then it can produce uh appropriate debt from it.
• 27:40
  SPEAKER_03: This is really cool.
• 27:42
  SPEAKER_03: Um, you know, I actually have to uh to mention based on what you said uh regarding the the clot skills, uh, that Astronomer um also released some uh cloud skills that you can use uh for authoring DAGs and so on.
• 27:55
  SPEAKER_03: Obviously, those skills are not you know tailored to your uh uh use cases, but for you know people that listen to you to us, uh they might also be useful if you want to author your DAGs following best practices, if you want to implement lineage and so on.
• 28:08
  SPEAKER_03: Might be also uh helpful for you um you know to take a look at.
• 28:12
  SPEAKER_03: It's an open source uh repo, it's astronomer agent, something like that.
• 28:16
  SPEAKER_03: Um but okay, this is great.
• 28:17
  SPEAKER_03: Actually, like I know that many people are using AI today, not within Airflow yet, but a lot of people are using AI for developing DAGs and so on.
• 28:28
  SPEAKER_03: Uh, because obviously it's you know, I mean, developing DAGs is it's fun at the beginning, but then it's not at some point.
• 28:34
  SPEAKER_01: In in nutshell, the my current experience with Cloud Skills and Cowork is more about you can use these tools to accelerate your efforts, but you cannot fully rely on them.
• 28:51
  SPEAKER_01: Right?
• 28:51
  SPEAKER_01: You probably need to apply your own due diligence on top of it, what been created on top of it.
• 28:57
  SPEAKER_01: And sometimes I also felt that okay, this is over-engineered.
• 29:01
  SPEAKER_01: Sometimes it it's uh it has some flaws into it.
• 29:05
  SPEAKER_01: So it's and and some of my engineers also complaining that it's it it was easier for us to write this code in 30 minutes or 40 minutes rather than reviewing this code for three hours.
• 29:22
  SPEAKER_03: Well, um I mean, well, uh breaking news, you still have to do what you are doing if you are using AI.
• 29:29
  SPEAKER_03: Otherwise, it's not gonna end uh not gonna end well.
• 29:32
  SPEAKER_01: Right.
• 29:32
  SPEAKER_01: So I think the hand whatever work what you do by your hands that is getting minimized, but whatever work you are doing with your cognitive ability, it is basically increased after using and stuff like that.
• 29:48
  SPEAKER_03: I agree, I agree, I agree, I agree.
• 29:51
  SPEAKER_03: The cognitive load is is is real, especially when you have like many agents and so on.
• 29:55
  SPEAKER_03: I don't know how people are able to use like 12, 20 sub-agents at the same time.
• 30:01
  SPEAKER_03: Uh, but anyway, that would be for another discussion.
• 30:04
  SPEAKER_03: Thank you so much.
• 30:05
  SPEAKER_03: Uh honestly, thank you so much, Sanket, for this uh discussion.
• 30:08
  SPEAKER_03: I think it was a wonderful one.
• 30:10
  SPEAKER_03: Uh, but last, very last question to you.
• 30:13
  SPEAKER_03: Uh, what is the best way for folks uh to get in touch with you?
• 30:17
  SPEAKER_03: You know, if they want to ask you questions or whatever, like is there any uh way to contact you or I think uh the best way to contact me is through LinkedIn.
• 30:28
  SPEAKER_01: They can reach me out through LinkedIn.
• 30:31
  SPEAKER_01: I can share my LinkedIn profile to you.
• 30:33
  SPEAKER_03: Yeah, I will put that under the description.
• 30:36
  SPEAKER_01: Yeah.
• 30:37
  SPEAKER_03: So people will be able to see.
• 30:38
  SPEAKER_01: So yeah, they can reach out to me on LinkedIn.
• 30:40
  SPEAKER_01: I'm really responsible on LinkedIn.
• 30:42
  SPEAKER_01: Uh, if somebody wants to know more about how they can um go for an astronomer, how they can use astronomer appropriately in compared to Apache R4.
• 30:55
  SPEAKER_01: Uh, I can help.
• 30:57
  SPEAKER_03: That's awesome.
• 30:58
  SPEAKER_03: Well, again, thank you so much, Sankit.
• 31:00
  SPEAKER_03: It was a wonderful pleasure to have you.
• 31:02
  SPEAKER_03: And uh have a wonderful day.
• 31:05
  SPEAKER_03: Yeah, thank you.
• 31:06
  SPEAKER_03: Thank you.
• 31:07
  SPEAKER_03: See you.
• 31:07
  SPEAKER_03: Thank you.
• 31:08
  unknown: Yeah.
• 31:09
  SPEAKER_02: Thank you for listening to the data flowcast.
• 31:12
  SPEAKER_02: Check the show notes for links to everything mentioned today.
• 31:16
  SPEAKER_02: If you're enjoying the podcast, subscribe so you never miss an episode.
• 31:21
  SPEAKER_02: And leave us a five star review.