Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating cloud data migrations at HCL Tech

Transcript

Orchestrating cloud data migrations at HCL Tech · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:08
  SPEAKER_01: Let's get into it.
• 0:10
  SPEAKER_00: Okay, hi everyone.
• 0:11
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:13
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Banu Prakash, who is the technical lead at HCL Tech.
• 0:19
  SPEAKER_00: Banu, how are you?
• 0:21
  SPEAKER_02: I'm doing great, Kenton.
• 0:22
  SPEAKER_02: Firstly, thanks a lot for inviting me to your show.
• 0:26
  SPEAKER_02: So I have been watching your shows from quite a bit of time now.
• 0:30
  SPEAKER_02: And I'm pretty happy to be here.
• 0:33
  SPEAKER_02: Thanks a lot for inviting me again.
• 0:35
  SPEAKER_00: Yeah, that's great.
• 0:36
  SPEAKER_00: It's so great to have you on.
• 0:38
  SPEAKER_00: We can go ahead and dive into it.
• 0:40
  SPEAKER_00: So for today's podcast, we're going to discuss how you, Vanu, and your team use Airflow to orchestrate large-scale data and platform migrations to the cloud.
• 0:49
  SPEAKER_00: I think this will be really insightful for all of our listeners.
• 0:53
  SPEAKER_00: And before we get into kind of the technical details, I want to start with some background.
• 0:58
  SPEAKER_00: So first tell me about what HCL Tech does.
• 1:01
  SPEAKER_02: So HCL Tech is a service-based company.
• 1:04
  SPEAKER_02: So it has like uh you know pretty big clients who are into healthcare domain, finance, uh, you know, any other uh leading players are the clients of HCL.
• 1:15
  SPEAKER_02: Basically, they provide software services and uh they also provide uh their own products, uh, you know, along with that.
• 1:23
  SPEAKER_02: So this is what they do.
• 1:25
  SPEAKER_00: Okay.
• 1:26
  SPEAKER_00: And tell me about what you and your team do specifically.
• 1:31
  SPEAKER_02: So my our team is uh uh purely into data engineering uh and uh big data.
• 1:38
  SPEAKER_02: So I am an SME with uh when it comes to Airflow, ETL, you know, orchestration in general.
• 1:45
  SPEAKER_02: So uh I lead the efforts, uh, get involved in the uh park natural discussions, uh, you know, uh develop uh uh solutions, go through DR based, DR based, uh, you know, getting things worn.
• 2:01
  SPEAKER_02: So I get involved uh right from design to delivery.
• 2:04
  SPEAKER_02: So uh yeah, that's what we do.
• 2:06
  SPEAKER_02: And uh I basically lead all these efforts uh as an SME.
• 2:11
  SPEAKER_00: So great.
• 2:12
  SPEAKER_00: So you must have seen a lot of Airflow implementations then.
• 2:17
  SPEAKER_02: Oh yeah.
• 2:18
  SPEAKER_02: From the past six years I've been working with Airflow, so I did see a lot of implementations.
• 2:23
  SPEAKER_00: Yeah, great.
• 2:24
  SPEAKER_00: And then tell me, so you mentioned six years with Airflow, that's quite some time.
• 2:28
  SPEAKER_00: The project has come a long way in that time.
• 2:30
  SPEAKER_00: Uh tell me what kinds of use cases you've worked on with Airflow.
• 2:37
  SPEAKER_02: So uh starting with data migration, uh work good on uh, you know, uh analytics, uh so or like scheduled analytics jobs.
• 2:46
  SPEAKER_02: For example, I've uh dealt with the marketing client where in the ask it for uh refreshered customer uh segmentation details and then uh build out uh insights and uh you know get get the goal layer data for their analytics purposes.
• 3:01
  SPEAKER_02: So uh these kind of things, apart from that, like uh in general the data processing and then uh uh picking out uh jobs, you know, in sync with uh the web applications.
• 3:13
  SPEAKER_02: So especially like when the web applications interact with the data, right?
• 3:17
  SPEAKER_02: All the data ops is uh taken care of by Airflow orchestrators.
• 3:21
  SPEAKER_02: So these are the uh you know projects that I've been wanting so far.
• 3:25
  SPEAKER_00: Yeah, that makes sense.
• 3:27
  SPEAKER_00: And tell me at a high level when you're working with clients, when do you recommend that they use Airflow?
• 3:35
  SPEAKER_02: So uh I would recommend Airflow in all the you know orchestration scenarios except for real time.
• 3:42
  SPEAKER_02: So with the uh you know kind of integrations that Airflow has.
• 3:47
  SPEAKER_02: So it can be integrated with many platforms, be data bricks, move lake, and then for deleting tools like uh uh service now, and then uh it uh easily gets uh integrated with uh message tools like Kafka, and it has like a huge amount of integrations, really available.
• 4:06
  SPEAKER_02: Um, so that's the biggest advantage, and then uh uh it's cloud cloud agnostic, which as it's an open source software, it can be easily migrated from one cloud to another.
• 4:17
  SPEAKER_02: So that's uh one of the biggest advantage.
• 4:20
  SPEAKER_02: So like companies don't want to uh you know cactus to one single platform, right?
• 4:24
  SPEAKER_02: Uh uh even though there are uh you know uh slight not like uh completely full-fledged alternatives to Airflow, but then still there are uh quite a few uh you know alternatives provided by these cloud platforms like uh Azure Data Factory, and uh we have Glue on AWS.
• 4:44
  SPEAKER_02: So uh despite of having these orchestrators, but uh we if we go with the me law, we are kind of tied to a single platform that with Airflow.
• 4:52
  SPEAKER_02: Uh we are cloud agnostic, it's easy to move, it's uh uh plight to it.
• 4:57
  SPEAKER_02: All we have to do is uh you know uh get the setup done, it needs a metadata database and then keep the server running, uh you know, spin up the worker process.
• 5:08
  SPEAKER_02: That's all it takes.
• 5:09
  SPEAKER_02: So and a message queue.
• 5:10
  SPEAKER_02: That's all it takes.
• 5:11
  SPEAKER_02: So it's uh quite uh quite a simple setup and um it's easy to manage uh because there are uh uh you know a lot less uh complexities involved.
• 5:21
  SPEAKER_02: Uh we know like uh uh in the event of an infrared, we if we are like uh almost sure in most of the cases that uh uh where exactly the error is popping out from.
• 5:32
  SPEAKER_02: So uh if you know in all uh you know uh cases where there's uh uh requirement, there's not there is no requirement of real-time data with uh size shareflow.
• 5:43
  SPEAKER_00: Yeah, definitely.
• 5:44
  SPEAKER_00: I think the flexibility is of course a huge uh draw for working with Airflow.
• 5:50
  SPEAKER_00: And I like that you're describing it as simple to set up because I think that's not always the impression, certainly.
• 5:56
  SPEAKER_00: And I guess in some cases it might be complex for uh you know large scale installation, but I do think at the basic level it is pretty straightforward.
• 6:04
  SPEAKER_02: Oh, yeah, yeah.
• 6:05
  SPEAKER_02: After having this SaaS tools like ADF, uh it it kind of sounds uh complex like that.
• 6:11
  SPEAKER_02: When we actually started back then, right, the the setup, the setup was uh simple compared to the other tools.
• 6:18
  SPEAKER_00: Yeah, yeah, I definitely believe that.
• 6:21
  SPEAKER_02: Yeah.
• 6:22
  SPEAKER_00: Cool.
• 6:23
  SPEAKER_00: Where I'd love to discuss where Airflow fits into your current project.
• 6:28
  SPEAKER_00: So can you walk me through what you're working on now?
• 6:31
  SPEAKER_00: I understand you've been working on this cloud migration.
• 6:34
  SPEAKER_00: Where does Airflow fit into that?
• 6:36
  SPEAKER_02: So for all the orchestration purposes, we are using uh Airflow here.
• 6:41
  SPEAKER_02: So all the uh data extraction jobs are orchestrated through Airflow.
• 6:45
  SPEAKER_02: Uh, we are extracting data from on-premises databases, so uh all the watermarking process on the source side, and then uh maintaining all the cross-reference tables uh uh you know to do batch wise uh transfer from on-prem to cloud.
• 7:02
  SPEAKER_02: So uh, and then uh the actual cloud movement itself, like uh the uh post-extraction, the cloud files.
• 7:09
  SPEAKER_02: I mean, data has to be moved to cloud, right?
• 7:11
  SPEAKER_02: So the migration uh process itself is uh again orchestrated uh through Airflow, and then all the subsequent process, the data processing happening on cloud, uh beat on data breaks or small, both of them are orchestrated through Airflow.
• 7:25
  SPEAKER_02: So the end-to-end orchestration is done in Airflow.
• 7:29
  SPEAKER_02: Uh apart from that, like uh also for uh incident management and uh uh for uh building out uh analytical dashboards, so building out the uh gold layer data.
• 7:42
  SPEAKER_02: We have basically schedule jobs which run on a periodic basis, and then they you know build out uh gold layer data for the dashboards to pick up.
• 7:51
  SPEAKER_02: So these can like it.
• 7:53
  SPEAKER_02: Most of the orchestration uh you know work is done using airflow.
• 7:59
  SPEAKER_00: Okay, interesting.
• 8:01
  SPEAKER_00: And what version of Airflow are you using to do this?
• 8:05
  SPEAKER_02: So we have started with 2.3, then later migrated to 2.11, and then now we are uh on to 3.2.
• 8:13
  SPEAKER_00: So have you had any challenges with the migration or upgrades from like an Airflow 2.3 all the way to 3.1?
• 8:23
  SPEAKER_02: So the biggest advantage with Airflow is that uh the code is for the most part, the code is backwards compatible.
• 8:29
  SPEAKER_02: So it was uh pretty easy for us to move from one version of Airflow to the other.
• 8:33
  SPEAKER_02: So as I said, we started with 2.3 and then we moved on to 2.21.
• 8:38
  SPEAKER_02: Like for the most part, it's AAD uh uh code migration.
• 8:43
  SPEAKER_02: So we have LL models in place and uh with the help of which we did the code migration.
• 8:50
  SPEAKER_02: So uh there are a few places where we had to uh you know uh make a few modifications, but 99% of the code still remain the same.
• 8:59
  SPEAKER_02: So we can say that uh with any other uh uh applications.
• 9:04
  SPEAKER_02: So even like uh for a uh you know legacy framework like Spring Boot, when uh uh we are moving from one version to the other, a lot of uh things need to be changed.
• 9:14
  SPEAKER_02: And it would uh you know pop up a lot of uh new vulnerabilities.
• 9:19
  SPEAKER_02: So that's not the case with uh airflow.
• 9:22
  SPEAKER_02: So that's the biggest strategy.
• 9:26
  SPEAKER_00: Yeah, I can see where that makes a huge difference in the type of work you do where you're working on all sorts of different projects, being able to recommend a tool consistently depends on uh the ability to not have to change the code all the time.
• 9:42
  SPEAKER_00: He would just be in constant sort of loops of reworking everything.
• 9:46
  SPEAKER_00: And I know the developers on the project try really hard with new versions to minimize any disruptions in terms of how the code works.
• 9:54
  SPEAKER_00: So that's great to hear that it's working for you.
• 9:57
  SPEAKER_01: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 10:06
  SPEAKER_01: From batch pipelines to training models and wiring up agents.
• 10:10
  SPEAKER_01: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 10:16
  SPEAKER_01: Check it out at astronomer.io.
• 10:20
  SPEAKER_00: And then tell me about how you're generating DAGs for these projects.
• 10:24
  SPEAKER_00: I guess, especially when you're working with a lot of different clients and you have you know different setups, everything.
• 10:31
  SPEAKER_00: I would imagine that manually writing the Python is not gonna work in every case.
• 10:36
  SPEAKER_02: So absolutely.
• 10:38
  SPEAKER_02: So for the most part, it's a design.
• 10:41
  SPEAKER_02: So we come up with the template firstly, and then uh we give out uh all the further variables that need to be uh modified.
• 10:49
  SPEAKER_02: Well, we have a uh you know tag generated engine.
• 10:52
  SPEAKER_02: Uh it's basically an utility which would uh generate the tags.
• 10:56
  SPEAKER_02: So we basically give out all the scripts uh that needs to be called, and the all the parameters are uh passed in the JSON file.
• 11:06
  SPEAKER_02: Basically, it picks out all the parameters uh uh you know from the JSON file and then attach them to the scripts.
• 11:13
  SPEAKER_02: So based on the operator that has been selected.
• 11:15
  SPEAKER_02: So it basically everything is automated, not just uh you know static DAG file generation, but also in even in the runtime, we are generating the DAGs.
• 11:25
  SPEAKER_02: So especially in cases where we need to do micro batch.
• 11:30
  SPEAKER_02: So uh in most of the applications, right?
• 11:33
  SPEAKER_02: Uh not all tables uh get subjected uh every single time.
• 11:38
  SPEAKER_02: So like let's take an example of an e-commerce application.
• 11:43
  SPEAKER_02: So uh whenever a financial transaction happens, um only the uh you know, more for the most part, only the accounting and the finance related uh tables will get modified.
• 11:53
  SPEAKER_02: The rest of the tables uh you know are for the most part, those are not modified.
• 11:58
  SPEAKER_02: So uh dynamically, we can will have to create a micro batch that then and there itself based on the process that had uh uh actually you know made the data appearance.
• 12:08
  SPEAKER_02: That that is like pretty easy with uh airflow.
• 12:11
  SPEAKER_02: We can in the runtime we can uh you know easily generate uh the uh tags uh with whatever micro batch in solution that we want to you know bring it up with.
• 12:23
  SPEAKER_02: Um apart from that, uh even the UI is uh uh pretty fast, uh you know the scheduler uh brings up the code uh in minutes time for the most part.
• 12:34
  SPEAKER_02: And uh, everything is uh easy, be it like static DAG generation and uh also the uh runtime-based uh dynamic uh DAG generation is also uh pretty easy with the air flow.
• 12:46
  SPEAKER_00: Well, that's a great example.
• 12:48
  SPEAKER_00: And do you find that setting up the DAG generation in this way helps the clients that you're working with manage Airflow on their own, like once your project has ended?
• 12:58
  SPEAKER_00: I assume they have to kind of continue working with this and managing it even if you're not there anymore.
• 13:04
  SPEAKER_02: Absolutely.
• 13:05
  SPEAKER_02: So here what we have done, we have made everything configurable.
• 13:08
  SPEAKER_02: So uh we have defined batches, and then uh in a config table, they they can set their own schedules.
• 13:15
  SPEAKER_02: Apart from that, like we have multiple triggering mechanisms.
• 13:19
  SPEAKER_02: The first one is like uh through static uh schedules, they've uh you know predefined in the config tables, and then we also have uh uh trigger-based approach.
• 13:30
  SPEAKER_02: We have both the pull and the push approach.
• 13:33
  SPEAKER_02: Pull as in a scheduled uh run push, as in when new data lands, uh the DAX would get triggered, a new micro batch would be uh created, and then uh in the runtime uh the DAG run happens.
• 13:48
  SPEAKER_02: So we're using both the approaches, pull and the push approach.
• 13:52
  SPEAKER_00: Okay, interesting.
• 13:54
  SPEAKER_00: And from a quality perspective, where do like testing and deployment or maybe CICD fit into this picture?
• 14:05
  SPEAKER_02: So we do have three environments.
• 14:07
  SPEAKER_02: We start with uh the Deva environment, all the development-related work uh goes over there, and then uh we have stage environment where we may have we also have the mocked up data where we do the actual uh uh data validations and everything as well with the mocker data.
• 14:24
  SPEAKER_02: Then finally, uh pre-pod and prod are the two production environments which we have.
• 14:30
  SPEAKER_02: So uh every time like it's the uh you know uh regular uh standard blue green development.
• 14:37
  SPEAKER_02: Slowly, once uh you know isolation is the DAG is finalized and stabilized, it would be uh you know uh part of the regular runs, and then the the previous version would be disabled.
• 14:49
  SPEAKER_02: So that this is the pattern that we follow.
• 14:51
  SPEAKER_00: Okay.
• 14:52
  SPEAKER_00: And do you typically use this pattern for all of your clients where you're implementing Airflow?
• 14:57
  SPEAKER_02: It changes from team to team, from project to project, even within uh uh my current client itself, uh, there are like uh multiple uh uh uh strategies that they follow.
• 15:07
  SPEAKER_02: So this is uh one key strategy that uh my current client follows.
• 15:12
  SPEAKER_00: Okay.
• 15:12
  SPEAKER_02: There are many other strategies.
• 15:14
  SPEAKER_00: Sure, that makes sense.
• 15:17
  SPEAKER_00: And then on a maybe slightly different topic, or maybe this is a different project than the one we've been speaking about.
• 15:23
  SPEAKER_00: But I understand that you have also used Airflow's Service Now integration to help with incident management.
• 15:29
  SPEAKER_00: Tell me about this use case.
• 15:32
  SPEAKER_02: Yeah, as we know it, uh uh Airflow has like uh uh plethora of connections readily available with the help of which uh you know many platforms can be easily integrated.
• 15:43
  SPEAKER_02: Not just service nov, uh even uh uh cloud integrations are pretty easy.
• 15:49
  SPEAKER_02: If you want to trigger a clue job or a uh ADF pipeline, it's uh pretty straightforward with Airflow.
• 15:56
  SPEAKER_02: So when it comes to service knob itself, I personally worked on uh uh an automatic uh presolution engine.
• 16:04
  SPEAKER_02: So as in when there is an uh uh you know failure in a dab, it immediately creates an incident, and then the immediate next step uh uh uh you know is triggered, which is uh to identify the context, to build the context of all the similar incidents and then uh actually do the initial debugging uh with the help of the LLM itself.
• 16:27
  SPEAKER_02: So instead of uh having to create a ticket and then assign it to an individual, and then uh only the debugging star process starts in the user scenarios, right?
• 16:37
  SPEAKER_02: Uh but here, like all the debugging is uh uh pretty I mean already done, and then the incident itself is uh updated with all the details.
• 16:46
  SPEAKER_02: So it now it has like pretty streamlined.
• 16:49
  SPEAKER_02: For the most part, we've also defined rules wherein even the fix is also applied uh then and there itself, but uh we should never alter the business uh logic, right?
• 17:00
  SPEAKER_02: The while we are doing the fixes or anything.
• 17:02
  SPEAKER_02: So we do have uh uh you know certain gates as well, uh, which would uh restrict uh the you know the kind of fixes uh that the auto-resolution engine applies.
• 17:11
  SPEAKER_02: So this is the pattern we are following.
• 17:13
  SPEAKER_00: Okay, interesting.
• 17:15
  SPEAKER_00: And now that you've implemented this, is there any future work or any kind of optimizations that you hope to make in the future?
• 17:24
  SPEAKER_02: So I would like to see one uh feature in Airflow.
• 17:28
  SPEAKER_02: Uh, I want it to be an AFS platform.
• 17:31
  SPEAKER_02: So, as it already has like all the integrations, right?
• 17:34
  SPEAKER_02: Uh so uh currently what is happening in the event of an error, we have to go to uh you know that particular platform where the error had occurred to you know dig deep into things.
• 17:44
  SPEAKER_02: Instead of that, as Airflow already has all the integrations, uh you know, they can uh you know get the logs from my different platforms and all the debugging can be done in one single platform.
• 17:57
  SPEAKER_02: So at my current workplace, we have actually built an utility, LLM added utility, which would uh integrate multiple platforms together to you know to help out in the debugging process in the event of an error.
• 18:10
  SPEAKER_02: But with Airflow, uh uh it can easily be done, right?
• 18:14
  SPEAKER_02: Because all the integrations are already there.
• 18:17
  SPEAKER_00: Interesting.
• 18:18
  SPEAKER_00: Okay.
• 18:18
  SPEAKER_00: And so is that part of the ServiceNow use case, or you're saying this would be something separate?
• 18:25
  SPEAKER_02: Uh this is something separate.
• 18:26
  SPEAKER_02: Uh I mean uh even the service now integration can ease utilized here.
• 18:30
  SPEAKER_02: Sure.
• 18:31
  SPEAKER_02: Yeah, because all the incident uh related data is in the service node database, right?
• 18:35
  SPEAKER_02: So we we are using uh also using the incidence data for further debugging.
• 18:41
  SPEAKER_02: But then I would like to see it uh well integrated in Airflow UA itself.
• 18:45
  SPEAKER_02: I want the debugging to happen in the UI itself.
• 18:48
  SPEAKER_00: Got it.
• 18:49
  SPEAKER_00: Okay, so if something is going wrong in your DAGs or you have an error being able to debug with other tools or okay.
• 18:56
  SPEAKER_00: Yeah, I can see where that would be very helpful, um, especially with the AI, you know, help and having AI help debug stuff, having that in the Airflow UI would be great.
• 19:07
  SPEAKER_02: Absolutely.
• 19:08
  SPEAKER_02: Because uh Airflow itself is an orchestrator, right?
• 19:11
  SPEAKER_02: That's where everything gets triggered.
• 19:13
  SPEAKER_02: That's where uh that's the central point.
• 19:15
  SPEAKER_02: So that's where AI has to sit in and uh with all the integrations in place, debugging can be uh made pretty easy.
• 19:23
  SPEAKER_00: Yeah, definitely.
• 19:25
  SPEAKER_00: I love that.
• 19:26
  SPEAKER_00: That's great.
• 19:27
  SPEAKER_00: Uh okay.
• 19:28
  SPEAKER_00: Well, Bonnie, thank you so much for going through that.
• 19:30
  SPEAKER_00: That's been it's been really great speaking with you.
• 19:32
  SPEAKER_00: Just to close us out, what is the best way for folks to get in touch with you?
• 19:36
  SPEAKER_02: LinkedIn is the best way, uh Kenton.
• 19:38
  SPEAKER_00: Okay, great.
• 19:39
  SPEAKER_00: We'll throw a link to your LinkedIn in the show notes.
• 19:42
  SPEAKER_00: Well, thank you so much for going through all of your use cases.
• 19:44
  SPEAKER_00: It's been really interesting hearing about them.
• 19:47
  SPEAKER_02: Thanks for having me, Kenton.
• 19:49
  SPEAKER_01: Thank you for listening to the Data Flowcast.
• 19:52
  SPEAKER_01: Check the show notes for links to everything mentioned today.
• 19:56
  SPEAKER_01: If you're enjoying the podcast, subscribe so you never miss an episode.
• 20:01
  SPEAKER_01: And leave us a five star review.