Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori

Transcript

From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_04: I want to connect to a database and then I want to get some results out of it.
• 0:04
  SPEAKER_04: So ideally, when you are doing a testing, you should not be doing it.
• 0:08
  SPEAKER_04: You need to mock it, mock as in similar instance, like create a function similar to that of what you are doing with the connect DB connection, small function, just uh get the result back.
• 0:17
  SPEAKER_04: Basically, you are returning the data that you are expected to return as you are doing only the unit test, and then just go to the next module.
• 0:25
  SPEAKER_04: Don't expect to connect to a DB and come back because it will delay what you are trying to test.
• 0:30
  SPEAKER_04: And when you are trying to test the actual connection, it's not needed.
• 0:39
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:50
  SPEAKER_00: Hey Airflow community, Airflow Theory is here and we are taking it on the road.
• 0:55
  SPEAKER_00: Join the Astronomer Road Toe for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 1:03
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:11
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:16
  SPEAKER_00: Register now.
• 1:17
  SPEAKER_00: The link is in the show notes.
• 1:20
  SPEAKER_02: Hello everyone.
• 1:21
  SPEAKER_02: Hello, everybody.
• 1:22
  SPEAKER_02: Welcome to the Data Flowcast.
• 1:24
  SPEAKER_02: I'm your host, Mark Lamerty, and today I'm very excited because I'm joined by Raviteja.
• 1:29
  SPEAKER_02: Super excited to have you in this episode, senior data engineer at Deloitte Digital.
• 1:35
  SPEAKER_02: How are you doing?
• 1:36
  SPEAKER_02: I'm doing good.
• 1:37
  SPEAKER_02: How are you?
• 1:38
  SPEAKER_02: Yeah, doing great.
• 1:39
  SPEAKER_02: It's nice to have you here because we are going to we are going to go through like very interesting questions about Airflow and also like your experience that you had with Airflow.
• 1:49
  SPEAKER_02: For today's podcast, Ravi Teja will share practical insights and hardworn lessons from implanting Airflow for orchestration at scale.
• 1:59
  SPEAKER_02: So I'm very excited.
• 2:00
  SPEAKER_02: And I think we can start by the very beginning, which is like how what was your uh initial motivation for adopting Airflow?
• 2:10
  SPEAKER_02: Like, how did you get started with Airflow?
• 2:13
  SPEAKER_02: Why did you choose Airflow?
• 2:14
  SPEAKER_04: Yeah.
• 2:15
  SPEAKER_04: So the first to answer your question, first I'll want to talk about the problem statement that we have faced uh before Airflow, before implementing Airflow.
• 2:24
  SPEAKER_04: So basically, when we started working on a couple of microservices that we had, so where we tried creating a customer data platform.
• 2:33
  SPEAKER_04: For that purpose, we have to do the identity resolution, we have to do very like rigorous cleansing jobs, standardize the data, standardized the addresses that we get, PIA data, everything.
• 2:43
  SPEAKER_04: We used to load a lot of data into the system, make sure that data is being processed with different layers.
• 2:50
  SPEAKER_04: And once the data is processed, we have to load it to either there is again a specific use case, either to load in a cloud or directly to the SQL Server or to a BigQuery or to Azure's Azure Synapse.
• 3:02
  SPEAKER_04: So the use case is very different.
• 3:04
  SPEAKER_04: The use case is cloud agnostic.
• 3:06
  SPEAKER_04: So initially, when we started building the product itself, we started working on the microservices.
• 3:12
  SPEAKER_04: And uh to integrate those microservices, initially we started working using the tools like ETL tools.
• 3:17
  SPEAKER_04: It might be the talent, it might be Metallian.
• 3:23
  SPEAKER_04: So we started working on those tools initially.
• 3:25
  SPEAKER_04: And for that purpose, also, there are like different rules, like whenever the file comes, you have to do some action.
• 3:31
  SPEAKER_04: So basically, all those things, trigger-based rules, like what should happen, how the orchestration should happen, all those things we used to manually do it in a pipe uh in an ETL process and then run this process in a VM or something like that.
• 3:44
  SPEAKER_04: So there it was again tedious because either we have to take a material uh uh VM separately, and which is very uh I can whether I can show it or not, it's costly.
• 3:55
  SPEAKER_04: And we are searching for an open source tool where we can write our own orchestration or things and perform some logic.
• 4:02
  SPEAKER_04: Okay.
• 4:02
  SPEAKER_04: And as I mentioned, it is very clear, uh data-driven solution that we that we are trying to build.
• 4:07
  SPEAKER_04: So initially, what we did was so I thought in 2019, back in 2019, I researched about the tool called AFflow.
• 4:15
  SPEAKER_04: Then I started doing small POCs, very small POCs, like how to trigger a file, how to initiate a Docker command, and then run that in the Docker pod.
• 4:25
  SPEAKER_04: And then how you have to initiate how to do if let's say if you have a very huge process to do that, you how to do it in the Kubernetes pod.
• 4:33
  SPEAKER_04: So I have very small use cases initially as part of POC, but just achieving that.
• 4:38
  SPEAKER_04: So once I understand how efficient it is, how useful it is, I started promoting, not promoting basically, I started talking to my managers, like leadership, saying, why can't we implement this?
• 4:50
  SPEAKER_04: As an open source tool, we can use it, and it's not that it's cost effective.
• 4:54
  SPEAKER_04: So I did have research and whatever orchestration that we used to do in Meteleon or Talent, because we serve different clients with different technologies.
• 5:04
  SPEAKER_04: So I tried creating a custom solution for entire orchestration that we did through in the airflow, and with some cost, we did the cost analysis as well.
• 5:14
  SPEAKER_04: So we found that this is cost effective, easy to use because it's Python-based, and you can create the dynamic DAX again.
• 5:21
  SPEAKER_04: When I say dynamic DAC, again, I don't want to go into details, but you have much flexibility here.
• 5:27
  SPEAKER_04: That means you can, and the our end goal was to be cloud agnostic.
• 5:32
  SPEAKER_04: So when I say cloud agnostic, again, it should like the data solution can be deployed as GCP, AWS, Azure.
• 5:38
  SPEAKER_04: Our orchestration should not be dependent on the cloud.
• 5:42
  SPEAKER_04: So we created the generic solution which will work on all the environments, irrespective of the cloud.
• 5:47
  SPEAKER_04: So that's the main motivation that we got to go through the aid flow.
• 5:50
  SPEAKER_04: And it happened back in 2019, and it's a very long time.
• 5:53
  SPEAKER_04: I started using it, implemented it.
• 5:55
  SPEAKER_02: Yeah.
• 5:56
  SPEAKER_02: Okay.
• 5:56
  SPEAKER_02: You you mentioned the two interesting things.
• 5:58
  SPEAKER_02: So you actually said that with Airflow you can generate dynamic DAGs.
• 6:02
  SPEAKER_02: Yeah.
• 6:02
  SPEAKER_02: You can dynamically generate tags.
• 6:05
  SPEAKER_02: Because it's not something that you can do with talent.
• 6:08
  SPEAKER_02: Yeah.
• 6:09
  SPEAKER_02: Okay.
• 6:10
  SPEAKER_04: Okay.
• 6:10
  SPEAKER_02: I have no idea because I've never touched the talent.
• 6:14
  SPEAKER_02: So that's why.
• 6:15
  SPEAKER_04: Yeah, it's an ETL tool again.
• 6:16
  SPEAKER_04: So it will be like um step-by-step procedure.
• 6:19
  SPEAKER_04: Okay, you have some XYZ thing happened.
• 6:22
  SPEAKER_04: How the data should be cleansed using some component, then what let's say if you are to use a doctor uh pod and initiate something, then you have to like basically drag and drop tool, and everything, for example, even I'll give you a very small example.
• 6:36
  SPEAKER_04: Even to uh change the schema, you have to go to the talent job, make the changes, again redeploy, which is hectic work.
• 6:44
  SPEAKER_04: So we want to build a solution which is not dependent on the schema.
• 6:47
  SPEAKER_04: So we want everything to be configured and with Airflow variables and also the environment that we have, the entire port setup, the we make we made use of all the configurations that are provided by Airflow as much as possible.
• 7:00
  SPEAKER_04: That we build an entire platform on top of that currently, it's going good.
• 7:06
  SPEAKER_02: Okay, so would you say, is it correct to say that you find you you found more flexibility with Airflow than with talent?
• 7:15
  SPEAKER_02: Like you you were able to do pretty much whatever you wanted with.
• 7:19
  SPEAKER_02: Yeah, it's interesting because like I guess like those tools like talent and others, I don't have alien, yeah, exactly.
• 7:26
  SPEAKER_02: And and even others, they seem to be like nice because they seem to be easy to use, maybe at the beginning, right?
• 7:34
  SPEAKER_02: Like it's a drag and drop stuff.
• 7:36
  SPEAKER_02: Drag and drop, yes.
• 7:38
  SPEAKER_02: But then at some point, when you want to do like more advanced things, you realize that you because of that, because you have this layer on top of the of all the technical things, then you cannot really go as far as you want.
• 7:51
  SPEAKER_02: Like you don't have as much flexibility as you need.
• 7:54
  SPEAKER_02: And and actually, I think many data engineers like Airflow because of that, because of all of the flexibility.
• 7:59
  SPEAKER_04: One more point is basically like a couple of tools.
• 8:02
  SPEAKER_04: Let's say talent might support Snowflake, might not support Snowflake.
• 8:05
  SPEAKER_04: I'm not uh like saying talent won't support, but the components come with uh enhancements.
• 8:10
  SPEAKER_04: Some at some point of time you need uh Snowflake components, it might not be available.
• 8:14
  SPEAKER_04: At some point of time, you might need BigQuery component, it might not be available.
• 8:17
  SPEAKER_04: So it will not see when uh the best thing that we can find with Airflow is like you can write your own components.
• 8:24
  SPEAKER_04: You can write your own, define your own task.
• 8:26
  SPEAKER_04: It's just a Python script.
• 8:27
  SPEAKER_04: At the end of the day, write what you want, it will be achieved.
• 8:31
  SPEAKER_04: That's a very good flexibility because you will not be dependent on a tool saying I want to do this thing, whether the component is available or not.
• 8:40
  SPEAKER_04: Yeah, with respect to Airflow, it's a Python code, write it, what you want to.
• 8:44
  SPEAKER_02: Yeah, yeah, yeah, yeah.
• 8:45
  SPEAKER_02: Yeah, actually, that's also why I adopted Airflow a long time ago.
• 8:49
  SPEAKER_02: And also you said something interesting because maybe some people will disagree with that.
• 8:53
  SPEAKER_02: You said that Airflow was easy to use.
• 8:56
  SPEAKER_02: It was easy to use.
• 8:57
  SPEAKER_04: Once you understand it.
• 8:58
  SPEAKER_04: Once you understand it.
• 9:00
  SPEAKER_04: So there is like a hard hill if you want to climb it.
• 9:03
  SPEAKER_04: If you will to climb it, once you climb up the hill, then there you know no one would stop her.
• 9:09
  SPEAKER_04: It will be straightforward.
• 9:10
  SPEAKER_04: Once you understand how it works, it is easy.
• 9:13
  SPEAKER_04: But for a new beginner, you have to hold.
• 9:17
  SPEAKER_04: Like example, if you see any watch, if you want to watch any series or something like that, there will be a threshold till that point of time it would be boring, or it will be difficult to go through the series, and then once you reach it, it will be easy for you to navigate it.
• 9:29
  SPEAKER_02: Any particular, like any specific concepts, maybe that you felt at that time was a bit not hard.
• 9:37
  SPEAKER_02: Yeah, not no, I wouldn't say not good, but hard to understand, maybe at first as a beginner.
• 9:41
  SPEAKER_04: Uh yeah.
• 9:42
  SPEAKER_04: So as a beginner, like back in 2019, orchestration is good, but in the back end, what is happening?
• 9:49
  SPEAKER_04: It's like I would say a bit complex.
• 9:51
  SPEAKER_04: Example, how the scheduler is making the work of pot spin-up.
• 9:55
  SPEAKER_04: Because at that point of time, I had two, four, three years of experience.
• 9:59
  SPEAKER_04: For me, it's too much to consume, like too much of information to consume.
• 10:03
  SPEAKER_04: So it took time for me to understand how the scheduler is working, how the workers are working, how you are using the metadata in the background, and then passing the information to one component to the other component.
• 10:15
  SPEAKER_04: So all that information, it will come with experience.
• 10:18
  SPEAKER_04: As a in the beginning, you might not feel comfortable with it, but once you get enough information about it, you'll feel good about it.
• 10:27
  SPEAKER_02: Yeah, yeah, true.
• 10:28
  SPEAKER_02: I guess one yeah, and also you mentioned that it was in 2019.
• 10:32
  SPEAKER_02: So I think it's important for listeners that in 20 in 2019 you didn't have that much information about Airflow.
• 10:39
  SPEAKER_02: Yeah, it was it was pretty much the beginning, honestly.
• 10:42
  SPEAKER_02: Yeah.
• 10:42
  SPEAKER_02: All right.
• 10:43
  SPEAKER_02: So I guess did you actually did you evaluate any other orchestrators or did you go with Airflow right away?
• 10:49
  SPEAKER_04: Evaluate, no.
• 10:51
  SPEAKER_04: At that point of time, I was searching for an open source store specifically.
• 10:55
  SPEAKER_04: I heard there was one more orchestration called Qflow.
• 10:58
  SPEAKER_04: I don't know why.
• 10:59
  SPEAKER_04: Like I and I want to do a data engineering aspect, not a data center aspect where you want to learn learn machine learning, everything.
• 11:06
  SPEAKER_04: Very basic research I did, and I came to know that this is more comfortable, not comfortable for my use case.
• 11:11
  SPEAKER_04: This is the what I want to use, not the people.
• 11:15
  SPEAKER_04: I did that research, but uh I went through it.
• 11:20
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:29
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 11:41
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 11:49
  SPEAKER_02: With all the experience that you have, obviously, with Airflow, I imagine that you've built a couple of DAGs since 2019.
• 11:56
  SPEAKER_02: Is there like any DAGs maybe that you remember where you where you had some really interesting challenges that you had to solve?
• 12:05
  SPEAKER_02: Maybe?
• 12:06
  SPEAKER_02: Can you tell me more about that?
• 12:07
  SPEAKER_04: So again, I like for a new newcomer to Airflow, it might not be useful, but people who are working in the Airflow, they might do late.
• 12:14
  SPEAKER_04: So at some point of time, whenever you are choosing the executor, you have a limitation.
• 12:18
  SPEAKER_04: Either you have to go to Kubernetes executor or salary executor.
• 12:22
  SPEAKER_04: In a specific use case, you can use Kubernetes salary executor.
• 12:25
  SPEAKER_04: So it's one of the updates that we got.
• 12:27
  SPEAKER_04: So, for example, when I'm writing a couple of steps in my DAC saying I have 10 tasks and I want to choose which executor I want to use.
• 12:35
  SPEAKER_04: If I have a very bulky job where I want to run in the separate port, in that instance, I can go for Kubernetes executor.
• 12:42
  SPEAKER_04: If I all my tasks have a specific use case to use Kubernetes pod, I'll choose Kubernetes executor.
• 12:48
  SPEAKER_04: If those are very minimal tasks, like very basic things, proceed a file, do some operation, cleanse it, and then send it back.
• 12:54
  SPEAKER_04: For that, I can go for uh cellular executor.
• 12:57
  SPEAKER_04: For us, it was both.
• 13:00
  SPEAKER_04: And the transition between the Kubernetes executor and celery executor is a bit tricky.
• 13:05
  SPEAKER_04: It always not efficient.
• 13:07
  SPEAKER_04: I feel it's not efficient because we used to face a lot of problems.
• 13:11
  SPEAKER_04: Example, the info, the information that was passed from Kubernetes executor, the pod or the task from the next executor, there is some delay.
• 13:18
  SPEAKER_04: Or else something in the background in the DB, something goes wrong, uh gets wrong, and that information is not passed to the next executor and it is delayed.
• 13:25
  SPEAKER_04: If pod is struck, the next information is got not getting passed to the central executor and there is a delay.
• 13:31
  SPEAKER_04: So this transition we phased a number of times.
• 13:34
  SPEAKER_04: At that point of time, we didn't have an answer.
• 13:36
  SPEAKER_04: I recently came to know that you can choose a specific executor for a specific task, which was good, but earlier we faced issues with that.
• 13:44
  SPEAKER_02: Okay.
• 13:45
  unknown: Yeah.
• 13:45
  SPEAKER_02: Yeah, I think I think whenever honestly, I think whenever you have to deal with Kubernetes, it's always gonna be uh complex.
• 13:52
  SPEAKER_02: But yeah, that's actually uh interesting what you said about executors and choosing the right one.
• 13:56
  SPEAKER_02: There is the there is this new feature in I think it came with 2.10 or 2.9 maybe with the hybrid executors.
• 14:04
  SPEAKER_02: Yeah, so maybe people don't know about that new feature, but basically now you can really at the task level define what which executor you want to do.
• 14:13
  SPEAKER_02: Executor you want, yeah.
• 14:14
  SPEAKER_02: Exactly.
• 14:14
  SPEAKER_02: So, for example, you can just decide to go with the salary executor or the Kubernetes executor.
• 14:19
  SPEAKER_02: Obviously, that implies that you have the infrastructure behind the scene, right?
• 14:23
  SPEAKER_02: And you cannot use the Kubernetes executor if you're not pointing it to our Kubernetes.
• 14:26
  SPEAKER_02: But this is a really cool feature.
• 14:27
  SPEAKER_02: So for those who are listening right now and want to like actually don't want to choose one or the others, then you can go with the hybrid executors, like in the I think in the configuration file you have to specify which executors you want to use, and then at the task level, you can just specify the executor.
• 14:42
  SPEAKER_02: So this is really cool.
• 14:44
  SPEAKER_02: Great.
• 14:44
  SPEAKER_02: I guess like how do you approach like testing and monitoring with Airflow?
• 14:48
  SPEAKER_02: Because this is actually, I mean, for me, like testing and monitoring is almost like cybersecurity, where we all agree to that, but it's not the most, it's not the sexiest thing, right?
• 14:59
  SPEAKER_02: Yes.
• 14:59
  SPEAKER_02: So can you tell us more about maybe how do you like let's start with how do you monitor your pipelines?
• 15:05
  SPEAKER_02: And it can be as simple as just like looking at DUI, right?
• 15:08
  SPEAKER_02: But if you have anything else, and also like how do you test your DAGs if you do it?
• 15:12
  SPEAKER_02: Because again, this is actually pretty pretty complex as workflows and interact with many things.
• 15:18
  SPEAKER_04: When you say complex, I'll tell you the reason why.
• 15:20
  SPEAKER_04: So basically, you need to understand the mocking mechanism.
• 15:24
  SPEAKER_04: Again, it's a new thing to example for a new beginner, uh beginner, it was a new thing to try implementing it.
• 15:30
  SPEAKER_04: And when you are dealing with multiple tasks, let's say connected to different components, DB, Kubernetes port, and everything, to mock those many components would be tricky.
• 15:42
  SPEAKER_04: Can you define mock for those who don't know?
• 15:45
  SPEAKER_04: So basically, mocking is like example, I want to connect to a database and then I want to get some results out of it.
• 15:51
  SPEAKER_04: So ideally, when you are doing a testing, you should not be doing it.
• 15:55
  SPEAKER_04: You need to mock it, mock as in similar instantly, like create a function similar to that of what you are doing with the DB connection, small function, just uh get the result back.
• 16:04
  SPEAKER_04: Basically, you are directly returning the data that you are expected to return as you are doing only the unit test, and then just go to the next module.
• 16:12
  SPEAKER_04: Don't expect to connect to a DB and come back because it will delay what you are trying to test.
• 16:17
  SPEAKER_04: And when you are trying to test the actual connection, it's not needed.
• 16:21
  SPEAKER_04: If you are just uh trying to functionality, just do that, not test the entire connection and everything in the back end.
• 16:28
  SPEAKER_04: So when we are doing it, as you said, it was complex, and uh, I'll be very frank, we haven't implemented 80, not even 80 percent of the test cases for airflow.
• 16:40
  SPEAKER_04: I'm I want to be very frank, I don't want to be exaggerating things.
• 16:43
  SPEAKER_04: It's like it's if it's working, it's good.
• 16:46
  SPEAKER_04: If it's fails, we'll fix it.
• 16:51
  SPEAKER_04: Yeah, yeah, yeah.
• 16:52
  SPEAKER_04: I think like you know, yeah, go ahead.
• 16:54
  SPEAKER_04: Yeah, sorry, and data engineers, right?
• 16:56
  SPEAKER_04: So you cannot expect everything to work at any point of time.
• 17:00
  SPEAKER_04: Yeah, you will get issues, you will get you need to fix things because when you are dealing with data, you have to understand you cannot cover all the scenarios.
• 17:09
  SPEAKER_04: So go with the flow where identify an issue, try to cover most of the scenarios.
• 17:15
  SPEAKER_04: There will always be edu cases.
• 17:17
  SPEAKER_04: When you come up with an edge case, try to fix them.
• 17:20
  SPEAKER_04: Don't revolve around, don't try to solve all the problems at once.
• 17:24
  SPEAKER_04: When you face it, solve it.
• 17:26
  SPEAKER_02: Yeah, no, absolutely.
• 17:28
  SPEAKER_02: And I think it's important to be honest.
• 17:30
  SPEAKER_02: And you are definitely not the I think like many data engineers are in the same uh case as you.
• 17:36
  SPEAKER_02: Obviously, it's pretty easy to test Python code, but as soon as you're gonna have bashed scripts to run, requests to run, all of that stuff, like interacting with Snowflake and so on, and all a workflow can be pretty hard, pretty complex again.
• 17:50
  SPEAKER_02: So if you have if you run like Python functions and if you rely on the Python operator a lot, then it might be easier.
• 17:56
  SPEAKER_02: You can run unitest like you will do with Python code.
• 17:59
  SPEAKER_02: But as soon as you're gonna have to make connections and so on to your point, you will need to mock that.
• 18:04
  SPEAKER_02: And if you want to know if your tasks work together, you will even have to run your DAG at some point, right?
• 18:11
  SPEAKER_02: So it's yeah, it's definitely tricky, but but thank you for like it is what it is.
• 18:16
  SPEAKER_02: We do what we can do.
• 18:17
  SPEAKER_02: Yes, I guess on the monitoring part, like do you have anything in place in addition to the Airflow UI?
• 18:24
  SPEAKER_02: Do you rely on any other systems like Grafana or things like that or not?
• 18:28
  SPEAKER_02: Yeah.
• 18:29
  SPEAKER_04: So for standard output, basically we use uh low-key as part of we use Prometheus for monitoring as I speak, right?
• 18:37
  SPEAKER_04: We were talking, we are dealing about using the Kubernetes pods.
• 18:41
  SPEAKER_04: So we need to monitor that pod memory pod CPU.
• 18:44
  SPEAKER_04: So we have even implemented Prometheus in the back end, and also we use Grafana also, Grafana dashboards to check everything, like what are the logs that need to be checked, what are the resources utilized by Airflow.
• 18:56
  SPEAKER_04: So all those things we have that in the back end deployments.
• 18:59
  SPEAKER_04: And uh one more thing is basically for monitoring the data flow, what we used to do is we have a system integrated within the DAGs or tasks where wherever we process something, we used to put all the stats of the file, load it as a Postgres DB within the Aflow.
• 19:17
  SPEAKER_04: We have created one more instance of the DB, and then we used to load all that manifest data into the table so that every run, whatever you are doing, how much data you have processing, what are the jobs that you are performing, how much data you are loading, all those stats we are making sure we are just placing it one place, and then we are showing that information on graphana dashboard.
• 19:38
  SPEAKER_02: That's pretty cool.
• 19:39
  SPEAKER_02: That's pretty cool.
• 19:40
  SPEAKER_02: Okay, I guess like running Airflow at scale, and you already mentioned that a little bit because most of the time the issues that you're gonna get are on the infrastructure side.
• 19:49
  SPEAKER_02: But yes, running airflow at scale, maybe did you what has been your experience so far with that?
• 19:54
  SPEAKER_02: Would you say I I don't want to say would you say it's easy because it's not, but like maybe what can you tell us more about what kind of issues did you get did you did you have when you started to run Airflow at scale?
• 20:08
  SPEAKER_02: Maybe like what kind of challenges I would say.
• 20:10
  SPEAKER_02: Not issues, but challenges.
• 20:12
  SPEAKER_04: Uh correct uh identifying the correct configurations of your scheduler and worker ports.
• 20:17
  SPEAKER_04: Again, this might be technical again, whether to have one scheduler or two schedulers, again, it matters.
• 20:24
  SPEAKER_04: So when you want to scale up the data processing, let's say you are using 10 DAX, 20 DAX, it's good basic resources that you are supposed to give it to the scheduler and it will handle it.
• 20:33
  SPEAKER_04: But let's say you have 100 DAX.
• 20:35
  SPEAKER_04: Since situations change, things change, then you have to think about having the replication sets on scheduler, repetition sets on the worker port, how the information is passed from one scheduler to other scheduler.
• 20:46
  SPEAKER_04: And if you are using it in a, let's say you have cloud native, like a cloud composer, uh there the things will change because everything is handled by GCP.
• 20:54
  SPEAKER_04: If you are using a specific tool with respect to AWS, things will change.
• 20:58
  SPEAKER_04: But if you want to create your own instance of Airflow and then run all the processes uh in Kubernetes, that's I would say it's hard.
• 21:08
  SPEAKER_04: It's hard to manage it because see, it's like you are taking the Airflow, which is an open source tool, and you are building a solution out of that.
• 21:15
  SPEAKER_04: So you have to manage what amount of resources that it needs to keep.
• 21:20
  SPEAKER_02: Yes, yes, it's free.
• 21:23
  SPEAKER_02: Yeah, yeah, yeah.
• 21:24
  SPEAKER_02: Actually, you you you made a very good point.
• 21:26
  SPEAKER_02: I think people need to realize like there is a big difference between like Airflow as an orchestrator and having to set up the infrastructure to run Airflow.
• 21:38
  SPEAKER_02: Because running Airflow on an EC2 instance with just the basics, it's okay.
• 21:42
  SPEAKER_02: Yeah, but running Airflow at scales, at scale, sorry, by your own is definitely something else.
• 21:48
  SPEAKER_02: Something else.
• 21:48
  SPEAKER_02: When you see like some competitors saying that, hey, look how it is easy to run uh to run our own solution at scale, blah blah blah.
• 21:57
  SPEAKER_02: Most of the time, what they don't tell you is yes.
• 22:00
  SPEAKER_02: On the cloud, obviously it works because you don't have to manage the infrastructure, right?
• 22:04
  SPEAKER_02: Like you don't have to manage the infrastructure.
• 22:06
  SPEAKER_02: But as soon as you have to manage the infrastructure, regardless of the tool you're gonna use, you will see the same challenges that that you will have for any other tools because you are trying to run that tool at scale.
• 22:19
  SPEAKER_02: And uh when you have like obviously, like when you want to run things at scale, you have like some scalability challenges.
• 22:25
  SPEAKER_04: There is always a thin layer.
• 22:27
  SPEAKER_04: Yes, it will work, yes, you are managing it good, but at what cost?
• 22:31
  SPEAKER_02: Yeah, exactly.
• 22:32
  SPEAKER_04: Cost you have to consider.
• 22:34
  SPEAKER_04: Example, if you are giving how much resources you do, you do the testing that you want, decide the resources, and it will use it, utilize those resources, and the amount of cost that it generates it is like billing is you have the control over it.
• 22:47
  SPEAKER_04: If let's say some extra XYZ solution is there and they are saying we can do wow factors, you also should also consider about its fact, its cost.
• 22:56
  SPEAKER_04: At what cost you are getting that solution, and at what cost you are making sure that you are running your own DAC.
• 23:01
  SPEAKER_04: So, cost factor is always an important thing where people will forget about talking this terminology, but even cost should be considered when you are serving the clients, you should not cost them much.
• 23:12
  SPEAKER_04: So less cost, more efficient, or more cost, okay, or more efficient.
• 23:17
  SPEAKER_04: It's it's always like a thin gap between what you want to choose.
• 23:20
  SPEAKER_02: Yeah, there is a trailer for sure.
• 23:22
  SPEAKER_02: Yeah, I agree with you.
• 23:23
  SPEAKER_02: So now looking at Airflow evolution over the years, because Airflow hopefully has evolved a lot since 2019.
• 23:31
  SPEAKER_02: Um, do you have any features or improvements in mind that for you has had the biggest impact on your work?
• 23:40
  SPEAKER_02: Is there anything that like when you got that feature, you were like, oh damn.
• 23:44
  SPEAKER_04: Finally, with true, yeah, with this latest release.
• 23:47
  SPEAKER_04: I haven't used that yet because we are using the old version where we have we have already integrated everything and everything is running smooth, so we don't want to directly migrate to newer version.
• 23:55
  SPEAKER_04: But the feature that I have sought, like I have saw in the latest book, I heard that whatever we have discussed earlier, where you can choose what executor you want.
• 24:05
  SPEAKER_04: That was a war factor for me.
• 24:07
  SPEAKER_04: Because I struggled a lot with that.
• 24:09
  SPEAKER_04: So I can relate much to that.
• 24:11
  SPEAKER_04: That's a what factor.
• 24:12
  SPEAKER_04: One good thing I can tell you is that dark mode, waiting for that.
• 24:15
  SPEAKER_04: I don't know whether it is valid or not.
• 24:17
  SPEAKER_04: But yeah, dark mode, because I I used to stare at the screen and number of things.
• 24:22
  SPEAKER_04: I don't want my I don't want it to be strained because of that.
• 24:25
  SPEAKER_04: So yeah, dark mode will love that.
• 24:27
  SPEAKER_04: Yeah, those are the two things I like about it, and also the UI.
• 24:31
  SPEAKER_04: When I say UI, the resources utilization and the dog, like there is one more page, right?
• 24:35
  SPEAKER_04: Apart from the dash, true.
• 24:36
  SPEAKER_04: There is one more page where you can see the consumption, resource consumption and everything.
• 24:41
  SPEAKER_04: That is also pure, like that is also good, rather than navigating it to some other to how much resources that Kubernetes port is doing, it's good to have it there.
• 24:51
  SPEAKER_02: Yeah, true.
• 24:52
  SPEAKER_02: Yeah, I I definitely agree with you.
• 24:53
  SPEAKER_02: Like the dark mode and the hybrid executors are definitely cool features.
• 24:57
  SPEAKER_02: Actually, I don't know about you for that one, but I I remember that when dynamic task mapping got released, I was so excited because finally you were able to truly like dynamically generate tasks based on an input that you don't know in advance.
• 25:13
  SPEAKER_02: If you get if you get five files on Monday, then you generate five tasks.
• 25:17
  SPEAKER_02: If you get six on Tuesday, you can generate six tasks.
• 25:20
  SPEAKER_02: So this like for me, that was really uh really nice feature that uh we got in air clue.
• 25:25
  SPEAKER_04: I don't really already doing a POC on that uh recently because we have that specific use case where we want to run different things for different uh set of files based on the schema and everything.
• 25:36
  SPEAKER_04: So rather than defining saying, okay, do this job, then like rather than doing it sequentially, we want to do it dynamically.
• 25:44
  SPEAKER_04: Once the file is there, decide see the schema and dynamically create its corresponding tables or something like that.
• 25:51
  SPEAKER_04: Yeah, do I'm still doing the POC, but yeah, it's a good feature to have.
• 25:55
  SPEAKER_02: Yeah, definitely.
• 25:57
  SPEAKER_02: Ravitra, uh it was awesome to have you for this episode.
• 26:00
  SPEAKER_02: We are almost at the end, but before leaving, thank you so much for joining.
• 26:06
  SPEAKER_02: And what is the best way for for us to for sorry to get in touch with you?
• 26:11
  SPEAKER_04: Yeah, LinkedIn is the best way.
• 26:13
  SPEAKER_04: It's Ravitja 0096, my username.
• 26:15
  SPEAKER_04: So feel free to talk, ask me questions in LinkedIn.
• 26:18
  SPEAKER_04: I even like a lot of people try to reach out, so I was like selected as an Airflow champion where the astronomer like did that.
• 26:27
  SPEAKER_04: So I was selected as an Airflow champion.
• 26:28
  SPEAKER_04: I did have written multiple uh blogs about it using Airflow, simplifying the architecture, my airflow journey.
• 26:35
  SPEAKER_04: So all those things, and I used to reach out to multiple, like people used to reach out to me for a specific use case.
• 26:41
  SPEAKER_04: One of a manager, like one of a manager from a specific company, I don't want to name the company, reached out to me saying, My team is facing issues with this airflow, and is how it is quantified.
• 26:52
  SPEAKER_04: Could you please jump on a call and try to help us?
• 26:55
  SPEAKER_04: Actually, I was not working for the organization, but I did, and uh I kind of debug, like sat with them for one hour and something like that.
• 27:02
  SPEAKER_04: Try to help them.
• 27:03
  SPEAKER_04: It was good.
• 27:04
  SPEAKER_04: Like people started reaching out, and you can also reach out to me on LinkedIn.
• 27:08
  SPEAKER_02: Here we go.
• 27:09
  SPEAKER_02: Again, thank you so much, uh, Ravitaja.
• 27:11
  SPEAKER_02: It was a pleasure to have you on this episode.
• 27:13
  SPEAKER_02: I'm sure that the listeners will enjoy it.
• 27:15
  SPEAKER_02: And uh thank you so much.
• 27:16
  SPEAKER_02: Take care and thank you so much.
• 27:18
  SPEAKER_02: Yeah, have a great day.
• 27:19
  SPEAKER_02: Bye-bye, guys.
• 27:20
  SPEAKER_02: Bye-bye.
• 27:20
  SPEAKER_02: Bye-bye.
• 27:23
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 27:30
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 27:33
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.