Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Why Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik

Transcript

Why Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We are talking about billions and billions of rows getting processed in like minutes.
• 0:04
  SPEAKER_02: And that's where I am currently now, okay, with that final one where in like it's an intersection of all the modern data stack and also scalable environment and then the resilient one and also the artificial intelligence, which is getting in the data platform integrated all together.
• 0:19
  SPEAKER_02: So that's my 20 years of journey.
• 0:21
  SPEAKER_02: That's the data, how it has evolved so far.
• 0:23
  SPEAKER_02: And if somebody wants to be in data engineering, it's not a simple thing.
• 0:27
  SPEAKER_02: You really have to understand how the data flows, and there are different kinds of data.
• 0:58
  SPEAKER_01: And I would like to start with a quick question for you, Arun.
• 1:01
  SPEAKER_01: How are you doing today?
• 1:03
  SPEAKER_02: I'm good.
• 1:04
  SPEAKER_02: Thanks, Mark.
• 1:05
  SPEAKER_02: Thanks for having me in this conversation.
• 1:07
  SPEAKER_02: Thank you.
• 1:08
  SPEAKER_01: A pleasure.
• 1:09
  SPEAKER_01: So today we are talking with Arun Kartik, as you know.
• 1:12
  SPEAKER_01: He has around two decades of experience in data space, which is a lot of experience.
• 1:17
  SPEAKER_01: And he is a seasoned engineer, has worked from primitive databases to the modern distributed data process tools.
• 1:24
  SPEAKER_01: So a lot of experience that Arun is going to share with us, and I'm sure you will learn a lot, especially about data tools and Airflow.
• 1:33
  SPEAKER_01: So without further ado, let's begin.
• 1:36
  SPEAKER_01: So, Arun, I have a one question for you.
• 1:40
  SPEAKER_01: Can you tell us about your background and how you got into data engineering?
• 1:44
  SPEAKER_01: Because I know that it's actually a very common question, especially for people that want to jump into the data engineering world.
• 1:51
  SPEAKER_01: So, what has been your journey over the past 20 years in this space and how everything started?
• 1:56
  SPEAKER_02: Thanks for the question because I get this question a lot.
• 1:59
  SPEAKER_02: Because people really with the booming data engineering space, data science, and everything coming in.
• 2:05
  SPEAKER_02: And when somebody gets to know, like I'm a data engineer guy.
• 2:08
  SPEAKER_02: So this question comes to me a lot.
• 2:11
  SPEAKER_02: Okay.
• 2:11
  SPEAKER_02: To be honest, it's really a very short span of time to summarize my 20 years of experience.
• 2:17
  SPEAKER_02: But still, I'll try to justify that.
• 2:19
  SPEAKER_02: Okay.
• 2:20
  SPEAKER_02: In the time when I started my career, which is 20 years back, somewhere around 2006, we had only a relational database.
• 2:27
  SPEAKER_02: We never had something called big data.
• 2:29
  SPEAKER_02: That's the time all the Hadoop and everything was coming in.
• 2:32
  SPEAKER_02: It's not like you had this and you big data started, and no.
• 2:36
  SPEAKER_02: So you had everything in a relational tables, entities, tuples.
• 2:39
  SPEAKER_02: That's what rows and columns which we are working on.
• 2:42
  SPEAKER_02: And tools like databases like Oracle, Cybase, and MySQL, they were the backbone for your entire enterprise.
• 2:48
  SPEAKER_02: And you run your enterprise completely with tables that being built on top of it.
• 2:53
  SPEAKER_02: So that's when these ETL tools were emerging, like Informatica.
• 2:57
  SPEAKER_02: I wouldn't say emerging, they were trying to put it in a relational world and then use that and then see how fast we can process.
• 3:05
  SPEAKER_02: And batch processing is the only way of getting the data.
• 3:08
  SPEAKER_02: And then we used all these files and tools to manipulate the records and then pull them into our RDBMS and then get it going.
• 3:16
  SPEAKER_02: And they were also seamlessly working on versions to integrate the data and then get them into automated reporting tools kind of system.
• 3:25
  SPEAKER_02: So that being said, this is something in terms like my first five years of my career, right?
• 3:30
  SPEAKER_02: I was able to get it.
• 3:31
  SPEAKER_02: And then that's when in 2010s to early 2010s, I would say like we were getting all these cloud and then AWS and all the high-scaling performance-oriented system, and then scalabilities, everything were coming in.
• 3:45
  SPEAKER_02: And people are moving from your on-prem to your cloud and then trying to see how good it is, how bad it is, and trying to address it, right?
• 3:52
  SPEAKER_02: With the ability to scale up and scale down your infrastructure.
• 3:55
  SPEAKER_02: So somewhere in the mid of 2015s, wherein like Spark came in with the distributed high-performance solutions wherein they were able to process it.
• 4:05
  SPEAKER_02: And the real modern architecture of data engineering started.
• 4:09
  SPEAKER_02: That's when the real role and the name of data engineering actually coined.
• 4:13
  SPEAKER_02: And you people started hiring data engineers, not just writing SQLs and store procedures.
• 4:18
  SPEAKER_02: And that being said, that's the era.
• 4:20
  SPEAKER_02: Like I would say, like we were all transitioning into the big data environment, and then we are having Skava, Spark, and everything coming into place.
• 4:27
  SPEAKER_02: So early 20s till till date, which is the mid-20s.
• 4:30
  SPEAKER_02: So that's the era of all these new modern tools, right?
• 4:34
  SPEAKER_02: So we've moved from the big data batch tables, batches, and then now we are in the real-time data monitoring and then real-time solutions that we look at in terms of or like near real-time, we call it.
• 4:45
  SPEAKER_02: So badge processing is getting managed, I would say.
• 4:48
  SPEAKER_02: So people are looking at data in terms at least like a one to two minutes delay rather than looking at it in terms of two hours, three hours delay, or like something like that.
• 4:56
  SPEAKER_02: Right.
• 4:57
  SPEAKER_02: So that being said, the observability has increased, and you are having the ability to see the lineage and where the data is flowing from and how the data, and we are talking about billions and billions of rows getting processed in like minutes.
• 5:10
  SPEAKER_02: Okay.
• 5:11
  SPEAKER_02: And that's where I am currently now, okay, with that final one wherein like it's an intersection of all the modern data stack and also scalable environment, and then the resilient one, and also the artificial intelligence, which is getting in the data platform, integrated all together.
• 5:28
  SPEAKER_02: So that's my 20 years of journey.
• 5:30
  SPEAKER_02: Okay, that's the data, how it has evolved so far.
• 5:33
  SPEAKER_02: And if somebody wants to be in data engineering, it's not a simple thing.
• 5:36
  SPEAKER_02: You really have to understand how the data flows, and there are different kinds of data and how you get into the data engineering, right?
• 5:42
  SPEAKER_02: It's an evolution on its own.
• 5:44
  SPEAKER_02: So maybe it's not a simple answer, I would say.
• 5:47
  SPEAKER_01: No, no, no, no.
• 5:48
  SPEAKER_01: It is.
• 5:48
  SPEAKER_01: And you know, like I didn't have the chance to start my data engineering career in 2006 or 2008.
• 5:54
  SPEAKER_01: I mean, I definitely remember that at least before Airflow, for me it was extremely painful.
• 5:58
  SPEAKER_01: Not saying that Airflow solved all of my issues, definitely not.
• 6:01
  SPEAKER_01: But even like before Spark, you had Hadoop and all of those tools.
• 6:05
  SPEAKER_01: And while they were working, right, it was actually uh not that convenient, I would say, not user-friendly at least to use those tools.
• 6:12
  SPEAKER_01: And I'm glad that right now we are moving towards tools that are easier to use, but also unfortunately, we have an explosion of different tools, right?
• 6:21
  SPEAKER_01: Like they are tools today in this uh modern data stacks.
• 6:24
  SPEAKER_01: But I but it's really nice that you went through all of those steps.
• 6:27
  SPEAKER_01: You are talking about the evolution of data tools and how it has matured.
• 6:32
  SPEAKER_01: How did you make sure to orchestrate your daily data needs and deliver the data to the users on time?
• 6:37
  SPEAKER_01: You know, like what was your uh go-to tool?
• 6:40
  SPEAKER_01: Because for me, I mean, before Airflow, I think we were using Jenkins, which was not meant for that.
• 6:49
  SPEAKER_01: It's not meant at all.
• 6:50
  SPEAKER_01: So we were using Jenkins.
• 6:52
  SPEAKER_01: I think we tried Uzi, but Uzi was, I mean, I think it was like in XML and so on, so we didn't go with it.
• 6:58
  SPEAKER_01: So tell me more about your droning here.
• 7:00
  SPEAKER_02: So I'll keep it generic when I want to answer this question because as the data processing evolved, even the scheduling has evolved, right?
• 7:08
  SPEAKER_02: Previously it was like a batch, I'm not sure if like batch processing, even you say junkins, right?
• 7:12
  SPEAKER_02: So people used to schedule it for a particular time of period.
• 7:15
  SPEAKER_02: So let's say uh my batch runs for 30 minutes.
• 7:18
  SPEAKER_02: People will actually say, I'll give you a 45 minutes batch slot.
• 7:21
  SPEAKER_02: So it starts at this time, so it ends by in 45 minutes, within that 30 minutes, if your batch completes, then it's you are happy, or if not, like it's killed or it's like it fails.
• 7:31
  SPEAKER_02: Okay, they then the next batch has to start.
• 7:34
  SPEAKER_02: So those kind of tools, when we evolved and then we moved on, and then trigger-based.
• 7:39
  SPEAKER_02: So instead of scheduler-based, this is a schedule-based, and when you have a trigger-based or an event-based scheduler, so wherein you can wait for your batch operator to send signals to actually trigger the next batch and then things like that.
• 7:52
  SPEAKER_02: So tools like cron, tools like control M, tools like other, as you said, like there are junk ins and you had other tools which were in the market.
• 8:00
  SPEAKER_02: Somewhere in the again, same in the mid-2010s, which is like 2015 or sometime when AFL actually came in, oh, gave it as an open source solution, it was awesome.
• 8:10
  SPEAKER_02: So when you say you were able to see something where you're graphically, you were able to see where is your task, you from one task to other task, and then whether it is a failure or a success, you click on that particular task and then go to the messagings of whatever errors that you have in your logs, you try to resolve it, it was easy to manage.
• 8:26
  SPEAKER_02: Okay.
• 8:27
  SPEAKER_02: That's when I would say I was using Airflow as my predominant go-to for scheduling.
• 8:33
  SPEAKER_02: And in terms of what do I say, data and data engineering, Airflow is the tool I would say.
• 8:38
  SPEAKER_02: There are other tools as well, but Airflow, as you said, it's more easy, more convenient.
• 8:42
  SPEAKER_02: And the programming language is Python.
• 8:45
  SPEAKER_02: So everyone knows Python, Python, everybody loves it.
• 8:48
  SPEAKER_02: I'm a lover of Python.
• 8:49
  SPEAKER_02: So it was easy for me to get it, get used to it.
• 8:51
  SPEAKER_02: Okay.
• 8:52
  SPEAKER_02: And later on from Airflow, we moved into Astronomer, which is again a predominantly Airflow-based solution.
• 8:58
  SPEAKER_02: On top of it, it's a fully managed Airflow, is what I would call it.
• 9:01
  SPEAKER_02: So we moved into Astronomer's Managed Service.
• 9:04
  SPEAKER_02: So yeah, that's how I would say like the tool also evolved from a basic scheduler to your operator instinct one.
• 9:12
  SPEAKER_01: I have to ask, so why did you choose Airflow?
• 9:15
  SPEAKER_01: Why did you pick Airflow?
• 9:17
  SPEAKER_01: There are different choices, but why did you pick Airflow over the others?
• 9:21
  SPEAKER_02: See, as I said, like initially, first, your programming language is basically the language that gives you the what do I say affinity towards what you are working on, right?
• 9:30
  SPEAKER_02: In data engineering world, Scala and Python is the go-to language.
• 9:35
  SPEAKER_01: So I forgot about Scala.
• 9:37
  SPEAKER_02: Yeah, Scala is the tool.
• 9:39
  SPEAKER_02: So Python, it was easy for us to work with it.
• 9:42
  SPEAKER_02: Okay.
• 9:42
  SPEAKER_02: And creating your DAGs using the Python and pure Python code, right?
• 9:46
  SPEAKER_02: And you also have the UI, which is very simple, but it is very powerful.
• 9:50
  SPEAKER_02: As I said, like you can go there and then you can click your own task and then write your code within that one and then exactly mention what you want to do, right?
• 9:58
  SPEAKER_02: And if you have any other, what I say, plug and play tools like APA calls, you can do it.
• 10:03
  SPEAKER_02: And then you don't really need to rewrite the entire operator or you don't need to orchestrate, but you have a plug-and-play option where you can do that.
• 10:10
  SPEAKER_02: And then you had the community power, right?
• 10:14
  SPEAKER_02: The Airflow community was giving you a lot of user guides, documents, and then it supported you with all the needs that you have.
• 10:23
  SPEAKER_02: And when you go ask a question in your community forum, then you at least get that answered within two to three hours.
• 10:28
  SPEAKER_02: So it was so active.
• 10:29
  SPEAKER_02: The user community was so good.
• 10:31
  SPEAKER_02: And also when you are a data engineer, right?
• 10:34
  SPEAKER_02: So as your Airflow, even you get start using your Airflow, it will be your bread and butter, and you will start working towards it, and it's like easy for you.
• 10:42
  SPEAKER_02: And it becomes your neural system wherein you have your triggers and everything orchestrated in a very fine manner.
• 10:48
  SPEAKER_02: So also you can have your own upstream and downstream relation task between each and every task that actually runs in the right order or not.
• 10:57
  SPEAKER_02: So the dependability that you can give.
• 10:59
  SPEAKER_02: So that was the go-to tool for me.
• 11:02
  SPEAKER_02: And also you can write your own monitoring system alerts.
• 11:05
  SPEAKER_02: You can write, and then if you want to send a mail for a particular action, if you want to do something, you can trigger your own mailer objects, and then it will give you alerts as well.
• 11:13
  SPEAKER_02: So you can write your own alerts as well, and then you can do it.
• 11:17
  SPEAKER_02: It's something that you will have to write it and then keep it right.
• 11:20
  SPEAKER_02: And also in terms of scalability, it is a scalable tool, I would say.
• 11:23
  SPEAKER_02: If you want to add some more pipelines, like say you have 50 DAGs running today and you want to add 100 more, and you need to increase your infra capacity and then do it, yes, Airflow, it can enable you with that as well.
• 11:36
  SPEAKER_02: That being said, with the happiness, as you said, like happiness to have the control of your job.
• 11:42
  SPEAKER_02: So that is the main reason why I moved into Airflow, I would say.
• 11:48
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 11:57
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:09
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:18
  SPEAKER_01: I don't know exactly what I can add to your answer because you said everything, but that's exactly why I decided to use Airflow.
• 12:26
  SPEAKER_01: When I actually discovered Airflow, the fact that everything was in Python, just that for me was enough.
• 12:31
  SPEAKER_01: But also like the UI and the community.
• 12:34
  SPEAKER_01: I think I started to use it in 2017 maybe, but even in 2017, the community was already there.
• 12:40
  SPEAKER_01: It was much harder to find the right documentation and everything.
• 12:44
  SPEAKER_01: But still, I don't know.
• 12:45
  SPEAKER_01: Like I truly felt that this tool would be big at some point because it was filling a gap in the data engineering industry.
• 12:51
  SPEAKER_01: And yeah, I mean, obviously, I'm glad that you picked that tool over the others, but I think there are many reasons why Airflow is great, and you mentioned a lot of them, so uh that's cool.
• 13:00
  SPEAKER_01: But you also decided to move from the open source version to uh astronomer.
• 13:07
  SPEAKER_01: Can you tell me more about why?
• 13:09
  SPEAKER_01: Like, why did you decide to go from the open source version to a fully managed version of Airflow, which is Astronomer?
• 13:17
  SPEAKER_02: Okay, so this is an open source tool, so let's be honest to begin with.
• 13:22
  SPEAKER_02: Like Airflow was open source, it is good and it has equally uh some heavy lifting in your side that you need to do.
• 13:30
  SPEAKER_02: Okay, so if you see like to setting up an initial environment of Airflow to get started, let's say you want to start something next week, so you will have to prepare now, and then you will have to set up your instance, you select it, and then you do your scheduler, all your web service and all your metadata, DB and everything that you need to set up, and then do your configuration and then testing it out, and then see your first DAG runs good.
• 13:56
  SPEAKER_02: So it is going to be an end-to-end effort that you need to have an engineer.
• 14:00
  SPEAKER_02: And you being a business first organization, right?
• 14:04
  SPEAKER_02: So, what happens is like you have to focus most of your time into your business rather than saying, hey, this software is going to be okay, I will manage it, I will do it.
• 14:12
  SPEAKER_02: So it's gonna be like at least one person whom you have to spend in it, and then they will have to work on it.
• 14:18
  SPEAKER_02: And also, in terms of if I have to take care of any upgrades which comes in the Airflow system, and if there is an upgrade that has to happen in the version, if there is some security certification that has to be done, and all your other server upgrades and other things that you need to do, okay, that's going to be a whole overhead for the engineer.
• 14:36
  SPEAKER_02: Okay.
• 14:36
  SPEAKER_02: And also when you are a small data ecosystem, that's fine.
• 14:40
  SPEAKER_02: But when your ecosystem is increasing to say 40 to 70 pipelines that you need to manage, what happens if your scalability is questioned and your logs, when it starts writing, your storage goes up, and then you you will really have to contain that storage.
• 14:56
  SPEAKER_02: And there are a lot of other complex things that you need to take care of, right?
• 14:59
  SPEAKER_02: Your scheduler performance, your task queue performance, and then your memory operator issues, and those are all like overhead, I would say.
• 15:08
  SPEAKER_02: So being in an open source tooling, using an open source tool is really good.
• 15:12
  SPEAKER_02: But what happens eventually is when you are moving from a small environment to a bigger growing business needs, then your things go, what do we say, overboard.
• 15:22
  SPEAKER_02: You cannot manage Airflow in your own way, right?
• 15:26
  SPEAKER_02: So it is always good to go ahead with someone who understands your business and then helps you out with your managed service solutions so that even when you want to see your search or error logs, right?
• 15:36
  SPEAKER_02: So in your open source, you will have your thousands and thousands of lines and you try to see where the exception is and then try to send the error.
• 15:44
  SPEAKER_02: No, you don't need to do that in your managed service.
• 15:46
  SPEAKER_02: One of my choice was Astronomer.
• 15:50
  SPEAKER_02: So they actually got it and they worked on top of it, and it is an awesome tool to use.
• 15:55
  SPEAKER_02: And the modern one that they have is also having all these data lineage and other things, which is coming up in their tools, right?
• 16:01
  SPEAKER_02: So that really helps us, right?
• 16:03
  SPEAKER_02: That's the real need for us to move into a managed solution than this one.
• 16:08
  SPEAKER_01: I keep smiling during your answer because everything you said is just like all the things that you actually don't want to deal with.
• 16:16
  SPEAKER_01: You know, especially you as a director of engineering, you don't want to have like an engineer spending all the time setting up the uh airflow infrastructure and then managing the scalability and all of that stuff.
• 16:26
  SPEAKER_01: It's important, but you don't want to spend time on that, right?
• 16:29
  SPEAKER_01: Like you don't care that much.
• 16:31
  SPEAKER_01: Actually, what you care is about the business, about the use cases that you run through Airflow, not like the underlying infrastructure.
• 16:36
  SPEAKER_01: You don't want to manage that because that's not your ROI, basically.
• 16:40
  SPEAKER_01: What you said is absolutely right.
• 16:41
  SPEAKER_01: Like when you go with Astronomer, obviously you will save a lot of time and even money, because again, engineers will be engineers timing exactly.
• 16:51
  SPEAKER_01: Now that you are on Astronomer, can you tell us more about your primary use cases today?
• 16:56
  SPEAKER_01: How you are leveraging airflow, and you know, how do you monitor that everything is working as expected?
• 17:03
  SPEAKER_02: So all the problems that I mentioned in this one, it is all gone.
• 17:07
  SPEAKER_02: Let's say I have implemented Astronomer, as you said, like the money is being saved with my engineering effort.
• 17:13
  SPEAKER_02: And also I have people on call and something goes wrong, so that's something that they have to help.
• 17:18
  SPEAKER_02: Yes, it is all done.
• 17:20
  SPEAKER_02: And that being said, now I have more observability of the data.
• 17:24
  SPEAKER_02: So, as I said, there are things that we can implement in this time and enhance the error loggings and then metric dashboards that's available, and then the alerting systems that we can build with the Slack, communicate it in the messages to the Slack, and then have my team on the board.
• 17:39
  SPEAKER_02: So these are all the additional nuances that you can get it from the astronomers tool and your plugins that you already have.
• 17:46
  SPEAKER_02: And when developers get time to work, right?
• 17:51
  SPEAKER_02: So you can look at other features of astronomers.
• 17:54
  SPEAKER_02: So now, as I said, like quality monitoring solutions and then data lineage and then the reporting workflows, and there are other AAML features and tools, and also now that we have DBT in our solutions, so we also can implement because currently DBT we are not having that much of monitoring.
• 18:12
  SPEAKER_02: Now we have what we say, a feature available in Astronomer, which will enable us to have that as well scheduling.
• 18:19
  SPEAKER_02: So that is what has empowered, or that is what, as an astronomer's primary use case and everything, helping us in implementing this.
• 18:28
  SPEAKER_01: There is one last question that I wanted to ask you because actually this is something that I would say that I don't know much about, which is regarding data governance.
• 18:37
  SPEAKER_01: Because as a the director of engineering, obviously you work with teams across the US, Europe, and many other countries.
• 18:44
  SPEAKER_01: How do you handle data governance challenges?
• 18:48
  SPEAKER_01: And maybe if you can start by defining what is data governance and then talk about the challenges and how you tackle them, and also like what role does your orchestration platform to manage the regulation?
• 19:00
  SPEAKER_02: So, as I said, like data engineering or the data solution part of it, right?
• 19:05
  SPEAKER_02: Data governance is key.
• 19:06
  SPEAKER_02: Okay, it is not an option, it is something like must-have in a modern ecosystem.
• 19:12
  SPEAKER_02: As there are many laws and legal to be followed in different, especially Europe has a very Singapore, has even more.
• 19:23
  SPEAKER_02: So people are getting into that.
• 19:24
  SPEAKER_02: And your revenue with the overall global revenue, you will have to pay a percentage of fines if something goes wrong.
• 19:30
  SPEAKER_02: And to be honest, data is the most costliest part in any organization.
• 19:36
  SPEAKER_02: To be honest, if you don't monitor your data infrastructure, you burn a lot of clusters over there and you spend a lot of storage, so you lose money.
• 19:44
  SPEAKER_02: And if you don't govern that data properly, and if something happens, then you are legally entitled to pay a hefty amount of fine.
• 19:52
  SPEAKER_02: So that being said, governance is not a complex thing, but unless and until you keep your data governance first, okay?
• 20:00
  SPEAKER_02: It starts from the way how you collect your data.
• 20:03
  SPEAKER_02: So the data governance starts at what all you're gonna collect, what is the cookie that you're gonna accept from your users, and then how you're gonna get your consent, how you're gonna do your data privacy of your user base, right?
• 20:15
  SPEAKER_02: How are you gonna keep your data?
• 20:17
  SPEAKER_02: So, what is your retention policy that you're gonna keep?
• 20:20
  SPEAKER_02: And how you're gonna classify it and then store it.
• 20:23
  SPEAKER_02: Okay, so how you're gonna govern the growing needs of your data?
• 20:28
  SPEAKER_02: So you have your business, people they really want to expand across the borders, right?
• 20:33
  SPEAKER_02: It's not just they're gonna sit in one particular location, so they're gonna do business across different multiple geography.
• 20:39
  SPEAKER_02: So that being said, it's going to be a mindset.
• 20:42
  SPEAKER_02: So the governance is going to be a mindset from the start.
• 20:46
  SPEAKER_02: So it is not an afterthought.
• 20:47
  SPEAKER_02: So whenever somebody is building a data pipeline, your data engineering ecosystem, so data governance should be the first thought, not an afterthought.
• 20:56
  SPEAKER_02: Okay, so that's when it starts, right?
• 20:59
  SPEAKER_02: That being said, so you will have to be cautious.
• 21:03
  SPEAKER_02: And then when you are trying to select your orchestration tool in particular, so now coming on, like what is that an orchestration tool can help you out.
• 21:11
  SPEAKER_02: In our system, we have a DAG-level modelization that can be done, right?
• 21:17
  SPEAKER_02: So each and every region, for example, when you're trying to process your data, so you modelize your DAG with respect to it's a reusable component wherein you can have your Europe, you can have your APAC, you have your US zone, so everything is stored in particular way.
• 21:31
  SPEAKER_02: So you modularize if there is anything that has to be applied on top of it, you have your ability to write your own, what I say, plugins accordingly, and then you run it.
• 21:40
  SPEAKER_02: So that's a DAG-level modelization that can be done on top of your orchestration.
• 21:45
  SPEAKER_02: And Airflow has got Airflow variables.
• 21:48
  SPEAKER_02: When you have your connectivity, you can store your connection variables accordingly, and then you can connect it with your backend, say, like in your secrets manager or in any other hashing key mechanism, you can use them and then give only particular users that particular access.
• 22:03
  SPEAKER_02: So you restrict them on that.
• 22:05
  SPEAKER_02: Okay.
• 22:06
  SPEAKER_02: And Airflow also has a role-based control.
• 22:10
  SPEAKER_02: So it has a role-based control wherein you can use your roles and then see who can access what kind of DAGs and who can access what kind of so that being said, all these things together, right?
• 22:20
  SPEAKER_02: Whenever you select your Airflow tool or like orchestration tool, you really have to think about your governance first.
• 22:26
  SPEAKER_02: As I said, who, what, how, what is your classification and what is the flexibility that you will give to someone.
• 22:33
  SPEAKER_02: So that is going to be your first.
• 22:35
  SPEAKER_02: And then you take about, think about the next thing, how you want to implement what you want to implement.
• 22:40
  SPEAKER_02: So that's how Airflow can help you out in terms of implementation as well.
• 22:44
  SPEAKER_02: Data governance, it's not an easy one.
• 22:47
  SPEAKER_02: So, again, as I said, if you automate your governance on top of orchestration tools like Airflow or Astronomer, it is going to help a lot.
• 22:54
  SPEAKER_01: Thank you so much.
• 22:55
  SPEAKER_01: Because that's actually the first time in all the APs that I've done so far that someone actually talked about data governance in relation to the orchestration tool.
• 23:06
  SPEAKER_01: So to me, I would have never thought about data governance before choosing my orchestration tool, but with everything that you said, actually, it makes a lot of sense.
• 23:15
  SPEAKER_01: So I'm sure that it would be like a wow moment for many data engineers or managers listening to this episode.
• 23:22
  SPEAKER_01: Thank you so much for joining, Arun.
• 23:23
  SPEAKER_01: It was a pleasure.
• 23:24
  SPEAKER_01: Can you tell us what is the best way for folks to get in touch with you?
• 23:28
  SPEAKER_02: Everyone is welcome to ask me questions.
• 23:31
  SPEAKER_02: I'll help you as much as possible.
• 23:32
  SPEAKER_02: I'm active in LinkedIn, so you guys can reach out to me in LinkedIn.
• 23:37
  SPEAKER_02: I will share you my LinkedIn bio so probably you can share it with the audience.
• 23:41
  SPEAKER_02: So that should be okay with me as well.
• 23:44
  SPEAKER_01: Okay.
• 23:44
  SPEAKER_01: Awesome.
• 23:45
  SPEAKER_01: Thank you so much.
• 23:46
  SPEAKER_01: Thank you again, Arun.
• 23:47
  SPEAKER_01: I hope to see you uh, who knows, again for another episode.
• 23:50
  SPEAKER_01: Otherwise, take care and see you.
• 23:52
  SPEAKER_02: Thank you, Mal.
• 23:53
  SPEAKER_02: See ya.
• 23:53
  SPEAKER_02: Bye bye.
• 23:54
  SPEAKER_02: Bye.
• 23:57
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:04
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 24:08
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.