Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade

Transcript

Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: How will each engineering team get the benefits of the infra changes?
• 0:04
  SPEAKER_02: It will be through library updates.
• 0:06
  SPEAKER_02: But if they are not deploying often, they won't get those library updates into their code.
• 0:12
  SPEAKER_02: So all the infra changes which companies investing it, people are not benefiting from it because they are not deploying.
• 0:19
  SPEAKER_02: So it becomes a vicious cycle.
• 0:21
  SPEAKER_02: And that's why these deployments is very critical because LinkedIn today, which has more than a billion users, internally we have about 6,000 services which powers LinkedIn.com.
• 0:38
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:48
  SPEAKER_00: Okay.
• 0:48
  SPEAKER_00: Hello, everyone.
• 0:49
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:51
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Rahul Gade from uh LinkedIn.
• 0:57
  SPEAKER_00: He is a staff software engineer.
• 0:59
  SPEAKER_00: Rahul, it's great to have you on.
• 1:01
  SPEAKER_00: How are you?
• 1:02
  SPEAKER_02: I'm good.
• 1:02
  SPEAKER_02: Thanks for having me at Kenton.
• 1:04
  SPEAKER_00: Awesome.
• 1:05
  SPEAKER_00: Are you gearing up for the holiday season here in the US?
• 1:09
  SPEAKER_00: I know we've got Thanksgiving just around the corner, at least when we're recording.
• 1:12
  SPEAKER_00: This might be released after that.
• 1:14
  SPEAKER_02: Yeah.
• 1:15
  SPEAKER_02: So yeah, I think holidays are always the fun part.
• 1:17
  SPEAKER_02: I have more decision makers in my life.
• 1:19
  SPEAKER_02: It's my wife and my kids.
• 1:21
  SPEAKER_02: They're growing up.
• 1:22
  SPEAKER_02: So making decisions become difficult.
• 1:24
  SPEAKER_02: Hey, what to do on holidays.
• 1:26
  SPEAKER_02: So I think we are still debating what to do.
• 1:28
  SPEAKER_02: Hopefully by in a week or so, we should be able to determine what our plan is.
• 1:32
  SPEAKER_02: But I think looking forward to it.
• 1:34
  SPEAKER_00: Nice.
• 1:34
  SPEAKER_00: How old are your kids?
• 1:36
  SPEAKER_02: I have two.
• 1:37
  SPEAKER_02: One is going to be seven in December, another would be four in Jan.
• 1:41
  SPEAKER_00: Okay.
• 1:41
  SPEAKER_00: I'm sure that's a very opinionated time for both of them, probably.
• 1:45
  SPEAKER_02: Yes.
• 1:46
  SPEAKER_02: And it's fun to see how they can be opinionated.
• 1:48
  SPEAKER_02: And most of the times they're not wrong.
• 1:50
  SPEAKER_00: Yeah, totally.
• 1:52
  SPEAKER_00: That's very fun.
• 1:53
  SPEAKER_00: Cool.
• 1:53
  SPEAKER_00: Let's go ahead and dive in.
• 1:55
  SPEAKER_00: So for today's podcast for our listeners, we're going to discuss how LinkedIn uses Airflow for continuous deployment.
• 2:02
  SPEAKER_00: And as always, I want to start with some background.
• 2:05
  SPEAKER_00: So I don't think we need to do a lot of background on LinkedIn and what LinkedIn does.
• 2:09
  SPEAKER_00: I think it's pretty ubiquitous and familiar to everybody.
• 2:13
  SPEAKER_00: But can you tell our listeners what your role is in particular and what your team does at LinkedIn?
• 2:18
  SPEAKER_02: Got it.
• 2:19
  SPEAKER_02: So I'll talk about what my team does, and then we'll talk about my role at the team.
• 2:23
  SPEAKER_02: So I'm in LinkedIn.
• 2:24
  SPEAKER_02: Our team is LinkedIn's continuous deployments.
• 2:27
  SPEAKER_02: Our goal for our team is we don't want humans in production environments because humans in production environments can be messy.
• 2:35
  SPEAKER_02: So we want to start limiting the access of production environments for humans.
• 2:39
  SPEAKER_02: And that starts with deployments, cluster provisioning, auto-resizing of clusters, doing chaos monkey testing and restarting of clusters and so on.
• 2:49
  SPEAKER_02: So with this motto, we built up a platform from ground up, which is LinkedIn's continuous deployment platform, which is built up on top of an orchestrator airflow, which most of you folks would be familiar with.
• 3:01
  SPEAKER_02: We also use other open source tools like OPA, which is Open Policy Agent, and that helps to define governance, like what policies should we use during different activities when we access production.
• 3:13
  SPEAKER_02: And these can be at different layers, like from team level to org level to company-wide policies, right?
• 3:20
  SPEAKER_02: We use Kubernetes.
• 3:21
  SPEAKER_02: Our stack has been built in Java and Python based on where it is.
• 3:25
  SPEAKER_02: We have our own custom UI for our platform, which is based off backstage.
• 3:30
  SPEAKER_02: So that's what our team does building up a platform from ground up to help continuous deployment at LinkedIn.
• 3:37
  SPEAKER_02: My role in the team is I'm the tech lead for LCD, LinkedIn Continuous Deployment Platform.
• 3:42
  SPEAKER_02: My role has been creating the vision for this platform, which started about two and a half years ago when LinkedIn had its own challenges with deployments, which we still have it as we are migrating to the new platform.
• 3:54
  SPEAKER_02: So came up with the vision, came up with the roadmap, got alignment with the stakeholders, which includes the management and the partner teams, because no platform can be built on its own.
• 4:03
  SPEAKER_02: So got our partners excited about this, got our customers, the most unhappy customers with the original platform involved in this process and trying to onboard them onto our new platform right now.
• 4:15
  SPEAKER_00: Okay, great.
• 4:16
  SPEAKER_00: Very interesting.
• 4:17
  SPEAKER_00: So let's dig in a little bit more on the deployments aspect of it.
• 4:21
  SPEAKER_00: You mentioned that a couple of times.
• 4:23
  SPEAKER_00: So what does deployments mean in the context of LinkedIn's business?
• 4:28
  SPEAKER_00: Kind of what are those?
• 4:29
  SPEAKER_02: Great question.
• 4:30
  SPEAKER_02: So before I talk about deployments, we all are engineers and engineers to SDLC, software deployment lifecycle, which involves like gathering requirements, then writing code, then getting it reviewed, and then merging it.
• 4:44
  SPEAKER_02: Now, the last part of the SDLC is deployments.
• 4:48
  SPEAKER_02: I call it as the last mile of the SDLC lifecycle.
• 4:52
  SPEAKER_02: And that's the most complicated because until and unless you deploy the code, the users, the customers will not get the real value of the work done by the engineers.
• 5:01
  SPEAKER_02: So that's where the value is delivered.
• 5:03
  SPEAKER_02: And in today's world, deployments are getting complicated.
• 5:06
  SPEAKER_02: For the reason being, we want services to be 24 by 7 available.
• 5:10
  SPEAKER_02: We don't want any downtime.
• 5:12
  SPEAKER_02: We want services to have lower latency.
• 5:14
  SPEAKER_02: That means services will be closer to the users.
• 5:18
  SPEAKER_02: That means services need to be deployed in different fabrics, different colos.
• 5:22
  SPEAKER_02: Now, if these deployments are being done, there are two ways of deployment.
• 5:28
  SPEAKER_02: So one is like a continuous deployment, where as soon as the code is merged, the platform will pick up and roll it out in different fabrics, colos, in a very safe way.
• 5:38
  SPEAKER_02: There is another way of doing it, which is manual deployments.
• 5:42
  SPEAKER_02: Manual deployment is when engineers are forced.
• 5:45
  SPEAKER_02: And I'm going to use the word force because no one enjoys this process.
• 5:48
  SPEAKER_02: The reason being now that engineer will have to make sure it is deployed in each fabric.
• 5:54
  SPEAKER_02: Environments can be different.
• 5:55
  SPEAKER_02: That means they might have to defollow different processes to deploy.
• 5:59
  SPEAKER_02: They may or may not do testing depending upon how much bandwidth they have, how much expertise they have.
• 6:05
  SPEAKER_02: And they might bypass a lot of policies, like people might deploy on Friday evening, right?
• 6:10
  SPEAKER_02: How often it has cost time there, right?
• 6:13
  SPEAKER_02: So a lot of bypasses happen when people do like these manual deployments, right?
• 6:18
  SPEAKER_02: And surprisingly, teams still do manual deployments.
• 6:21
  SPEAKER_02: At LinkedIn, when we started, the number was only 8% of services at LinkedIn were doing continuous deployment.
• 6:31
  SPEAKER_02: Wow.
• 6:32
  SPEAKER_02: And that's not only another company, I won't take their name.
• 6:35
  SPEAKER_02: They also went through a similar journey.
• 6:37
  SPEAKER_02: Their number was also about 8%.
• 6:40
  SPEAKER_02: And when they migrated, created a new platform, yes, that was one of the goals.
• 6:45
  SPEAKER_02: Right?
• 6:46
  SPEAKER_02: And this causes a lot of human toil to the engineers because they had to do this manually.
• 6:51
  SPEAKER_02: It is also error prone.
• 6:52
  SPEAKER_02: We talked about it.
• 6:53
  SPEAKER_02: And it also impacts a lot of infra teams.
• 6:57
  SPEAKER_02: Now we don't think about that, but an example would be one of our partner teams was introducing GRPC.
• 7:04
  SPEAKER_02: Now, how will each engineering team get the benefits of the infra changes?
• 7:09
  SPEAKER_02: It will be through library updates.
• 7:11
  SPEAKER_02: But if they are not deploying often, they won't get those library updates into their code.
• 7:17
  SPEAKER_02: So all the infra changes which companies investing it, people are not benefiting from it because they are not deploying.
• 7:24
  SPEAKER_02: So it becomes a vicious cycle.
• 7:26
  SPEAKER_02: And that's why these deployments is very critical because LinkedIn today, which has more than a billion users, internally we have about 6,000 services which powers LinkedIn.com.
• 7:38
  SPEAKER_02: And we have about 10K engineers, more than 10K engineers.
• 7:42
  SPEAKER_02: So the main goal of our platform was to improve deployments, was to improve the continuous deployment adoption.
• 7:49
  SPEAKER_02: And our end goal is to get about to 80% continuous deployment, like 80% of the services should be able to reach continuous deployment.
• 7:58
  SPEAKER_02: The last 20% is a North Star, which eventually we'll get to 100%.
• 8:03
  SPEAKER_02: But it's going to be the most difficult fees to get those 20% also onto continuous deployments.
• 8:08
  SPEAKER_02: So that's the overall aspect of these deployments.
• 8:10
  SPEAKER_02: And that's why we wanted to invest in it.
• 8:13
  SPEAKER_02: And it's also a challenging problem, mainly because whenever you do these infra changes, there's no direct dollar amount associated.
• 8:21
  SPEAKER_02: If you look for an ad steam, like it's very easy.
• 8:23
  SPEAKER_02: If I add this feature, I will save this, I will increase the revenue by this number.
• 8:28
  SPEAKER_02: Whereas for infra changes, it's very difficult.
• 8:31
  SPEAKER_02: So a lot of time it management always overlooks, but it was one good thing that management was able to see the problem through and was able to fund this project and get us through it.
• 8:43
  SPEAKER_00: Yeah.
• 8:44
  SPEAKER_00: Wow.
• 8:44
  SPEAKER_00: That's the 8% is honestly a striking figure to me, especially considering I came up as a data engineer kind of in the time of adopting SDLC best practices, including CICD for data as well as software.
• 8:59
  SPEAKER_00: So that's obviously a very low number for a company operating as high scale as LinkedIn.
• 9:05
  SPEAKER_00: So it makes sense that you would need some guardrails in some process to make 10,000 engineers successful at their jobs and providing a platform for over a billion people.
• 9:16
  SPEAKER_00: That's huge.
• 9:17
  SPEAKER_00: So tell me how Airflow fits into this, because you use Airflow in what I would consider a unique way in terms of what we often hear for Airflow usage.
• 9:27
  SPEAKER_02: Great question.
• 9:28
  SPEAKER_02: And again, before answering the question, I'll give you the history behind it, like how we get up with Airflow.
• 9:34
  SPEAKER_02: So when we were designing this platform, first thing we talked about the personas.
• 9:39
  SPEAKER_02: Who are the personas who will be interacting with the platform?
• 9:42
  SPEAKER_02: We had three personas in mind.
• 9:44
  SPEAKER_02: One was the admins, which is us, who would be managing the platform.
• 9:47
  SPEAKER_02: What we wanted was a Lego architecture.
• 9:51
  SPEAKER_02: Any piece in our platform should be easily pluggable and at the same time easily removable.
• 9:57
  SPEAKER_02: Because in five years from now, if we want to, let's say, move on to something else, it should not need a two or three-year migration cycle.
• 10:05
  SPEAKER_02: It should be easy enough without impacting others.
• 10:08
  SPEAKER_02: So that was the goal number one.
• 10:09
  SPEAKER_02: The second persona we had in mind was our partners' persona.
• 10:13
  SPEAKER_02: We are building a platform.
• 10:15
  SPEAKER_02: If people are not able to add features on top of it in a pluggable fashion, the platform is not of any use because we have a lot of partners who write like validation tools.
• 10:25
  SPEAKER_02: Hey, I want to do blue-green deployment.
• 10:27
  SPEAKER_02: So there's some validation will try, team will write a blue-green deployment task, which can be integrated with our platform.
• 10:34
  SPEAKER_02: So partners should be easily, should be able to easily plug their stuff into our platform.
• 10:40
  SPEAKER_02: That means a lot of things should be agnostic enough for that.
• 10:44
  SPEAKER_02: And the third thing is the user's personal.
• 10:47
  SPEAKER_02: Platform should be as easy.
• 10:48
  SPEAKER_02: They should not worry about what is the underlying technology, and they should just use it as a no-code, low-code platform.
• 10:55
  SPEAKER_02: We have these three personas in mind.
• 10:57
  SPEAKER_02: With these personals, we also looked at our existing complex customers who were unhappy with our current LinkedIn platform.
• 11:05
  SPEAKER_02: We talked to them.
• 11:06
  SPEAKER_02: The most unhappy customers were stateful teams.
• 11:09
  SPEAKER_02: Stateful teams is an example like Database Service, who have like huge clusters and their deployments span for about two weeks.
• 11:17
  SPEAKER_02: Like they take about two weeks to roll out the whole code onto the whole feet.
• 11:22
  SPEAKER_02: So what we did is based on all these personas, the architecture requirements, the vision, and the requirements from the complex slash unhappy customers.
• 11:30
  SPEAKER_02: The once we had the requirements, the few things that stood out and that actually impacted the decision was we need something that is a long-running pipeline, which can run up to two weeks, three weeks.
• 11:44
  SPEAKER_02: Deployments is also not a compute-intensive because when we want to do a deployment, what we'll do is we'll tell the underlying scheduler, like a Kubernetes server, that hey, I want to deploy something.
• 11:55
  SPEAKER_02: And that server can take up to, depending upon the size of the cluster, minutes to hours.
• 12:00
  SPEAKER_02: So we'll start the deployment and then we'll keep polling.
• 12:03
  SPEAKER_02: Are you done?
• 12:03
  SPEAKER_02: So we wanted an async operation rather than a compute-intensive.
• 12:07
  SPEAKER_02: So we wanted async support for all these operations.
• 12:10
  SPEAKER_02: We also wanted multi-language support because we said I talked about partner persona who would be writing stuff on top of platforms.
• 12:17
  SPEAKER_02: So we didn't want it to be stuck with a particular language.
• 12:20
  SPEAKER_02: So what we wanted was a platform that can support a running Docker containers.
• 12:26
  SPEAKER_02: And last but not the least was we wanted to build a custom UI on top of any orchestrator that we use.
• 12:34
  SPEAKER_02: The reason being deployments is one aspect, but we wanted to extend its functionality beyond deployments in the future.
• 12:40
  SPEAKER_02: So we wanted to be a driver of our own destiny.
• 12:43
  SPEAKER_02: So with all these requirements, we had a couple of options already available at LinkedIn.
• 12:48
  SPEAKER_02: Our CI, continuous integration team, was already using GitHub Actions.
• 12:53
  SPEAKER_02: And it's also something that Microsoft owns, so it was a natural fit.
• 12:58
  SPEAKER_02: And we also had Airflow, which was being used for data pipelines.
• 13:02
  SPEAKER_02: Based on these requirements, what we wanted was a solid orchestrator which would be able to orchestrate our pipelines.
• 13:09
  SPEAKER_02: What we did is based on our requirements, we started running scale tests.
• 13:14
  SPEAKER_02: As I said, we have 6,000 services.
• 13:16
  SPEAKER_02: So what we said is hey, in next five years, let's double the number of services.
• 13:20
  SPEAKER_02: Let's do 12,000 services.
• 13:23
  SPEAKER_02: We wanted to run one-third of the services in parallel at a time.
• 13:27
  SPEAKER_02: Like one-third of the services are deploying.
• 13:29
  SPEAKER_02: So that many pipelines will be running in parallel.
• 13:33
  SPEAKER_02: And these will be running for days, or some of them will be running for weeks.
• 13:37
  SPEAKER_02: So we did all the scale tests and everything.
• 13:40
  SPEAKER_02: And based on those requirements, uh, Airflow had an edge, mainly because it has the async support.
• 13:45
  SPEAKER_02: It also supported Docker, you Docker support for like you can use Kubernetes pod operator.
• 13:50
  SPEAKER_02: And with that, you can run Docker containers.
• 13:54
  SPEAKER_02: Based on those requirements, we were able to see Airflow had an advantage for our requirements, and we decided to move with Airflow.
• 14:01
  SPEAKER_02: There were also talks about hey, if you're using Airflow, why not use Temporal?
• 14:05
  SPEAKER_02: Why not use Cadence?
• 14:07
  SPEAKER_02: Like we have tens of different orchestrators in industry.
• 14:10
  SPEAKER_02: But one thing that I have learned from my previous even Salesforce, where I was trying to bring a new open source technology, it's not that whenever you use an open source technology, it's easy to bring it up and make it work.
• 14:23
  SPEAKER_02: But in big industries, that the challenge is how do you bring this open source technology and fit in in the existing ecosystem?
• 14:31
  SPEAKER_02: Like how do you fit in with the security?
• 14:34
  SPEAKER_02: Is this highly available?
• 14:36
  SPEAKER_02: Is it available in all the fabrics and everything, right?
• 14:39
  SPEAKER_02: There was a complexity.
• 14:40
  SPEAKER_02: There is a lot of cost involved in bringing an open source technology in a big company.
• 14:46
  SPEAKER_02: And Airflow was already there.
• 14:48
  SPEAKER_02: It did fit the bill of whatever we needed.
• 14:51
  SPEAKER_02: And it passed the scale test.
• 14:53
  SPEAKER_02: And we were happy with it.
• 14:55
  SPEAKER_02: And that's where we ended up with Airflow.
• 14:58
  SPEAKER_00: Yeah, very cool.
• 14:59
  SPEAKER_00: I love this use case because you said before, I think it's not as common as like the data orchestration running ETL pipelines that we often hear about.
• 15:09
  SPEAKER_00: But through that story, you touched on a couple of features that I would consider maintenance of Airflow that make it really widely applicable.
• 15:18
  SPEAKER_00: Like it's really flexible, it can integrate in with all the tools you already have in your ecosystem.
• 15:23
  SPEAKER_00: It's built to scale, has the flexibility of running on Kubernetes and features like deferable operators that allow you to leverage cost savings for long-running jobs.
• 15:34
  SPEAKER_00: So that's really interesting.
• 15:37
  SPEAKER_00: How did you go about setting up Airflow?
• 15:41
  SPEAKER_00: So you mentioned this is something that your users are not interacting with it directly.
• 15:46
  SPEAKER_00: I assume there's a layer between Airflow and them that you've customized that they are interacting with.
• 15:52
  SPEAKER_00: So what does that look like under the hood and for the developers interacting with it?
• 15:58
  SPEAKER_02: So let's go back to the personas that we had.
• 16:00
  SPEAKER_02: So the first persona is the user persona.
• 16:02
  SPEAKER_02: User personas all the 10,000 engineers who want to do deployments.
• 16:07
  SPEAKER_02: Now, one goal was we don't want them to learn Airflow because not that they should not learn, it's a good tool to know, but it adds to the tribal knowledge that you have for people because engineers move from one company to another and we always have new engineers coming at LinkedIn.
• 16:24
  SPEAKER_02: If they have to learn a new tool to do deployments, it just adds it to their ramp up time.
• 16:31
  SPEAKER_02: So what we wanted to do was build an abstraction.
• 16:33
  SPEAKER_02: And the way we did abstraction was using no code, low-code workflows.
• 16:37
  SPEAKER_02: So we have a custom YAML that we expose to the users where they basically tell us two things like, hey, I want to deploy this service, I want to deploy it in these fabrics.
• 16:49
  SPEAKER_02: That's it.
• 16:50
  SPEAKER_02: We take that, and behind the scenes, we have a DAG generator which takes these inputs and based on the best deployment practices that we have defined, we start mapping up their pipeline.
• 17:06
  SPEAKER_02: So what we do is we also look at the current traffic patterns.
• 17:09
  SPEAKER_02: So if you have a service which runs in three different prod fabrics or colos, we look at the current scenario in each of the fabric.
• 17:18
  SPEAKER_02: If some of the colo is down for some reason, we will say, hey, you should not be deploying to that at this moment.
• 17:24
  SPEAKER_02: Let's move on to the second one.
• 17:25
  SPEAKER_02: So we built up a lot of intelligence.
• 17:27
  SPEAKER_02: We also have a lot of testing practices like canaries.
• 17:32
  SPEAKER_02: Once the canary is done, we do analysis on the metrics, on the logs to see if everything is fine.
• 17:37
  SPEAKER_02: So a user is well abstracted from this.
• 17:40
  SPEAKER_02: They don't really need to worry about how is deployment, what are the best practices and anything.
• 17:46
  SPEAKER_02: They are busy with writing code.
• 17:48
  SPEAKER_02: Once the code is merged, they can go back to the feature gathering again.
• 17:52
  SPEAKER_02: So we just removed one step in the SDLC and we the platform took on.
• 17:56
  SPEAKER_02: So that's how the user persona is abstracted from Airflow.
• 18:00
  SPEAKER_02: That being said, there will always be like 1% or 5% of the users who come back and say, hey, this does not fit the bill.
• 18:08
  SPEAKER_02: We want to be the power users of Airflow.
• 18:11
  SPEAKER_02: For them, yes, things are open.
• 18:14
  SPEAKER_02: We can always expose the Airflow DAGs.
• 18:16
  SPEAKER_02: You can write your own custom DAGs and we'll get you what you want.
• 18:20
  SPEAKER_02: But the goal is to generalize so that the learning curve is as little as possible for everyone.
• 18:25
  SPEAKER_02: But if you are power users who cannot fit the platform use case, yes, we'll open up to the power users.
• 18:31
  SPEAKER_02: So that's persona one users.
• 18:33
  SPEAKER_02: The second persona was our partners who would be writing newer tools to make deployments safer.
• 18:41
  SPEAKER_02: Like one of our teams is like a canary analysis tool inside LinkedIn, which once a canary canary is nothing but you will deploy the new version of a code on a limited number of hosts.
• 18:52
  SPEAKER_02: This is the new version.
• 18:53
  SPEAKER_02: We don't know if it is safe, so they'll put deploy it only on one or a couple of hosts.
• 18:58
  SPEAKER_02: And once it's deployed, they will run some analysis on the old version and the new version.
• 19:02
  SPEAKER_02: And that tool determines if it is safe to move forward or not.
• 19:06
  SPEAKER_02: So those are the partner teams, and we have a few more partner teams who write more of these validation tools.
• 19:13
  SPEAKER_02: They should be able to plug in into this Airflow architecture.
• 19:16
  SPEAKER_02: And what we did was we use Docker containers.
• 19:19
  SPEAKER_02: So what they do is they just write their task in a Docker and give us the Docker and we plug it in our DAGs, and that's how we use it.
• 19:29
  SPEAKER_02: So another good thing is we don't want to, but if we have to move away from Airflow, it becomes very simpler because partners didn't have to worry about Airflow.
• 19:38
  SPEAKER_02: They didn't have to learn Airflow, they just do the Docker and give it to us.
• 19:42
  SPEAKER_02: We integrated it and now we can use it.
• 19:45
  SPEAKER_02: So again, the you partners don't have to worry about us.
• 19:49
  SPEAKER_02: Now the third thing is us.
• 19:51
  SPEAKER_02: We had to learn Airflow.
• 19:52
  SPEAKER_02: We need to know how Airflow works and those things.
• 19:56
  SPEAKER_02: And what we did, another thing was we wanted to run Airflow in an HA model.
• 20:01
  SPEAKER_02: Like we wanted to run multiple airflow clusters at a given time.
• 20:04
  SPEAKER_02: For the reason being, if one of the colo is down, we still want the other colour airflow to be running.
• 20:12
  SPEAKER_02: So what we have done is we run uh airflow in multiple colos.
• 20:16
  SPEAKER_02: Sorry, I use colo and fabrics interchangeably because in LinkedIn it's fabric, uh, whereas in industry it's colour so we will have airflow running in multiple colos at a given time and both are active.
• 20:26
  SPEAKER_02: For that to do, we had to build another abstraction, which is like a gateway service, which whenever someone says, Hey, I want to deploy something, the request ends up being onto the gateway service.
• 20:38
  SPEAKER_02: Gateway service will do the routing of which Airflow cluster this request needs to go.
• 20:43
  SPEAKER_02: And then one particular Airflow cluster will serve the request till the end.
• 20:48
  SPEAKER_02: So that's how we run the pipelines.
• 20:50
  SPEAKER_02: And this also helps us with redundancy, like in case if an entire colo is wiped out, we still are able to do deployments using our gateway service.
• 20:59
  SPEAKER_02: So gateway service acts as a stateful cache of Airflow.
• 21:03
  SPEAKER_02: So all the pipeline details, pipeline history and everything, we are able to cache it in a different service.
• 21:10
  SPEAKER_02: So we don't have to go and query Airflow all the time.
• 21:13
  SPEAKER_02: But that's how we built up our Airflow architecture on top of our deployment platform.
• 21:21
  SPEAKER_00: Yeah, interesting.
• 21:22
  SPEAKER_00: And so for the users who are not actually interacting with Airflow themselves, but are using it as a deployment tool, I think, at least in my experience with a lot of CI CD tools, you are interacting with the tool in the sense that like you're receiving the logs back from a deployment so that if a test fails, you know what happened.
• 21:42
  SPEAKER_00: And so for those users who are like writing a YAML file to create their pipeline, how does that work for them?
• 21:49
  SPEAKER_00: How do they get that feedback?
• 21:55
  SPEAKER_02: So what happens is we use pre execute post execute.
• 21:58
  SPEAKER_02: So what happens is all All we defined actions.
• 22:02
  SPEAKER_02: We call it as actions internally.
• 22:04
  SPEAKER_02: So let's deployment pipeline is a DAG.
• 22:07
  SPEAKER_02: Deployment pipeline in LinkedIn is airflow DAG.
• 22:11
  SPEAKER_02: Pipeline is made up of actions, and actions is nothing but airflow tasks.
• 22:16
  SPEAKER_02: What we do is in the actions, we have a pre-execute, post execute.
• 22:20
  SPEAKER_02: So that will be common for everybody.
• 22:22
  SPEAKER_02: So people don't have to worry about it.
• 22:24
  SPEAKER_02: So what we do is whenever an action starts, that pre-execute sends a status back to our another service called as deployment transparency that, hey, by the way, I started.
• 22:35
  SPEAKER_02: And the post execute will say, Hey, by the way, I ended.
• 22:38
  SPEAKER_02: And that end can be success, failure, or a custom message that needs to be sent back.
• 22:44
  SPEAKER_02: Moreover, each action is free to call over another API to talk to that service, which pre and post execute sends.
• 22:52
  SPEAKER_02: So with that, each action is able to send the status back to our deployment transparency service, which is what powers our UI.
• 23:02
  SPEAKER_02: So with that, and that also helps us with the multi-airflow cluster, right?
• 23:06
  SPEAKER_02: With that, if a pipeline is running on cluster one, pipeline is running on cluster two, all the information goes to a single deployment transparency service, and the UI can just poll that and get all the information.
• 23:18
  SPEAKER_02: Actions have the ability to send custom messages when they talk to the deployment transparency.
• 23:24
  SPEAKER_02: With that, the UI gets populated.
• 23:27
  SPEAKER_02: So whenever a deployment happens, people can go on to the UI, they can look at the status of the pipeline.
• 23:33
  SPEAKER_02: If they want to drill down, they can look at the status of the tasks slash action, and they would know what's going on.
• 23:40
  SPEAKER_00: Got it.
• 23:41
  SPEAKER_00: That's a really advanced customization.
• 23:43
  SPEAKER_00: That's really cool.
• 23:44
  SPEAKER_02: Yes, yes.
• 23:45
  SPEAKER_02: I think uh this is one of the uh projects that I'm pretty proud of with how we were able to achieve it, given a lot of different circumstances.
• 23:52
  SPEAKER_02: But yeah, this is turning out to be good now.
• 23:55
  SPEAKER_00: Yeah, it sounds like it's been really impactful in terms of democratizing kind of deployments within LinkedIn across so many different engineers.
• 24:03
  SPEAKER_00: As you mentioned, not all of them are going to be familiar with Airflow.
• 24:06
  SPEAKER_00: So you've managed to still make it accessible to them without that knowledge barrier.
• 24:11
  SPEAKER_02: And I think it's very important, like whenever you bring new open source tools, uh we need to be mindful.
• 24:18
  SPEAKER_02: Should we expose it as is, or we need enough abstractions that people don't have to understand the tribal knowledge?
• 24:24
  SPEAKER_02: Because 90% of the engineers really don't want to understand those.
• 24:28
  SPEAKER_02: They are like, I want to get my stuff done and move on, because they that time can be spent in feature development.
• 24:35
  SPEAKER_02: They have to learn like every different tool that you have in your company.
• 24:39
  SPEAKER_02: It adds to distractions, and hey, I have to learn a new tool every time.
• 24:45
  SPEAKER_02: And every tool needs a lot of knowledge that we need to prevent users from having.
• 24:51
  SPEAKER_00: Yeah, totally.
• 24:53
  SPEAKER_00: This is such an interesting use case.
• 24:55
  SPEAKER_00: We could keep talking about it for a long time, but I want to be respectful of your time.
• 24:58
  SPEAKER_00: Ruhu, my final question for everybody who comes on is what would you most like to see from the Airflow project in the near future?
• 25:06
  SPEAKER_02: I think there was one feature, but I think Airflow 3.0 is already coming with that, was multi-language support.
• 25:13
  SPEAKER_02: Because I think one of the challenges I always see with Airflow is whenever the conversation starts Airflow, one of the pushbacks I have always got is, oh, but it only supports Python.
• 25:23
  SPEAKER_02: And for engineers, I always say it, this can be controversial, but language is like religion for a lot of engineers.
• 25:29
  SPEAKER_02: My is the best, right?
• 25:32
  SPEAKER_02: But that's not true.
• 25:33
  SPEAKER_02: But that's the case.
• 25:34
  SPEAKER_02: And we and that's why we did the Dockerization, so that if someone wants to write a go action for us, go create a Docker container and we'll run it, right?
• 25:43
  SPEAKER_02: And with this multi-language support, I think the adoption will increase because a lot of people are like, oh, I just go, I just rust.
• 25:51
  SPEAKER_02: Why for their own reasons?
• 25:52
  SPEAKER_02: But hey, this works well.
• 25:54
  SPEAKER_02: You should use it.
• 25:55
  SPEAKER_02: Given we have multi-language support, I think people will be more open to it.
• 25:59
  SPEAKER_02: Yeah, okay, I can start using this.
• 26:00
  SPEAKER_02: The learning curve will be reduced.
• 26:02
  SPEAKER_02: And I think that is one feature that I am looking forward from Airflow, but I think it's already out coming out there.
• 26:08
  SPEAKER_00: Nice.
• 26:08
  SPEAKER_00: Yeah, that is in development for Flow3, which should come out kind of springtime next year.
• 26:13
  SPEAKER_00: Are there, I know it'll be you know a slow rollout over time with an initial set of the languages that are supported, and then obviously they'll be added as they go forward.
• 26:23
  SPEAKER_00: Are there any in particular that you or LinkedIn would be very excited about?
• 26:28
  SPEAKER_02: Yeah, I think Go is something that is looking forward to it.
• 26:31
  SPEAKER_02: And I think Java, most of the engineers still use Java.
• 26:35
  SPEAKER_02: So I think going with the pattern Java and Go would be something that will really help uh in the adoption.
• 26:41
  SPEAKER_00: Yeah.
• 26:42
  SPEAKER_00: I think last time I heard Go was going to be the first one, one of the ones that'll be right along with that 3.0 rollout.
• 26:48
  SPEAKER_00: That's great.
• 26:49
  SPEAKER_00: Glad to hear more community feedback on that one.
• 26:52
  SPEAKER_00: Um, okay, Rahul, it's been a pleasure talking to you.
• 26:56
  SPEAKER_00: Thank you so much for coming on.
• 26:57
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 27:00
  SPEAKER_02: LinkedIn.
• 27:01
  SPEAKER_02: I think that is one I can also uh support my own company who pays my paycheck and we can get in touch.
• 27:09
  SPEAKER_00: Yes, LinkedIn is a great, this is the great promo for it.
• 27:11
  SPEAKER_00: It's the perfect tool for that.
• 27:13
  SPEAKER_00: It's almost always what people say in response to that question.
• 27:16
  SPEAKER_00: So it really is the best way to get in touch with people.
• 27:20
  SPEAKER_00: Okay, this was a great conversation, Rohul.
• 27:22
  SPEAKER_00: Thanks again for coming on.
• 27:24
  SPEAKER_02: Thank you, Kent, and thank you for your time.
• 27:28
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 27:35
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 27:39
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.