Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building a Unified Data Platform at Pattern with William Graham

Transcript

Building a Unified Data Platform at Pattern with William Graham · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: I can then determine whether or not other tasks are run or if they're skipped or if they fail.
• 0:04
  SPEAKER_02: That's been really important for setting up our DAG.
• 0:07
  SPEAKER_02: So we've been really happy with that.
• 0:08
  SPEAKER_02: I don't know if there's any one thing I would say necessarily as far as future development.
• 0:14
  SPEAKER_02: We've just been really happy with what's there and really happy that there is a big community.
• 0:18
  SPEAKER_02: A lot of us are active on the Slack Airflow channel and we go there to ask questions and get input.
• 0:25
  SPEAKER_02: And it's always been like a really positive community to be a part of.
• 0:29
  SPEAKER_02: And we're really grateful for that.
• 0:36
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:46
  SPEAKER_00: Here, Airflow community, we've got exciting news.
• 0:49
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:52
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow and grab a free Airflow 3 fundamentals or DAG Authoring certification worth$150 each.
• 1:03
  SPEAKER_00: Plus, you can enter to win an exclusive workshop with Mark Lumberti.
• 1:07
  SPEAKER_00: Don't miss out.
• 1:08
  SPEAKER_00: The survey link is in the show notes and closes November 21st.
• 1:14
  SPEAKER_00: Hi everyone.
• 1:15
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:17
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Will Graham, who is a senior data engineer at Pattern.
• 1:24
  SPEAKER_00: Will, how are you doing?
• 1:25
  SPEAKER_02: Doing well.
• 1:26
  SPEAKER_02: Thanks, Kenton.
• 1:27
  SPEAKER_00: That's great.
• 1:28
  SPEAKER_02: How are you doing?
• 1:29
  SPEAKER_00: Yeah, I'm good.
• 1:30
  SPEAKER_00: I'm good.
• 1:30
  SPEAKER_00: Enjoying the last bits of summer here in the Pacific Northwest.
• 1:34
  SPEAKER_02: Oh, yeah, that's great.
• 1:35
  SPEAKER_02: It's been still really hot here, but I'm looking forward to the slightly cooler weather.
• 1:41
  SPEAKER_00: Okay.
• 1:41
  SPEAKER_00: Yeah, I think by the time this comes out, we should be well on our way to fall weather, hopefully everywhere.
• 1:48
  SPEAKER_00: But we'll see.
• 1:49
  SPEAKER_00: All right.
• 1:49
  SPEAKER_00: Well, let's go ahead and dive in.
• 1:51
  SPEAKER_00: So for today's podcast, we're going to discuss how pattern decouples scheduling and orchestration.
• 1:57
  SPEAKER_00: Before we get into all that, let's start with some background.
• 2:00
  SPEAKER_00: Can you tell us what Pattern does?
• 2:02
  SPEAKER_02: So pattern is what the industry calls an e-commerce accelerator.
• 2:08
  SPEAKER_02: And so basically we partner with a lot of different brands and we help them grow.
• 2:15
  SPEAKER_02: Similar to maybe an agency or something of that nature.
• 2:20
  SPEAKER_02: But often what we do is we will buy products from these brands.
• 2:24
  SPEAKER_02: And then so the brand gets to focus on the product itself.
• 2:29
  SPEAKER_02: And then we take it and we market it, we advertise it, we do all the logistics and we distribute it throughout the world.
• 2:37
  SPEAKER_02: So that's more or less what pattern is there.
• 2:40
  SPEAKER_00: Okay, great.
• 2:41
  SPEAKER_00: And what does your team do?
• 2:43
  SPEAKER_02: Yeah, so I work in the data organization, data team.
• 2:48
  SPEAKER_02: And right now we're split up into three different teams.
• 2:52
  SPEAKER_02: We do, we have a data acquisition team, we have a more traditional data engineering team, and also a data platform team.
• 3:01
  SPEAKER_02: And so I have been working on the data platform and acquisition teams.
• 3:06
  SPEAKER_02: So we're responsible for actually acquiring all of the data that's used across our business to make data-driven decisions for the brands that we partner with.
• 3:17
  SPEAKER_02: And then the platform team is largely to craft a really good user experience for everyone who uses that data, make it easy to access, make it easy to use across all the different applications that we have, and overall just make the experience better for those who are interacting with all the data we collect.
• 3:38
  SPEAKER_00: Okay, great.
• 3:39
  SPEAKER_00: And I imagine since it's split into several different teams with different responsibilities, that you're working at a pretty large scale with the data that you're managing.
• 3:49
  SPEAKER_02: We collect a lot of data.
• 3:50
  SPEAKER_02: We operate on, I think, around 60 different online marketplaces.
• 3:56
  SPEAKER_02: And so we're constantly collecting data from all of those different marketplaces.
• 4:02
  SPEAKER_02: I think in our last conference, they said somewhere around 35 million data points or something like that.
• 4:10
  SPEAKER_02: I haven't done the math exactly, so you can't hold me to that.
• 4:13
  SPEAKER_02: But it is a lot of data that we collect that provides the foundational layer for basically all of the applications that we have that we make accessible to our brands.
• 4:24
  SPEAKER_00: Yeah, makes sense.
• 4:25
  SPEAKER_00: And where does Airflow fit into that?
• 4:27
  SPEAKER_02: So Airflow, as you can imagine, we run a lot of batch jobs, particularly here at Patent.
• 4:35
  SPEAKER_02: And so Airflow provides a really clean interface for being able to schedule and run those jobs.
• 4:43
  SPEAKER_02: So we've been using Airflow for probably about three years now.
• 4:47
  SPEAKER_02: And it really has been game-changing for us.
• 4:50
  SPEAKER_02: It scaled very quickly here when people started using it.
• 4:54
  SPEAKER_02: I think the user experience of being able to schedule jobs in a central location was popular among a lot of the teams here.
• 5:03
  SPEAKER_02: And so we went from like a single Airflow instance to several pretty quickly and a lot of teams adopting it.
• 5:12
  SPEAKER_02: It's played a pretty integral part in orchestrating the jobs that we're running here.
• 5:16
  SPEAKER_00: That totally makes sense.
• 5:17
  SPEAKER_00: So I want to dig in a little bit to the decoupling that you've done on the scheduling and orchestration piece, because I think for a lot of teams, Airflow serves both use cases.
• 5:29
  SPEAKER_00: But I know you've implemented this architecture where you use Airflow in combination with an internal tool that you've built so that those combined are meeting that scheduling and orchestration need.
• 5:40
  SPEAKER_00: So tell me a little bit more about this architecture and how it works.
• 5:43
  SPEAKER_02: Yeah, and thanks for giving us this opportunity.
• 5:45
  SPEAKER_02: So we built a tool that we've called Heimdall, and we've open sourced it.
• 5:49
  SPEAKER_02: So it's now hopefully not just an internal tool, but you can go check it out on GitHub and play around with it.
• 5:57
  SPEAKER_02: And essentially what Heimdall is, it's a lightweight orchestrator that abstracts the infrastructure away from the client and the users, and also provides a very unified interface for all of the users and clients to access that infrastructure.
• 6:20
  SPEAKER_02: And so essentially what that has done is it's made it, it's given Airflow a narrower but just as important role as the scheduler within our framework.
• 6:31
  SPEAKER_02: And so now with these two technologies combined, the way we're orchestrating jobs is we use Airflow almost exclusively as the scheduler.
• 6:43
  SPEAKER_02: So it answers the question of when.
• 6:50
  SPEAKER_02: And then Airflow will call Heimdall through a series of small operators, and then Heimdall determines the what, the where and the how of that job being executed.
• 7:05
  SPEAKER_02: So it decouples those responsibilities a little bit.
• 7:08
  SPEAKER_02: Actually, the inspiration came from a project that Netflix has worked on called Genie, that is also a type of orchestrator.
• 7:18
  SPEAKER_02: There are a few key differences in how we built Time Doll that make it slightly different in its implementation and use.
• 7:26
  SPEAKER_02: But the overall concept, the high-level idea, is that Airflow does the scheduling, and then you have a different orchestrator that does the actual distribution of the workload to the different services that you want to communicate with.
• 7:42
  SPEAKER_00: Interesting.
• 7:43
  SPEAKER_00: And so before you built and implemented Heimdall, did you use Airflow for both scheduling and orchestration originally?
• 7:51
  SPEAKER_02: Yeah.
• 7:52
  SPEAKER_02: So we used Airflow.
• 7:53
  SPEAKER_02: It was a one-stop shop for both scheduling and orchestration.
• 7:58
  SPEAKER_02: We had a really wide use case for how we were using Airflow across our organization.
• 8:06
  SPEAKER_02: And I don't think there's necessarily one solution or one answer to how to run the infrastructure or your data platform at any one company.
• 8:15
  SPEAKER_02: It's really dependent on what your needs are and what problems you're facing individually.
• 8:21
  SPEAKER_02: But for us, we felt like the decoupling, there were some key reasons for us to decouple that were critical and would help us elevate the user experience of interacting with data at pattern.
• 8:34
  SPEAKER_02: That was the reason we decided to change that architecture.
• 8:39
  SPEAKER_02: But yes, before we were using it as for both.
• 8:42
  SPEAKER_00: Got it.
• 8:43
  SPEAKER_00: And are you able to dig in a little more to those reasons?
• 8:46
  SPEAKER_00: What were the issues that you were having that kind of prompted you to make this decoupling?
• 8:52
  SPEAKER_02: Sure, yeah.
• 8:53
  SPEAKER_02: There are a lot of reasons.
• 8:54
  SPEAKER_02: I'll talk about a few of them that are most relevant.
• 8:58
  SPEAKER_02: Probably one of the top ones was credential management.
• 9:02
  SPEAKER_02: For example, if we have five different teams and they all have Airflow instance that they're working on and they're all interacting with, say, like 15 or 20 different services, whether that's Spark or Dynamo or whatever service, they all have to have some kind of connection or credential connection in order to interact with those services.
• 9:26
  SPEAKER_02: And so what ends up happening usually is those connections are brought down onto the client or onto each person's laptop in order to run Airflow locally to say, even if you have a staging instance of your application or a staging a Spark cluster or whatever, you're still bringing those credentials onto your local and making a call in that way.
• 9:51
  SPEAKER_02: And so one of the ways that Heimdall solves that is that there's now a single entry point into all of add-on infrastructure, and that's through Heimdall.
• 10:04
  SPEAKER_02: And so when someone makes a request from local, all it does is it sends their user or their identity.
• 10:12
  SPEAKER_02: So in my case, it would say, Hey, I'm running this local DAG, and my name's Will.
• 10:17
  SPEAKER_02: And it sends that to Heimdall.
• 10:18
  SPEAKER_02: And then as that single point of entry, Heimdall can then say, Oh, let's look at what you have access to.
• 10:25
  SPEAKER_02: Yeah, you can access the ad hoc cluster from local, but you can't access the production cluster.
• 10:31
  SPEAKER_02: And it makes it so that I no longer have to keep any credentials to access those resources.
• 10:38
  SPEAKER_02: And Heimdall is able to tell me a based on your identity, these are the resources that I want you to access.
• 10:47
  SPEAKER_02: And same from a service perspective, right?
• 10:50
  SPEAKER_02: So if it were a service that needed to access Spark or uh some other deployed resource that we have, then that service could communicate its identity to Iand all, which could then say, hey, you do or don't have access.
• 11:06
  SPEAKER_02: And it's a service, so it can have a credential on its machine, but for anyone who's running it locally, then we eliminate the need to be passing around all of the relatively secure information.
• 11:18
  SPEAKER_02: So it's a security problem that it solves as well as a simplification for not having to pass around those credentials everywhere.
• 11:25
  SPEAKER_02: So that's one of the reasons it's made that process a lot more simple.
• 11:31
  SPEAKER_02: Another big reason I would say is the resources that we access via Airflow are also resources that we access in our other applications.
• 11:42
  SPEAKER_02: So if I have like a piece of software that we've written and it needs to run a query against a warehouse or something else, then it still has to know how to query those resources.
• 11:57
  SPEAKER_02: And Airflow, which is also a client, also has to know how to query those resources.
• 12:01
  SPEAKER_02: Because of that, now I have two clients or more, however many applications that need to talk to a service.
• 12:08
  SPEAKER_02: They all have to know what are my credentials to access that service, how do I interact with that service?
• 12:14
  SPEAKER_02: There's a bunch of questions basically that client has to understand in order to make the communication clear with our resource.
• 12:22
  SPEAKER_02: When you have a service like Time Doll, Airflow just becomes one of the clients that talks with that service.
• 12:31
  SPEAKER_02: And now, if somebody needs to query Spark from Airflow, or they need to query it from CMI, this other application that I've built, then they use the exact same interface and they're really just talking to one place, which is Heimdall, and then Heimdall is the only thing that needs to know how to interact with those services.
• 12:53
  SPEAKER_02: So that became a particular pain point, I think probably does for most people as you're scaling, and you have a lot of different applications that are needing to share the same resources, and it becomes difficult or even a learning curve for every application, having to rebuild this process of being able to communicate with that resource.
• 13:16
  SPEAKER_02: And so onboarding and being able to facilitate that communication comes more easily now because we have a system like Heimdall.
• 13:26
  SPEAKER_02: Uh, so those are a couple of the reasons.
• 13:29
  SPEAKER_02: Another one is I'm sure you're aware, and as you've talked to people, whether or not people follow best practice when writing DAGs, but I've seen some pretty hairy, like gnarly operators out there that people have written and really complex DAGs that are really hard to understand.
• 13:48
  SPEAKER_02: One of the beautiful things about having a system like Heimdall, and that makes working with Airflow so much nicer, is we have that common interface.
• 13:58
  SPEAKER_02: So if you can imagine an inheritance pattern where the lowest level operator that every other operator inherits from is just the Heimdall operator, and it provides the interface that says this is how the API structure is.
• 14:12
  SPEAKER_02: And then with every operator that we add to Airflow, it's only like 10, maybe 15 lines of code, and it's just changing, tweaking little parts of that, saying, Oh, instead of running this on Spark, I want to run this on Trino, or I want to run it in ClickHouse or some other resource.
• 14:31
  SPEAKER_02: So now when we go to our Airflow repository, it's this really clean.
• 14:35
  SPEAKER_02: Each operator is only 10 or 15 lines of code, and onboarding a new operator to access a different resource becomes really straightforward.
• 14:48
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 14:57
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 15:09
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 15:17
  SPEAKER_00: So in terms of the pipeline authors who are looking to schedule something in Airflow, they're just making use of some heimdall operators that you have created that do everything needed behind the scenes so that they can just, I assume, provide like the logic of whatever if it's Python or SQL or something that they're running.
• 15:36
  SPEAKER_02: Is that right?
• 15:37
  SPEAKER_02: Yeah, that's right.
• 15:38
  SPEAKER_02: And I think someone listening could say, are you just changing the workload from writing a complex operator to writing a complex plugin in Heimdall, for example?
• 15:47
  SPEAKER_02: It's a pluggable architecture, right?
• 15:49
  SPEAKER_02: So if I want to communicate with some resource, then I have to go build a plugin in Heimdall that will know how to communicate there.
• 15:56
  SPEAKER_02: And that's true.
• 15:57
  SPEAKER_02: But once you build it in Heimdall, now you have access to it in other applications as well.
• 16:04
  SPEAKER_02: So for example, if I have a real-time application, it's really unlikely that I'm gonna hit the Airflow API to kick off some DAG that's going to return a result to me.
• 16:17
  SPEAKER_02: That's just not how Airflow is structured and it's not supposed to, and it was built like that for a reason.
• 16:22
  SPEAKER_02: But once that logic for how to communicate with that resource is in Heimdall, how I can communicate with it through Airflow.
• 16:32
  SPEAKER_02: If I want to run like a batch schedule job, or I can use that same logic, that reusable component, and communicate with it through any other service that I build.
• 16:44
  SPEAKER_02: So you're still building the communication logic like that the operator would be, but it's a little bit more flexible and provides more options for that communication.
• 16:56
  SPEAKER_00: Yeah, I totally see the benefit of that.
• 16:59
  SPEAKER_00: And now that you've had the solution, I imagine for some period of time, is it stable?
• 17:05
  SPEAKER_00: Has it been fully implemented, or are there like optimizations you're looking to make?
• 17:09
  SPEAKER_00: Or where are you kind of in that timeline?
• 17:12
  SPEAKER_02: Yeah, we're pretty early, obviously.
• 17:14
  SPEAKER_02: I think we open sourced timed all, you know, maybe like a month or two ago.
• 17:19
  SPEAKER_02: So we're trying to gain a little bit of traction.
• 17:21
  SPEAKER_02: We'd love for people to go check it out, become contributors if they have resources that they're talking to, that to build a plugin that will support that communication.
• 17:34
  SPEAKER_02: As far as our implementation here at Pattern, it's definitely growing.
• 17:38
  SPEAKER_02: We're probably using five or six different plugins in production.
• 17:43
  SPEAKER_02: So different resources that we're communicating with through Heimdall scheduled via Airflow.
• 17:49
  SPEAKER_02: And it's been a really great process.
• 17:51
  SPEAKER_02: We've also already had some contributions from a few different companies to the Heimdall project.
• 17:57
  SPEAKER_02: And so we're really excited to see that.
• 17:59
  SPEAKER_02: And it's another one of the great things about open source is a lot of the work that we would need to do anyway, someone else is likely to also need.
• 18:09
  SPEAKER_02: And so we get to support the community, and the community gets to support us and hopefully accelerate some of the work that we want to do.
• 18:19
  SPEAKER_00: That's great.
• 18:20
  SPEAKER_00: I love that you open sourced it.
• 18:22
  SPEAKER_00: What a cool thing to contribute back to the community.
• 18:25
  SPEAKER_00: Uh, I'm curious, where did the name Heimdall come from?
• 18:28
  SPEAKER_02: Well, I have to mention, so my manager, Stan Dagrine, if you don't already know him, you should go look him up on LinkedIn or something.
• 18:37
  SPEAKER_02: This was his inspiration.
• 18:39
  SPEAKER_02: And Heimdall is I think a Norse mythic god.
• 18:45
  SPEAKER_02: I don't know if you've seen like Avengers or I'm sure he's portrayed in a lot of movies, but the guy who like sits at the entry to and gets the bifrost or or something like that.
• 18:55
  SPEAKER_02: But he's basically the guardian of the gate to be able to get into this community.
• 19:01
  SPEAKER_02: And so the idea is similar that Heimdall is the gate through which all of our users and systems go in order to access all of the infrastructure and the different systems that we have in place for them to use.
• 19:19
  SPEAKER_02: And it's really a beautiful thing when you have like a one way to get in.
• 19:24
  SPEAKER_02: There's a lot of really interesting things that you can start doing when you have a single entry point.
• 19:31
  SPEAKER_02: One that I'll mention that is in production now, but will hopefully enhance it later, is we can monitor exactly the kind of load that all of our clusters are experiencing and from which applications pretty easily now that everything's going through this single entry point.
• 19:54
  SPEAKER_02: So what that allows us to do is say if I receive this request, I can now strategically route it to a cluster depending on maybe the health of the cluster, or maybe the load of the cluster, or maybe I have certain weights in place depending on the user or the service that wants to make that request.
• 20:17
  SPEAKER_02: All of that becomes visible to me now that I can have all of that in a single location.
• 20:23
  SPEAKER_02: So I think that's probably where the idea of High-N came in.
• 20:27
  SPEAKER_02: And it's not particularly novel.
• 20:30
  SPEAKER_02: I was doing some research a while back, and there's a lot of security companies and other similarly structured projects that are called this for that reason, I think.
• 20:42
  SPEAKER_00: Interesting.
• 20:43
  SPEAKER_00: Cool.
• 20:43
  SPEAKER_00: This has been really fun to learn about, and I'm excited to continue to follow it and see where it goes.
• 20:49
  SPEAKER_00: In the interest of time, to direct a little bit back towards the scheduling piece, I always end these episodes with a standard final question for everybody.
• 20:57
  SPEAKER_00: And that's what would you most like to see from the Airflow project in the near future?
• 21:01
  SPEAKER_02: That's a great question, to be completely honest.
• 21:04
  SPEAKER_02: We have been really happy with Airflow as it stands.
• 21:07
  SPEAKER_02: We haven't actually upgraded yet to 3.0.
• 21:10
  SPEAKER_02: We're letting it simmer a little bit, but we've heard a lot of really great things about some of the features that Airflow 3.0 has come out with.
• 21:18
  SPEAKER_02: But from a scheduling perspective, I think we've all been really happy, really flexible with how we can do the scheduling, the kinds of dependencies we can set up have been really important.
• 21:28
  SPEAKER_02: So, like trigger rules as one example, where we have this task, and if this task, depending on the outcome of this task, I can then determine whether or not other tasks are run or if they're skipped or if they fail.
• 21:43
  SPEAKER_02: That's been really important for setting up our DAG.
• 21:46
  SPEAKER_02: So we've been really, really happy with that.
• 21:48
  SPEAKER_02: I don't know if there's any one thing I would say necessarily as far as future development.
• 21:56
  SPEAKER_02: We've just been really happy with what's there and really happy that there is a big community.
• 22:01
  SPEAKER_02: A lot of us are active on the Slack Airflow channel, and we go there to ask questions and get input.
• 22:10
  SPEAKER_02: And it's always been like a really positive community to be a part of.
• 22:15
  SPEAKER_02: And we're really grateful for that.
• 22:17
  SPEAKER_02: We know that Astronomer has done a lot to advance airflow in general and contribute to the open source project.
• 22:28
  SPEAKER_02: And so really grateful for that and hopefully returning some of that to the community by producing another project that can work well and tandem with airflow.
• 22:39
  SPEAKER_00: I love it.
• 22:39
  SPEAKER_00: I think you're my first guest who has answered that question with nothing.
• 22:44
  SPEAKER_00: And I'm actually really happy about it.
• 22:46
  SPEAKER_02: Maybe I need to dive in a little bit more and I'm sure there's things, but really we're happy, especially when we've decided, hey, we're gonna really focus in on just the scheduling piece, and we're not looking to try and use like all of the bells and whistles that we know Airflow is capable of using.
• 23:04
  SPEAKER_02: As far as scheduling, we think it's pretty unrivaled in orchestration space.
• 23:08
  SPEAKER_02: And so it wasn't a difficult decision for us to say, like, yeah, Airflow's the right tool for us.
• 23:13
  SPEAKER_00: Yeah, that's great.
• 23:14
  SPEAKER_00: I think no requests is a completely valid response.
• 23:18
  SPEAKER_00: All right.
• 23:19
  SPEAKER_00: Well, thanks so much for coming on.
• 23:21
  SPEAKER_00: It's been great speaking with you.
• 23:22
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 23:25
  SPEAKER_02: You can reach out on LinkedIn, anyone with a data title.
• 23:30
  SPEAKER_02: We really have a fantastic team here at Patterns.
• 23:32
  SPEAKER_02: So if you have any questions about Heimdall in general or how we use Heimdall with airflow, we'd be more than happy to answer those questions.
• 23:41
  SPEAKER_00: Okay, great.
• 23:42
  SPEAKER_00: Thanks again for chatting today.
• 23:44
  SPEAKER_00: This has been super interesting.
• 23:45
  SPEAKER_02: Yeah, thank you so much, Kenton, for having me on.
• 23:50
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:56
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:00
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.