Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / How Airflow orchestration decisions impact Spark performance

Transcript

How Airflow orchestration decisions impact Spark performance · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: You're listening to the data flow.
• 0:04
  SPEAKER_00: And we'll data and AI around orchestrate free online conference and September AI.
• 0:15
  SPEAKER_00: Engineering data and we are high.
• 0:20
  SPEAKER_00: AI work for the new Astronomer Certified AI Orchestration Fundamentals exam.
• 0:28
  SPEAKER_00: Attendees get a discount code for a free certification.
• 0:32
  SPEAKER_00: A$150 value.
• 0:34
  SPEAKER_00: Go to astronomer.link slash data minus flowcast minus OE.
• 0:39
  SPEAKER_00: Link in the show notes.
• 0:41
  SPEAKER_00: Now let's get into the show.
• 0:44
  SPEAKER_02: All right.
• 0:44
  SPEAKER_02: Hi everyone.
• 0:45
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:47
  SPEAKER_02: I'm your host, Kenton Danis, and today I'm joined by Manny Schmilly, who is a co-founder and CEO at Data Flint.
• 0:54
  SPEAKER_02: Many, how are you?
• 0:55
  SPEAKER_01: Lingwell.
• 0:56
  SPEAKER_01: What's about you?
• 0:57
  SPEAKER_02: I'm good.
• 0:58
  SPEAKER_02: Thanks.
• 0:58
  SPEAKER_02: Yeah, it's great to have you here.
• 1:00
  SPEAKER_02: I'm excited to chat today.
• 1:02
  SPEAKER_01: Me too.
• 1:03
  SPEAKER_02: Awesome.
• 1:04
  SPEAKER_02: Well, let's go ahead and dive into it.
• 1:06
  SPEAKER_02: So for today's podcast, we're going to discuss uh how Airflow and Spark fit together.
• 1:11
  SPEAKER_02: So I know a very common topic uh that I hear come up a lot with Airflow users.
• 1:16
  SPEAKER_02: We'll talk about some of the orchestration decisions that shape performance and cost for Spark workloads.
• 1:22
  SPEAKER_02: Uh before we get into that, I want to start with some background.
• 1:24
  SPEAKER_02: So first tell me about Data Flint.
• 1:28
  SPEAKER_01: Yeah, so Data Flint will build production over a agent for Apache Spark, what it means.
• 1:33
  SPEAKER_01: In practice, when you have an Apache Spark job that runs on hundreds of thousands of machine and process petabytes of data, uh you can use a cloud or Kessler to ask it like how you're doing, is it slow?
• 1:44
  SPEAKER_01: Is it like expensive?
• 1:45
  SPEAKER_01: If you have like a crashing or like something to Tabush 2M, how can I solve it?
• 1:50
  SPEAKER_01: Um basically bring this word and context into AI agents.
• 1:55
  SPEAKER_02: Okay, that's great.
• 1:57
  SPEAKER_02: I have very limited Spark experience myself from when I worked as a data engineer.
• 2:01
  SPEAKER_02: I only dabbled into it a little bit, and this was many years ago.
• 2:05
  SPEAKER_02: But I found it sometimes quite difficult to work with, so especially difficult to optimize.
• 2:09
  SPEAKER_02: So I can see where what you're describing would be very useful.
• 2:14
  SPEAKER_02: Yeah, that's great.
• 2:15
  SPEAKER_02: Uh and tell me a little bit about your background.
• 2:18
  SPEAKER_02: And since you're a co-founder, how did you come to start the company?
• 2:22
  SPEAKER_01: Yeah, so basically, also was a sort of data engineer, worked in the data and AI space, and uh we saw a lot of the time of the company.
• 2:30
  SPEAKER_01: Me personally was an Apache Spark, and especially about uh monitoring, performance, troubleshooting things in 2AM and uh Apache Spark and distributed computing in in general, like the art form of taking unstructured data from object storage and processing it on hundreds of machines is really difficult and very complex, uh, and was excited enough about this problem uh to start a company around it.
• 2:54
  SPEAKER_02: Okay, yeah, great.
• 2:56
  SPEAKER_02: And then I'd love to get into more about your experience working with Spark and Airflow together.
• 3:03
  SPEAKER_02: You mentioned you were a data engineer, so I imagine you have used both of these tools.
• 3:07
  SPEAKER_02: Again, I mentioned at the beginning, I think it's very common to hear about them uh being used together and a common pattern where Airflow is used as the primary orchestrator and you know, heavier data processing is offloaded to another tool like Spark.
• 3:22
  SPEAKER_02: Uh, but I'm curious today at Data Flint, where do you see um these two things fitting in like for your customers?
• 3:31
  SPEAKER_02: And you know, why do you think that that combination is so common?
• 3:35
  SPEAKER_01: Yeah.
• 3:36
  SPEAKER_01: So um as we said, like we are uh optimizing Apache Spark.
• 3:40
  SPEAKER_01: Apache Spark is the most popular big data engine around the world.
• 3:44
  SPEAKER_01: If we're you're moving data to place to place and you have data pipeline, in most cases it's Apache Spark under the world, it's what's powered data breaks.
• 3:52
  SPEAKER_01: Um and um, but this this is an engine.
• 3:56
  SPEAKER_01: This is uh like a piece of software that know to move data to place to place.
• 4:00
  SPEAKER_01: It doesn't uh handle when to run it or in which order or what to do with retries.
• 4:05
  SPEAKER_01: For that, we have uh workflows or orchestration tools.
• 4:08
  SPEAKER_01: And Apache Airflow is the most popular one there is today.
• 4:11
  SPEAKER_01: So in most of our customers, they're using Apache Spark for big data processing and using Apache Airflow or uh our astro customers uh to decide which data pi intend to run well in which order, what to paralyze, what to run sequentially, and so on.
• 4:27
  SPEAKER_02: Yeah, totally makes sense.
• 4:28
  SPEAKER_02: Kind of using both things for what they're strongest at.
• 4:32
  SPEAKER_02: So yeah, that's uh definitely an attractive combo there.
• 4:36
  SPEAKER_02: Uh okay, and then I want to get into a little more about the performance aspect that I mentioned at the beginning because I, as I was alluding to, this was certainly something that I found quite challenging uh when working with Spark.
• 4:49
  SPEAKER_02: And I know you have spoken about how the orchestration decisions maybe that you might be making on the Airflow side can actually impact things like Spark performance and cost.
• 5:00
  SPEAKER_02: Uh so what kind of orchestration decisions are you talking about there?
• 5:05
  SPEAKER_01: Yeah, so again we're orchestrating with Apache Airflow, but the actual data processing in running different engines in Apache Spark, but it's one data pipeline.
• 5:14
  SPEAKER_01: It's one piece of uh process that move data from place to place.
• 5:18
  SPEAKER_01: And affecting Apache Airflow in the orchestration level actually could affect the uh performance and what happened in the Apache Spark level.
• 5:26
  SPEAKER_01: So we had the one customer that uh decided to paralyze multiple tasks in uh Airflow because they weren't like uh connected with each other, so why not paralyze it?
• 5:36
  SPEAKER_01: It will run faster, right?
• 5:37
  SPEAKER_01: Maybe a bit more expensive, but faster.
• 5:40
  SPEAKER_01: Um but what ends up happening is that it actually ran slower, it was unstable and it was much, much more expensive.
• 5:47
  SPEAKER_01: Um and what happened is that in each task they're they're all using the same resources.
• 5:53
  SPEAKER_01: So when you run distributed computing, like multiple distributed computing in parallel on the same resources, uh each processing they need to move a lot of data to place to place doesn't have enough resources, don't have enough memory.
• 6:05
  SPEAKER_01: What happened is crashes, is processes moving to use instead of memory disk, which is much slower, and general less instability, where in this case running each uh computation sequentially and giving him all the resources that the cluster had uh could be much faster because everything is in memory and was uh even if there is no dependencies between the different tasks, it was much better.
• 6:28
  SPEAKER_01: On the other end, uh we had customers that run stuff sequentially when they could have paralyzed, and because of each task in the package part is not inefficient enough and not using all the resources that they can, then it's like it's it's both uh slower because insequentially and it's more expensive because there are resources that we are not using, right?
• 6:49
  SPEAKER_01: Calls like just um money we well spend on the calls that are not actually being used.
• 6:54
  SPEAKER_01: So in this case, uh uh parallelizing the parts to to like uh push as much as uh queries and distributed tasks into the resources pool we have is actually much faster, it's much cheaper.
• 7:09
  SPEAKER_02: Okay, interesting.
• 7:10
  SPEAKER_02: Yeah, that makes sense.
• 7:11
  SPEAKER_02: Maybe not necessarily an intuitive trade-off there, but uh yeah, I can see where that would cause some issues.
• 7:20
  SPEAKER_02: How do you recommend that teams you know monitor or benchmark their performance or pipeline performance when they're using Airflow and Spark together?
• 7:29
  SPEAKER_01: Yeah, I think here's the number one mistake that we see is just looking on the Apache Airflow side, just looking in the on your workflow and see if it trends like for two and a half hours or three hours, uh, and not looking on the underlying like what's like when you offload to Apache Spark, what it did and how much it cost.
• 7:47
  SPEAKER_01: So I think that's like critical when you're benchmarking or optimizing or looking at the workflow that using Airflow and spanning Spark is to understand, like have both context and understand not only that this workflow took two hours, that now it's take like one and a half hours, it might now like instead of 100 machines and span like 100, like it costs$100, it's now spanning 1000 machines costs$1,000.
• 8:12
  SPEAKER_01: So I think that's that's really, really critical when working with Apache Airflow.
• 8:17
  SPEAKER_02: Yeah, that makes sense.
• 8:18
  SPEAKER_02: And as a follow-up to that, just curious to clarify.
• 8:21
  SPEAKER_02: So, are you recommending that people do that monitoring within Airflow, or does that happen within Data Flint?
• 8:27
  SPEAKER_02: Is that something that you provide to your customers?
• 8:31
  SPEAKER_01: Yeah, so you could uh try uh build something yourself, like a lot of companies do.
• 8:35
  SPEAKER_01: You can pass your uh Airflow tagging, like in which workflow you are, which run you are through Spark, and then uh build monitoring like back again that tells, oh, for each workflow how much it costs in your Phoenix system, so custom monitoring.
• 8:51
  SPEAKER_01: Uh a data flint, we we have that built in.
• 8:54
  SPEAKER_01: We have uh uh integration for Apache Airflow and for Astro.
• 8:58
  SPEAKER_01: We have Astro customers and we have a native integration for that.
• 9:01
  SPEAKER_01: And then we we know to say for each workflow you have and which workflow how much it costs, how much data processing it was, um was like for each task of Apache Spark was it efficient?
• 9:14
  SPEAKER_01: And what is more exciting is that we know to bring all this data to AI.
• 9:17
  SPEAKER_01: So all the dilemmas and use case that we talked about, we can actually ask AI, like this is two runs, please benchmark me them in both in duration and cost.
• 9:26
  SPEAKER_01: Please like help me with this dilemma of orchestration and uh make uh my Apache Airflow pipeline much better.
• 9:34
  SPEAKER_00: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
• 9:43
  SPEAKER_00: From batch pipelines to training models and wiring up agents.
• 9:47
  SPEAKER_00: Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
• 9:54
  SPEAKER_00: Check it out at astronomer.io.
• 9:57
  SPEAKER_02: So you mentioned that just now that Dataflint has an integration with Airflow or with Astro.
• 10:02
  SPEAKER_02: Um and so other than the monitoring and AI piece that you just mentioned, what does that integration give a team that is orchestrating Spark workflows with Airflow?
• 10:15
  SPEAKER_01: So it gives the ability to like be in your uh code repo and uh with your both like Airflow manifest uh or Astro manifest and uh your Spark code, it's usually the same workflow and be able to give uh your infrastructure and tell like AI, please get the production context from like my actual production run to my AI agent and uh look at all this dilemma and optimize it.
• 10:38
  SPEAKER_01: And once you optimize to be able to say, oh, please compare me this run with the like uh the optimized run and tell me what happened, what did improve, what didn't improve, do we need another iteration?
• 10:49
  SPEAKER_01: Move a lot of the offload uh of this uh optimization, look like it's a holistic view to AI.
• 10:56
  SPEAKER_02: Okay, yeah, that sounds great.
• 10:59
  SPEAKER_02: Uh and then I'm curious to dig in a little more to the AI side of this.
• 11:05
  SPEAKER_02: So uh you've mentioned that Dataflow can compare cost and duration.
• 11:09
  SPEAKER_02: So that might be, I guess, before or after a workflow change.
• 11:13
  SPEAKER_02: Uh just more in in general, kind of backing up a bit, how do you think AI is changing how teams are managing their orchestration and the performance of it?
• 11:23
  SPEAKER_01: Yeah, so what we saw is that the first team is using AI to try to optimize their uh Apache Airflow pipelines.
• 11:30
  SPEAKER_01: So for example, when I worked as a data engineer, we always talked about, oh, we have all these like hundreds of pipelines and they all spawn in like uh 12 pm, and like we have a spike of machines and done, and it would be amazing if someone could like look holistically on our uh Apache Airflow pipelines and optimize like the when we run where to optimize the cost and performance, and not just spans like thousands of machines at 12 pm and then shut them down.
• 11:55
  SPEAKER_01: So that's stuff that the AI make possible, and you can like see the holistic picture and uh uh make the decision.
• 12:02
  SPEAKER_01: And the second thing we see is that uh uh Apache Airflow by itself, like just looking at the duration and what runware is uh is is one context.
• 12:11
  SPEAKER_01: So yes, use uh like Astro MCP, but um there are more contexts or more MCPs you need to connect to real AI to have holistic view.
• 12:20
  SPEAKER_01: So you need uh like context for Apache Spark, for your FinOps, for your cloud environment.
• 12:26
  SPEAKER_01: Um like you need context for multiple places for production to see like one holistic view and uh uh for AI to be able to do this optimization and troubleshooting for you.
• 12:37
  SPEAKER_02: Yeah, yeah, very interesting.
• 12:40
  SPEAKER_02: Uh that's great.
• 12:41
  SPEAKER_02: Um, have you seen with your customer base kind of a like I don't know, some benchmarks on how much this has improved for them from a cost and performance standpoint?
• 12:53
  SPEAKER_02: You don't have to share specific numbers if you can't, but just curious.
• 12:58
  SPEAKER_01: Yeah, we we had a customer that used like uh um fully automated uh migrating from uh Databricks to EMR.
• 13:06
  SPEAKER_01: Part of it was also to change all the orchestration, all the operator of Apache Airflow.
• 13:10
  SPEAKER_01: They managed to build a full uh automation of this process, meaning take the manifest change like the like the runs from uh Databricks to EMR, then um run the pipeline.
• 13:24
  SPEAKER_01: Uh they're actually like not only using uh Apache Airflow, but uh they're using Astro deployment.
• 13:29
  SPEAKER_01: Um then they uh looked at the performance with data sprint or like what happened before and after, if there were any error, fixed it, and then like uh compared the before and after.
• 13:39
  SPEAKER_01: We released with them a use case where we managed to uh reduce cost by 80% in their workflow as part of this uh migration process.
• 13:49
  SPEAKER_01: So like you can get amazing.
• 13:53
  SPEAKER_01: We also like have use cases in the same company.
• 13:55
  SPEAKER_01: We got 100x.
• 13:57
  SPEAKER_01: That's the most important impressive part of uh big data optimization.
• 14:01
  SPEAKER_01: Like once you understand the underlying compute, you can do like uh like and and actually optimize the distribution part.
• 14:08
  SPEAKER_01: It could be sometimes 10x, 100x.
• 14:10
  SPEAKER_01: Like there is no limit how much you can optimize stuff.
• 14:13
  SPEAKER_02: Yeah, that's wild.
• 14:14
  SPEAKER_02: Uh it's also it's crazy to kind of think about that if you're just writing these pipelines without knowing anything about this, like just how much really the cost impact can be if if you're not doing this optimization.
• 14:27
  SPEAKER_02: That's crazy.
• 14:29
  SPEAKER_01: Definitely.
• 14:29
  SPEAKER_01: There are not a lot of like big data is super complex.
• 14:33
  SPEAKER_01: There are not a lot of uh apartispark uh experts.
• 14:36
  SPEAKER_01: Like I just met with one of the like Apartis Spark maintenance and Francis, like there are probably hundreds that know, like really to to do this kind of stuff.
• 14:43
  SPEAKER_01: And uh if we could you leverage AI to to help us with that and actually take a look and uh be able to see all these uh dilemmas and optimize for us, like it's like uh the sky's the limit.
• 14:56
  SPEAKER_02: Yeah, that's great.
• 14:57
  SPEAKER_02: Um, very, very cool.
• 14:59
  SPEAKER_02: And then it's kind of related.
• 15:02
  SPEAKER_02: I'm gonna do my final question in two parts this time.
• 15:05
  SPEAKER_02: So I'm I'm curious if there's anything you think about like the airflow and spark story together that could be better overall than it is today.
• 15:15
  SPEAKER_02: And maybe that's related to the AI um piece or maybe not.
• 15:21
  SPEAKER_01: Yeah.
• 15:22
  SPEAKER_01: So in the end, like what we want as like a like users is data pipeline.
• 15:28
  SPEAKER_01: And the fact we have two products, one like doing the processing Spark and one is doing the orchestration is nice.
• 15:34
  SPEAKER_01: But uh as discussed in this uh like uh here, like a lot of times the dilemmas are uh intertwined.
• 15:40
  SPEAKER_01: So the Apache Spark community and like uh the maintainers in Spark 4.1 uh released a feature called uh Spark Decterapy Decterapy Pipeline.
• 15:49
  unknown: Uh-huh.
• 15:49
  SPEAKER_01: It's a cool feature that Spark's trying to start doing some orchestration or what to like in which order to run things in like with understanding of the distributed computing aspect.
• 16:01
  SPEAKER_01: Uh it's still using Airflow, like it's still better together, but a bit different.
• 16:06
  SPEAKER_01: And what I hope to see is uh the same thing from the Airflow side.
• 16:09
  SPEAKER_01: Could Airflow understand peritasks and get from the Apache Spark engine some metrics or some insights?
• 16:16
  SPEAKER_01: So these two like amazing open source projects that basically runs all of the data pipelines in the world can work a bit better together and give us uh better results.
• 16:28
  SPEAKER_02: Yeah, that totally makes sense.
• 16:30
  SPEAKER_02: I can see where that would be so useful.
• 16:33
  SPEAKER_02: I I guess in my experience, it's relatively rare that people aren't doing big enough data processing that they need something or at least have to manage that in some way beyond sort of just natively doing it on their normal Airflow worker.
• 16:48
  SPEAKER_02: So I think that would have a huge impact.
• 16:51
  SPEAKER_02: Uh and then related, so my standard final question that I asked to all my guests is what would you most like to see from the Airflow project in the near future?
• 16:59
  SPEAKER_02: So maybe that's just a little more detail on what you just said on what that would mean within the Airflow project.
• 17:05
  SPEAKER_01: Yeah.
• 17:06
  SPEAKER_01: I think Apache Airflow understanding better Spark, and I know like Apache Airflow maintenance working on that.
• 17:12
  SPEAKER_01: Like understanding that it's run, like because it's so popular, it's like basically runs the entire world.
• 17:17
  SPEAKER_01: Like better understanding of Apache Airflow when it runs Spark, like what's happening under the hood.
• 17:23
  SPEAKER_01: I think that could be a real game changer.
• 17:25
  SPEAKER_02: Yeah, definitely.
• 17:26
  SPEAKER_02: That's great.
• 17:27
  SPEAKER_02: Well, lots of exciting things to look forward to in the Airflow project.
• 17:31
  SPEAKER_02: We can add that one to the list.
• 17:33
  SPEAKER_02: Uh okay.
• 17:34
  SPEAKER_02: Well, many, thank you so much for going through that.
• 17:36
  SPEAKER_02: This has been really interesting.
• 17:38
  SPEAKER_02: Uh again, it's a topic that I haven't spoken about very much, um, at least on the podcast, and don't dabble into myself that much anymore.
• 17:46
  SPEAKER_02: So it's been really cool to hear about.
• 17:48
  SPEAKER_02: Uh, just to close us out, what is the best way for folks to get in touch with you?
• 17:53
  SPEAKER_01: Yeah, you can uh DM me at uh LinkedIn.
• 17:55
  SPEAKER_01: It's uh Manish Moe Dataflame.
• 17:58
  SPEAKER_01: Uh you can uh like uh contact us via the website, dataflint.io.
• 18:03
  SPEAKER_01: And the best way if you're uh joining the Airflow Summit in September is uh like to meet in person.
• 18:08
  SPEAKER_01: Like we have a booth.
• 18:09
  SPEAKER_01: I'm giving uh a talk about the Apache Airflow and Apache Spark.
• 18:13
  SPEAKER_01: So please say hi.
• 18:15
  SPEAKER_02: Yeah, that's great.
• 18:16
  SPEAKER_02: Huge plug for the Airflow Summit.
• 18:18
  SPEAKER_02: Uh tickets are available.
• 18:20
  SPEAKER_02: I think they will still be available at the time that we're publishing this.
• 18:23
  SPEAKER_02: Uh so we will throw a link to those in the show notes.
• 18:26
  SPEAKER_02: That's gonna be in Austin, Texas, at the very, very tail end of August, early September this year.
• 18:31
  SPEAKER_02: So I'm very excited to see you there.
• 18:33
  SPEAKER_02: I will be there as well giving a workshop um also on AI and Airflow.
• 18:37
  SPEAKER_02: So uh yeah, that's great.
• 18:39
  SPEAKER_02: Okay, many, well, thank you so much again for joining.
• 18:42
  SPEAKER_02: It's been really great speaking with you.
• 18:46
  SPEAKER_00: Thank you for listening to the Data Flowcast.
• 18:49
  SPEAKER_00: Check the show notes for links to everything mentioned today.
• 18:52
  SPEAKER_00: If you're enjoying the podcast, subscribe so you never miss an episode.
• 18:57
  SPEAKER_00: And leave us a five-star review.