Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Orchestrating AI video intelligence and evaluation pipelines at Firework

Transcript

Orchestrating AI video intelligence and evaluation pipelines at Firework · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: Uh you also need to know whether the model output was actually good because a lot of times the model gives you very um confident but incorrect answers.
• 0:11
  SPEAKER_03: Uh and those are extremely difficult to detect.
• 0:15
  SPEAKER_02: You're listening to the Data Flowcast, the podcast about Apache Airflow and the world of data and AI around it.
• 0:23
  SPEAKER_02: Let's get into it.
• 0:25
  SPEAKER_00: All right.
• 0:25
  SPEAKER_00: Hi, everyone.
• 0:26
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:28
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Sean Feng, who is the head of data at Firework.
• 0:34
  SPEAKER_00: Sean, how are you?
• 0:36
  SPEAKER_03: Good, good.
• 0:36
  SPEAKER_03: Thanks for having me.
• 0:38
  SPEAKER_00: Yeah, thanks for joining us.
• 0:39
  SPEAKER_00: It's really great to have you on.
• 0:41
  SPEAKER_00: I'm very excited for this conversation.
• 0:43
  SPEAKER_00: I think we're gonna cover a pretty hot topic in the world of Airflow and orchestration.
• 0:48
  SPEAKER_00: Uh, we'll discuss how Firework implements AI workflows for Airflow.
• 0:52
  SPEAKER_00: Um, so something I think a lot of heads of data and data teams are grappling with right now.
• 0:58
  SPEAKER_00: Uh, but yeah, I guess before we get into the nitty-gritty all of that, let's, you know, back up a bit, start with some background.
• 1:05
  SPEAKER_00: Uh, maybe first you can tell me a little bit about the company you work for, Firework, and what you do.
• 1:11
  SPEAKER_03: Sure, yeah.
• 1:12
  SPEAKER_03: Yeah, absolutely.
• 1:12
  SPEAKER_03: Um, so so Firework is a um um leading video commerce um platform that supports hundreds of brands and retailers globally.
• 1:22
  SPEAKER_03: Uh, we help brands bring short-form shop videos um and live stream experience directly onto their own websites and apps.
• 1:30
  SPEAKER_03: So you can think of, you know, instead of sending users to platforms like TikTok or Instagram, uh, we basically bring the same level of engagement back to where the transactions actually happen.
• 1:43
  SPEAKER_03: Uh, and more recently, we've been uh investing very heavily in video intelligence and conversational experiences to make those interactions more personalized, contextual, and also interactive.
• 1:57
  SPEAKER_00: Okay, very cool.
• 1:58
  SPEAKER_00: And just from my own understanding, so I guess to summarize, you're helping companies host videos within their own sites, but obviously that takes a lot of technical backend compared to a lot of web design.
• 2:10
  SPEAKER_00: Is that fair categorization?
• 2:13
  SPEAKER_01: Yeah, yeah.
• 2:14
  SPEAKER_00: Okay, yeah, very cool.
• 2:16
  SPEAKER_00: Um, I can see where that would be super useful.
• 2:19
  SPEAKER_00: We have managed some of that stuff ourselves with the discussion of you know hosting videos elsewhere versus on our own platform.
• 2:26
  SPEAKER_00: So yeah, cool.
• 2:27
  SPEAKER_00: Uh and what do you you and your team do specifically?
• 2:31
  SPEAKER_03: Yeah, so so um I lead the data in AI platform team at Firework.
• 2:35
  SPEAKER_03: Uh so we are responsible for for pretty much the uh full stack of data uh from data ingestion, storage, all the way to transformation, uh, both real-time and batch, and then of course into uh BI analytics and and the reporting stacks as well.
• 2:52
  SPEAKER_03: Uh on top of that, we also work on applied AI, uh things like integrating AI into our products, building data pipelines to support those systems and making sure everything is measurable and improving over time.
• 3:06
  SPEAKER_03: So it's a mix of platform analytics as well as AI, um, but all built on, you know, top of a pretty strong data foundation.
• 3:16
  SPEAKER_00: Okay.
• 3:16
  SPEAKER_00: Yeah, interesting.
• 3:18
  SPEAKER_00: And tell me a little bit more about where Airflow fits into all of that.
• 3:22
  SPEAKER_03: Sure, yeah, yeah, that's a good question.
• 3:24
  SPEAKER_03: So um, yeah, just a little bit uh background history.
• 3:29
  SPEAKER_03: We actually adopted Airflow pretty early on, uh, not long after it came out as an open source project.
• 3:36
  SPEAKER_03: Uh since then, it's you know really become our main orchestration layer for all of our batch transformation workflows.
• 3:43
  SPEAKER_03: Um, and that includes you know, things like business insights and analytics transformations, uh, data platform automations, and and also more recently, a lot of our AI-related workflows as well, uh, like conversational insights, um, knowledge-based building, uh, and many automated evaluation pipelines as well.
• 4:05
  SPEAKER_00: Interesting.
• 4:06
  SPEAKER_00: So you've been using Airflow since the 1.x days then.
• 4:11
  SPEAKER_01: Yes.
• 4:12
  SPEAKER_00: Yeah, cool.
• 4:13
  SPEAKER_00: Uh, probably even longer than I have.
• 4:15
  SPEAKER_00: I started in like 2019.
• 4:17
  SPEAKER_00: I think it had been around for a couple of years at that point.
• 4:20
  SPEAKER_00: So I still consider that pretty early now in the uh history of the project.
• 4:24
  SPEAKER_00: But uh yeah, that's cool that you guys are such longtime users.
• 4:29
  SPEAKER_00: Well, with that said, I would love to dig in a little bit more, as I mentioned before, to the AI workflows that you've implemented with Airflow.
• 4:37
  SPEAKER_00: Of course, this is, as I said, a really hot topic and an emerging use case that we see kind of more and more uh use for within the Airflow community.
• 4:47
  SPEAKER_00: Uh, I guess to first start off, you you talked a little bit before about in your use cases where kind of AI workflows fit in, but maybe tell me in a little more depth, kind of what is the business use case for any AI workflows that your team is managing?
• 5:02
  SPEAKER_03: Yeah, of course.
• 5:03
  SPEAKER_03: Um, so so ultimately, um our main business use case is really around uh improving um the end user experience uh as well as the purchase conversion for our clients.
• 5:16
  SPEAKER_03: Um so we use AI to make the shopping experience more interactive and personalized.
• 5:22
  SPEAKER_03: Um, for example, um, you know, our conversational agents um helps users discover products or systems that generate insights from user interactions.
• 5:33
  SPEAKER_03: Um, but to make all that work well, it's not just about the model itself.
• 5:39
  SPEAKER_03: Um, you also need to have a reliable way to sort of like continuously process new data, um, update knowledge sources, and and also evaluate whether the system is actually improving.
• 5:51
  SPEAKER_03: Um so the workflows we've built are really about closing that loop from data to the um you know model behavior and and then to the evaluation, um, and then repeat, right?
• 6:04
  SPEAKER_03: Um so so that's that's what um really allows us to improve these uh systems in a consistent and measurable way.
• 6:14
  SPEAKER_00: Yeah, interesting.
• 6:15
  SPEAKER_00: I as you might imagine, my team does a lot of talking about uh data orchestration in the context of AI workflows right now.
• 6:22
  SPEAKER_00: And um Volker Hans, who is on my team, uh has recently given a talk uh where he had a slide on it that is like the AI model is like the small part.
• 6:31
  SPEAKER_00: So kind of like uh in the old like ML op slides where they would say like the model is a small part, now it's like the AI is also the small part, and there's all this stuff.
• 6:39
  SPEAKER_00: Right.
• 6:40
  SPEAKER_00: Yeah, exactly what you said the data quality, the orchestration, the feedback loops, the human in the loop, all that good stuff.
• 6:48
  SPEAKER_00: So there's a lot that goes into it beyond beyond just uh yeah, that's great.
• 6:53
  SPEAKER_00: Well, then tell me a little bit more about what the technical implementation looks like for this use case.
• 6:59
  SPEAKER_00: Uh, I imagine, given our conversation that you're using Airflow, at least to some extent, but I would imagine there's also other tools involved.
• 7:06
  SPEAKER_00: So yeah, what does that look like?
• 7:08
  SPEAKER_03: Yeah, so um at the very high level, um, we pretty much treat air uh AI workflow uh very similar to other uh data pipelines, um, with just a few additional layers.
• 7:22
  SPEAKER_03: Uh so it usually starts with the traditional ingestion uh pipeline uh to handle all the interaction, user interaction data, things like uh you know the user behavior, conversations, and also the content, video, you know, um like content consumption histories, uh all that good stuff.
• 7:44
  SPEAKER_03: Uh and then from there we process and structure that data and use it to update the personalized user profile.
• 7:51
  SPEAKER_03: Um, and and uh also the embeddings that powers the AI systems.
• 7:57
  SPEAKER_03: Um so on the side, we also run evaluation pipelines, um, which is dispatched from third-party eval systems that we use internally to test the different prompts, configurations, uh, measuring performance, and then comparing that with the previous baselines.
• 8:15
  SPEAKER_03: Um, all of that is orchestrated through Airflow, um, not within the same DAG, but separate, uh, which handles all the dependencies between each step.
• 8:27
  SPEAKER_03: Uh on the tooling side, we use Snowflake pretty heavily, um, including their Cortex AI capabilities, along with some internal services for model executions and data processing.
• 8:38
  SPEAKER_03: Um so yeah, I'd say like Airflow is really coordinating all these systems and making sure the pipeline runs end-to-end reliably.
• 8:49
  SPEAKER_02: This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
• 8:57
  SPEAKER_02: From batch pipelines to training models and wiring up agents.
• 9:02
  SPEAKER_02: Astro handles the infrastructure upgrades and scaling so you can focus on building.
• 9:08
  SPEAKER_02: Check it out at astronomer.io.
• 9:13
  SPEAKER_00: Yeah, yeah, very interesting.
• 9:15
  SPEAKER_00: And tell me a little bit more about why you chose Airflow.
• 9:18
  SPEAKER_00: I mean, I'm at you said you've been using Airflow since nearly since the project started.
• 9:23
  SPEAKER_00: So certainly long before there were AI workflows that you had to manage.
• 9:28
  SPEAKER_00: Um, but obviously you had established use at this point.
• 9:31
  SPEAKER_00: But yeah, tell me about um when you were putting together the stack, what made you think that Airflow would be a good choice for the orchestrator?
• 9:38
  SPEAKER_03: Yeah, that's a that's an excellent question.
• 9:41
  SPEAKER_03: Um, at first it was like uh very um exploratory um at the very beginning.
• 9:48
  SPEAKER_03: Um, but we stay with uh Airflow.
• 9:51
  SPEAKER_03: I think the main reason is because of the flexibility and also the reliability.
• 9:56
  SPEAKER_03: Um I always uh say that Airflow is one of those tools that never really goes out of uh fashion.
• 10:03
  SPEAKER_03: Um it's it's flexible enough to handle almost all kinds of automation and schedule workflow.
• 10:08
  SPEAKER_03: Um you know, whether that's a SQL transformation, uh a Python, you know, um distributed job or AI-related tasks.
• 10:18
  SPEAKER_03: Um before Airflow, we actually had, you know, a lot of like menu cron jobs set up everywhere.
• 10:26
  SPEAKER_03: Uh we sort of brought all that into Airflow after we um start really liking the platform.
• 10:32
  SPEAKER_03: Um yeah, and then compared to a lot of newer tools that tend to change pretty quickly, um, Airflow has been very stable for us.
• 10:41
  SPEAKER_03: Uh it's something we adopted early and it's continued to scale with the company as our use case evolved over time.
• 10:49
  SPEAKER_03: Um we we also already had a strong foundation with it on the data platform side.
• 10:55
  SPEAKER_03: So extending it into Air Workflow was a pretty natural step for us.
• 11:00
  SPEAKER_03: Um, and most importantly, it gives us like a very clear visibility into dependencies and execution, uh, which becomes really critical when you're chaining together multiple steps across AI and uh data systems.
• 11:16
  SPEAKER_00: Yeah, absolutely.
• 11:17
  SPEAKER_00: That makes sense.
• 11:18
  SPEAKER_00: I I think it's so cool having been part of the project for a long time, having seen Airflows start to be used more and more for different use cases that are, you know, kind of more, I guess, emerging than a lot of the bread and butter stuff.
• 11:33
  SPEAKER_00: But every time that I've asked somebody who is using Airflow for AI workflows or something new like that why they're choosing it, it's always the same answers.
• 11:41
  SPEAKER_00: Like I would have given myself when I was a data engineer choosing it for ETL workflows.
• 11:46
  SPEAKER_00: And that's the same things you just mentioned, the flexibility, the reliability.
• 11:50
  SPEAKER_00: It's just really standardized now.
• 11:52
  SPEAKER_00: Uh so yeah, I think I think that's really cool.
• 11:55
  SPEAKER_03: Yeah.
• 11:56
  SPEAKER_00: Yeah.
• 11:57
  SPEAKER_00: Um now that you have been working with Airflow for a while in this way, or have these kind of established or productionized AI workflows, I'm curious if there are any challenges that you've encountered when implementing them.
• 12:10
  SPEAKER_00: And that could be maybe with Airflow or or maybe not, maybe it's just more broadly.
• 12:15
  SPEAKER_03: Yeah, definitely.
• 12:16
  SPEAKER_03: Definitely.
• 12:17
  SPEAKER_03: Um, I mean, I wouldn't say these are Airflow issues specifically, uh, because even without Airflow, we like any sort of application built on top of AI typically experience very similar challenges.
• 12:32
  SPEAKER_03: Uh, most of our challenges really come from the nature of air workflow themselves.
• 12:39
  SPEAKER_03: Um, I think Airflow just makes some of these gaps more visible because uh it's you know fundamentally built around deterministic uh schedule-based pipelines.
• 12:50
  SPEAKER_03: Um so so um uh the traditional data pipelines are um really like predictable.
• 12:58
  SPEAKER_03: So if the input data is correct, uh you would expect the output to also be correct.
• 13:05
  SPEAKER_03: Uh but with AI workflows, especially LLM related systems, outputs can be much harder to validate automatically.
• 13:13
  SPEAKER_03: Um so one major challenge is um the observability and evaluation.
• 13:19
  SPEAKER_03: Uh it's not enough to know whether a task uh succeeded technically.
• 13:25
  SPEAKER_03: Uh, you also need to know whether the model output was actually good, because a lot of times the model gives you very um confident but incorrect answers.
• 13:37
  SPEAKER_03: Uh, and those are extremely difficult to detect.
• 13:41
  SPEAKER_03: Um, so that means building additional evaluation layers, um, traceabilities, regression testing, and and the feedback loop into the pipeline itself becomes more critical than ever before.
• 13:55
  SPEAKER_03: Um, I think another challenge that we have is the uh cost management.
• 14:01
  SPEAKER_03: Uh so uh as a lot of people already know that air workflows can be, you know, can become very expensive uh fairly quickly.
• 14:12
  SPEAKER_03: So uh especially if you're running embedding jobs, evaluations, or any sort of large-scale prompt experimentation repeatedly.
• 14:21
  SPEAKER_03: Uh like so so we have had to become much more intentional around caching, uh, you know, think from a batching perspective, uh, selective uh evaluations, and understanding really like which workflows actually create business value.
• 14:41
  SPEAKER_03: Because if you're not careful, the evaluation cost could actually quite exceed the computation cost itself.
• 14:48
  SPEAKER_03: Uh so that's you know a big lesson for us because every time we need to evaluate the quality of the output, um, you need to have multiple layers of setup, different variables, different configuration that multiplies very, very, very quickly.
• 15:10
  SPEAKER_03: Uh in order to have one job done to validate the result, you may need to run like four or five other jobs in parallel.
• 15:20
  SPEAKER_03: Uh so the the costs do add up pretty quickly.
• 15:25
  SPEAKER_00: Yeah, absolutely.
• 15:26
  SPEAKER_00: I think those things, the two things you mentioned are both uh a very good summary of a lot of the issues that folks are running into when trying to productionize these workflows.
• 15:37
  SPEAKER_00: And I'd love to double-click a little bit more about the first one because I think this is such a big challenge, this sort of lack of determinism or deterministic output or lack of item potency in your tasks, which is always something that we've talked about, you know, as a best practice with using Airflow, is always make sure your tasks are itempotent and you really can't with a lot of the ones that have LLM involvement.
• 15:59
  SPEAKER_00: Uh, we were just dealing with this earlier today on my team, somebody preparing a demo for an event and saying, like, we don't really know what it's gonna spit out like on the day that we actually run it.
• 16:11
  SPEAKER_00: And so in that particular case, when it's a demo, it just makes it kind of fun.
• 16:15
  SPEAKER_00: You get to sort of like go with the flow of the output, but obviously in a production workload, it's a little more challenging.
• 16:21
  SPEAKER_00: Uh tell me a little bit more about what you're doing for the observability piece.
• 16:26
  SPEAKER_00: Since you mentioned because of these challenges, that's so important.
• 16:29
  SPEAKER_00: And you also mentioned the confident and wrong uh sounding output of the model, which uh I've had other guests talk about too.
• 16:36
  SPEAKER_00: I think that is uh definitely a big challenge.
• 16:39
  SPEAKER_00: But uh yeah, what what do you do to make those things easier for your team to manage?
• 16:44
  SPEAKER_03: Um yeah, that's uh a very good question.
• 16:48
  SPEAKER_03: Um so so we have already started implementing some of uh uh these observability and cost management um uh initiatives internally, but realistically with time and resource constraints, especially in the startup world, uh it's it's hard to prioritize every platform improvement at once.
• 17:09
  SPEAKER_03: Uh so our approach has been really focused on building the most critical pieces first, uh, and the things that give us the biggest impact on reliability, visibility, and then the operational stability uh side of things.
• 17:24
  SPEAKER_03: So uh my my founder uh um uh always joke about things that uh uh there are two things that that we definitely cannot do.
• 17:35
  SPEAKER_03: One is uh bankrupt the company, second is uh putting him to the jail.
• 17:42
  SPEAKER_00: Those are good guidelines for sure.
• 17:46
  SPEAKER_03: Yeah, so um um so yeah, for example, uh you know, among all the uh priorities that we have, we've make sure that uh we actually have uh safeguards and constraints set in the system so that uh we're always getting alerted and the job gets aborted if the the cost exceeds a certain threshold.
• 18:08
  SPEAKER_03: So things like that typically gets prioritized first.
• 18:12
  SPEAKER_03: Um but a lot of things uh within the ecosystem are still trying to catch up.
• 18:18
  SPEAKER_03: Um but but uh um um you know that's sort of the also the improvement that we hope to to get from from Airflow community.
• 18:27
  SPEAKER_03: That uh uh so not everything needs to be built internally.
• 18:32
  SPEAKER_00: Yeah, absolutely.
• 18:34
  SPEAKER_00: That's a great answer.
• 18:35
  SPEAKER_00: Uh well, Sean, thanks so much for going through that.
• 18:38
  SPEAKER_00: I think this will be so valuable to others in the community who, again, are I know often being tasked with implementing these type of workflows.
• 18:45
  SPEAKER_00: And we certainly hope that Airflow ends up being kind of the first choice for orchestration for a lot of people looking to productionize AI.
• 18:54
  SPEAKER_00: Uh, just in the interest of time, I always like to close with a standard final question.
• 18:59
  SPEAKER_00: Uh, and that is what would you most like to see from the development of the Airflow project in the near future?
• 19:05
  SPEAKER_03: Yeah, that's a that's a good question.
• 19:07
  SPEAKER_03: I think um I I already mentioned um earlier, I think um, you know, one thing we'd love to see is uh stronger building support for AI workflows um and AI native operation patterns.
• 19:20
  SPEAKER_03: Uh right now, a lot of teams are still building many of those capabilities themselves, uh, including including us.
• 19:27
  SPEAKER_03: Uh things like evaluation tracking, uh prompt experimentation, model observability, um, or just more event-driven AI orchestration patterns.
• 19:38
  SPEAKER_03: I think those would be great to see more of that become first class uh within the ecosystem over time.
• 19:45
  SPEAKER_03: Um on top of that, there's also a smaller note.
• 19:49
  SPEAKER_03: Um, you know, I'll be very honest.
• 19:52
  SPEAKER_03: I uh personally liked some aspect of Airflow UI before um version three uh bit more.
• 19:59
  SPEAKER_03: So the the newer versions uh definitely added a lot of capabilities, but I think there are still room to improve um the overall usability and developer experience.
• 20:10
  SPEAKER_03: Uh and I think my my team members also uh share the same uh feedback uh about the the UI and the UX that comes with the version three airflow.
• 20:20
  SPEAKER_00: Okay.
• 20:20
  SPEAKER_00: Yeah, I think those are both great answers.
• 20:22
  SPEAKER_00: I guess on the first piece, the uh more functionality for AI workflows.
• 20:28
  SPEAKER_00: I'm curious if you have seen anything about the new common AI provider that was just at least at the time we're recording this release, maybe a month or so ago.
• 20:37
  SPEAKER_00: Uh I don't think it has the things that you mentioned specifically, but it's definitely where those that type of functionality is being added in.
• 20:44
  SPEAKER_00: So it's nice.
• 20:46
  SPEAKER_00: Uh okay.
• 20:47
  SPEAKER_00: Yeah, it's it's definitely the most progress that Airflow has made in terms of LLM workflows currently.
• 20:52
  SPEAKER_00: And I think it's pretty exciting.
• 20:54
  SPEAKER_00: So maybe one to watch out for.
• 20:57
  SPEAKER_00: Uh yeah, and you're certainly not the first person to talk about UI updates.
• 21:01
  SPEAKER_00: I I think it's so tricky every time you change a UI, some people are gonna be unhappy about the changes.
• 21:07
  SPEAKER_00: I do know that there's uh work going on, I think, for 3.3 right now to kind of fill in the last remaining gaps of at least functionality that existed in the Airflow 2 UI that temporarily we'll say didn't exist in Airflow 3.
• 21:23
  SPEAKER_00: Uh, but yeah, I think um the folks who are developing Airflow are always interested in hearing UI feedback, and that you know iterates on almost every minor release.
• 21:32
  SPEAKER_00: So definitely would love for yeah, your team to uh, you know, submit a GitHub issue, someone will probably look at it.
• 21:39
  SPEAKER_03: Perfect.
• 21:40
  SPEAKER_03: Yeah.
• 21:40
  SPEAKER_03: We'd happy to do so.
• 21:42
  SPEAKER_00: Yeah, that's great.
• 21:43
  SPEAKER_00: Okay, Sean.
• 21:44
  SPEAKER_00: Well, it's been really great speaking with you.
• 21:46
  SPEAKER_00: Super interesting to hear more about your use case.
• 21:49
  SPEAKER_00: Uh, to close us out, what is the best way for folks to get in touch with you?
• 21:53
  SPEAKER_03: Um, um, feel free to connect me uh with me on on LinkedIn.
• 21:58
  SPEAKER_03: Um, I'm always happy to chat about airflow data platforms as well as AR workflows.
• 22:03
  SPEAKER_03: Um, I think that's the yeah, best way to to connect.
• 22:07
  SPEAKER_00: Okay, great.
• 22:08
  SPEAKER_00: Well, we will throw a link to that in the show notes.
• 22:10
  SPEAKER_00: Uh Sean, thank you again.
• 22:12
  SPEAKER_00: It's been a pleasure speaking with you.
• 22:13
  SPEAKER_00: We really appreciate you taking the time.
• 22:15
  SPEAKER_03: Yeah, likewise.
• 22:16
  SPEAKER_03: Thank you so much.
• 22:17
  SPEAKER_02: Thank you for listening to the Data Flowcast.
• 22:21
  SPEAKER_02: Check the show notes for links to everything mentioned today.
• 22:24
  SPEAKER_02: If you're enjoying the podcast, subscribe so you never miss an episode.
• 22:29
  SPEAKER_02: And leave us a five star review.