Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers

Transcript

Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_04: We listen to people, try and understand the problems that they have and try and figure out if we can build a generalizable solution to that.
• 0:06
  SPEAKER_04: And so we've embraced that pretty much from the last eight years.
• 0:09
  SPEAKER_04: It's always are there ways in which we can build a broader framework rather than solve like a very specific problem?
• 0:14
  SPEAKER_01: And we are also lucky enough to have a dedicated infrastructure team within Intercom who takes care of a lot of AWS heavy lifting.
• 0:21
  SPEAKER_01: So if we need a new EC2 fleet for airflow, then we can just go to an internal lab say that hey, give me a fleet.
• 0:27
  SPEAKER_01: I want to scale it between one and ten, and this is the instance type I1.
• 0:36
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering in AI.
• 0:49
  SPEAKER_03: Hello everyone, welcome to the Data Flowcast.
• 0:51
  SPEAKER_03: I'm your host, Marco Merti, and today I'm joined by Paul Vickers and Andras Kombochi from Intercom.
• 0:59
  SPEAKER_03: I would like to ask, how are you both doing today?
• 1:02
  SPEAKER_02: You're doing fine.
• 1:03
  SPEAKER_02: Thank you very much.
• 1:04
  SPEAKER_02: How are you?
• 1:05
  SPEAKER_03: So I must say that's the first time that I see a setup like yours.
• 1:09
  SPEAKER_03: Everything is perfect, the microphone is perfect.
• 1:11
  SPEAKER_03: For those who are listening to the podcast, I highly encourage you to look at the video because this is professional setup here.
• 1:18
  SPEAKER_03: Alright, so today we are talking with Paul and Andras, who lead Intercom's data infrastructure and analytics engineering efforts.
• 1:26
  SPEAKER_03: And they have built one of the most sophisticated airflow deployments we've seen, with over 11,000 stags, I believe, running across three global regions, which is pretty incredible, supporting everything from basic data ingestion to AI-powered customer analysis.
• 1:43
  SPEAKER_03: So I'm sure many people will enjoy that episode.
• 1:46
  SPEAKER_03: But first, let's start with your backgrounds and how you came to Airflow.
• 1:51
  SPEAKER_03: So, Paul, can you tell us more about your Airflow journey?
• 1:54
  SPEAKER_04: Sure.
• 1:54
  SPEAKER_04: So I joined Intercom in around 2017.
• 1:58
  SPEAKER_04: And so at that point, like Intercom's a startup, it was like reasonably mature at that point.
• 2:03
  SPEAKER_04: So we had a data warehouse, which was Redshift, and we had a bunch of ingestion pipelines to ingest data, telemetry from the product, different events, production data, and so on into Redshifts for product analytics.
• 2:17
  SPEAKER_04: The kind of issues were that a lot of these pipelines were all independent.
• 2:22
  SPEAKER_04: So they were written in different languages by different product engineers over the years.
• 2:27
  SPEAKER_04: And some of them were in EMR, very difficult to debug when things went wrong.
• 2:32
  SPEAKER_04: You had to set up a tunnel SSH into an EC2 instance to look at logs.
• 2:37
  SPEAKER_04: You need to be an expert in a bunch of different languages.
• 2:40
  SPEAKER_04: And it was quite overwhelming.
• 2:42
  SPEAKER_04: And also for the data consumers who were trying to use this data to make decisions or share insights with the product teams, they didn't know when the data was updated.
• 2:52
  SPEAKER_04: A lot of times it wasn't updated.
• 2:53
  SPEAKER_04: We had like lots of production issues and so on.
• 2:56
  SPEAKER_04: And so people were like just very confused.
• 2:58
  SPEAKER_04: As the kind of original data engineers, we were pretty like frustrated with the systems, the observability, and so on.
• 3:05
  SPEAKER_04: And we were on the lookout for alternatives to what we had.
• 3:09
  SPEAKER_04: And around that time, because I was pretty new to the data space, I'd been working on a little bit of Microsoft before that, but I read this blog post called The Rise of the Data Engineer.
• 3:19
  SPEAKER_04: And so like a bunch of people read it and it resonated with a lot of people.
• 3:23
  SPEAKER_04: One of the things we talked about was like the need for like good orchestration.
• 3:26
  SPEAKER_04: And in Microsoft, I had a tool that we kind of used that was like visual, you could see things.
• 3:31
  SPEAKER_04: And so in Intercom, we did have a little bit of orchestration with Luigi, which I think was built by Spotify.
• 3:37
  SPEAKER_04: But even that, all the different kind of systems used different orchestration and things were all like mistimed.
• 3:44
  SPEAKER_04: And so it was just a pretty confusing time.
• 3:46
  SPEAKER_04: We had always been pretty inspired by Airbnb in Intercom.
• 3:51
  SPEAKER_04: We had a few different contacts there over the years, and they used to blog a lot.
• 3:54
  SPEAKER_04: I don't know if they still do, about the kind of data platforms that they built out.
• 3:59
  SPEAKER_04: And one of the things they talked about was Airflow.
• 4:01
  SPEAKER_04: I kind of lucked out that my manager, she got me a contact with someone in Airbnb and they said, Is there anyone that you'd like to meet?
• 4:09
  SPEAKER_04: And I was like, Yeah, personally created Airflow.
• 4:11
  SPEAKER_04: And Sasha, the other data engineer at the time, we traveled over to San Francisco and met up with Max in the Airbnb office.
• 4:18
  SPEAKER_04: And if you've ever been there, every kind of meeting room is set up like someone's housed in Airbnb.
• 4:24
  SPEAKER_04: And so we were in this Victorian sitting room, sitting at this big mahogany table talking about whether Airflow would scale to intercom's needs.
• 4:32
  SPEAKER_04: It was weird.
• 4:33
  SPEAKER_04: But ultimately, he kind of nodded and said it works for Airbnb, it'll probably work for us too.
• 4:38
  SPEAKER_04: So we headed back to Dublin and we decided that we were going to go all in on Airflow.
• 4:43
  SPEAKER_04: And we'd done a little bit of the research too.
• 4:45
  SPEAKER_04: And so at that point, we decided that our kind of main goals here were to orchestrate or put all of the orchestration in one place and try and put all of our data tooling into one repo so that you could go to this one place and see like how all the data is being processed and ultimately be able to orchestrate all those different ingestion and processing pipelines that that happened for some time.
• 5:08
  SPEAKER_03: So that's pretty amazing.
• 5:10
  SPEAKER_03: That's the first time that actually someone tells me that we decided to start Airflow because one of the reasons is that we met the creator of Airflow, which is pretty cool at Airbnb.
• 5:20
  SPEAKER_03: So that's pretty amazing to hear.
• 5:22
  SPEAKER_03: I guess Andrash, do you share the same journey?
• 5:24
  SPEAKER_03: Did you join Intercom at the same time as Polo?
• 5:27
  SPEAKER_01: No, I joined Intercom three years later.
• 5:29
  SPEAKER_01: Oh, okay.
• 5:30
  SPEAKER_01: Okay.
• 5:30
  SPEAKER_01: In 2020.
• 5:31
  SPEAKER_01: Paul was my first manager.
• 5:34
  SPEAKER_03: Oh, okay.
• 5:36
  SPEAKER_02: Yeah.
• 5:37
  SPEAKER_01: His brain would be based in on all the admin required.
• 5:40
  SPEAKER_01: So no, I came in three years later from kind of solution architecture and data background.
• 5:46
  SPEAKER_03: Okay, and so Airflow was already there when you joined?
• 5:49
  SPEAKER_01: Airflow was already here.
• 5:50
  SPEAKER_01: Oh yeah.
• 5:51
  SPEAKER_01: That's amazing.
• 5:52
  SPEAKER_03: So I guess now if we deep dive into your uh current setup.
• 5:56
  SPEAKER_03: So today you are running around, again, like 11,000 stags, maybe more, across three global regions.
• 6:02
  SPEAKER_03: So you are truly running Airflow at scale.
• 6:04
  SPEAKER_03: Can you walk us through your infrastructure approach and how you have built a self-service platform where teams can deploy their own pipelines?
• 6:13
  SPEAKER_03: I know it's actually a pretty complex question because the first thing is how do you actually run Airflow at scale?
• 6:19
  SPEAKER_03: But also making Airflow a self-service is something that many companies are trying to do right now.
• 6:24
  SPEAKER_03: And I'm sure you have plenty of insightful things to say about.
• 6:28
  SPEAKER_04: Sure.
• 6:28
  SPEAKER_04: So, like in Intercom, we have the data engineering team back in like 2017 when we started at here was like a pretty small team.
• 6:35
  SPEAKER_04: There's two of us.
• 6:36
  SPEAKER_04: Okay.
• 6:36
  SPEAKER_04: And there was a lot more analysts and other people across the company that wanted to do stuff with data.
• 6:42
  SPEAKER_04: So we knew pretty early on that we were going to be a like really big bottleneck for all those people if they required us to make any change that they needed to process their data.
• 6:52
  SPEAKER_04: We decided to try and embrace this kind of self-serve kind of approach to building out the tooling.
• 6:58
  SPEAKER_04: Sasha, who was like like our primary data engineer, he really kind of liked this idea of building kind of generalized DAGs and building configuration files that people could express what they wanted, how they wanted their data moved, their process, and so on in like nice, easy, almost human readable way.
• 7:16
  SPEAKER_04: And then these generalized DAGs would then generate DAGs that would do that work for them.
• 7:21
  SPEAKER_04: And so what we started doing was getting people comfortable using GitHub and Git.
• 7:27
  SPEAKER_05: Okay.
• 7:27
  SPEAKER_04: But then so that people could clone our single repo down into their local machines, make changes, configure the changes that they want.
• 7:34
  SPEAKER_04: And then we would have a bunch of CI tests that would make sure that they work okay.
• 7:38
  SPEAKER_04: And then they would get shipped to production and they would have then a pipeline running in Airflow on a daily basis.
• 7:44
  SPEAKER_04: And then over time, more and more people got involved in creating these pipelines.
• 7:49
  SPEAKER_04: And so we started off with kind of small things.
• 7:52
  SPEAKER_04: And we started building out this kind of transformations framework, which turned out to be DBT, but we didn't know about DBT back then.
• 7:59
  SPEAKER_04: So we built like our own kind of clone of it.
• 8:01
  SPEAKER_04: Okay.
• 8:02
  SPEAKER_04: Like a similar version.
• 8:04
  SPEAKER_04: Later on, we ended up just like switching to DBT because there was more people developing it and we gave a lot of things for free.
• 8:10
  SPEAKER_04: But we had our transformations framework.
• 8:12
  SPEAKER_04: And effectively, people could write SQL files and they could submit those into the repo.
• 8:16
  SPEAKER_04: We would then run some tests against them to make sure they hadn't broken anything.
• 8:19
  SPEAKER_04: And then they'd get pushed out to production and they would be part of our daily ETL pipeline.
• 8:24
  SPEAKER_04: But the kind of the issues that we had with this over time was that when everything is self-serve, you need some sort of like guardrails and guidance to make sure that people are doing things in a consistent way and made a few mistakes there in not really doing that.
• 8:38
  SPEAKER_04: But ultimately, we kind of really embraced that kind of culture of we listen to people, try and understand the problems that they have, and try and figure out if we can build a generalizable solution to that.
• 8:49
  SPEAKER_04: And so we've embraced that pretty much from the last eight years, it's always are the ways in which we can build a broader framework rather than solve like a very specific problem.
• 8:59
  SPEAKER_01: And we are also lucky enough to have a dedicated infrastructure team within Intercom who takes care of a lot of AWS heavy lifting.
• 9:06
  SPEAKER_01: So if we need a new EC2 fleet for airflow, then we can just go to an internal app, say that hey, give me a fleet, I'll just scale it between one and ten, and this is the instance type I want.
• 9:18
  SPEAKER_01: And a minute or so later I'll get back the endpoint for the ARB.
• 9:22
  SPEAKER_01: And this helped a lot.
• 9:24
  SPEAKER_01: So a lot of AWS heavy lifting is abstracted away from us.
• 9:28
  SPEAKER_01: So we just need to focus on data.
• 9:30
  SPEAKER_03: This is a luxury, guys.
• 9:31
  SPEAKER_03: This is a true luxury.
• 9:37
  SPEAKER_04: Intercom is like quite opinionated.
• 9:40
  SPEAKER_04: And so like teams own running their stuff and managing it themselves.
• 9:44
  SPEAKER_04: But we do have some, I wouldn't quite say centralized teams, but teams who are like experts at things, and they try and build like the tooling that allows other teams to be able to unblock themselves quickly.
• 9:56
  SPEAKER_04: So rather than us having to open a ticket to some other team and say, hey, can you go and provision us all these things?
• 10:02
  SPEAKER_04: They have a repo, it's all infrastructure, it's coloured, you write a PR, it builds a plan, they like it, it looks good, gets shipped, we're ready to go.
• 10:11
  SPEAKER_04: That's amazing.
• 10:12
  SPEAKER_04: Quite refreshing, maybe.
• 10:14
  SPEAKER_02: Yeah.
• 10:14
  SPEAKER_02: It also introduced chaos engineering into our world because it's still that team managing the patching of these C2 instances and the AMI rollouts.
• 10:23
  SPEAKER_02: So we had to cater for all of the retries and basically the whole clusters being pulled from underneath our flow.
• 10:31
  SPEAKER_02: So it's fun.
• 10:32
  SPEAKER_03: I bet it is.
• 10:33
  SPEAKER_03: And I need to go back to what you said all about the guard rivals that you and you made some mistakes.
• 10:40
  SPEAKER_03: Can you tell us a little bit more about those mistakes?
• 10:42
  SPEAKER_03: Because obviously, as we learned from mistakes, can you tell us more about that?
• 10:46
  unknown: Yeah.
• 10:47
  SPEAKER_04: Thinking back, so a lot of the kind of initial work we did was to take a whole bunch of like disparate queries that people were running for themselves that was counting millions or billions of records and pointing out distinct and so on, and trying to centralize those things and run them once per day so that everyone can get the value of those things being pre-calculated.
• 11:07
  SPEAKER_04: Standard kind of data warehousing things we do today.
• 11:10
  SPEAKER_04: And so as we kind of mentioned before, we didn't want like us, the data engineering team, to be like the blockers for things and people say, Hey, I need this new data model that does X.
• 11:20
  SPEAKER_04: And so we'll be like, oh, we'll try and get to that.
• 11:22
  SPEAKER_04: So like the analysts were working with very closely with the product teams to understand like the product usage.
• 11:28
  SPEAKER_04: And so they were able to figure out the right SQL that they needed to put together to answer the questions that people had.
• 11:34
  SPEAKER_04: But when we built this transformations framework, it made it very easy for then for people to be able to roll out their own data models, probably a little bit too easy.
• 11:44
  SPEAKER_04: And like we gave them the ability, but without any real kind of run book or rule book or guidance on like when to do what.
• 11:52
  SPEAKER_04: And so as a result, what would happen is somebody would might look in the data warehouse, see a table, look at that and be like, oh, it's not quite exactly what I need.
• 12:02
  SPEAKER_04: So I'll just go and add my own one.
• 12:04
  SPEAKER_04: And rather than go in and just change the existing one or augment it in some way, it was just easier.
• 12:09
  SPEAKER_04: I'll just build my own.
• 12:10
  SPEAKER_04: The idea here was that we were trying to get like a central source of truth, but now we had five or six central sources.
• 12:16
  SPEAKER_04: But the lack of like general like oversight or like a vision of like how things should be done meant that we started getting into this kind of data sprawl as happens when there's not some sort of generalized oversight or somebody who's keeping tabs on what's being done here.
• 12:33
  SPEAKER_03: Thank you so much for showing that.
• 12:34
  SPEAKER_03: I'm sure it will resonate for many people.
• 12:37
  SPEAKER_04: It's taking us many years to undo all of that.
• 12:39
  SPEAKER_04: And like literally this week, I think Andre on the team is doing the last of the kind of clearing out of all that legacy code base.
• 12:47
  SPEAKER_04: So it's certainly if you can avoid making that mistake, it's worth doing so.
• 12:52
  SPEAKER_01: We are down to 2,700 models from 4,000 family start this project.
• 12:57
  SPEAKER_03: So wow, it's pretty incredible.
• 13:00
  SPEAKER_03: All right, so one of your most interesting frameworks is the batch LLM processing system.
• 13:05
  SPEAKER_03: We talked about that in our preparation call.
• 13:08
  SPEAKER_03: This framework is used by teams to submit their YAML configurations and Airflow orchestrates the entire workflow, from reading data in Snowflake to processing it through AI models and writing the result back.
• 13:20
  SPEAKER_03: But can you explain how this works and what problems it solves for your teams?
• 13:25
  SPEAKER_03: Because you know, LLMs, I mean, we know that, right?
• 13:27
  SPEAKER_03: AI and stuff, very popular right now.
• 13:29
  SPEAKER_03: I'm sure many people wonder how to concretely use AI and LLMs in their company.
• 13:35
  SPEAKER_03: But it looks like you are doing that right.
• 13:37
  SPEAKER_03: So can you tell us more about this?
• 13:39
  SPEAKER_01: Yeah, we're not just using AI, obviously, to empower Finn, like our own products, but we try to empower every single user within into completely use LLM where it's warranted.
• 13:50
  SPEAKER_01: We are experimenting with new tools as well.
• 13:52
  SPEAKER_01: And they're simply just use cases which really, really, really can benefit from LM usage.
• 13:59
  SPEAKER_01: And if you have a single transcript of a call, you can very easily just put it into Gemini or Cloud or whatever.
• 14:05
  SPEAKER_01: But if you want to summarize that or to take the sentiment analysis, for example, but if you are working with a relatively large input data set of hundreds or thousands of records, and that can be gone calls from your customers, or that can be Reddit comments from your community pages and the likes, and then you want to create like a more holistic vision of what's happening around you, then this kind of manual copy pasting won't scale.
• 14:34
  SPEAKER_05: Right.
• 14:35
  SPEAKER_01: If you want to kind of drive these through the real-time LM APIs, that can also cost a lot.
• 14:42
  SPEAKER_01: It's just expensive.
• 14:43
  SPEAKER_01: The other side of this problem was that analysts who need this sort of qualitative analysis, they don't necessarily have Python developers, they don't necessarily have expertise to write Python code.
• 14:55
  SPEAKER_01: So we decided to create this framework to abstract all the Python stuff away from them.
• 15:00
  SPEAKER_01: The framework literally expects a YAML file from an analyst.
• 15:04
  SPEAKER_01: That YAML consists of a Snowflake C4, but they can test immediately from the local development environment through the Snowflake MCP, what we have in place.
• 15:15
  SPEAKER_01: Then it also requires the prompt, what the analyst wants to run against every single record of the input data.
• 15:22
  SPEAKER_01: Then they can specify their LLM providers, their preferred models as well, and then the target schema where they want to see the eventual results, and then the schedule they want to run it.
• 15:35
  SPEAKER_01: And then the framework DAG basically enumerates all these YAML files, creates individual DAGs, schedules them accordingly, and then all the API processing, talking to the batch APIs of the individual providers, all of that is abstracted away from them.
• 15:52
  SPEAKER_01: We have the owner information, so they submit this job, and then when the job is done, then they get a Slack message that hey, your results are in that Snowflake table.
• 16:02
  SPEAKER_01: And the whole development of the YAML file itself is AI assisted as well.
• 16:08
  SPEAKER_01: We double down on cloud code and terminal, so everything is just nice and streamlined and as cost-effective as possible.
• 16:18
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:27
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:39
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:48
  SPEAKER_03: That's pretty amazing the how you leverage AI for actually helping your users internally.
• 16:53
  SPEAKER_03: Like the way they can just use uh cloud and different LLMs to actually write the even the YAML files that they need and then get the queries that get the data that they need from Snowflake.
• 17:04
  SPEAKER_03: That's actually the first time that I hear that, I think.
• 17:07
  SPEAKER_04: So yeah, it's actually quite a while back when I suppose kind of Copilot came out in the early days of like Blood and Cursor, it was like very clear that there's like a lot of value here for us, but also for like our kind of data scientist partners.
• 17:22
  SPEAKER_04: So we did do a bunch of work to just really make our kind of developer environment.
• 17:27
  SPEAKER_04: So like our Interflow is our repo.
• 17:30
  SPEAKER_04: It's like a mixture of intercom and airflow together.
• 17:32
  SPEAKER_04: We put Inter in front of everything.
• 17:34
  SPEAKER_04: And when people talk about like our data platform internally, we talk about Interflow and it's the repo which we deploy Airflow from, but also our DBT models are there and our notebooks and everything.
• 17:44
  SPEAKER_04: And we worked really hard to have a very straightforward and easy setup so that you can clone and run it locally.
• 17:50
  SPEAKER_04: And from there, then you can do AI assisted coding.
• 17:54
  SPEAKER_05: Yeah.
• 17:55
  SPEAKER_04: But we try and ensure that everybody is set up and using the same kind of plugins and processes so that we can easily kind of share like best practices and everybody gets the value of the same kind of setup and plugins and things set up in the tools, so that if one person has figured out a good way to do it, everybody can emulate that.
• 18:14
  SPEAKER_04: And we have the same with CloudMD files and things like that in the repo so that the LLM is like opinionated or thinks the way we want it to think.
• 18:24
  SPEAKER_04: You're absolutely right, Paul.
• 18:26
  SPEAKER_03: That's very good.
• 18:27
  SPEAKER_03: And going back to the guardrails that you put before for the transformations, do you have the same kind of guardrails for your uh LLM usage?
• 18:36
  SPEAKER_03: Because we know that it can cost a lot of money.
• 18:39
  SPEAKER_03: Do you have anything in place for uh controlling the cost of using LLMs internally?
• 18:43
  SPEAKER_03: I'm wondering because obviously at Astronomer we use also LLMs like in many companies, but I was just wondering if you have anything in place today to control whatever people are doing with the LLMs.
• 18:53
  SPEAKER_04: Yeah, so a lot of our DAGs as well, we have a lot of operators that we've built ourselves on HUX.
• 18:59
  SPEAKER_04: And typically most of our DAGs that are these generator DAGs that are built from config files, log a bunch of metrics.
• 19:06
  SPEAKER_04: So that like each time the DAG is run, we'll like log, for instance, if it there's a DAG for the LLM processing, it'll be like the number of records that were sent in this particular thing, the name of the DAG, the config file, and so on.
• 19:18
  SPEAKER_04: So we've got lots of dashboards and data dog where we can see which are the things that are driving the volume here.
• 19:23
  SPEAKER_04: Within the organization, then we have various like cost monitoring programs and who are looking at spikes and we can kind of trace things back to particular jobs.
• 19:32
  SPEAKER_04: It's also another thing that we try to like feed into our Claud MD file and things like that is to try and help people like for it to ask the questions of the person who's writing or telling the AI what to do.
• 19:44
  SPEAKER_04: So that when it comes to the PR time, when like a human is looking at this, like somebody's already considered whether this is like a high volume thing and whether it's the appropriate size model and so on.
• 19:54
  SPEAKER_04: So like it's still a bit of gut as well.
• 19:57
  SPEAKER_04: You have to try and remind people about thinking about costs and so on.
• 20:01
  SPEAKER_04: But we prefer to kind of react quickly rather than to have people think that they can't do these things.
• 20:07
  SPEAKER_04: And that's the model that's typically been pushed by intercoms.
• 20:10
  SPEAKER_04: Do the thing, we'll tell you if it becomes expensive, and then we'll figure out how to make it cheaper.
• 20:14
  SPEAKER_04: Good.
• 20:15
  SPEAKER_01: On kind of corporate level, the intercom training fin and the new models, and then serving Finn.
• 20:22
  SPEAKER_01: So we have kind of corporate agreements in place with most of the LLM providers.
• 20:27
  SPEAKER_01: So we can apologize, we can piggyback on those contracts as well.
• 20:32
  SPEAKER_03: Okay, that makes sense.
• 20:33
  SPEAKER_03: So thank you so much for sharing this use case with AI.
• 20:36
  SPEAKER_03: But I think there is another one that is actually very interesting that I would love to talk about with you guys, which is the way you can help the sales team to actually prepare for calls that they have.
• 20:49
  SPEAKER_03: I think you use AI for that as well.
• 20:51
  SPEAKER_03: Can you walk us through the pipeline and how it works, basically?
• 20:55
  SPEAKER_03: Yeah, and we're saying basically that doesn't mean it is basic, but yeah.
• 20:59
  SPEAKER_04: So this started off with the data science team doing some work with some customers, like some prospects that we had, where we wanted to help them like figure out if FIN was right for them.
• 21:10
  SPEAKER_04: We think it is, or kind of data suggests that it is, but it we could the data science team could run some analysis on a sample of their conversations for them and be able to kind of show that actually there's a certain class of conversations that can be answered really easily with AI because they just require like maybe a good knowledge base to be set up, or certain types of conversations that require an AI agent to go and look up something in another system.
• 21:34
  SPEAKER_04: It's a little bit more complicated.
• 21:35
  SPEAKER_04: And then there's maybe a third where the AI agent might need to take some action in some other system.
• 21:40
  SPEAKER_04: So these first one, the kind of informational ones, we think are like easy to answer.
• 21:44
  SPEAKER_04: And so we wanted to be able to give the sales team kind of tools to be able to show them the types of conversations that they're having and how suitable to get immediate value from thing.
• 21:53
  SPEAKER_04: So the data science team built out a prompt where you could feed in some conversations and they would give you like this kind of taxonomy.
• 22:00
  SPEAKER_04: Me output and it was all being run manually, and it would take quite a bit of work to set up.
• 22:05
  SPEAKER_04: So we wanted to be able to scale this up more.
• 22:07
  SPEAKER_04: And so we took that code and ported across to Airflow.
• 22:11
  SPEAKER_04: And so we had a bunch of reusable operators.
• 22:14
  SPEAKER_04: And so one of the things we did was added an additional one which would allow us to use Snowflake Cortex.
• 22:20
  SPEAKER_04: So in Snowflake, there's a Cortex AI.
• 22:23
  SPEAKER_04: So you can just call your LLMs from within SQL.
• 22:26
  SPEAKER_04: And using that versus some of the like before we were using OpenAI and Cloud directly, it's a little bit more clunky because you've got to have a big mixture of Python and SQL code.
• 22:38
  SPEAKER_04: And for like ease of development, we found that using Cortex and Snowflake was really good for us.
• 22:43
  SPEAKER_04: So we built a DAG that would allow you to like input some parameters.
• 22:48
  SPEAKER_04: So we've actually started using this quite a lot in Airflow, having parameterized DAGs.
• 22:52
  SPEAKER_04: And when you hit the trigger button, it brings up a nice piece of UI with the various parameters that you can enter in.
• 22:57
  SPEAKER_04: And so we started by just building a parameterized DAG.
• 23:00
  SPEAKER_04: So a data scientist could go in and input the parameters of the customers and it would go and then run this analysis.
• 23:07
  SPEAKER_04: And then over time, we wanted to make this more self-serve.
• 23:11
  SPEAKER_04: And so we built a UI on top of it.
• 23:13
  SPEAKER_04: That UI was built in a separate tool, you streamlit for that.
• 23:17
  SPEAKER_04: But ultimately, it would write data to a table, and then we would have a DAG that would read that table and see that, okay, this request has come in.
• 23:24
  SPEAKER_04: And so it would then go and trigger this DAG to go and run this processing.
• 23:29
  SPEAKER_04: And so it would feed some data into Cortex, get the results out of that, and store that in a table.
• 23:36
  SPEAKER_04: And then it would like take that and turn it into a deck that which is like consumer-friendly that the salesperson could see with a bunch of charts and so on, that they could then show the customer this is the value that you can get from Finn.
• 23:49
  SPEAKER_04: And so we've scaled this process out there in front of like hundreds and hundreds of decks, and customers like value this.
• 23:56
  SPEAKER_04: In fact, I suppose in some ways it was like the very early kind of like prototype to what became like the AI insights product that's in today.
• 24:03
  SPEAKER_03: I truly love this workflow.
• 24:05
  SPEAKER_03: So the deck that you get, because I'm very interested in the deck, actually.
• 24:08
  SPEAKER_03: Like when you say it is like human relable, I mean in a friendly way, what do you mean?
• 24:13
  SPEAKER_03: Is it like a PDF or uh it's a Google Sheet slide?
• 24:17
  SPEAKER_04: Yeah, build a slide.
• 24:19
  SPEAKER_03: It generates that using an AI.
• 24:21
  SPEAKER_04: So the AI does the processing and it outputs the data in a format, and then we have like a DAG, part of our DAG has a function that will take like this raw data and will basically turn it into a set of like charts and things that get inserted into a deck.
• 24:39
  SPEAKER_05: Okay.
• 24:39
  SPEAKER_04: That process, there's still like humans involved in like writing AI assisted code to do that.
• 24:45
  SPEAKER_04: So the reality is that AI has allowed us to build much more complex DAGs that can do like much more cool things than before somebody would say, hey, we need this data move from A to B.
• 24:56
  SPEAKER_04: And so somebody would have to go and read how does this API work.
• 24:59
  SPEAKER_04: And like all of that kind of boring stuff is done by things like FiveTran and so on.
• 25:04
  SPEAKER_04: And instead, we have like data scientists who like we can partner, we build them the building blocks, like the hooks and the operators, some templates, and then they're able to like go ahead and create like these DACs that can do like really exciting and interesting things.
• 25:19
  SPEAKER_04: And part of it is it will run this processing, but it also generates a deck.
• 25:23
  SPEAKER_04: It also like fires a message into Slack to the salesperson to let them know that their deck is ready.
• 25:28
  SPEAKER_04: They can go and set up a call with the customer straight away and share this information and make loads of money.
• 25:33
  SPEAKER_03: Honestly, like I wish I had an internship at Intercom before because I would love to see how things are going in your company because I love you know all those use cases.
• 25:42
  SPEAKER_03: I'm sure it's super interesting.
• 25:44
  SPEAKER_02: Let me know when you want to join, Mark.
• 25:46
  SPEAKER_02: Have you for a tour?
• 25:47
  SPEAKER_01: Yeah, and like this all goes back to a philosophical decision what we took around five, six years ago, and that's when we decided that Airflow will not be just the scheduling tool, but we will handle that as an integration ecosystem.
• 26:01
  SPEAKER_01: So everything we needed to do in Airflow, instead of just writing raw Python code, we created operators, we created custom hooks, and then eventually your toolkit just grows so big that you can really put these complex use cases together with relative ease?
• 26:19
  SPEAKER_03: Yeah.
• 26:20
  SPEAKER_03: Actually, I have to ask a question regarding that.
• 26:22
  SPEAKER_03: All the building blocks that you have created around Airflow and everything, do you feel that today it is an issue regarding migrating Airflow to another version?
• 26:32
  SPEAKER_03: Like for example, I don't know what is your current version.
• 26:34
  SPEAKER_03: I see Andras, you are smiling.
• 26:37
  SPEAKER_03: But is it easy today to migrate?
• 26:39
  SPEAKER_03: I mean it's never easy, but would you say that the building blocks that you have now on top of Airflow, everything you built, make the migration harder, or is it still okay?
• 26:51
  SPEAKER_03: Or maybe it makes that easier actually, because some people actually build abstraction on top of Airflow, and that makes the migration from Airflow two to three, for example, easier.
• 27:01
  SPEAKER_04: Yeah, I think one of the things is that we've always like this has been built out by like software engineers, and from the very start, we added tests, tests for everything.
• 27:12
  SPEAKER_04: And so everything comes with tests.
• 27:14
  SPEAKER_04: And so what that allows us to do is that we have a separate staging environment.
• 27:18
  SPEAKER_04: The only thing we use it for is for like doing our airflow upgrades.
• 27:21
  SPEAKER_04: Um so we'll push a branch out of there and then we'll switch to like the newer version and see what breaks.
• 27:26
  SPEAKER_04: And our tests will tell us like what's broken, what's not working, and then we just work on fixing those tests until the point that we now know, okay, if all these tests are passing, there's a good chance that we'd be able to switch over to this because we try and ensure that we have really good test coverage.
• 27:41
  SPEAKER_04: It's something that like it's been a bit of a pain sometimes writing tests for things, but it does pay off a lot when you come to do migrations and things like that.
• 27:50
  SPEAKER_03: Okay.
• 27:51
  SPEAKER_03: Andras, anything you you wanted to add?
• 27:54
  SPEAKER_01: Yeah, no, I'm I fully second what Paul said.
• 27:56
  SPEAKER_01: We are protecting our code quality ferociously and we are spending a lot of time reviewing PRs, making sure we have the appropriate test coverage.
• 28:04
  SPEAKER_01: We are shying away from the 3.0 upgrade a little bit, but that's more because of the kind of philosophical change of data set-driven orchestration rather than potential problems with our existing code base.
• 28:18
  SPEAKER_04: Yeah, I think in the past we kind of used to go a bit like bleeding edge when maybe we're like a little bit like we had fewer people that relied on the data that we're producing.
• 28:29
  SPEAKER_04: But nowadays what we have running in Airflow is used across the company, it like feeds into the product and or billing systems.
• 28:36
  SPEAKER_04: Like some of that data is like processed by Airflow.
• 28:38
  SPEAKER_04: So it's incredibly important to the company that this is highly reliable.
• 28:42
  SPEAKER_04: Um, so we take that very seriously.
• 28:45
  SPEAKER_04: Every week there's metrics reviews where we look at like the problems that we've encountered, like the reliability of our like nightly ETL, like whether there's like bugs or just delays and so on.
• 28:55
  SPEAKER_04: We're like pretty rigorous in ensuring that it's like reliable.
• 28:58
  SPEAKER_04: So the next major version upgrade will be a big kind of project for us, or at least we'll make sure that we siphon off a good bit of time to invest and be like very confident that it won't be an issue for us.
• 29:09
  SPEAKER_04: We try and stay a few releases behind, but not too far behind that it becomes so arduous to kind of do an upgrade then.
• 29:16
  SPEAKER_03: I'm glad that you are not on one.x anymore.
• 29:19
  SPEAKER_04: I don't know, like we did upgrade from one.x quite a it was probably a little bit late.
• 29:24
  SPEAKER_01: I think we are on the latest two point something.
• 29:26
  SPEAKER_04: 21.3, I think is 210.3, yeah.
• 29:29
  SPEAKER_04: So we're probably a little bit behind now.
• 29:31
  SPEAKER_03: Okay.
• 29:31
  SPEAKER_03: I have one last question.
• 29:32
  SPEAKER_03: Looking at your journey, and again, thank you so much for sharing your airflow journey with us today.
• 29:37
  SPEAKER_03: You know, you obviously moved from scattered crown jobs and spark clusters and so on in 2017.
• 29:43
  SPEAKER_03: As you said, Paul, it was pretty hard at that time before you had Airflow, and now you are running again thousands of DAGs today.
• 29:50
  SPEAKER_03: What are the key lessons you've learned about running airflow at scale?
• 29:55
  SPEAKER_03: And what advice would you give to teams looking to build similar self-service uh data platforms?
• 30:01
  SPEAKER_01: That's a very good question.
• 30:03
  SPEAKER_01: Invest in training.
• 30:04
  SPEAKER_01: Okay.
• 30:04
  SPEAKER_01: There's a guy called Mark Lambert here on Udemy, and he does like really nice courses.
• 30:09
  SPEAKER_03: I don't know that guy.
• 30:11
  SPEAKER_01: Having a champion, having not even necessarily as an individual, but as a team who owns this whole ecosystem is super, super important.
• 30:20
  SPEAKER_01: And for multiple reasons, communication just to tell people what's available in the platform and that stops duplications, that just helps to streamline any new workload.
• 30:33
  SPEAKER_01: The next bit is getting your end users to get used to GitHub-based workloads because we like self-serving, but we also like self-serving with proper guardrails, and our guardrails we like them as code as much as possible.
• 30:50
  SPEAKER_01: So that means a proper CI pipeline with testing and GitHub PR reviews.
• 30:57
  SPEAKER_01: So we have full accountability, full testability.
• 31:00
  SPEAKER_01: And it's sometimes not necessarily an easy thing.
• 31:03
  SPEAKER_01: If people are not used to the GitHub-based workloads, that's a hurdle you have to just jump over.
• 31:08
  SPEAKER_01: And yeah, OPEX, like general operational excellence, instrumentation, to know as much about your airflow deployment as humanly possible, every single metric about the scheduler, about five processing.
• 31:22
  SPEAKER_01: So you can spot spikes, you can spot scaling problems, and just have a deep understanding of what's happening under the hood.
• 31:30
  SPEAKER_01: That's basically how we operate or what we've learned.
• 31:33
  SPEAKER_03: Any other advice is at all?
• 31:35
  SPEAKER_04: Yeah, I think I'd echo most of that.
• 31:37
  SPEAKER_04: Over the years, we have encountered like strange things where you know, the morning we come in and for some reason they are like overnight ETL hasn't run.
• 31:47
  SPEAKER_04: And we're like starting, what did we do here?
• 31:49
  SPEAKER_04: And so we built out like extensive dashboards that measure all the different kinds of usage, what's going on, built health checks to make sure the scheduler's running and to like alarm and alerts and page us in the middle of the night if there's like a problem.
• 32:03
  SPEAKER_04: Unfortunately, it hasn't happened for a long time.
• 32:05
  SPEAKER_04: But no, like effectively, you're operating a platform for your company.
• 32:08
  SPEAKER_04: And ideally, we've been lucky that we've been able to like build this one platform that the entire company so we don't have people competing with us, but it's a big responsibility making sure that's up all the time.
• 32:20
  SPEAKER_04: And also when people are like using your platform, you need to understand like how they're using it to make sure that like it can handle like the growth and so on.
• 32:29
  SPEAKER_04: And we just had a massive amount of adoption, and with adoption comes things like costs and so forth.
• 32:35
  SPEAKER_04: So you need like understanding like how it's being adopted, how it's being used, and whether that's in line with what you expect, and try and embrace these things and figure out ways to like enable them, but just don't be blindsided because that doesn't look good to anybody.
• 32:50
  SPEAKER_01: And from organizational point of view, we had an on-call engineer who was keeping an eye on things but also working on project work.
• 32:59
  SPEAKER_01: Right now we kind of evolved into a space where the on-call engineer doesn't do project work that there is all purposes, like during the daytime, to actually monitor the platform health and then be the user interface for the users of the analytics platform because it's basically a full-time job.
• 33:16
  SPEAKER_03: Yeah, yeah, I'm sure it is.
• 33:18
  SPEAKER_03: Thank you so much for all of those slices.
• 33:19
  SPEAKER_03: And honestly, I'm sure it will help many data engineers and even managers.
• 33:23
  SPEAKER_03: And thank you so much for sharing all of those insights with us.
• 33:26
  SPEAKER_03: Last but not least, what is the best way for folks to get in touch with you if they have some you know questions at midnight to ask you, like how they can reach out to you?
• 33:37
  SPEAKER_02: I'd say LinkedIn would be probably okay.
• 33:40
  SPEAKER_03: Okay, so they should look for Paul Vickers and then Andras Gomboji.
• 33:45
  SPEAKER_03: Yeah, it sounds good.
• 33:47
  SPEAKER_03: All right, perfect.
• 33:48
  SPEAKER_03: Again, thank you so much both for being on the show today.
• 33:51
  SPEAKER_03: And I truly hope to see you again, maybe I don't know, as soon as possible because I'm sure you are doing plenty of things that I still don't know about.
• 33:59
  SPEAKER_02: So we're having us, Mark.
• 34:01
  SPEAKER_02: This is crazy.
• 34:02
  SPEAKER_03: Thank you so much.
• 34:02
  SPEAKER_03: Take care.
• 34:05
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 34:12
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 34:16
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.