Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Adopting Airflow at Netlify

Transcript

Adopting Airflow at Netlify · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Hi there, and welcome to the Airflow Podcast, produced by Astronomer.
• 0:03
  SPEAKER_02: If you're new to the show, in this podcast we explore core fundamental concepts in data engineering, architectures for operating modern data platforms at scale, and the process of maintaining and operating a popular open source software project in Apache Airflow.
• 0:19
  SPEAKER_02: This week we got together with the folks at Netlify.
• 0:22
  SPEAKER_02: Netlify is one of the most rapidly growing tools out there for building and deploying Jamstack sites.
• 0:28
  SPEAKER_02: They provide hosting, CI, DNS, authentication, and manage back-end tools that help users run and operate static sites at scale.
• 0:38
  SPEAKER_02: Netlify recently adopted Airflow to help manage the ever-increasing volume of application and marketing data that they need to transfer to maintain robust and reliable systems around administration and user experience.
• 0:50
  SPEAKER_02: In the episode, we'll chat with Brian DeLamotte, senior data engineer, and Florian Hines, principal engineer, on the key motivating factors that drove them to adopt Airflow and what the onboarding process has been like over the past few months.
• 1:03
  SPEAKER_02: Hope you enjoy Brian and Florian, welcome to the Airflow podcast.
• 1:12
  SPEAKER_02: So to get started, do you guys mind telling us a little bit about yourselves, your backgrounds, and what you do at Netlify?
• 1:17
  SPEAKER_02: Brian, maybe you can start and Florian can follow.
• 1:20
  SPEAKER_01: Uh sure.
• 1:21
  SPEAKER_01: So I'm Brian Delamont, a senior data engineer at Netlify.
• 1:25
  SPEAKER_01: I got into big data about the beginning of 2014, just when like everyone was uh talking about how great Hadoop and uh HDFS were.
• 1:37
  SPEAKER_01: And then before that, doing a little bit of web development and firmware engineering.
• 1:42
  SPEAKER_01: And over here at Netlify, I'm on the data team and uh gonna do all sorts of fun stuff here.
• 1:50
  SPEAKER_00: I'm Florian Hines.
• 1:51
  SPEAKER_00: I've been at Netlify for just over a year.
• 1:55
  SPEAKER_00: I'm a data infrastructure engineer.
• 1:56
  SPEAKER_00: I kind of took a little bit of a winding path to get here.
• 1:59
  SPEAKER_00: I started out life as a SRE and then most recently spent like a bunch of time at a private equity firm.
• 2:06
  SPEAKER_00: And then I joined about a year ago to kind of like start helping with the data infrastructure.
• 2:14
  SPEAKER_02: Sure.
• 2:14
  SPEAKER_02: So in the broader context of Netlify, what is the data team tasked to do?
• 2:18
  SPEAKER_02: What type of stuff are you focused on managing for Netlify as a whole?
• 2:24
  SPEAKER_01: So kind of our, I guess, 10,000-foot view is the data team exists to empower the entire organization to make the best decisions possible by providing like accurate, timely, and then useful insights.
• 2:37
  SPEAKER_01: But really, what that means is kind of rolling up your sleeves and getting into the nitty-gritty details at times and you know, monitoring our data infrastructure, making sure the data is flowing right, making sure we're actually measuring things right.
• 2:51
  SPEAKER_01: And then just surfacing that to the company.
• 2:56
  SPEAKER_03: Awesome.
• 2:57
  SPEAKER_03: So, you know, before we dive into the specifics, I'd love if you could just kind of paint me a 10,000-foot overview on what does your data stack really look like in Netflix right now?
• 3:07
  SPEAKER_03: You know, everything from the tools you're using to some of the tools you've tried out and so on.
• 3:10
  SPEAKER_03: Just what does the environment look like?
• 3:13
  SPEAKER_00: Yeah, it's still pretty basic today.
• 3:15
  SPEAKER_00: We start out an all-in-one Spark platform called Databrex, which was a good fit at the time.
• 3:20
  SPEAKER_00: And so it kind of handles like tries to be like one end-to-end solution.
• 3:24
  SPEAKER_00: And then on top of that, we have a bunch of Go microservices that like you know ship logs or move data around for us because we tend to have some pretty large volumes sometimes.
• 3:34
  SPEAKER_00: And then more recently, we started kind of like putting um airflow in the mix too.
• 3:41
  SPEAKER_03: Gotcha.
• 3:42
  SPEAKER_03: And you know, we love your product, right?
• 3:45
  SPEAKER_03: We are shameless Netlify users, and I assume that you've had a lot of user growth.
• 3:49
  SPEAKER_03: As you have you kind of gone through the whole thing where you kind of mentioned it a little beforehand, but the tools you're using were great for what you were doing at the time.
• 3:56
  SPEAKER_03: And now you're kind of dealing with how you can take things to the next level.
• 4:00
  SPEAKER_03: And that kind of introduces uh certain limitations for things that weren't around before.
• 4:06
  SPEAKER_00: Yeah, exactly.
• 4:07
  SPEAKER_00: And it's not just because of user growth, but I think it's also just like team growth, right?
• 4:11
  SPEAKER_00: So when Metalify first rolled out to Databricks, it was a good solution because the data team was still really small.
• 4:20
  SPEAKER_00: I think it might have just been only like our uh data scientist at the time.
• 4:24
  SPEAKER_00: And so in that scenario, something like a platform like that where it's an all-in-one solution where you don't necessarily need a full data engineering team to support you, works out really well because they can kind of get up and running and they can write some notebooks and they can start giving out like recommendations to the business, all that kind of good stuff.
• 4:42
  SPEAKER_00: But then as you grow and you add people and you add processes and you add volume and stuff, uh all-in-one kind of solutions like that kind of start to get a little painful.
• 4:53
  SPEAKER_02: So, for folks who are less familiar, obviously, as Varaj mentioned, we've been big Netlify users and Netlify fans for a long time.
• 5:00
  SPEAKER_02: What has the headcount growth been like on the data team at Netlify over the last year or two?
• 5:06
  SPEAKER_02: And at what point did you hit that inflection point where you had to become a little bit more, or I'd say less specialized and less tied into that Databricks ecosystem with respect to your data stack?
• 5:18
  SPEAKER_00: So when I joined, I wasn't, I'm not technically on the data team, I'm actually on the infrastructure side.
• 5:24
  SPEAKER_00: In part because I think the idea is I don't necessarily deal with like the actual data infrastructure, like I'm not like running Spark for us or anything like that.
• 5:33
  SPEAKER_00: I'm just kind of neither here nor there.
• 5:34
  SPEAKER_00: But our data team was basically two folks a data scientist and like a data engineer.
• 5:40
  SPEAKER_00: And then since then, we've added another data scientist, and then more recently, Brian and also another analyst, and now an actual engineering manager too.
• 5:50
  SPEAKER_00: And I think the inflection point was actually before I joined, I think we just maybe didn't realize it yet.
• 5:56
  SPEAKER_00: And part of that was just because as the original um data scientist and engineer started building out more stuff and started having to like you know service more asks from the business, things started getting complicated, like as far as like management of the platform and like the notebooks and the environment and the jobs they were running.
• 6:18
  SPEAKER_00: I think it's one of those things where it's just like sometimes you don't notice right away that like you've outgrown the platform.
• 6:23
  SPEAKER_02: So, at which point did Airflow actually come into the picture over there to help abate some of that growing complexity?
• 6:30
  SPEAKER_00: We started talking about it in January, but it was one of those things like things where I didn't have any airflow experience because like I'm not traditionally like a uh a data person.
• 6:38
  SPEAKER_00: So then it was as we started getting um some head count when we snagged Brian, we kind of knew it was time.
• 6:45
  SPEAKER_02: And Brian, you mentioned that you had Airflow experience prior to joining Netlify.
• 6:49
  SPEAKER_02: Can you talk to us a little bit about what the process was like joining and evangelizing Airflow?
• 6:54
  SPEAKER_01: Yeah, part of the the interview uh process at Netlify is to do a little take-home project.
• 7:01
  SPEAKER_01: And during that, my kind of just said, hey, in a production environment, we'd actually want to schedule these things, this pipeline to be managed by Airflow.
• 7:10
  SPEAKER_01: And I think that key word really got uh some people's attention.
• 7:14
  SPEAKER_01: Oh, Airflow, we I think we need that.
• 7:16
  SPEAKER_01: I you know, we want that.
• 7:17
  SPEAKER_01: So as soon as I started, it was kind of the expectation that I'd be you know building out a proof of concept and and demoing at that, and then really pushing for it.
• 7:27
  SPEAKER_01: And it really I didn't have to push that hard because Florian was already convinced, and uh was really just hey, the the first project I'm gonna work on, I'm gonna you know, airflowize it from the get-go.
• 7:40
  SPEAKER_03: Awesome.
• 7:41
  SPEAKER_03: So I don't mean to belabor on the past too much, but we really know just to set context.
• 7:46
  SPEAKER_03: What were kind of the shortcomings of the way you all were handling orchestration previously before rolling out airflow?
• 7:52
  SPEAKER_03: We know we talked about some team limitations, but anything specific I think would be really interesting.
• 7:57
  SPEAKER_00: Um team-wise, I think it's more uh I think some of the platforms like Databricks maybe don't necessarily out of the box enforce a lot of like operational rigor.
• 8:06
  SPEAKER_00: So like it's easy, it's way too easy for somebody to go, okay, there's this one small bug here in this notebook, I'm just gonna make this change really quick, or you know, things like that.
• 8:14
  SPEAKER_00: And I think things like that kind of they're okay when you're starting out, right?
• 8:19
  SPEAKER_00: Because like you're moving fast and you're breaking stuff.
• 8:22
  SPEAKER_00: But then as the pipelines you've string together get more complicated and you're doing more important things, and maybe like some of your stuff that was originally just you spelunking through the data suddenly are load-bearing, right?
• 8:35
  SPEAKER_00: That is kind of like more of the problem, I think, on the team side.
• 8:38
  SPEAKER_00: Because it's like it doesn't really enforce, I think, the kind of operational regular that like somebody like Brian and I would like like.
• 8:47
  SPEAKER_03: Yeah.
• 8:48
  SPEAKER_03: You want your data workflows to be like software, right?
• 8:51
  SPEAKER_03: Like you want everything code controlled and not anyone can just push change up or something along those lines.
• 8:56
  SPEAKER_00: Yeah, and it's not platforms like this tend to do, they do tend to have it's not like there's no version control, right?
• 9:00
  SPEAKER_00: But it's built into the product.
• 9:01
  SPEAKER_00: It's not GitHub is a first-class citizen, which makes like a lot of like the traditional workflows like more complicated.
• 9:07
  SPEAKER_00: So like we did start going down a path where we do have some of the Databricks notebooks and pipelines and GitHub and running through GitHub actions, but it's just it's kind of a pain just in general, which because you're using this all-in-one platform that you're kind of trying to shoehorn in like a third-party like CI CD system, right?
• 9:27
  SPEAKER_03: Totally.
• 9:28
  SPEAKER_03: Gotcha.
• 9:29
  SPEAKER_03: And you know, I'd love to know a little bit about what some of the use cases that Airflow is powering for your team today, and what does it really allow you to do?
• 9:42
  SPEAKER_01: Yeah, I mean, for starters, it just gives us a better confidence in our data pipelines.
• 9:47
  SPEAKER_01: With scheduling our notebooks with data bricks, it was all still like cron-based, but there's no dependencies that you can really tie in from like one job to another.
• 9:58
  SPEAKER_01: So if there really is a uh workflow that needs to happen where notebook one needs to run, and then once that's finished, notebook two needs to run.
• 10:06
  SPEAKER_01: There's no tight coupling you can do.
• 10:08
  SPEAKER_01: You can only say this one starts at you know, midnight UTC, and this one should start at 1 a.m.
• 10:14
  SPEAKER_01: UTC, and there's no coupling to where if one fails, the other one knows about it.
• 10:20
  SPEAKER_01: So already we have an immediate benefit of just using those existing notebooks, but just having them scheduled in airflow.
• 10:28
  SPEAKER_01: So, yeah, strong dependencies is a big one.
• 10:31
  SPEAKER_01: And then, you know, I think just scheduling things, and then the other one that we are really hoping to use is still kind of a work in progress, but actually having easier visibility of our SLAs and our runtimes for just our entire uh data pipelines.
• 10:46
  SPEAKER_01: And then coming on board, just we want to we're growing rapidly, you know.
• 10:50
  SPEAKER_01: Netlify has, I think, doubled in size in the past year with the number of employees.
• 10:56
  SPEAKER_01: So we really want to make it easier for the next data engineer to come on board.
• 11:00
  SPEAKER_01: And I think it's much easier to look at a high-level DAG view of these pipelines and look at the README inside of Airflow and see when it runs and see kind of almost like you look at that page and go, Oh, okay, like I get what's going on here.
• 11:16
  SPEAKER_01: Much harder to do when you're just staring at a notebook.
• 11:19
  SPEAKER_00: And yeah, there's no real like dependency graph or anything, like you can easily see.
• 11:23
  SPEAKER_00: And sure.
• 11:24
  SPEAKER_02: How many employees are you guys at today?
• 11:27
  SPEAKER_01: I think we hit 120 last week or something like that.
• 11:30
  SPEAKER_01: Okay.
• 11:31
  SPEAKER_02: Got it.
• 11:31
  SPEAKER_02: And just for maybe a little bit of a broader context on what you're actually doing in those Spark notebooks and what types of jobs you're scheduling.
• 11:39
  SPEAKER_02: Are you tasked with managing all production application data, all marketing data, or is it just everything falls to your team?
• 11:46
  SPEAKER_02: Because I know, I mean, like there must be a ton of stuff coming from all of Netlify's web assets, right?
• 11:52
  SPEAKER_02: Like there, you guys have so much stuff out there.
• 11:55
  SPEAKER_02: Um, and there are so many different ways to engage with Netlify as well, whether that's via functions or um just kind of static site hosting or authentication or whatever.
• 12:02
  SPEAKER_02: Is all of that being managed by that same Spark cluster or how do you delineate?
• 12:06
  SPEAKER_03: For the record, Pete's the one who got us using Netlify internally.
• 12:09
  SPEAKER_03: So he's given this pitch a few times to our CTO and other people on board.
• 12:14
  SPEAKER_03: Nice.
• 12:15
  SPEAKER_00: So it's a little bit of a mix.
• 12:16
  SPEAKER_00: Obviously, there's like all of the like business level metrics that like you'd always kind of expect like product managers and data science folks to want and need.
• 12:26
  SPEAKER_00: A lot of times that's stuff that's shipped towards us, I'll say, like in easy to ingest formats.
• 12:31
  SPEAKER_00: And sometimes it's us having to go out and snag it from whatever data source lives in.
• 12:36
  SPEAKER_00: And then there's also there's no direct product features, I guess, powered by the Spark clusters today, but it is like a major load-bearing piece for our building infrastructure.
• 12:47
  SPEAKER_00: So a lot of the metered building components still traverse that Spark cluster today.
• 12:51
  SPEAKER_00: For example, it might be something like um uh, you know, when you snag an image and it traverses the CDN network, obviously we log that and then eventually that makes it to a spark cluster, and we count like how many bytes it costs to transfer that, and then that image ends up getting pushed to um uh a data store that like then we use to query for building stuff.
• 13:11
  SPEAKER_00: Those are really the only kind of like load-bearing directly into product ones.
• 13:16
  SPEAKER_02: Yeah, I have to imagine the entire billing infrastructure over there has to be egregiously complex just because there are literally so many variables.
• 13:25
  SPEAKER_00: Yeah, and also the billing pipeline, people tend to underestimate the volume, right?
• 13:29
  SPEAKER_00: So it's easy when you start out with a product and you're maybe just billing for like seats or something like that.
• 13:35
  SPEAKER_00: And then even as you start adding metered components, but like at our scale, also things like trying to just um track actual bytes transferred starts getting like hard.
• 13:46
  SPEAKER_02: I can imagine that is really challenging.
• 13:49
  unknown: Yeah.
• 13:49
  SPEAKER_00: So is that it's a challenge because of the volume, not because it's like technically hard, right?
• 13:54
  SPEAKER_02: Sure, sure.
• 13:55
  SPEAKER_02: And is that running on Airflow now or is that something running still running natively in Databricks or Spark?
• 14:01
  SPEAKER_00: Right now it's still in in Databricks and Spark.
• 14:06
  SPEAKER_02: And are there plans to port that process over to Airflow at some point?
• 14:10
  SPEAKER_00: Definitely be nice to start having Airflow manage those jobs.
• 14:14
  SPEAKER_00: Because those are the ones like that are really tricky where like if for some reason ingestion fails happens downstream.
• 14:20
  SPEAKER_00: Like we definitely want to be able to like independently retry other pieces of the billing pipeline, that kind of good stuff.
• 14:25
  SPEAKER_02: Sure.
• 14:26
  SPEAKER_02: For what it's worth, we run all of our SAS products billing internally on Airflow.
• 14:31
  SPEAKER_02: And it actually is very nice because as you mentioned, if if we get an alert of a failure of some kind, because we do ingest quite we we have a similar billing model to you guys.
• 14:39
  SPEAKER_02: We probably don't have the volume of users that Netlify has, but we it's quite complex under the hood because we do need to bill based on resource consumption, right?
• 14:46
  SPEAKER_02: Um and if for whatever reason something isn't picked up, it's very good to be able to jump on that and backfill it as soon as possible and be alerted of that as soon as possible.
• 14:54
  SPEAKER_02: All of it kind of for us lands in a downstream Snowflake warehouse.
• 14:58
  SPEAKER_02: So like we want when it hits that Snowflake warehouse, we want the data to be clean and good, so we don't need to you know rerun the entire pipeline.
• 15:03
  SPEAKER_02: So Airflow has helped us tremendously in that mission.
• 15:09
  SPEAKER_03: We're dock footing our own product.
• 15:12
  SPEAKER_03: You gotta do it, right?
• 15:13
  SPEAKER_03: Yeah, absolutely.
• 15:15
  SPEAKER_03: I'd love to talk a little bit about how you know you decided you're moving to Airflow, or uh it sounds like Brian, you kind of uh advocated for it internally.
• 15:23
  SPEAKER_03: But I'd love to know about the rollout process a little bit, you know, especially at a company that's dealing with the type of problems that y'all are.
• 15:29
  SPEAKER_03: You know, like how did you get it up and running and how are you kind of helping new users jump on?
• 15:35
  SPEAKER_01: Yeah.
• 15:36
  SPEAKER_01: So initially this was just trying out the like Terraform module for using uh GCP Composer and just saying, hey, I want an Airflow Composer instance.
• 15:48
  SPEAKER_01: Didn't even configure it, just said, hey, go create a new project and give me Composer and let's just start playing with it.
• 15:55
  SPEAKER_01: I did all my development with Airflow locally, and so by the time everything was kind of working, I had about four or five jobs written out for one of these major projects, four or five different DAGs.
• 16:07
  SPEAKER_01: And I said, Okay, I want this to run, not on my laptop anymore.
• 16:11
  SPEAKER_01: It's time to get this off of my laptop and ship somewhere and running reliably.
• 16:17
  SPEAKER_01: So we initially just kind of rolled that out and started scheduling these jobs, but then we kind of ran into uh some trouble where we wanted to start migrating other jobs into Airflow and off of Databricks, specifically talking to like other firewalled data sources.
• 16:37
  SPEAKER_01: We wanted to talk to a MongoDB cluster and a Cloud SQL instance that were an entire entirely different GCP project, you know, networked off, everything's only accessible by private IP.
• 16:50
  SPEAKER_01: And so we're starting to go down the path of we got to leverage like the Cloud SQL proxy, and then we need to develop a solution actually to make the MongoDB proxy.
• 16:59
  SPEAKER_01: And the idea is we're gonna be pulling massive amounts of information from MongoDB into Spark to do transformations and aggregations, and we didn't want to go over the public internet if we can just do that you know in the same data center too.
• 17:14
  SPEAKER_01: And then so it'd save some time if we could all do that on the private network and I think save some costs since we're not like going out the out to the internet to actually grab that data and bring it in.
• 17:25
  SPEAKER_01: That's been incredibly uh complicated to do and set up.
• 17:30
  SPEAKER_01: We've had a lot of pain points and working with the SREs, they're doing a great job on it, but it's just a lot of plumbing to get working.
• 17:38
  SPEAKER_01: So that's kind of the pain point right now, because we're still actually rolling that part out of making everything flow to where Airflow can really connect to all these data sources that we need to with a minimum amount of friction.
• 17:53
  SPEAKER_01: Since we're still actually having that as a work in progress, we decided to spin up a second Airflow instance just this week.
• 18:02
  SPEAKER_01: We are now on Astronomer for our second Airflow Cluster, and we decided we're gonna move all the pipelines over to Astronomer that don't require that connection to Mongo or Cloud SQL and just get all our other pipelines going and start scheduling all these other jobs that were scheduled on Databricks.
• 18:19
  SPEAKER_01: We're gonna just move them over on Airflow with Astronomer.
• 18:21
  SPEAKER_01: And yeah, so it's three days in using Astronomer, never used it before, and it's been awesome.
• 18:28
  SPEAKER_01: It's been a pleasant surprise.
• 18:30
  SPEAKER_03: I don't know about you.
• 18:30
  SPEAKER_02: I had no idea that yeah, I was gonna say that's a surprise to us.
• 18:34
  SPEAKER_02: I we saw a support bot come in or something, and we figured it was something to do with we're building our new website infrastructure out on Netlify.
• 18:40
  SPEAKER_02: This is a sidebar right now, I probably won't include this, but I honestly you guys said we're trying this out.
• 18:46
  SPEAKER_02: Um that's cool.
• 18:48
  SPEAKER_02: We're happy to talk about that separately, but anyway.
• 18:51
  SPEAKER_02: Um back to the regularly scheduled programming.
• 18:55
  SPEAKER_02: Raj, I think you were gonna dig in there, yeah.
• 18:57
  SPEAKER_03: Yeah.
• 18:58
  SPEAKER_03: Just kind of digging in a little bit, right?
• 19:00
  SPEAKER_03: You know, one thing we're always curious to see is what's the first good workflow to move on to Airflow.
• 19:04
  SPEAKER_03: So you mentioned when you started off your kind of a four or five DAGs to get things started.
• 19:08
  SPEAKER_03: What did those really do?
• 19:09
  SPEAKER_03: And what made them good candidates for like first workflow?
• 19:13
  SPEAKER_01: Yeah.
• 19:14
  SPEAKER_01: So my first big project when I was hired was to develop this cost of goods sold project.
• 19:19
  SPEAKER_01: We call it the COGS project.
• 19:21
  SPEAKER_01: And we knew it was gonna be you know multiple months long.
• 19:25
  SPEAKER_01: The idea behind it is to grab all of our cloud billing data, like all the invoice line items from every single cloud provider, and really put that into a common model to answer questions like how much is it costing Netlify to transfer you know one gig of data for our CDN?
• 19:43
  SPEAKER_01: Or you know, how much is our you know, our our build process costing, you know, for one build minute?
• 19:49
  SPEAKER_01: And then breaking that out into dimensions of maybe it's location, maybe it's cloud provider, all sorts of different dimensions like that.
• 19:58
  SPEAKER_01: And so we knew from the get-go it's gonna be kind of complicated because we need to pull in all these different data sources and really get into one common data model.
• 20:08
  SPEAKER_01: And some of these providers they only have an API, some like Google just throws it in BigQuery, so you got to pull it out of BigQuery.
• 20:16
  SPEAKER_01: AWS lands all their data in S3, like Packet and DigitalOcean, they got APIs.
• 20:22
  SPEAKER_01: So, like how we collect the data is all different, and then the data is at a different granularity, and you know, all the columns are completely different, right?
• 20:31
  SPEAKER_01: So I decided to just start building it out by leveraging like a lot of the things that were already there.
• 20:38
  SPEAKER_01: There's the S3 to Redshift operator.
• 20:41
  SPEAKER_01: I'm like, okay, that makes some things easy, right?
• 20:43
  SPEAKER_01: To at least ingest this over to Redshift, and then you know, Python operator just go call a plane uh method just to you know I scrape the API, get all this data, and uh land that in a like a CSV, upload it to S3, and then ingest it into Redshift.
• 21:02
  SPEAKER_01: So it's kind of pulling all this stuff in.
• 21:04
  SPEAKER_01: And so yeah, we got one DAG for kind of each cloud provider pulling that in, getting that data landed like raw into Redshift, and then once it's there, then we use DBT and do all our transformations up all the way to kind of the analytics model that other people in the company would use.
• 21:22
  SPEAKER_02: So I want to dig into the DBT piece in just a minute, but just to take a step back and maybe this is a clarifying question for myself when you deploy a site to Netlify, where does the web server actually live?
• 21:33
  SPEAKER_02: Is that variable or is that hosted statically on some major cloud provider?
• 21:37
  SPEAKER_02: Are you able to choose where that runs?
• 21:39
  SPEAKER_02: It's variable.
• 21:40
  SPEAKER_02: Okay.
• 21:41
  SPEAKER_02: So is that what you're talking about?
• 21:42
  SPEAKER_02: All of that stuff coming from all of the different cloud providers basically.
• 21:45
  SPEAKER_02: It's all of these web servers that are Yeah, exactly.
• 21:48
  SPEAKER_00: So it's it's sort of like how we basically deploy pops in multiple regions, like points of presence.
• 21:53
  SPEAKER_00: So like we can spread it across uh multiple cloud providers, like all the different cloud providers have different regions, right?
• 21:59
  SPEAKER_00: Let's
• 22:05
  SPEAKER_02: Yeah.
• 22:05
  SPEAKER_02: So so Brian, basically when you're calculating things like cost of goods sold, right?
• 22:10
  SPEAKER_02: Like you basically need to figure out what those web servers are costing you on each of the clouds where they're being hosted and figure out how you can factor that into your pricing model so that your margins can be consistent across the cloud providers.
• 22:26
  SPEAKER_02: Is that an accurate statement?
• 22:27
  SPEAKER_01: Exactly.
• 22:28
  SPEAKER_01: Yeah.
• 22:29
  SPEAKER_01: So I mean just to go like one step further, you know, like I'm just gonna pick AWS because I'm I'm most familiar with AWS, but um like we need to figure out for for all these CDN nodes, like what's the EC2 cost, what's the bandwidth cost coming through, like how much are we getting charged for transferring bandwidth?
• 22:48
  SPEAKER_01: You know, what about the attached public IPs on it, elastic IPs, and then what about the EBS volume?
• 22:54
  SPEAKER_01: So like literally we're taking all the costs that we're getting billed for delivering this service and really trying to figure out how okay, how much does it cost to do uh this action for the company?
• 23:06
  SPEAKER_01: Sure.
• 23:07
  SPEAKER_02: And so so you're orchestrating all of that with Airflow, and then you're doing the M warehouse transformation with DBT, and presumably you're launching a DBT job from your Airflow DAG or something like that.
• 23:17
  SPEAKER_02: Does that sound right?
• 23:18
  SPEAKER_01: We're getting there.
• 23:19
  SPEAKER_01: That's another thing I was gonna bring up is I had so many pain points of trying to get DBT, just the right Python modules on uh Composer.
• 23:29
  SPEAKER_01: I spent like a half a day just trying to play with these you know Python pip strings of saying, okay, well, what if I loosen this dependency?
• 23:37
  SPEAKER_01: Can I get things to work?
• 23:38
  SPEAKER_01: And it just aired out every single time.
• 23:41
  SPEAKER_01: And uh I probably spent more time than I should have.
• 23:44
  SPEAKER_01: And then uh just for kicks, two days ago on Astronomer, I was like, well, let's see what happens if I just throw this in the requirements.txt and you know go ahead and and do a deployment.
• 23:55
  SPEAKER_01: And it installed it like first try.
• 23:58
  SPEAKER_01: I was like, oh, maybe I can get you know Airflow and DBT working together side by side.
• 24:03
  SPEAKER_01: So yeah, it's kind of still a work in progress.
• 24:05
  SPEAKER_01: Ideally, like I want the Airflow job to do all of the transformations and you know, I mean not all the transformations, but all of the loading, extracting, and loading is happening in Airflow.
• 24:18
  SPEAKER_01: And then I'd love for it to at the end of the DAG say, okay, now go call DBT.
• 24:22
  SPEAKER_01: And so end-to-end, I want it to kind of manage everything.
• 24:29
  SPEAKER_02: Yeah, we see that quite a bit with our customers, actually.
• 24:31
  SPEAKER_02: We at Astronomer really like DBT as a tool for managing the SQL transformations at the end of an ELT pipeline, you know?
• 24:39
  SPEAKER_02: But adding the airflow layer in to visualize the end-to-end process can be massively valuable for folks that really liked being able to see all of that extract and load alongside the presumably more complex transformation logic that happens downstream.
• 24:54
  SPEAKER_02: So I guess now, Viraj, I'm gonna throw it over to you to talk a little bit about the future.
• 25:00
  SPEAKER_03: Yeah, so one thing we always love hearing about, uh especially it relates to the use cases, is um where do you want to see airflow go towards in the future?
• 25:08
  SPEAKER_03: What kind of feature sets, use cases, et cetera?
• 25:11
  SPEAKER_03: If you could kind of snap your fingers and just have Airflow be a lot better at certain things or do certain things, where does that finger snapping happen?
• 25:20
  SPEAKER_01: So something that happened yesterday, uh part of rolling out like you know, astronomer and I'm getting my like DAGs uploaded, setting all the variables, setting the connections up, and then I actually wanted to rerun my DAG, I forgot a variable.
• 25:36
  SPEAKER_01: So I had 20 tasks or uh DAG failures, right?
• 25:40
  SPEAKER_01: And so when I go into uh Airflow and I try to like delete those runs, I was hoping it would just rerun them all, like figure out oh, you deleted them, now I need to go rerun them because I had that backfill or I can't remember what that exact tag was, but when you say go run the past and then kind of catch up.
• 25:58
  SPEAKER_01: Uh, even if I deleted them, it didn't seem to like trigger everything.
• 26:02
  SPEAKER_01: I had to go into my DAG task and delete each individual one.
• 26:08
  SPEAKER_01: I had to clear each DAG run one by one through the UI, which just was a pain.
• 26:13
  SPEAKER_01: It just seemed like it took forever.
• 26:15
  SPEAKER_01: I was like, there's got to be a better way.
• 26:16
  SPEAKER_01: So maybe there is, maybe I'm just uh uh airflow noob on on some of these things, but that's like the biggest one I thought is if you could like clear a date range and just say, hey, for from here to here, like clear all of it.
• 26:28
  SPEAKER_01: That way you're not just continually clicking on the UI.
• 26:32
  SPEAKER_03: Yeah, yeah.
• 26:33
  SPEAKER_03: There's a ton of, I think, with Airflow 2 and kind of the revamped API, that'll kind of make all of that all ideally make a lot of those pain points feel uh a lot less apparent.
• 26:45
  SPEAKER_02: So before we wrap up here, um, and this has been great, guys.
• 26:47
  SPEAKER_02: Is there anything that you want to plug at Netlify?
• 26:50
  SPEAKER_02: Um, any roles you have open, anything that you'd like to say?
• 26:54
  SPEAKER_01: Yeah, definitely.
• 26:55
  SPEAKER_01: So, I mean, Netlify, we're trying to revolutionize the way organizations and developers build websites.
• 27:02
  SPEAKER_01: It's really cool.
• 27:03
  SPEAKER_01: I feel like we're kind of the underdog right now, but we have a really strong community behind us pushing the Jam stack, and it's really exciting.
• 27:13
  SPEAKER_01: There's really awesome things beginning to take shape and kind of standardize around this.
• 27:18
  SPEAKER_01: So, you know, we're still kind of maybe in our infancy of using both Airflow and DBT in production, but that is the vision for our data team.
• 27:28
  SPEAKER_01: So we're hiring more data engineers and more data analysts, and then some analytics engineers as well who can kind of specialize and and kind of just get right into that DBT code, start start working with that.
• 27:44
  SPEAKER_02: Love it.
• 27:44
  SPEAKER_02: And not to be the dead horse, I promise I'm not getting paid to advertise you guys, but huge fan of Netlify, both for personal projects and you know, professional projects.
• 27:53
  SPEAKER_02: Definitely my default for deploying any site or web application to the internet.
• 28:00
  SPEAKER_00: We definitely love to hear that.
• 28:04
  SPEAKER_02: Thanks so much to Brian and Florian for coming on.
• 28:07
  SPEAKER_02: It was such a pleasure chatting with them.
• 28:09
  SPEAKER_02: Obviously, I'm a huge Netlify fan, as you can probably tell from that episode.
• 28:12
  SPEAKER_02: Um, so I was super stoked to be able to interview them and hear about how they're um approaching their their data infrastructure.
• 28:19
  SPEAKER_02: Uh please do email me at Pete P E T E at astronomer.io if you have anything you'd like us to discuss in an upcoming episode, any guests that you think would be a good fit for the show.
• 28:29
  SPEAKER_02: Um happy to entertain any ideas, just please do reach out.
• 28:32
  SPEAKER_02: Appreciate you listening, and we'll see you next time.