Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Use Cases

Transcript

Use Cases · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Hi there, and welcome to episode two of the Airflow podcast produced by Astronomer.
• 0:05
  SPEAKER_02: Over the past six months, we've taken a complete deep dive into the data engineering space and have heard from some extremely interesting people about their use of Apache Airflow.
• 0:13
  SPEAKER_02: We're extremely excited to keep the ball rolling in our mission to solidify airflow as the gold standard for ETL.
• 0:20
  SPEAKER_02: If you didn't catch the last episode, my name is Pete DeJoy and I'm a product specialist here at Astronomer.
• 0:25
  SPEAKER_02: I conducted these interviews with Viraj Perek, one of Astronomer's data engineers.
• 0:29
  SPEAKER_02: While you'll likely hear from me occasionally, you'll probably hear more of Viraj's voice asking interview questions, as he's the real airflow expert.
• 0:37
  SPEAKER_02: For reference, he sounds something like this.
• 0:40
  SPEAKER_07: Hi, I'm Viraj.
• 0:42
  SPEAKER_07: I like the New York Knicks, long walk on the beach, and talking about data infrastructure.
• 0:47
  SPEAKER_02: This week we'll be discussing airflow use cases.
• 0:50
  SPEAKER_02: We have an awesome group of guests lined up, including Patrick Atwater from Argo Labs, Maxime Petrosky from the City of San Diego, Scott Halgram from Zapier, Boca DeBruin from ING, Chris Riccomini from WiiPay, and Ben Gregory from Right Here at Astronomer.
• 1:04
  SPEAKER_02: Without further ado, let's dive into episode two of the Airflow podcast.
• 1:08
  SPEAKER_02: Use cases.
• 1:10
  SPEAKER_02: Just a quick disclaimer before we get started with this episode.
• 1:13
  SPEAKER_02: This episode is a bit more fragmented than our last one.
• 1:15
  SPEAKER_02: We included interviews with six different people, and the transitions are a little bit rough, so in order to make it a little bit easier to understand, you'll hear a sound like this when an interview starts and ends.
• 1:31
  SPEAKER_02: We'll get things kicked off with a conversation with Patrick Atwater, Water Data Project Manager at Argo Labs.
• 1:37
  SPEAKER_02: Argo is a startup nonprofit that builds, operates, and maintains pioneering data infrastructure to transform how water reliability, street quality, and other basic public services are delivered.
• 1:47
  SPEAKER_02: They're doing some really cool stuff, but I'll let Patrick dive in further.
• 2:04
  SPEAKER_02: Patrick, mind kicking us off by telling us a little bit about yourself and Argo?
• 2:08
  SPEAKER_01: Yeah, so I serve as water data project manager for Argo Labs, a public data infrastructure nonprofit.
• 2:15
  SPEAKER_01: We power a coalition of California water utilities and integrated urban water use data across the entire state, delivering Governor Brown's historic framework to make conservation a way of life and benchmark, the first available estimates of water efficiency across the entire state.
• 2:33
  SPEAKER_07: Oh that's sick.
• 2:40
  SPEAKER_01: So I had a bit of a meandering career.
• 2:51
  SPEAKER_01: Went back to grad school in New York, the Center for Urban Science and Progress, which is where met Maroon.
• 2:57
  SPEAKER_01: Bunch of us kind of all had shared a similar kind of dream about modernizing uh local government operations.
• 3:04
  SPEAKER_01: Uh some municipalities were interested in our work, you know, spun this up and yeah, been doing it for about three years now.
• 3:13
  SPEAKER_02: Very cool.
• 3:14
  SPEAKER_02: We haven't actually spoken to too many folks that are using Airflow in the public sector, so really excited to hear what you have to say.
• 3:19
  SPEAKER_02: So kind of going off of that a little bit, you mind just diving in and telling us a little bit about your specific use case at Argo?
• 3:24
  SPEAKER_01: Yeah, so I mentioned um we delivered this kind of water efficiency benchmarks for Donor Broad's new historic framework to make conservation a way of life.
• 3:34
  SPEAKER_01: So California just fresh off the worst drought in 600 years uh with future uncertainty.
• 3:42
  SPEAKER_01: There's a lot of hydrologic records that suggest the 20th century was abnormally wet, even before you talk about something like climate change.
• 3:49
  SPEAKER_01: So kind of common sense to get our docs in a row about how we can use uh scarce, frequently scarce resource uh widely efficiently.
• 3:58
  SPEAKER_01: So we do that, you know, the governor's performance is basically assigning uh a water budget across all of California's communities.
• 4:06
  SPEAKER_01: And what that means is seeing what is reasonable for people like residential youth to be able to use indoor and outdoor.
• 4:13
  SPEAKER_01: But there's a lot of fun we really focused on this kind of first iteration on that residential piece.
• 4:19
  SPEAKER_01: So to do the indoor, that required getting uh population data across all the states, which is fairly straightforward.
• 4:26
  SPEAKER_01: Um, and then also on the outdoor, we had to get uh land use information so we know which which parcels are residential, um, and then integrate that with landscape area data.
• 4:35
  SPEAKER_01: So that required um classification of remote sensing of aerial imagery.
• 4:41
  SPEAKER_01: Um and then so that landscape area data to figure out the water requirements, we had to integrate that with evapotranspiration um data, so like how much water evaporates, how much transpires through plants.
• 4:52
  SPEAKER_01: Um, so we had to integrate all those different data streams and then perform some calculations uh to enable this kind of visual visualization planning tool.
• 5:01
  SPEAKER_01: Um and all that happens automatedly and continues to pipe into kind of a live dashboard.
• 5:07
  SPEAKER_01: Um and then on the other end, we have to get the actual urban use.
• 5:10
  SPEAKER_01: So we're kind of integrating all these diverse data streams.
• 5:13
  SPEAKER_01: There's tons and tons of little transforms that make them uh kind of link up together.
• 5:18
  SPEAKER_01: Um and uh it's sort of a you know, it's it's it's a it the result is a visualization tool that's kind of uh simple and easy to use and elegantly kind of abstracts away from all that back-end stuff that has to happen.
• 5:39
  SPEAKER_02: And there you have it, the first of a handful of Airflow use cases that we're gonna be discussing on this episode.
• 5:44
  SPEAKER_02: Be sure to check out Patrick at PATWater on Twitter.
• 5:47
  SPEAKER_02: He's always posting some really interesting stuff about public data and tech, and he's a great follow.
• 5:52
  SPEAKER_02: Next up, continuing with the trend of Airflow in the public sector, we have Maxime Petrosky, CDO of the city of San Diego.
• 5:58
  SPEAKER_02: Maxime's gonna walk us through his setup and give us a higher-level overview of what he's able to do for in-government employees using Airflow.
• 6:10
  SPEAKER_02: Maxime, mind kicking us off by giving us a little bit of a background about yourself?
• 6:15
  SPEAKER_00: Um, yeah, so um I'm the chief data officer for the city of San Diego.
• 6:20
  SPEAKER_00: Um, and I am I run the data and analytics team in performance and analytics.
• 6:26
  SPEAKER_00: So what we do is we help people get and use data across the city.
• 6:31
  SPEAKER_00: So we make sure that you know they know it exists, they can discover it, they can understand it, um, and they can ask intelligent questions, and then they can move to a decision.
• 6:42
  SPEAKER_02: Very cool.
• 6:42
  SPEAKER_02: So, what does your path look like?
• 6:44
  SPEAKER_02: How have you ended up in the position that you're in now?
• 6:48
  SPEAKER_00: Sure.
• 6:48
  SPEAKER_00: Uh so my background is uh my background is in software engineering.
• 6:52
  SPEAKER_00: I worked at a few startups, I worked at a dev shop.
• 6:56
  SPEAKER_00: Um, what kind of got me in here was I was a fellow at Code for America in 2014 and I worked with Puerto Rico.
• 7:05
  SPEAKER_00: And um, you know, I kind of got to see there, you know, how how much impact um technology applied correctly can make to real people's lives.
• 7:17
  SPEAKER_00: And so, you know, so after that, you it was like an 11-month fellowship, and so after that when I was looking around, I kind of saw this pop up and kind of thought they wanted a bureaucrat.
• 7:27
  SPEAKER_00: I was wrong.
• 7:29
  SPEAKER_00: Um so here I am.
• 7:31
  unknown: Awesome.
• 7:33
  SPEAKER_02: Very cool.
• 7:33
  SPEAKER_02: So mind diving in a little bit into your airflow stack and how you're using it for the city of San Diego.
• 7:40
  SPEAKER_00: Mm-hmm.
• 7:41
  SPEAKER_00: So what we do is uh one of the things that my team does is we run data.sandiego.gov, which is um San Diego's open data portal.
• 7:51
  SPEAKER_00: And what generally the practice is with open data is you stand up a portal and then you figure out um some sort of process where people in departments have to go pull a report from a system and upload it.
• 8:03
  SPEAKER_00: Um when we were thinking about this and setting up and figuring out how to set this up, we decided it was not the most efficient use of people's time because the reason that you have open data is to reduce the friction that people have when accessing data, whereas you know, what I just mentioned actually creates more work.
• 8:23
  SPEAKER_00: And so we decided to go full on automation.
• 8:28
  SPEAKER_00: And at the city of San Diego, we have you know, we have hundreds of different data sources, a very wide variety of data sources.
• 8:34
  SPEAKER_00: So we have Excel files on Samba ShearDrives, we have Access Databases on FTP, we have STP HANA, we have three versions of Oracle, SQL Server, uh two ArcJ, two SRI data warehouses.
• 8:47
  SPEAKER_00: So we have a variety of different types of data sources.
• 8:50
  SPEAKER_00: And so we needed the flexibility to connect to them all and be able to do ETL out of those data sources, um, clean it up, um enrich all of that stuff, and push it out to data.sandiego.gov.
• 9:08
  SPEAKER_00: So that was kind of the initial use of it.
• 9:11
  SPEAKER_00: And then as you know, as we continued on, we started, you know, if you think of Airflow not just as a data automation system, but just like as an automation system, right?
• 9:21
  SPEAKER_00: Well now I can send you emails based on data, or I can send you emails when certain rules get hit, um, or I can notify you when things are going wrong, and I have you know, I have the entire Python ecosystem at my exposal or my disposal to do that.
• 9:38
  SPEAKER_00: So that that's kind of what makes it powerful for us.
• 9:41
  SPEAKER_07: Yeah, for sure.
• 9:42
  SPEAKER_07: Um I think one of its biggest advantages is that you have the entire Python ecosystem behind it.
• 9:48
  SPEAKER_07: Um can you say a little about how or the process you had to go through to adopt it?
• 9:52
  SPEAKER_07: Like what made what about Airflow made you choose it over the other tasks schedulers out there?
• 9:59
  SPEAKER_00: So the other one that we mainly looked at was um Safe FME.
• 10:05
  SPEAKER_00: Um and then, you know, and Safe FME falls into the same product because not everyone seems familiar with it.
• 10:10
  SPEAKER_00: It falls into kind of that same family of products as Pentaho or Altix or one of those you know, clicky, draggy, droppy um ETL things.
• 10:21
  SPEAKER_00: Um and what we quickly realized was one, um there's no code.
• 10:28
  SPEAKER_00: So just like with any WizardWig editor of anything, right?
• 10:31
  SPEAKER_00: Like whenever you want to change something or remember what you did, it's really hard when there's no code.
• 10:36
  SPEAKER_00: Um and also that no code, or there is code actually, generates XML, but I mean it's just like crazy.
• 10:44
  SPEAKER_00: Um it's like it's kind of like what the um Xcode UI builder does, but crazier.
• 10:50
  SPEAKER_00: Um and what was everything I was gonna say?
• 10:53
  SPEAKER_00: Oh, and then you know, no code means no version control.
• 10:56
  SPEAKER_00: So if I have to have multiple people working at my ETL jobs, that becomes really hard.
• 11:01
  SPEAKER_00: And then on top of that, I have to learn like an entirely new thing just to be able to add additional connectors, and sometimes I can't because it's a proprietary system and I can't contribute them back, and it just like becomes a you know, it's too much, too much of a headache.
• 11:15
  SPEAKER_00: Um whereas airflow, you know, it's just I mean, really, it's just a flask one app, right?
• 11:21
  SPEAKER_00: Um and so everything's kind of predictable, everything's been done before.
• 11:26
  SPEAKER_00: Um it's it's yeah, there's a there's things you have to learn to understand, like DAGs, but I mean that took about, you know, I don't know, two weeks, two weeks, maybe a week of immersion and reading the documentation, and I'm probably overestimating that to really understand how it all comes together.
• 11:43
  SPEAKER_00: Um and then from there, you know, we run it.
• 11:47
  SPEAKER_00: Um the way we run it is we run it on a uh Docker compose configuration.
• 11:52
  SPEAKER_00: So we we maintain stereo, we maintain parity with Dev and prod.
• 11:57
  SPEAKER_00: Um, we make sure that our credentials are separate outside of the code base, so we make sure it's secure.
• 12:04
  SPEAKER_00: Um and so yeah, and so so actually going back to your question, sorry, I wanted to expand about how we actually deployed it.
• 12:14
  SPEAKER_00: So and well, I guess that was the evaluation process, right?
• 12:18
  SPEAKER_00: And then how we went about actually deploying it is so I couldn't put something out, so we have a VPN, we have a closed network, and there's a bunch of databases that run on that network that have different levels of security and passwords and things like that.
• 12:32
  SPEAKER_00: So it was you know, and I and I work really closely with IT and the CISO and all that, and so uh we kind of decided from the very beginning there was a pretty bad idea to put Airflow on an EC2 server outside of the network.
• 12:47
  SPEAKER_00: And we didn't have um, you know, we didn't want to do all the crazy reverse proxy stuff.
• 12:52
  SPEAKER_00: We thought that was too complicated, and there's really no need to.
• 12:55
  SPEAKER_00: So um IT just gave me uh Suzy box on our network in our data center, um, and that's where I brought up the Docker containers for Airflow, and that's basically where all the connections to the different databases are made.
• 13:10
  SPEAKER_07: Yeah, sounds awesome.
• 13:11
  SPEAKER_00: So could you go and and I'll add one more thing, and in terms of government procurement, it was awesome because anytime you have to procure something in government, it's a pain.
• 13:20
  SPEAKER_00: But because Airflow is open source, it made uh it let us focus more time on the development of the jobs and additional connectors rather than going through procurement.
• 13:34
  SPEAKER_07: Oh yeah, that definitely seems like it would be a huge plus.
• 13:37
  SPEAKER_07: I can't even imagine what that's like.
• 13:39
  SPEAKER_00: I've I've been through it and they still have nightmares.
• 13:44
  SPEAKER_07: Yeah.
• 13:44
  SPEAKER_07: So uh could you dive a little more into your use cases for it?
• 13:48
  SPEAKER_07: Um you say you use it to move data and schedule jobs, but uh kind of give me just more details on that and how exactly it affects like everyone across the city of San Diego's department.
• 13:58
  SPEAKER_00: Cool.
• 13:59
  SPEAKER_00: So I'll give you examples.
• 14:00
  SPEAKER_00: So we um we had this project.
• 14:03
  SPEAKER_00: Um it was kind of like you know, quick win type of project as we're supposed to be at the beginning.
• 14:09
  SPEAKER_00: And uh it's it's streets.sandiego.gov, it's still up.
• 14:13
  SPEAKER_00: Um, it's way more used and popular than I ever thought it would be.
• 14:17
  SPEAKER_00: But it basically is just a map of all the streets that we have paved, um, all the streets that we're going to pave, um, and the overall condition index for every street in the city.
• 14:28
  SPEAKER_00: So there's a truck that drives around every four years with a bunch of sensors on it and spits out a number for every street segment that we have.
• 14:36
  SPEAKER_00: Um and so when we were building this, um, so the for the mayor, um, it's a really big initiative with street paving.
• 14:45
  SPEAKER_00: He's been paving a ton of streets during his tenure.
• 14:49
  SPEAKER_00: Um, so that was, you know, it was kind of there's like a little bit of a political motivation there too.
• 14:53
  SPEAKER_00: But even with that, when we were working with the department, um the person, the data person that we were working with the department, he was not very happy with us at all.
• 15:03
  SPEAKER_00: And what we were basically asking him to do is I'm all right, I'm gonna get really kind of detailed here, so just stop me if it's if it's boring.
• 15:13
  SPEAKER_00: Yeah.
• 15:14
  SPEAKER_00: Um for sure.
• 15:15
  SPEAKER_00: So um, so whenever the city has to, or whenever the city goes and paves a street, right?
• 15:20
  SPEAKER_00: It's it's the so pothole filling or street paving is done by the transportation and stormwater department.
• 15:26
  SPEAKER_00: But there's other departments that deal with stuff on the street, right?
• 15:30
  SPEAKER_00: So you have uh the water department that may need to replace a pipe.
• 15:34
  SPEAKER_00: Um, you have the electric company that may need to go and replace some wires underground.
• 15:39
  SPEAKER_00: Um, you you know, you may have a public works project that needs to route something to a building, right?
• 15:45
  SPEAKER_00: And so all these departments need they need to coordinate because what you don't want to have happen is you know, have the water department dig up um, you know, a street on the first week of March, and then have the transportation department come and dig up the same street on the second week of March, right?
• 16:00
  SPEAKER_00: Um and so this the reason we have a thing, it's called a conflict mitigation system or typical government acronym, IMCAT.
• 16:10
  SPEAKER_00: And the way that IMCAT is fed is there's a person that pulls um a report from the street paving management system, um, does a bunch of manual processing to it, and then puts it into uh actually aggregates it with geospatial data and puts it into IMCAT.
• 16:29
  SPEAKER_00: And so that's really the data we were asking for.
• 16:31
  SPEAKER_00: We were just asking it from him in a predictable format, and we were asking it uh from him a little bit more often than once a month.
• 16:40
  SPEAKER_00: So, but this guy hated us, right?
• 16:43
  SPEAKER_00: And so we're trying to figure out why does he not like us?
• 16:46
  SPEAKER_00: And the reason is is because to get this data to get this report, he has to manually pull a query out of an application.
• 16:57
  SPEAKER_00: Um and by query I don't actually mean SQL query, it's like a superimposed language by the application.
• 17:04
  SPEAKER_00: Um so he has to pull that out and then he has to put it into Excel, and there's about 40-ish manual steps that he has to do in Excel every single time um to make the data compatible with IMCAT and make sure that the numbers match.
• 17:20
  SPEAKER_00: Um you know, removing all these various caveats like you know, like test data and things like that.
• 17:27
  SPEAKER_00: Um and so we were basically asking him to do it more often than he already has to do it.
• 17:32
  SPEAKER_00: And this guy's not this guy's job is not pulling data from a thing, it's you know, figuring out where to pay the streets.
• 17:39
  SPEAKER_00: Um and so we came in and what we did is we just wrote an airflow job.
• 17:46
  SPEAKER_00: Um we you know had him write down all the steps, we figured out what all the sequences of the steps were, we wrote some Python pandas code, um, and then we put it into Airflow.
• 17:56
  SPEAKER_00: So now this guy saves a bunch of time.
• 18:00
  SPEAKER_00: Our streets application is continuously updated, our data portals continuously updated, and then if anybody ever wants the correct, most up-to-date street paving data, they can go and get that, right?
• 18:13
  SPEAKER_00: Um, and then go and get that on data at sandago.gov, which means that this entire thing not only saved the streets engineers' time, but it also saved the time of anybody in the city or outside of the city that needs to get street paving data.
• 18:27
  SPEAKER_00: And it's also has significantly decreased the amount of inaccuracies that um that would happen in a reporting, and not like not malicious inaccuracies, but there's nobody in this world that can do 40 manual steps in Excel correctly at the same time.
• 18:42
  SPEAKER_07: Yeah.
• 18:42
  SPEAKER_00: Right.
• 18:43
  SPEAKER_00: Um so take that and multiply that by a variety of different people and a variety of different data sources.
• 18:50
  SPEAKER_00: Um, and that's the kind of value that we were able to bring with Airflow.
• 18:54
  SPEAKER_07: Yeah, that's awesome.
• 18:55
  SPEAKER_07: Um, definitely see the like is kind of how that make everyone's life easier.
• 19:00
  SPEAKER_07: Um mentioned last time we spoke that you set up some really cool alert system too, where different employees at different levels in the city get notifications based on certain KPIs.
• 19:10
  SPEAKER_07: Uh could you dive into that a little bit?
• 19:12
  SPEAKER_00: Exactly.
• 19:13
  SPEAKER_00: So um so we code named this thing sonar because we like code name, um, and sonar sounds cool.
• 19:21
  SPEAKER_00: Um and basically what it is is um, and right now we're kind of just working with one department, but the idea is hopefully to scale it up to multiple places across the city.
• 19:33
  SPEAKER_00: Um, but the way that um we see this working is so um so there's a job that runs that street pavement job, for example.
• 19:44
  SPEAKER_00: And um when that job runs, we just have another um we have another task in the DAG um that takes several aggregations from that data, like how many streets were paved, um, whatever, whatever, in the past 30 days, right?
• 20:01
  SPEAKER_00: So now I know that I have the correct data.
• 20:03
  SPEAKER_00: Let me just do some aggregations on it.
• 20:05
  SPEAKER_00: And um I shove it into Keen, Keen.io.
• 20:10
  SPEAKER_00: Um and then I have another set of tasks that runs that pulls out the latest things I shoved into a Keen.io, um sends them over as an API called the send grid, which then sends them out through mail gun to a person, right?
• 20:28
  SPEAKER_00: Um, but the next step of that would be to take, so for example, say that you know we paved, you know, as of this job right now today, the job runs once it runs once every day.
• 20:39
  SPEAKER_00: In the past 30 days, um, we have paved 420 miles of streets.
• 20:45
  SPEAKER_00: But our goal is to pay 450 miles of streets per day.
• 20:50
  SPEAKER_00: So now we're 30 miles below our goal.
• 20:52
  SPEAKER_00: So then now the director gets an email, right?
• 20:54
  SPEAKER_00: You don't bother the director until there's like a 30 mile gap.
• 20:58
  SPEAKER_00: Um, and then you know, say there's a 100-mile gap, so something's going really wrong, right?
• 21:04
  SPEAKER_00: And that's when you know the mayor or the CEO gets an email.
• 21:09
  SPEAKER_00: So this kind of tiered alerting system, that's what I'm hoping we can get to.
• 21:15
  SPEAKER_07: Yeah, that's awesome.
• 21:16
  SPEAKER_07: Um so jumping right in a little bit, uh, you mentioned using Keen and SendGrid.
• 21:20
  SPEAKER_07: Um, kind of talk to me about some of the other tools you're using in auxiliary airflow to kind of get around some of its shortcomings or even fit it more to your use case.
• 21:29
  unknown: Cool.
• 21:29
  SPEAKER_00: Actually, no, it's not sendgrid.
• 21:31
  SPEAKER_00: Um it's um it's called SendWithUs, send with us host our templates, and then mail gun is the back end that actually sends the emails.
• 21:41
  SPEAKER_07: Okay.
• 21:42
  SPEAKER_00: Um so yeah, so we use keen pretty heavily for instrument in terms of like what jobs are running, um, how long they're taking to run, what's failing, things like that.
• 21:51
  SPEAKER_07: Um how do you set that up?
• 21:54
  SPEAKER_07: Keen?
• 21:55
  SPEAKER_07: Yeah, um, how do you set up that keen monitors which jobs are running and how the how they've
• 22:01
  SPEAKER_00: So what actually happens is this is um it's a task that gets added to every DAG.
• 22:09
  SPEAKER_00: Um and at the end of the actually no it's not a task that gets added to every DAG.
• 22:15
  SPEAKER_00: It's something that runs at the end of every task.
• 22:18
  SPEAKER_00: And at the end of every task, we you know how there's a tag object or a task object that gets passed around?
• 22:24
  SPEAKER_00: So we send a bunch of metrics from that object as an API called a keen every single time it runs.
• 22:31
  SPEAKER_00: And because Keen is basically an unstructured database, so you know it doesn't have really, I think it's it's like hosting Cassandra.
• 22:40
  SPEAKER_00: Um so you don't have to conform to a schema.
• 22:43
  SPEAKER_00: We basically just shove our data in there and then ask questions about it.
• 22:50
  SPEAKER_02: Awesome, and thanks so much to Maxime for coming on the show.
• 22:53
  SPEAKER_02: He was an awesome dude to talk to, and we actually have had multiple conversations with him since about airflow and about what he's doing.
• 22:58
  SPEAKER_02: He did a guest blog post with us.
• 23:00
  SPEAKER_02: Uh you can find him on Twitter at Mr.
• 23:02
  SPEAKER_02: Maximize, and you can find his blog at mistermaximize.com.
• 23:06
  SPEAKER_02: We'll link to all of those things right in the description of this podcast so you can check them out.
• 23:10
  SPEAKER_02: Next up we have Scott Halgram, data engineer at Zapier.
• 23:13
  SPEAKER_02: We had a great chat with Scott, and he gave us a quick high-level overview about the problems that Zapier faces and how they're resolving them using Airflow.
• 23:23
  SPEAKER_07: Right.
• 23:24
  SPEAKER_07: Yeah, cool.
• 23:26
  SPEAKER_07: Um Yeah, so kind of let's just kind of start off with a little intro.
• 23:29
  SPEAKER_07: Uh you want to tell us a little bit about uh what you do and how you got there, Scott?
• 23:34
  SPEAKER_05: Yeah.
• 23:34
  SPEAKER_05: Um I am uh I'm a data engineer at Zapier, and um uh how did I get here?
• 23:42
  SPEAKER_05: I applied and I got the job.
• 23:44
  SPEAKER_05: No, um the history of the the data department at uh Zapier is they started off with a single data scientist and then they added a second data scientist, and as those things I think typically go, um the data scientists ended up doing a lot of data engineering.
• 23:59
  SPEAKER_05: So um they wrote me out about seven months ago to offload um uh some of that work and they had airflow up and running at that point.
• 24:08
  SPEAKER_05: Um so and I uh kind of wrapped my arms around it because that seemed to be where the most um the most of their DE work was going into maintaining airflow.
• 24:18
  SPEAKER_05: And so I wanted to free them up to do more DS work.
• 24:21
  SPEAKER_05: Um and uh let's see, since then now the the data team has seven members um and we have uh uh me and another uh data engineer working.
• 24:35
  SPEAKER_02: Very cool.
• 24:35
  SPEAKER_02: I'm glad to hear that your team's expanding a little bit, Scott.
• 24:38
  SPEAKER_02: Uh mind just diving in and giving us an overview of your use case for Airflow at Zapier.
• 24:42
  SPEAKER_05: I don't think I'm speaking out of turn to say that you know part of the pain they were feeling when they decided they needed to hire a data engineer was that a lot of times the the tree view of Airflow would look red in the morning when they started up.
• 24:56
  SPEAKER_05: Um because it was, you know, again, you know, it was a startup, it was fast growth, and so they were they were um changing the the DAGs in Airflow a lot and they were pretty frequent frequently.
• 25:10
  SPEAKER_05: Um usually it wasn't um high uh hugely important data that um a bunch of time again would be uh eaten up by the data engineers each day or the data scientists each day doing data engineering.
• 25:25
  SPEAKER_05: So that's what it was like when I came in.
• 25:29
  SPEAKER_05: And it was it was pretty close to being stable when I came in, but there was still some you know breakage.
• 25:36
  SPEAKER_05: And that's how I learned a lot about how it works is just by uh um getting in there and and uh fixing these uh these DAGs and and the the tasks really and uh more specifically.
• 25:50
  SPEAKER_05: Um so we got we got that stabilized and we also brought in a few more um products.
• 25:57
  SPEAKER_05: Uh we got Matillion, we brought in Looker and phased out Periscope.
• 26:01
  SPEAKER_05: Um and then the the next big thing we did for Airflow was um was we we created a test environment.
• 26:09
  SPEAKER_05: So we uh we have a replica of almost everything.
• 26:12
  SPEAKER_05: So Redshift, RDS, Redis, um everything in EC2.
• 26:18
  SPEAKER_05: Um and then we I have some uh uh code under the hood that uh writes to different S3 buckets if it's running on a test box.
• 26:26
  SPEAKER_05: So now we at least have a pathway to check into one repo, and then I I have a delay task.
• 26:33
  SPEAKER_05: It it runs at like 11 a.m.
• 26:37
  SPEAKER_05: Central instead of you know midnight ETC.
• 26:39
  SPEAKER_05: So we can get a we can get a a full run of a full run of the daily day again before we uh push any changes out to production that night.
• 26:47
  SPEAKER_05: And now we're working on now we're focused on um continuous integration, continuous development, um stuff like that, just trying to make things uh even more locked down.
• 26:59
  SPEAKER_05: Um one of the cool things I recently did was uh we now have I set up a I set up a DAG to look through a directory of config files, and basically uh I set up these config files to be like you can just a few parameters in there.
• 27:12
  SPEAKER_05: One is a Python callable and then the other is like a SQL query.
• 27:17
  SPEAKER_05: And so you basically so the the DAG cycles through that directory and creates a task for each config file and finds uh for that Python callable and SQL statement, it runs the SQL statement.
• 27:29
  SPEAKER_05: Basically just says, was I supposed to return zero rows?
• 27:32
  SPEAKER_05: Was I supposed to return one row?
• 27:34
  SPEAKER_05: Was I supposed to return one row with one column whose value is zero or something like that?
• 27:39
  SPEAKER_05: We do a lot of that stuff to check for duplicate and keys and stuff like that.
• 27:49
  SPEAKER_02: Huge thanks to Scott for coming on and giving us a high-level overview of how they're using Airflow at Zapier.
• 27:53
  SPEAKER_02: We'll post his information in the description as well.
• 27:55
  SPEAKER_02: He's an awesome guy, and we've talked to him a bunch over here at Astronomer.
• 27:58
  SPEAKER_02: He's really helped us out uh with a couple things that we've been working on.
• 28:02
  SPEAKER_02: Next up, we have Boca De Bruin, head of advanced analytics technology at ING.
• 28:06
  SPEAKER_02: Boka's one of Airflow's top contributors, and he's the chief maintainer of the open source project nowadays.
• 28:11
  SPEAKER_02: He's committed a ton and he was hugely involved with the 1.9 release.
• 28:16
  SPEAKER_02: He's an incredibly brilliant guy, and it was great to talk to him.
• 28:19
  SPEAKER_02: Really excited to share an interview with you all.
• 28:23
  SPEAKER_02: So, Bokeh, mind starting us off by giving us a little bit of an introduction for yourself.
• 28:29
  SPEAKER_06: Um I'm currently the CTO of Analytics for also banking and uh for ID Bank.
• 28:38
  SPEAKER_06: Uh IG Bank is a global uh global funding services organization.
• 28:43
  SPEAKER_06: I think on the also sort of in 45 countries.
• 28:47
  SPEAKER_06: Um and we're now actually getting to a point where we're able to create either products around advanced logic or are able to uh enable our business users uh better uh with the so um basically I'm co-handing together with one of um a team of data scientists, data engineers, business developers, uh, and so on.
• 29:15
  SPEAKER_06: And there are 35 people right now, but growing up to 90 people next year, and there are more and more groups projected for the for the um for the next year.
• 29:29
  SPEAKER_07: Yeah, absolutely.
• 29:30
  SPEAKER_07: Yeah, uh it's a big thing that we have uh uh as well.
• 29:35
  SPEAKER_06: Uh well you're not we're not the 90 yet, and it's like let's say the 90 is a budget, not as target, um, because we will definitely face some challenges along the way in you know getting those people in and they need to be at a certain level, obviously.
• 29:48
  SPEAKER_06: Um so we're kind of scared about it to be honest.
• 29:51
  SPEAKER_06: And we don't think we'll make the exact 90, uh, but at least it gives us some uh headroom uh for growth, especially large corporations.
• 30:01
  SPEAKER_06: You're mostly managed on budgets, and budgets are also with people.
• 30:05
  SPEAKER_06: So sometimes you'll have the money, but you can't spend it because you cannot hire the people for it.
• 30:09
  SPEAKER_06: So this is uh fortunately for now it's you know it's not there, and we'll have a little bit of uh uh headroom.
• 30:16
  SPEAKER_06: Um and yes, we're one of the bigger ones that we know of as well.
• 30:20
  SPEAKER_06: Uh, I know there are bigger ones, for example, in uh ALN in the UK.
• 30:25
  SPEAKER_06: Um, and there are, I would say, with the tech companies around, they're mostly bigger as well.
• 30:30
  SPEAKER_02: Very cool.
• 30:31
  SPEAKER_02: Well, we're super excited to talk to you.
• 30:33
  SPEAKER_02: We monitor your commits pretty regularly over here at Astronomer.
• 30:36
  SPEAKER_02: We're big fans of all the stuff that you do with Airflow.
• 30:38
  SPEAKER_02: So, what do you guys use Airflow for at ING?
• 30:43
  SPEAKER_06: Back end to front end.
• 30:44
  SPEAKER_06: So imagine that we'll have a lot of transactional systems that deliver information or real transactions basically.
• 30:52
  SPEAKER_06: So any kind of transaction that you do in a shop that's electronically there, um, we'll ingest that, put it on a platform, make sure that it's cleaned and that we understand that this transaction is this transaction, uh, and we're a bank for since the 70s, and uh, this means that we also have a lot of legacy systems.
• 31:11
  SPEAKER_06: So, what we consider a transaction in one system is not a transaction in another system, and it will be even different in the same kind of systems across the globe.
• 31:20
  SPEAKER_06: Um, and we try will try to get that into kind of all uh one thing and then put tests on that, uh, and then do our analysis or our models on that, and then move it out to either uh well, it will be to front-end systems, and sometimes sometimes these are systems like you that allow for business intelligence or are direct end user applications.
• 31:44
  SPEAKER_06: Um, and even they'll go um not right now, but they'll go pretty soon to external um external uh clients as well.
• 31:54
  SPEAKER_07: Oh, cool.
• 31:55
  SPEAKER_07: Um, I think it's uh I would love to hear more about how you said you're using it as an intermediary between conciling records from legacy systems, legacy systems versus modern systems.
• 32:05
  SPEAKER_07: Um, I don't think we've heard too much about that use case yet.
• 32:08
  SPEAKER_07: So if you could say a few a little bit more on that, that'd be really great.
• 32:11
  SPEAKER_06: I don't know, I I'm not sure if you want to know because these kinds of things are pretty complex, and uh my um the team wouldn't be too happy every time if they have to do it the whole time.
• 32:20
  SPEAKER_06: But it's it's uh uh imagine that we have mainframes that supply uh or uh our accounting system sometimes that you know have their records stored in FCDIC format, uh, and we need to move that over to a Hadoop platform uh uh while translating it to a to a more uh let's say common format that we still are able to understand.
• 32:43
  SPEAKER_06: And then you'll have uh a software that's you know often Oracle databases where you ingested information from, or where the systems that are involved there you cannot directly get in touch with because they will not allow or they will not you know um uh understand the load that you put on them on them.
• 33:04
  SPEAKER_06: Um so that you have to put like intermediaries in between to make sure that those systems can handle the load that you put at them and then move it over to those are uh to those big data systems.
• 33:15
  SPEAKER_06: Uh that's a large part of the work that we do.
• 33:18
  SPEAKER_06: Uh although large corporates will talk about data lakes and so on.
• 33:22
  SPEAKER_06: And if you talk to the Airbnb guys, they started you know with a with a big system, uh well, Hadoop system in this case as well, I think, uh, where they put everything together.
• 33:32
  SPEAKER_06: Well, that's kind of a luxury for us.
• 33:34
  SPEAKER_06: Uh we'll have to go through a lot of hoops before we get it.
• 33:37
  SPEAKER_06: The bank has a lot of information, but it's stored at so many locations.
• 33:40
  SPEAKER_06: Uh, we have to have more than 1200 applications, and those are still being used, and that's only on the wholesale side.
• 33:47
  SPEAKER_06: So you'll have it in different countries on the retail side as well.
• 33:50
  SPEAKER_06: And data does not belong into one of the silos of what you know what is in a bank.
• 33:55
  SPEAKER_06: So uh data being generated by companies will also be seen by retail customers.
• 34:00
  SPEAKER_06: Um, but the bank is organized in a siloed way, uh, so this is uh there's a lot of politics uh involved sometimes too.
• 34:08
  SPEAKER_07: Yeah, I can imagine that'd be pretty tough.
• 34:11
  SPEAKER_07: Um, yeah, I haven't even thought about the use cases of Airflow when you're not you don't have an easy-to-go data lake because internally we just dump everything in S3 and just pull and pull from it almost like it's not even a concern for us.
• 34:23
  SPEAKER_07: We're so much more concerned about all the other things.
• 34:25
  SPEAKER_07: But there you go.
• 34:26
  SPEAKER_06: So we're basically with Airflow, it allows us to create that data lake.
• 34:31
  SPEAKER_07: Yeah, that's super interesting.
• 34:33
  SPEAKER_07: Um, kind of moving right along.
• 34:35
  SPEAKER_07: Um, can you talk about some of the stack you're using around Airflow to let it do these tasks that you're talking about?
• 34:42
  SPEAKER_06: Um, so for us, we'll basically use uh the the standard Hadoop stack for and that's uh because we cannot go to cloud.
• 34:49
  SPEAKER_06: We're a European uh European bank, uh, we're a bit afraid of the Patriot Act in the in the in the in the US or any kind of way that you know the the US government likes to access uh records from uh from European customers uh somehow, some way.
• 35:05
  SPEAKER_06: Uh that's you know true or not true, but that's the uh the in the eye of the beholder.
• 35:10
  SPEAKER_06: Um so we'll be using standard Hadoop stack with Spark on top of that.
• 35:16
  SPEAKER_06: We'll use flink, use uh Spark structured streaming for anything that we the things that we put out for Airflow.
• 35:22
  SPEAKER_06: We'll be using Spark mainly for the things that you know for our data processing because it's just more efficient.
• 35:29
  SPEAKER_06: Um, we'll even launch uh uh jobs with TensorFlow.
• 35:33
  SPEAKER_06: Uh you know, just anything that you know is can be put into a job that's kind of batch oriented, they'll run that with Airflow for us on our side.
• 35:42
  SPEAKER_06: So there's a you know, we're just one part of the bank.
• 35:44
  SPEAKER_06: I know some parts of the bank are using some IBM, uh RBM components, uh but for the commercialization side, which which I'm in, um, we'll use Airflow for that.
• 35:58
  SPEAKER_07: All right, cool.
• 36:00
  SPEAKER_06: Yeah, so can you say one maybe what's interesting what we're looking at now, um, because that's probably particular to on-premise Hadoop systems, although you might consider it in the cloud as well, because uh if you have to grow go close region, um, is that well we're looking at how to use Airflow to have an active active setup for our big data systems, so that allows us to have like multiple jobs running across multiple uh multiple uh big data clusters while using them at the same time.
• 36:36
  SPEAKER_06: And this is something where we are investigating how we can make that work with Airflow.
• 36:41
  SPEAKER_06: It will require patches and it will require updates to it, but this is what we're uh researching currently.
• 36:47
  SPEAKER_07: Cool.
• 36:48
  SPEAKER_07: I would uh love to kind of hear more about that.
• 36:50
  SPEAKER_07: Um is that kind of how you're are you just putting are you envisioning putting the scheduler on almost a separate box and having that just coordinate between the different clusters?
• 36:59
  SPEAKER_07: Or are you thinking about using some more like a more complex setup?
• 37:02
  SPEAKER_06: No, there will be a little bit more complex like that because for us being um uh uh the typical CIA ratings that we'll use are confidentiality, integrity, availability.
• 37:13
  SPEAKER_06: Um, if uh if we want to go to a so availability of two for us, that's an immediate requirement that we can do that cross-data center basically.
• 37:23
  SPEAKER_06: Um, and uh in this case means that we need to be able to pull the plug on one side of Airflow and still be able to run those tasks in the other location, but with a reduced capacity at that moment.
• 37:38
  SPEAKER_06: Uh, when it comes back, it needs to be able to run again and together.
• 37:42
  SPEAKER_06: So basically, you would look at high available databases at the back end for it with an MQ that's also highly available, with a scheduler that is highly available, and then uh making sure that across those clusters you're going to be uh able to shift the data at the end of a job to another place and that you know that that you know that that situation is completed.
• 38:04
  SPEAKER_06: This gives Airflow gives you the complete uh has this task completed and has it fully completed by copying the data over to uh the other uh to the other uh to the other system.
• 38:16
  SPEAKER_06: Um availability parts that you definitely need to put in still.
• 38:22
  SPEAKER_07: Okay, that makes sense, super interesting.
• 38:25
  SPEAKER_07: Um, you also mentioned that you're using Apache Spark and Apache Flank almost in tandem.
• 38:30
  SPEAKER_07: Uh can you talk about that a little bit?
• 38:33
  SPEAKER_06: Um so Spark is being used for the batch processing, obviously.
• 38:37
  SPEAKER_06: Uh but imagine that uh and until um the micro batches in Spark streaming at that time, and although structure streaming came into that, it's still a bit micro batch based, as far as I know.
• 38:50
  SPEAKER_06: Um uh it doesn't allow the sub-second uh so so the sub um uh under under 100 milliseconds basically doesn't allow you to go there nowadays with structure streaming, it might actually do that.
• 39:03
  SPEAKER_06: But flink is just a little bit easier and a lot of areas for this kind of thing.
• 39:08
  SPEAKER_06: So uh if we look at financial markets uh where um streaming data is uh is everywhere, uh then uh then we'll use uh Apache Flink for this mostly because that subsecond part is actually pretty important.
• 39:23
  SPEAKER_06: Or if you want to do a transaction, uh so if you do a transaction in the shop, for example, and you swipe your card uh or you put your card in, you also want to have that done pretty quickly.
• 39:33
  SPEAKER_06: You don't want to wait till you know that uh that 0.5 seconds that uh star Spark Streaming gave you.
• 39:40
  SPEAKER_06: Uh and uh structured streaming maybe uh will improve upon that.
• 39:43
  SPEAKER_06: But Flink is for us uh a bit of standard in that area.
• 39:53
  SPEAKER_02: Very cool.
• 39:54
  SPEAKER_02: Thanks so much to Bokeh for coming on.
• 39:56
  SPEAKER_02: It was really great chatting with him, and we can't wait to see what he does with Airflow going forward.
• 40:01
  SPEAKER_02: Next up, we have Chris Riccomini, principal software engineer at WiiPay.
• 40:05
  SPEAKER_02: Chris also was heavily involved in the 1.9 Airflow release and had a lot of really interesting things to say.
• 40:14
  SPEAKER_02: Chris, mind giving us a little bit of an introduction for yourself?
• 40:18
  SPEAKER_08: Uh yeah, I'm I'm Chris Riccomini.
• 40:21
  SPEAKER_08: I work at uh WePay.
• 40:22
  SPEAKER_08: Um, I've been here for about two and a half years, um, and that is where I first got introduced to Airflow.
• 40:29
  SPEAKER_08: Um prior to WePay, I was at LinkedIn for about six and a half years, um, where I did a number of things, um mostly uh at both WePay and LinkedIn in the infrastructure space.
• 40:40
  SPEAKER_08: So at Wii Pay, I'm in charge of the data infrastructure and um uh let me rephrase it.
• 40:48
  SPEAKER_08: I'm I at WePay, I'm in charge of the data infrastructure, um, which kind of breaks down into two areas.
• 40:54
  SPEAKER_08: One is our offline ecosystem, so batch processing, data warehousing, ETL pipeline.
• 40:58
  SPEAKER_08: Uh, and the other one is our streaming infrastructure, uh, which is mostly Apache Copca and a bunch of data integration and connectors for it.
• 41:07
  SPEAKER_07: Um so that that's what I do now.
• 41:11
  SPEAKER_07: Um can you talk to me a little bit about uh you can stay as high level as you want, but some of the specific tasks you have DAGs running for and kind of the technical side and the use case and the business side of those.
• 41:23
  SPEAKER_08: Yeah.
• 41:24
  SPEAKER_08: Um so let me start with the business side first, or maybe the um the use case side rather than the implementations of it.
• 41:33
  SPEAKER_08: Um we have a number of different tasks, but they break or DAGs rather, but they break down into a few buckets.
• 41:41
  SPEAKER_08: One of them is ETL.
• 41:42
  SPEAKER_08: So we need to get data out of our OLTP uh production databases and get it into various systems downstream.
• 41:49
  SPEAKER_08: Um so we we use it for uh for that.
• 41:52
  SPEAKER_08: And as part of as part of that, we um we have a bunch of data quality check DAGs as well that validate that the data in the destination database matches what's in the source database.
• 42:04
  SPEAKER_08: So you don't have missing rows and you don't have, you know, perhaps you you might miss an update, and so the field value in the source is different than in the destination database.
• 42:12
  SPEAKER_08: Um the so that's one category, sort of ETL.
• 42:16
  SPEAKER_08: The the second category is about report generation and materialization.
• 42:20
  SPEAKER_08: So this is essentially once we have the data in our data warehouse, we want to generate reports based off of the data.
• 42:26
  SPEAKER_08: So this could be everything from like financial reconciliation stuff to uh use it, usage of our product, to um uh even in some cases, like uh not really monitoring, but um looking at logs in our data warehouse and seeing what's going on uh there.
• 42:45
  SPEAKER_08: Um so that's a second use case.
• 42:49
  SPEAKER_08: Um the third use case is really around machine learning.
• 42:52
  SPEAKER_08: So this is uh we want to train models, we want to um uh uh basically explore the data, do feature uh engineering for the for the models, uh, and so on.
• 43:04
  SPEAKER_08: So our data scientists um interact with Airflow a bit to do the training.
• 43:10
  SPEAKER_08: And so essentially how it works is they start off, they're fiddling around, maybe they're doing ad hoc queries eventually to get to a point where uh they train a model and it looks good.
• 43:19
  SPEAKER_08: Um we need to productize the model so that it um you know it runs always, not just one time and not just on somebody's laptop.
• 43:27
  SPEAKER_08: And so for that, we uh we run the um the DAGs in Airflow.
• 43:33
  SPEAKER_08: And that one in particular, it's really nice to have Airflow as a Python, you know, the DAGs as Python and the tasks as Python because a lot of our data scientists spend time in Python, so it's pretty natural for them to interact with Airflow.
• 43:45
  SPEAKER_08: Um so that's the third uh large bucket.
• 43:49
  SPEAKER_08: Um the fourth bucket is something that we're just getting into, and this is more uh to get back to your your cron on crack mention, um we we were uh Heavily uh cron-based a few years ago.
• 44:02
  SPEAKER_08: Um payments by nature, a lot of it is asynchronous processing after the you know credit card swipe happens.
• 44:08
  SPEAKER_08: And so um uh we had a bunch of stuff that was running in cron.
• 44:14
  SPEAKER_08: And uh you know, you bump up against all the problems that Airflow tends to solve, like retrying and alerting and all that kind of stuff, scheduling and not allowing overlapping jobs.
• 44:23
  SPEAKER_08: Um these are all problems that uh I think anyone who uses cron quote unquote at scale uh can relate to.
• 44:29
  SPEAKER_08: Um and so we are migrating a lot of our cron jobs out of um cron and into Airflow.
• 44:36
  SPEAKER_08: And the model there is is much more of um uh the task kicks off basically a call to a web service, and the web service then triggers some logic versus where it was before where like cron would would execute um the logic itself.
• 44:54
  SPEAKER_08: Um and I can get into the implementation details about that, but those are the four use cases, right?
• 45:02
  SPEAKER_08: So um cron, machine learning, reporting, and uh ETL.
• 45:07
  SPEAKER_08: Um now you wanted me to go into implementation a little bit, or did you want to uh do something else?
• 45:17
  SPEAKER_08: Yeah, so implementation on the um on the ETL side is pretty uh pretty, I would say pretty normal.
• 45:24
  SPEAKER_08: Um it's you know, select star from data over the last 15 minutes, do that every 15 minutes, load it into the destination database.
• 45:32
  SPEAKER_08: We've increasingly been moving away from uh Airflow for the core ETL pipeline in favor of uh streaming solutions, but we still use it for some uh things.
• 45:42
  SPEAKER_08: And namely, uh some of our um some of our MySQL databases are still using Airflow, um where we we take the data from MySQL and load it into uh our data warehouse, um, which in our case is Google Cloud BigQuery.
• 45:56
  SPEAKER_08: Um But uh we the second part that I mentioned I think is a little less common is to run these data quality checks, and we do two kinds of checks.
• 46:05
  SPEAKER_08: One is just a row level count.
• 46:06
  SPEAKER_08: So how many rows are in this table, how many rows are in that table?
• 46:09
  SPEAKER_08: If they match, things are good.
• 46:11
  SPEAKER_08: Um there's some slight nuance to it.
• 46:13
  SPEAKER_08: Um, for example, if you have hard deletes in your tables, it gets a little wonky.
• 46:17
  SPEAKER_08: Um in general, um, that's one data quality check we do that's sort of uh easy and relatively cheap to do.
• 46:27
  SPEAKER_08: The second one is more expensive but more thorough, and that is where we will actually do row by row, column by column checks.
• 46:34
  SPEAKER_08: And we do this less frequently, um sort of on a you know, um maybe a weekly basis, uh, would be uh more the the schedule by which we would do this, is we'll actually take you know a sliver of a table and compare it to the exact same sliver of the table in the destination database and validate that every single row and column is identical.
• 46:55
  SPEAKER_08: And that that basically lets us sleep at night because it knows it means that we know that as of you know within a week, that we would know if there was any uh bad data in the system at all.
• 47:05
  SPEAKER_08: And and that's important when you're doing things like you know generating board reports or generating reports that you would want to send your customers or generating reports that you know maybe for uh financial purposes, like all that kind of stuff, you really have to be able to trust your data.
• 47:19
  SPEAKER_08: Um so doing the data quality checks is something that gets us there.
• 47:22
  SPEAKER_08: Now, scalability-wise, some of this stuff may not uh be feasible for everybody or for all tables, but um it's definitely worth doing in cases where you can do it.
• 47:32
  SPEAKER_08: Um so that's the ETL implementation part of it.
• 47:36
  SPEAKER_08: The um reporting part uh I guess breaks down into two areas.
• 47:41
  SPEAKER_08: One are these sort of materialized reports that just get made into CSV files, and those get implemented uh emailed rather uh using like an airflow email operator, say once a day, and they just get blasted out to whoever wants to consume them.
• 47:57
  SPEAKER_08: This is something that I think um specifically some of our non-engineering teams like because they tend to consume things in like you know, CSV files or Excel spreadsheet kind of thing, and um, getting just a report every day with the stuff they need in a in an email attachment um makes them happy.
• 48:13
  SPEAKER_08: Um, the second path in this um use case bucket is that we have our business analyst folks generating um these materialized uh data warehouses for various um teams.
• 48:26
  SPEAKER_08: So, for example, you might want a risk data warehouse, you might want a product data warehouse or a sales data warehouse.
• 48:31
  SPEAKER_08: So they do that.
• 48:32
  SPEAKER_08: So they take all of our um the data that we load into our data warehouse and they you know join, filter, munge uh the stuff into something that's more consumable for the various teams or more relevant to the various teams.
• 48:43
  SPEAKER_08: We we tend to follow kind of a uh extract load transform pattern rather than extract transform load pattern, um, which means that my team, the data infrastructure team, pretty much just takes the data out and loads it into uh the downstream systems.
• 48:57
  SPEAKER_08: It doesn't do a lot of transformation as it's loading.
• 49:00
  SPEAKER_08: Um, so the the um analysts, the business analyst folks tend to do some of the transformation to make the data a little more consumable to non-engineers.
• 49:09
  SPEAKER_08: Um and then some of that stuff gets fed into our visualization tier where we you know can make lovely little line charts and pie charts and so on.
• 49:19
  SPEAKER_08: Um now, machine learning-wise, um the implementation is um it's basically a series of uh very large and sophisticated DAGs that do um some feature engineering and then uh a bunch of model uh training and then uh some validation as well to make sure that things are performing reasonably well against the holdout set.
• 49:46
  SPEAKER_08: Um and then the last part of that is pushing the data from uh our data environment into the production environment where where it can be used by the modeling system.
• 49:56
  SPEAKER_08: Um for more details on this, like we do have blog posts.
• 49:59
  SPEAKER_08: Basically, everything I'm talking about, we have blog posts on.
• 50:01
  SPEAKER_08: So I'm gonna kind of leave it at a relatively high level, but we have like pretty detailed blog posts about um how that stuff works.
• 50:07
  SPEAKER_08: But the general development flow is, as I mentioned before, uh data scientists um playing around, maybe playing with a new library or uh trying to add a new feature to an existing model, uh trying a new model, um, initially starts locally on their laptop.
• 50:25
  SPEAKER_08: Um and one of the things that we do that that I think is really pretty cool is um with with Google Cloud Airflow integration, you can actually spin up Airflow locally on your laptop and run it locally, um, but you can authenticate uh as yourself, which means uh in Google Cloud, which means you have uh Google Cloud access to uh the data that you need to do the model training and so on.
• 50:50
  SPEAKER_08: Um so they do a bit of that.
• 50:53
  SPEAKER_08: Um sorry, I'm rambling.
• 50:57
  SPEAKER_08: That's exactly what we want it.
• 50:59
  SPEAKER_08: Uh where was I going with this?
• 51:03
  SPEAKER_08: Um I'm gonna validate I was talking about the the development lifecycle for for model training and stuff.
• 51:11
  SPEAKER_08: Um anyway, so they they do some uh training either locally or uh in a Google Cloud project on some higher-end machines, maybe with GPUs.
• 51:23
  SPEAKER_08: Um eventually they um they take the stuff that they've written and uh make it into an airflow DAG, and that's what actually gets run by us on the airflow uh systems that we have.
• 51:34
  SPEAKER_08: Um the last one was cron, which I mentioned, and this one is a little bit um I'd say not experimental, but uh it's our our newest use case.
• 51:46
  SPEAKER_08: Um I think you basically have two options when it comes to cron replacement.
• 51:50
  SPEAKER_08: One is to take the logic that cron was triggering uh locally and run it locally in an airflow um task or series of tasks, right?
• 52:01
  SPEAKER_08: So this would be, for example, if you had a um if you had a cron script that was say querying a database and then uh updating a bunch of rows based on some custom logic, um you could you could have a task that would that would do that um in Airflow.
• 52:21
  SPEAKER_08: And the disadvantage to that approach is that you're basically doing all the logic locally on the task instance that is uh running the logic, which is sort of a bit of an Airflow anti-pattern.
• 52:37
  SPEAKER_08: You want in general to have as much logic being executed outside of the Airflow worker itself.
• 52:42
  SPEAKER_08: Um so, for example, if you were putting a database, or rather, if you were doing some logic, you would you would want another machine to do the logic.
• 52:50
  SPEAKER_08: Um so the way we implemented it was not to just directly port the scripts that we had in cron and copy them in as tasks in Airflow, but actually to um to have Airflow kind of be the scheduler, but be the um have this have a web service or series of web services do the execution.
• 53:06
  SPEAKER_08: So in our model, what that means is the Airflow uh DAGs kickoff and the tasks uh essentially issue a curl request or an HTTP request to a web service, um, and that web service is responsible for executing uh the logic that needs to get executed.
• 53:24
  SPEAKER_08: Um, the trade-off here is like if it if it takes a long period of time for the web service to do the work, you end up having to manage the connection between Airflow and the web service over a long period of time.
• 53:34
  SPEAKER_08: If there's network flips, you have to reconnect.
• 53:35
  SPEAKER_08: So you end up having to build kind of like job execution state into the web service.
• 53:40
  SPEAKER_08: Um so that's the the downside.
• 53:42
  SPEAKER_08: But the upside is that the tasks in Airflow become very dumb and and therefore very uh very easy to operate because they're not using any CPU, they're not using any disk, they're not um you know using any memory, they're just kind of sitting there triggering the execution of the logic.
• 53:58
  SPEAKER_07: Sounds like keep that pretty item potent too, which sounds nice.
• 54:01
  SPEAKER_07: Sounds nice.
• 54:02
  SPEAKER_08: Yeah, yeah.
• 54:02
  SPEAKER_08: That again, that's sort of like uh airflow golden rule.
• 54:06
  SPEAKER_08: You should be able to run the tasks in DAGs over and over again without breaking things.
• 54:14
  SPEAKER_02: And have no fear, we'll have an Airflow best practice guide episode released next, so that will be two weeks from now.
• 54:20
  SPEAKER_02: Thanks so much to Chris for coming on.
• 54:21
  SPEAKER_02: Again, also great chatting with him, as it has been with everybody else.
• 54:25
  SPEAKER_02: Really had a lot of interesting things to say.
• 54:27
  SPEAKER_02: And now we have our final interview of this episode, and it's with one of our own folks here at Astronomer.
• 54:32
  SPEAKER_02: Ben Gregory was astronomer's first employee and has self-taught himself into a data engineer role.
• 54:37
  SPEAKER_02: Natural self-starter, Ben has been absolutely integral to the core of our team and is responsible for a lot of the internal reporting we've set up using Airflow and is an expert on all things ETL.
• 54:47
  SPEAKER_02: Fun fact is that Ben Virage and I are all at Astronomer through a fellowship program called Venture for America.
• 54:53
  SPEAKER_02: Definitely check them out at ventureforamerica.org if you're interested in hiring some aspiring entrepreneurs.
• 54:59
  SPEAKER_02: Quick disclaimer we veered away from use case a bit towards the end of this conversation, but we really enjoyed our discussion and felt as if it should be included in the episode.
• 55:10
  SPEAKER_07: All right.
• 55:10
  SPEAKER_07: So uh now to talk about astronomers' internal reporting, we're gonna talk to Ben Gregory, one of our data engineers.
• 55:19
  SPEAKER_07: Ben, do you want to introduce yourself a bit?
• 55:22
  SPEAKER_09: Uh sure.
• 55:23
  SPEAKER_09: So uh like Raj said, I work on the uh data team Astronomer.
• 55:28
  SPEAKER_09: I've been with Astronomer for about three and a half years now and uh slowly transition over to the engineering side uh after being more customer-facing and account management focused.
• 55:39
  SPEAKER_09: So uh I've worked on a lot of the internal reporting that we have at Astronomer that ranges a number of databases we use and uh different APIs for SaaS tools as well.
• 55:50
  SPEAKER_09: So happy to talk with you all.
• 55:52
  SPEAKER_07: Well, yeah.
• 55:53
  SPEAKER_07: What Ben is super modest about is Ben was actually Astronomer's first employee way back in the day.
• 55:58
  SPEAKER_07: And in addition to all the great things he's done, he's had his fair share of mishaps, including hiring me.
• 56:05
  SPEAKER_09: Yeah, and then you hired Pete, so look what we have now.
• 56:09
  SPEAKER_07: Mistakes all around.
• 56:11
  SPEAKER_07: Yep, they just kind of build on top of each other.
• 56:14
  SPEAKER_07: Oh, yeah.
• 56:15
  SPEAKER_07: So no mistake.
• 56:19
  SPEAKER_07: Do you want to give us a quick rundown of so obviously Astronomer offers Airflow as a managed service and as an enterprise edition, but you want to kind of give us a quick rundown of how we use Airflow internally?
• 56:32
  SPEAKER_09: Yeah, yeah.
• 56:33
  SPEAKER_09: So uh so we use Airflow for really three different use cases.
• 56:38
  SPEAKER_09: Um the first is pulling in stats from our different SaaS tools, um, just like via their usually REST APIs.
• 56:46
  SPEAKER_09: Um for that, in the past, we've used HubSpot and Salesforce.
• 56:51
  SPEAKER_09: Um, you know, depending on how we do our uh account management and uh like sales cycle.
• 56:59
  SPEAKER_09: We've kind of you know fluctuated between tools, but we've built pipelines for both of their reporting APIs.
• 57:05
  SPEAKER_09: Um we're also working on uh building out some operators for Stripe and Intercom, which is uh two tools, the two tools we're using more and more now.
• 57:15
  SPEAKER_09: Um so this is really like the idea.
• 57:18
  SPEAKER_09: The idea of this is really building like a holistic picture uh of our customer across all the tools.
• 57:23
  SPEAKER_09: So uh, you know, what how many conversations are we having with them?
• 57:27
  SPEAKER_09: What are our touch points?
• 57:29
  SPEAKER_09: Uh we forward all customer-facing emails to Intercom.
• 57:32
  SPEAKER_09: So by pulling from Intercom uh into Redshift, which we standardize as our data warehouse, uh we're able to kind of see, okay, what are their touch points?
• 57:41
  SPEAKER_09: Uh are they behind on payments?
• 57:43
  SPEAKER_09: Have they added a credit card?
• 57:44
  SPEAKER_09: Uh have we talked to them at all?
• 57:46
  SPEAKER_09: Some you know, we just uh set up self-service, uh like a self-service part of our platform.
• 57:52
  SPEAKER_09: So there's a chance that you know we have customers who are paying us and we're not even really aware of them.
• 57:58
  SPEAKER_09: Uh so that's the first.
• 58:00
  SPEAKER_09: The second, um, oh, also GitHub, I would say.
• 58:04
  SPEAKER_09: It's another SaaS tool that we're using.
• 58:06
  SPEAKER_09: Uh we use that pretty heavily in terms of our own kind of productivity reporting.
• 58:12
  SPEAKER_09: Uh we manage, I think at this point, over 200 repositories on GitHub across three organizations, I think.
• 58:21
  SPEAKER_09: Maybe we may have consolidated to two, but uh essentially, you know, how many commits are everyone is everyone doing, what repos are people working on, uh how many outstanding issues are out there, how many pull requests do we have, uh, what's the average lifespan of these issues and pull requests, you know, how long is a pull request going without being either closed or merged?
• 58:44
  SPEAKER_09: Uh, this just helps us make sure that from a high level we're kind of maintaining like a lean engineering structure.
• 58:51
  SPEAKER_09: Um, so that's the first.
• 58:53
  SPEAKER_09: The second uh is what we use for our own like database reporting.
• 58:58
  SPEAKER_09: We use a lot of different types of databases uh to manage like different elements of our platform, given that we have like a microservice architecture.
• 59:08
  SPEAKER_09: So uh at the very base level, we pull data from Mongo into Redshift um to help enrich the reporting we already have from Intercom, Stripe, Salesforce.
• 59:20
  SPEAKER_09: Uh the Mongo reporting or the Mongo data has to do mainly with just app creation, account creation.
• 59:27
  SPEAKER_09: So you know, we're not seeing every touch point that they have, but we do see uh you know the major milestones.
• 59:35
  SPEAKER_09: So, you know, what's the name of the app that created, how many users are associated with that app, how many integrations have they activated with that app, uh, what are our most popular integrations, how many average integrations per customer do we have, that kind of stuff.
• 59:50
  SPEAKER_09: Uh we also uh are pulling in a lot of data around Qlickstream.
• 59:57
  SPEAKER_09: So we have the uh we have the standard like Clickstream, Analytics.js uh library wired up to our app website and Houston API.
• 1:00:11
  SPEAKER_09: Houston is just the name of our internal API that we use to manage all the different services that we have.
• 1:00:18
  SPEAKER_09: Uh so that's that's kind of interesting, but we don't see a ton of activity uh just through our app website, given that you know, usually like we're a set it kind of platform where you, you know, if everything's working, you're not logging in all the time and reconfiguring stuff.
• 1:00:36
  SPEAKER_07: Yeah.
• 1:00:36
  SPEAKER_07: So uh on a broad picture, we kind of use Airflow to take our customer journey all the way up from when they discover us through being an ad or hubs coming on intercom or something, all the ways through how they're using our platform, right?
• 1:00:49
  SPEAKER_07: With how many integration are they writing, how many apps are they making, all the way through.
• 1:00:54
  SPEAKER_09: Right.
• 1:00:54
  SPEAKER_09: So so it's like every every touch point outside of our platform.
• 1:00:58
  SPEAKER_09: And so that could be you know Hubsloader Salesforce, Intercom, Stripe.
• 1:01:01
  SPEAKER_09: Um, and then what's happening within our platform.
• 1:01:05
  SPEAKER_09: So not only the NLXGS, uh like Clickstream touch points, but also app creation via Mongo.
• 1:01:12
  SPEAKER_09: Uh and also importantly, like what even though they're not logging into our platform, doesn't mean that their apps or accounts aren't very active.
• 1:01:21
  SPEAKER_09: And for that, we use Kairos, which is a time series database uh built on top of something, I think.
• 1:01:28
  SPEAKER_09: It's Cassandra.
• 1:01:30
  SPEAKER_09: Cassandra?
• 1:01:30
  SPEAKER_09: Yeah, pretty sure.
• 1:01:32
  SPEAKER_09: Okay, yeah.
• 1:01:32
  SPEAKER_09: So so we use the the better metric for us in terms of you know, is a customer enjoying our platform and having success with our platform is not necessarily uh, you know, how many tickets are they opening up in intercom or conversations.
• 1:01:46
  SPEAKER_09: It's not necessarily how many times are they logging in or how many times are they creating apps?
• 1:01:50
  SPEAKER_09: Over time, that kind of evens out.
• 1:01:53
  SPEAKER_09: I mean, like you you're the math major, what is it, a logarithm where it's just like you kind of reach a ceiling.
• 1:01:59
  SPEAKER_09: Um like over time, like it goes, you know, in the in the beginning, a lot of a lot of app creation, a lot of logging in and fiddling with different integrations, a lot of conversations on intercom, you know, but once a contract's signed, once an app's configured, once they have a good idea of our platform, really the better metric at that point is how much data they're sending through our system so that we use Kairos.
• 1:02:26
  SPEAKER_09: Uh so for Kairos, we've automated via Airflow um two parts of a reporting.
• 1:02:33
  SPEAKER_09: Like Kairos is great for storing data, but it's not great at retrieving data at scale if you haven't pre-processed it.
• 1:02:40
  SPEAKER_09: So we do uh Kairos roll-ups uh on a second, minute, hour, day, and month basis or day, week, and month basis.
• 1:02:51
  SPEAKER_09: So for that, we can, you know, if we want to know how many events have has this app processed in the past hour, we've already kind of rolled up that stat, so we don't need to like process again all of the raw event data.
• 1:03:06
  SPEAKER_09: Um so yeah, so that's really great.
• 1:03:09
  SPEAKER_09: And then also we use Airflow to like move that roll up every hour to Redshift, because that's what we use for our warehouse.
• 1:03:17
  SPEAKER_07: So when a customer comes onto Astronomer, they're probably gonna interact with four different microservices, four different services, whether it be intercom, whether it be our analytics.js, whether it be something in Salesforce, they're gonna they're gonna use two separate modules, our Airflow module and our Clickstream module.
• 1:03:33
  SPEAKER_07: And they're gonna be in Mongo, Kairos.
• 1:03:35
  SPEAKER_07: So there's probably like nine different sources that a customer is gonna be across.
• 1:03:40
  SPEAKER_07: And you've set up our you've set the model up so that Airflow acts as a common orchestrator that kind of gives you a 360-degree view all across our nine different sources, right?
• 1:03:50
  SPEAKER_09: Yeah, so so let's ignore all the SaaS tools like the HubSpot and Salesforce stuff.
• 1:03:55
  SPEAKER_09: Just within our platform, when a user interacts with our products, we have records in Mongo, Kairos, and two Postgres databases.
• 1:04:04
  SPEAKER_09: Uh and the Postgres databases are for Airflow.
• 1:04:08
  SPEAKER_09: Um we use like we use other databases, uh like um Prometheus, uh, but that's not really as customer phasing, that's more just like um stuff.
• 1:04:23
  SPEAKER_09: Yeah, yeah.
• 1:04:24
  SPEAKER_09: And and you know, we've I think we've talked about like moving that into Redshift as well.
• 1:04:29
  SPEAKER_09: It's just for that use case to really more real time than like longitudinal reporting.
• 1:04:36
  SPEAKER_09: Like all of this stuff is customer-centric, right?
• 1:04:38
  SPEAKER_09: So we want to know how it's evolving over time.
• 1:04:40
  SPEAKER_09: If the platform's down, you don't want to know that like in an hour.
• 1:04:43
  SPEAKER_09: You want to know it like that second.
• 1:04:45
  SPEAKER_09: So in that case, Airflow is not necessarily as good of a use case.
• 1:04:49
  SPEAKER_09: Um, but we do we are doing kind of like a meta airflow and airflow, um, where we're pulling like airflow metadata out of the underlying Postgres uh database for Airflow using Airflow.
• 1:05:03
  SPEAKER_09: Um so like and we're like moving that to Regidge as well, because yeah, like you said, like there are two two main kind of modules in our platform right now.
• 1:05:12
  SPEAKER_09: Click shouldn't be one of them.
• 1:05:13
  SPEAKER_09: We've talked about that with Kairos.
• 1:05:15
  SPEAKER_09: Um, the other half is Airflow, and that's all backed by Postgres.
• 1:05:19
  SPEAKER_09: So for that, we we kind of like pull the same stats, like how many task instances have been run, how many DAG runs are there, how many failures, successes, what's the average time?
• 1:05:30
  SPEAKER_09: Uh like Airflow gives you some basic stats in the UI, but you always you always want more.
• 1:05:35
  SPEAKER_07: Yeah.
• 1:05:35
  SPEAKER_07: So you have all these resources that are all updated almost randomly if you look at it, right?
• 1:05:39
  SPEAKER_07: Because it's based on event interaction and user interaction, and Airflow kind of coordinates batching them together based on its own schedule.
• 1:05:46
  SPEAKER_07: So it kind of lives at a layer of abstraction above all the other resources, all the other reporting tools.
• 1:05:52
  SPEAKER_09: Yeah, I mean, at any given point, if you look at you know, if you look at Salesforce and Intercom and GitHub.
• 1:06:00
  SPEAKER_09: Um you know, our different databases, like they're all in different states.
• 1:06:04
  SPEAKER_09: Uh so you know it's really useful that to coordinate them into one like centralized warehouse.
• 1:06:11
  SPEAKER_09: I think that's why like stuff like BigQuery, Snowflake, Redshift are so like popular these days that you need like a single source of truth.
• 1:06:19
  SPEAKER_09: And when you're working like we are with four different databases and you know five or six like important from a reporting standpoint, SaaS tools, you know, you don't want like you don't want to be logging into all those tools, and also you're always getting a fractured bit view of what's happening with your customers if you're only ever looking at like one or two at a time.
• 1:06:41
  SPEAKER_07: Yeah.
• 1:06:42
  SPEAKER_07: So uh you mentioned before this, uh before switching to engineering team, you were on kind of the more account management side.
• 1:06:49
  SPEAKER_07: So uh before you started like getting into Airflow and building out our kernel recording, what was your engineering background like?
• 1:06:56
  SPEAKER_09: Um my engineering background was mostly from an analyst perspective.
• 1:07:01
  SPEAKER_09: It wasn't really like hard like software engineering.
• 1:07:05
  SPEAKER_09: It was um it was mostly a lot of Python, a lot of R, a lot of like statistical computing.
• 1:07:12
  SPEAKER_09: Um I studied economics in college and then got a work got work as a data analyst uh for a startup after college and was using Excel to do some like behavioral modeling on people's uh like people's behavior in our app.
• 1:07:28
  SPEAKER_09: Like we were kind of what we're doing today, I was just like pulling stats on how people were using our app and I was using Excel to like model behavior and and give out reporting for um our team and investors.
• 1:07:39
  SPEAKER_09: But I was using data sets that are so large I was crashing Excel like 10 times a day.
• 1:07:44
  SPEAKER_07: Uh so you know you'd switch, right?
• 1:07:47
  SPEAKER_07: If you crash Excel, it's satellite Python.
• 1:07:49
  SPEAKER_09: No, I I mean I would like compulsively save um like ever like every half hour, just be and it would take probably five minutes for the worksheet to save.
• 1:07:58
  SPEAKER_09: Um just because like the what I was building, not only were the datasets large, but I was building like complex enough models that when you have a like 800 megabyte data set and you're building like some pretty heavy transformations or models in in Excel, and you don't like at the time I think my laptop had like four gigs of memory, uh like it's yeah, it's just like it's gonna crash all the time.
• 1:08:21
  SPEAKER_09: Um and so like was frustrated with that, so just kind of had done some like work in R in college, um, but kind of was reading that Python was the way to go.
• 1:08:31
  SPEAKER_09: Uh like more and more of what you could do in R was being replicated in Python.
• 1:08:35
  SPEAKER_09: Uh and Python had like more widespread functionality beyond just like analytics reporting.
• 1:08:42
  SPEAKER_09: So uh did a uh nano degree via Udacity in like data analytics, uh to kind of just get better uh coding shops, and then you know kept doing kept work in Python and then eventually uh through work at Astronomer, kind of went full time on Python development.
• 1:09:01
  SPEAKER_07: Yeah, but uh I think it's fascinating that uh came into it with some Python experience, but not like by no means a seasoned dev and just through what's available to the open source community and how the how logically the airflow uh the airflow workflows are kind of planned out.
• 1:09:20
  SPEAKER_07: Sorry, I'm gonna say that part again.
• 1:09:21
  SPEAKER_07: Just so coming in from with a light Python background and just using what's available in the community to and uh combining that with how logically like Airflow lays out workflows, you'll be able to get pretty freaking good at Airflow just by bringing in some basic skills to the table.
• 1:09:38
  SPEAKER_09: Yeah, I would even say that you know working in Python in Airflow is a great kind of primer to being a Python dev in general.
• 1:09:46
  SPEAKER_09: Because if you think of every you know API you're hitting, every DAG you're building as like its own microservice, you're you can build a pretty complex uh not just reporting operation, but like a kind of a if this then that logic, um, you know, with trigger DAGs and different um sensors that you could essentially like proxy an entire app like without just without the front end to make it.
• 1:10:12
  SPEAKER_07: Is that is that gonna be the name of your uh medium article?
• 1:10:14
  SPEAKER_07: Every API is a microservice.
• 1:10:17
  SPEAKER_09: Yeah, I mean, like I mean, if you look at I think it's like you know, you look behind the scenes at how some apps work, and like a lot of it is exactly what you're doing with Airflow, is just like using cron um or jinkens um to execute rather than a tool like Airflow.
• 1:10:34
  SPEAKER_09: But I have seen and am working with certain customers now where essentially like they're OEMing our well, not OEMing because they're not reselling us, but like they're not just building, they're not just using astronomer and airflow to like enhance their own reporting, like they're using it to actually power the back-end logic of their app.
• 1:10:57
  SPEAKER_07: Yeah, because at its core, Airflow is just orchestrating tasks, right?
• 1:11:00
  SPEAKER_07: And yeah, what you need when you're orchestrating tasks is just access to a bunch of services.
• 1:11:05
  SPEAKER_09: Yeah, so I would say if you if you're looking, if you're a new Python dev and you're looking for a lot of experience and just like how does backend development works, you know, Airflow is a great way to start.
• 1:11:18
  SPEAKER_07: Cool.
• 1:11:18
  SPEAKER_07: Yeah.
• 1:11:19
  SPEAKER_07: Um yeah, that was all that I had.
• 1:11:21
  SPEAKER_07: Um, is there anything else that you want to pub or any kind of like short story you want to give?
• 1:11:27
  SPEAKER_09: Um feed edit it is uh I think I'll say, you know, for me personally, what gets me excited about um you know what we're building at Astronomer is not just what you can accomplish with you know click stream collection and airflow reporting, um, but really it the groundwork of what we're building gives way to so many more different modules and functionality that uh make using a lot of open source tools a lot easier to scale and productionize.
• 1:12:00
  SPEAKER_09: Uh, you know, Jupyter Lab just came out with like the public release of like Lab, the next development of notebooks.
• 1:12:08
  SPEAKER_09: Um, they have Jupyter Hub, they have like a KubeSpawner to run Jupyter on Kubernetes.
• 1:12:15
  SPEAKER_09: Uh there's a ton that you can do with that if you have a team to like scale it and build it and monitor it.
• 1:12:23
  SPEAKER_09: Uh you know, there's so many, there's so many tools that are so many organizations that have given so much, um, but they're like a mile away from the finish line or from like helping people get to the finish line.
• 1:12:36
  SPEAKER_09: In some cases, like Confluent or Databricks, you know, they have hosted versions, and in a lot of cases, they don't.
• 1:12:43
  SPEAKER_09: Uh, so I see Astronomer as like that last mile uh to help companies like take advantage of a lot of very generous open source contributions and realize them into a production-ready environment.
• 1:12:57
  SPEAKER_07: Yeah, um, I think it's been fascinating that through like all both of our time here, especially like we've probably pivoted on vision and stack like a thousand times, but Grand Vision's always been one platform to get you all of your data.
• 1:13:10
  SPEAKER_07: And uh this seems like the way it's going for now, right?
• 1:13:13
  SPEAKER_09: Yeah, I mean, everyone's like everyone's talking about open source.
• 1:13:20
  SPEAKER_09: I think when you like if you have if you ask people to be really honest about open source, like it's a lot harder to or not harder to use, it's it's not adopted nearly as much, I think, at an enterprise level as people may realize, just because uh you know, it doesn't have like it doesn't have the support of like going with an IBM contract, which gets you like someone on the phone, it doesn't you know have like push key activation or push them activation, which just like spins up like a beautiful product that's like ready to use out of the box.
• 1:13:55
  SPEAKER_09: Like there's a lot to be said for that.
• 1:13:57
  SPEAKER_09: Uh and if you are in an enterprise environment, you can't like you may the developer and you are the engineer and you may like want to advocate for open source, uh, but anytime you try to adopt a new product, you're putting yourself out the dry, uh, potentially if it doesn't work out.
• 1:14:13
  SPEAKER_09: There's that saying, like, no one ever got fired for hiring IBM.
• 1:14:17
  SPEAKER_09: You know, IBM may not be the right choice in every case, but you know, it has a name behind it, it has known support behind it.
• 1:14:25
  SPEAKER_09: So I think if Astronomer can bridge that gap and make like no one ever got fired for higher for you know using open source via astronomer, uh you know, that'll be really cool to see.
• 1:14:41
  SPEAKER_02: And that's it for our second episode of the Airflow Podcast on use cases.
• 1:14:45
  SPEAKER_02: Tune in two weeks from now to hear our next episode on best practices.
• 1:14:48
  SPEAKER_02: We have a ton of great content that we're really excited to push out there.
• 1:14:51
  SPEAKER_02: As always, leave us a comment if you have any feedback, are interested in hearing about anything in specific, or are interested in being a guest.
• 1:14:58
  SPEAKER_02: We're still looking for new guests.
• 1:14:59
  SPEAKER_02: We love interviewing people and chat with them about airflow, so all are welcome.
• 1:15:02
  SPEAKER_02: We've also recently open sourced a library of airflow plugins on GitHub, and that's at GitHub.com slash airflow-plugins.
• 1:15:09
  SPEAKER_02: We'd love to have more people from the community commit, so just go there, fork it, and submit a PR and we'll merge it in.
• 1:15:15
  SPEAKER_07: Finally, if any of these use cases can be applied at your company, contact us about our spacecamp program, the fastest way to get your team up and running on Airflow.
• 1:15:24
  SPEAKER_02: If you want to contact us about spacecamp, just head on over to www.astronomer.io and drop us a note in the web chat.
• 1:15:31
  SPEAKER_02: Thanks for listening, and we'll see you in two weeks for the episode on Airflow best practices.