Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero

Transcript

Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: I think we've seen really data centric things come up with the support of data sets as a way to trigger pipelines.
• 0:08
  SPEAKER_01: And I think that solves a very domain specific problem.
• 0:11
  SPEAKER_01: But I'd love to see it get even more in general as we go forward.
• 0:15
  SPEAKER_02: I think one of the things that I've seen recently that I think would be very beneficial for most open source projects to lean into is I saw this really interesting project recently, right?
• 0:24
  SPEAKER_02: And you gotta think again, like what is the cause of most outages today?
• 0:27
  SPEAKER_02: It's configuration change, the op flow.
• 0:34
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:44
  SPEAKER_03: Hey Airflow community, Airflow3 is here and we are taking it on the road.
• 0:49
  SPEAKER_03: Join the Astronomer Road Show for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 0:58
  SPEAKER_03: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:06
  SPEAKER_03: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:11
  SPEAKER_03: Register now.
• 1:12
  SPEAKER_03: The link is in the show notes.
• 1:15
  SPEAKER_03: Hi everyone.
• 1:16
  SPEAKER_03: Welcome to the Data Flowcast.
• 1:18
  SPEAKER_03: I'm your host, Kenton Danis, and today I'm joined by Corey O'Daniel, who is a CEO and co-founder of Mass Driver, as well as Jake Ferrero, who is a senior software engineer at Astronomer.
• 1:30
  SPEAKER_03: How are you guys doing?
• 1:32
  SPEAKER_03: I'm doing pretty good.
• 1:34
  SPEAKER_03: Good.
• 1:34
  SPEAKER_03: Corey, I forget.
• 1:36
  SPEAKER_03: Remind me where you're located.
• 1:38
  SPEAKER_02: I am in Pasadena, California.
• 1:40
  SPEAKER_03: Okay, great.
• 1:41
  SPEAKER_03: So we're all West Coasters.
• 1:43
  SPEAKER_02: Yep.
• 1:44
  SPEAKER_02: And these are my these are my neighbors' dogs barking in the background.
• 1:47
  SPEAKER_02: Apologize.
• 1:48
  SPEAKER_03: Zoom is very good at filtering all of that out.
• 1:51
  SPEAKER_03: So at least I can't hear anything.
• 1:53
  SPEAKER_03: I have stopped apologizing for my dog going nuts on calls because I kept doing it and people would be like, What are you talking about?
• 1:59
  SPEAKER_03: I'm like, oh my gosh, it's so loud, but nobody else could hear it.
• 2:04
  SPEAKER_02: I had my door opened for one of my podcasts recently and had a squirrel come into the office, which was interesting in the middle of a podcast.
• 2:09
  SPEAKER_02: Yeah, that was crazy.
• 2:10
  SPEAKER_03: Oh my gosh.
• 2:11
  SPEAKER_03: That is crazy.
• 2:12
  SPEAKER_02: I have never had anything nearly that exciting happen in a little bit at least it wasn't a coyote, and I'll take a squirrel any day.
• 2:18
  SPEAKER_03: That is super funny.
• 2:20
  SPEAKER_03: All right.
• 2:20
  SPEAKER_03: Well, I'm not sure.
• 2:21
  SPEAKER_03: I guess I'll hope for like no wildlife encounters during our recording today, although I would get a kick out of that, to be honest.
• 2:28
  SPEAKER_03: So we'll see what happens.
• 2:29
  SPEAKER_03: All right, but let's go ahead and dive in.
• 2:31
  SPEAKER_03: So for today's podcast, we're gonna discuss the biggest challenges that data teams face with data platform management, which is a huge topic.
• 2:39
  SPEAKER_03: And we'll, of course, talk about how Airflow fits in.
• 2:41
  SPEAKER_03: But I want to start with some background on both of you.
• 2:44
  SPEAKER_03: Corey, let's start with you.
• 2:45
  SPEAKER_03: And can you tell us a little bit about yourself and what Mass Driver does?
• 2:49
  SPEAKER_02: Yeah, so I'm a caught operations engineer, been in this space since the early 2000s.
• 2:54
  SPEAKER_02: Background was originally data centers and healthcare was one of the first companies to like move a full data center to AWS when they were launching EC2 way back when.
• 3:08
  SPEAKER_02: So I've been doing that for the past 20-ish years.
• 3:11
  SPEAKER_02: And I don't know, maybe six years ago, I was working with a little company called Google Cloud and a PSO or that's where I met Jake originally.
• 3:19
  SPEAKER_02: And that's where I got the idea for Mass Driver.
• 3:21
  SPEAKER_02: So I'd worked with a bunch of smaller companies and I saw people constantly struggling with managing infrastructure and like using these tools that I use day in, day out.
• 3:30
  SPEAKER_02: And I'm like, I don't understand what the problem is.
• 3:31
  SPEAKER_02: These are all super easy to use.
• 3:33
  SPEAKER_02: Like, why are we not adopting more infrastructure as code?
• 3:36
  SPEAKER_02: Like, why aren't we doing more self-service?
• 3:38
  SPEAKER_02: And then as I got to work with some of these bigger companies, like I could see the same things that were causing these smaller companies to struggle with adoption also happening at these larger organizations.
• 3:47
  SPEAKER_02: And the big root cause of all of this is developers, ML engineers, et cetera, were making those way faster than we're making operations, SREs, DevOps engineers.
• 3:58
  SPEAKER_02: And so you have this imbalance of people that have experience automating the cloud and cloud management on Teams just completely outbalanced by the number of engineers on their teams that need access to it.
• 4:10
  SPEAKER_02: And so the idea with Mass Driver was to make infrastructure automation easier for developers.
• 4:15
  SPEAKER_02: So they shouldn't have to learn the operations team's tooling to be able to get cloud, right?
• 4:21
  SPEAKER_02: That's the core idea.
• 4:22
  SPEAKER_02: And at the same time, operations engineers who are already underwater don't have the time to learn TypeScript to go build some fancy UI for developers to make accessing infrastructure as code easier.
• 4:34
  SPEAKER_02: So our idea was make it something developers can access.
• 4:37
  SPEAKER_02: They understand diagramming, they do it every day.
• 4:40
  SPEAKER_02: Make the platform extendable using operations tooling.
• 4:42
  SPEAKER_02: So everything in the platform is extendable by either writing Docker or infrastructure as code.
• 4:47
  SPEAKER_02: And so neither team has to learn a new tool set.
• 4:49
  SPEAKER_02: And hopefully we can be this membrane or API layer between your developers and operations to make it a bit more accessible.
• 4:56
  SPEAKER_03: Okay, that's great.
• 4:57
  SPEAKER_03: Thanks for that background.
• 4:58
  SPEAKER_03: And we'll definitely dig in a lot more to the infrastructure management side of things.
• 5:02
  SPEAKER_03: Certainly very relevant for a lot of Airflow users.
• 5:06
  SPEAKER_03: Jake, from your side, so I think a lot of our listeners will be relatively familiar with Astronomer and what we do, at least our involvement in the Airflow project and as a managed service.
• 5:16
  SPEAKER_03: But can you tell everybody a little bit about your role and your work here?
• 5:20
  SPEAKER_01: So I've been an engineer at Astronomer for two years, a six months hiatus with Corey at Mass Driver, and then about another two years back at Astro.
• 5:30
  SPEAKER_01: And my focus has really been on the back end of how airflow gets provisioned and how we monitor, auto-scale, et cetera, your airflow deployments.
• 5:45
  SPEAKER_01: So I mostly have expertise, if you're really familiar with Astronomer, in on the data plane side of things rather than the control plane side.
• 5:55
  SPEAKER_01: Going back further than that, my career was largely built in Google Cloud, PSO on their data analytics team.
• 6:02
  SPEAKER_01: So I had a lot of experience with many different flavors of organizations struggling with the same problems of building a data platform.
• 6:12
  SPEAKER_01: And that's part of what got me originally so excited about Airflow, because it seems to be like one of these real instrument technologies.
• 6:20
  SPEAKER_03: Yeah, that totally makes sense.
• 6:21
  SPEAKER_03: And I think for any of our listeners who have used Astro as a managed airflow service, I think one of the great benefits of that is that you don't have to know as much about Airflow infrastructure.
• 6:32
  SPEAKER_03: So a lot of your work is exactly why we have that piece of the product, is so that nobody else has any idea how it works and you don't have to as much.
• 6:41
  SPEAKER_03: Although, as we're going to get into when you're working with Airflow, you're never going to be able to ignore the infrastructure and stuff entirely.
• 6:47
  SPEAKER_03: So yeah, that's really helpful background.
• 6:50
  SPEAKER_03: And I think good segue into our next topic.
• 6:53
  SPEAKER_03: So I promised that we would talk about some of the frequent challenges that data platform engineers run into, which again is a really big topic, but let's kind of level set with some context first and talk about just what we mean when we say data platform and again how Airflow fits into it.
• 7:08
  SPEAKER_03: So at a high level, what is that to both of you?
• 7:12
  SPEAKER_01: So to me, like the data platform is an entity that supports many different like stakeholders and glues together what whoever your end users are and how we collect data from those end users, whether they be like IoT devices on machine shop floor or social media users or patients going through a healthcare system.
• 7:39
  SPEAKER_01: Those are like our like data sources.
• 7:41
  SPEAKER_01: And that's like the left edge of the data platform.
• 7:46
  SPEAKER_01: Then in the middle, you have all of these kind of like data analysis pipelines ranging from things that run over and over again orchestrated by something like Airflow or a lot of ad hoc analysis done by your like internal analysts.
• 8:03
  SPEAKER_01: And then as you move further to the right on the data platform, you have these, okay, how do we make sense of all this data?
• 8:10
  SPEAKER_01: What is the result of all these ad hoc analysis or recurring pipelines?
• 8:17
  SPEAKER_01: I'm thinking like the dashboard that your CEO reviews to see how things are going.
• 8:23
  SPEAKER_01: I'm thinking the outputs of an ML model that are then like served on your website or something like that.
• 8:33
  SPEAKER_01: So there's a lot of different types of people, a lot of different like sorts of expertise throughout the data platform.
• 8:41
  SPEAKER_01: The your analysts, your like data engineers, your like business stakeholders, your end users, and then the like often unsung heroes, which is the data platform engineer, which is usually someone on an infrastructure team that's responsible for getting all the bits and bobs that connect all of these people together in place.
• 9:06
  SPEAKER_02: Yeah.
• 9:07
  SPEAKER_02: So when I think of a data platform, I think of the underside of it and how painful it is.
• 9:11
  SPEAKER_02: Being that data platform engineer, right?
• 9:13
  SPEAKER_02: It's wild when you think about the types of engineers and orgs.
• 9:16
  SPEAKER_02: Like if you just classify people like there's the operations-esque people, there's the ML people, and then there's like your back-end, front-end engineers, right?
• 9:24
  SPEAKER_02: I think when you look at the ML space, it is probably the most intimidating space to manage infrastructure as code in.
• 9:34
  SPEAKER_02: And I think when you look at all the stuff you have to bring together, all the things you have to bring together to run that data platform is in and of itself a lot of work.
• 9:43
  SPEAKER_02: Look at Astro, right?
• 9:44
  SPEAKER_02: Like you guys exist because that part is so hard.
• 9:47
  SPEAKER_02: But then beyond that, you end up in this constellation of all of these data sources that you need to bring in.
• 9:54
  SPEAKER_02: And immediately for most companies, this becomes a multi-cloud problem, right?
• 9:59
  SPEAKER_02: Like I've got my data here, but we have some data in some other store that I cannot access over my VPC and I've got to traverse the internet, right?
• 10:08
  SPEAKER_02: Where does this data like there's so much compliance in DevOps, like bits and glue, like holding the underside of this together?
• 10:15
  SPEAKER_02: And I think like that, that right there, like for an operations person, like that is one of the hardest spots to be able to provide support for your organization because it's generally cutting edge.
• 10:27
  SPEAKER_02: There's database administrators that have inputs.
• 10:30
  SPEAKER_02: There's ML engineers that have inputs, like your stakeholders are much larger than I have a back-end developer who needs Postgres and S3 bucket.
• 10:37
  SPEAKER_02: He needs a relational and needs and blob store.
• 10:39
  SPEAKER_02: That's trivial at the end of the day.
• 10:41
  SPEAKER_02: Like when you think about it compared to ML ops, and teams are still struggling with that.
• 10:45
  SPEAKER_02: Right.
• 10:45
  SPEAKER_02: And so, like data plane, like this is by far one of the most important things that we have today in engineering with all the stuff that we're doing in ML.
• 10:54
  SPEAKER_02: And it's one of the places where I think, as many organizations sit, it is one of the least underserved parts of the engineering org as far as DevOps and platform attention.
• 11:02
  SPEAKER_01: Yeah, Gloria, I'm so glad you mentioned compliance because that was a persona I didn't mention.
• 11:08
  SPEAKER_01: And this would all be so easy if we didn't have concerns like PHI, we didn't have concerns of like data privacy or like geo restrictions on data.
• 11:20
  SPEAKER_01: That is like so many of the requirements actually come from the legal team, and the data engineer does not care, he just wants access to his data.
• 11:29
  SPEAKER_02: We want to do cool things, like I don't know, like did that data leave like a little geo boundary?
• 11:34
  SPEAKER_02: Maybe it did, maybe it didn't.
• 11:36
  SPEAKER_01: Yeah, something you mentioned before is oh, we don't want the like data teams to have to learn the infrastructure teams tool team.
• 11:43
  SPEAKER_01: And I know you and I have seen firsthand when you give a data engineer like terraform powers, they're cowboys, they want to deploy all the time, they want to just like tweak something until it finally works for from their perspective.
• 12:00
  SPEAKER_01: And I think one of the powers of a tool like Mass Driver, or generally a really strong data platform story, is how to democratize access to new and varied infrastructure in a way that is really standardized, where like you can't be a cowboy that colors outside the lines.
• 12:21
  SPEAKER_02: Yeah.
• 12:22
  SPEAKER_02: And it's hard.
• 12:22
  SPEAKER_02: I mean, it's one of those things that's hard too, right?
• 12:24
  SPEAKER_02: It's like you you start getting into a place where maybe the tools that ops traditionally recommends you to use for managing cloud, like also just don't make sense from the ML engineer's point of view.
• 12:37
  SPEAKER_02: I can I can recall a point in time.
• 12:40
  SPEAKER_02: Maybe this will sound familiar to you, Yank, where a team was very adamant about using Terraform.
• 12:45
  SPEAKER_02: And it was like that was the way to manage all things cloud.
• 12:48
  SPEAKER_02: And then all of a sudden a team found themselves managing their BigQuery tape schemas using Terraform, which is torture, is true torture, right?
• 12:55
  SPEAKER_02: And this is one of those places where it's like it's it is again hard, right?
• 12:59
  SPEAKER_02: As an operations engine, you're saying, hey, we gotta keep, we gotta have parity, we gotta have reproducibility, we gotta have audit trails and all this infrastructure stuff.
• 13:06
  SPEAKER_02: Terraform is how we do it.
• 13:07
  SPEAKER_02: Here, ML team, like manage your database schemas with this.
• 13:10
  SPEAKER_02: And people are like, that's cuckoo bananas to quote my son, right?
• 13:13
  SPEAKER_02: But like this is the ops team's like, I'm not familiar with your tools for doing this.
• 13:17
  SPEAKER_02: And so they offer, hey, this is how we manage cloud.
• 13:20
  SPEAKER_02: And then you know these teams were like, they have this really rough sense of like how to work with cloud resources because you're using the wrong pool for the job.
• 13:26
  SPEAKER_02: And honestly, like the whole plane, whether it's just traditional back-end services, networking services, or ML ops, kind of suffers from this.
• 13:34
  SPEAKER_02: Like ops teams will look around and say, hey, there's this tool that I really know.
• 13:37
  SPEAKER_02: We this is the one that we all know internally.
• 13:39
  SPEAKER_02: This is the tool that we're using.
• 13:40
  SPEAKER_02: And it's like we all have very different jobs.
• 13:42
  SPEAKER_02: We need to figure out how we can have these platforms that are able to infra orchestrate our infrastructure and tools that are familiar with us, the end user, not tools that are familiar with someone on my team.
• 13:54
  SPEAKER_02: And so that's one of the things that we're like, we're trying to push for is like creating a platform that's easy enough for someone to use without necessarily becoming an expert in an additional setup tooling, which these engineers most of the sort of don't have time to do, right?
• 14:07
  SPEAKER_02: I've got 13 different databases with different query languages I have to figure out how to bring together to produce some value for the business.
• 14:14
  SPEAKER_02: Like, is fiddling with the IAM role for accessing Postgres a valuable use of my time or thinking about the security groups between like where my ML workload is and Postgres a valuable use of my time?
• 14:25
  SPEAKER_02: No, it's not.
• 14:26
  SPEAKER_02: But is it valuable?
• 14:27
  SPEAKER_02: Yes.
• 14:28
  SPEAKER_02: Yes, it is, right?
• 14:29
  SPEAKER_02: So it's a bit of a conundrum, right?
• 14:31
  SPEAKER_01: Yeah, I think we've done a really like nice job kind of overviewing the like people of the data platform and like kind of the people problem.
• 14:40
  SPEAKER_01: And I think what you did just there was really underscore how many different technological bits there are.
• 14:45
  SPEAKER_01: And especially for ML workloads, how many varied kinds of compute you're looking for.
• 14:51
  SPEAKER_01: Sometimes it's a little workload on a cube cluster, sometimes you're still using like a managed Spark cluster, sometimes you're running a DVT cloud workload.
• 15:01
  SPEAKER_01: Like there are so many different ways and so many different technological bits that have to be stitched together.
• 15:08
  SPEAKER_01: And I think what's really interesting to put airflow in context here is Airflow defines a lot of the logic of okay, this workload runs here, this workload runs there, I get access to data here, I produce data there.
• 15:24
  SPEAKER_01: So it defines a lot of the workflow logic.
• 15:28
  SPEAKER_01: But what it doesn't define is how all of those bits got created in the first place and like how it actually accesses that.
• 15:36
  SPEAKER_01: So like a line in airflow in an airflow DAG really just means, okay, we go from this step to that step.
• 15:43
  SPEAKER_01: In Master Driver, we drive like draw like similar pictures, but a line means, okay, here's a network connection, an IAM role, all of those things.
• 15:53
  SPEAKER_01: Um and it's like you need both pictures, the picture of the infrastructure and the way the data flows between all the bits to really get a meaningful data platform.
• 16:06
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:15
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 16:27
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 16:35
  SPEAKER_03: Jake, I'm glad you brought this up because I think it's so relevant for a lot of Airflow users.
• 16:40
  SPEAKER_03: Because to your point, Airflow is purposefully unopinionated in terms of how those connections are being made and how Airflow interacts with those other systems.
• 16:48
  SPEAKER_03: It's just there to provide the structure for orchestrating your logic.
• 16:53
  SPEAKER_03: And to your discussion earlier about all of the different personas involved and who is managing these platforms, I think there's often this sort of picture presented of this binary of like the people working with Airflow just write Python code for their DAGs, and then somebody else is doing all of the data platform management and like making Airflow run for them.
• 17:14
  SPEAKER_03: And I think in all of my time working in this space, I have never seen it work that simply just because of the nature of like how these tools work and like Airflow is not used in a vacuum.
• 17:26
  SPEAKER_03: It has to connect to all of these other things.
• 17:28
  SPEAKER_03: And so there is that both managing all of the expectations amongst the different teams and just physically making it all work.
• 17:36
  SPEAKER_03: I liked your comment earlier about how data engineers are like cowboys, they're just used to problem solving and being like, oh, I can hack my way into getting Airflow to connect to this other system.
• 17:46
  SPEAKER_03: But it may not be what your security team wants you to do to connect to that other system.
• 17:51
  SPEAKER_01: Yeah.
• 17:51
  SPEAKER_01: And to circle back to something Corey mentioned, a core part of the people problem here is the impedance mismatch of like how many like data engineers, ML engineers, apps engineers, like our like workforce has, and how few like DevOps engineers our workforce has.
• 18:11
  SPEAKER_01: Because one of these things is pitched as a lot sexier than the other.
• 18:15
  SPEAKER_03: Yeah, for sure.
• 18:15
  SPEAKER_03: Corey, is that something you encounter a lot like at Mass Driver?
• 18:18
  SPEAKER_03: I assume you're often trying to bridge the gap between those different personas as well as between all the different technologies just within your platform itself.
• 18:28
  SPEAKER_02: Yeah, I'm a big fan of all the acronyms that end in DD.
• 18:31
  SPEAKER_02: So I like test-driven development, I like behavioral-driven development, domain-driven development.
• 18:36
  SPEAKER_02: What I do not like is interrupt-driven development, which I feel like is what most mostly what we do, right?
• 18:40
  SPEAKER_02: So it's like thinking from that data engineer's point of view, it's like, okay, I've got a team over here that's got this great web service.
• 18:46
  SPEAKER_02: They got a whole bunch of data that I would love to get out of their Postgres instance.
• 18:50
  SPEAKER_02: I know where it is.
• 18:51
  SPEAKER_02: I got the host name of it, right?
• 18:52
  SPEAKER_02: Okay, great.
• 18:53
  SPEAKER_02: Let's get that data into my pipeline.
• 18:54
  SPEAKER_02: How do I get it in there?
• 18:56
  SPEAKER_02: My Airflow is got to connect this data source, he's got to get this, gotta get this Postgres over here.
• 19:00
  SPEAKER_02: Does our company use Vault?
• 19:02
  SPEAKER_02: Like, how do I get my credentials?
• 19:04
  SPEAKER_02: Like, where are the credentials?
• 19:05
  SPEAKER_02: First off, like who I have to ask for them?
• 19:08
  SPEAKER_02: Do I ask the lead on this team?
• 19:09
  SPEAKER_02: Or is it like a DBA admin?
• 19:11
  SPEAKER_02: Some players just gonna give me these credentials.
• 19:13
  SPEAKER_02: Once I find them, how do I get them?
• 19:14
  SPEAKER_02: Do I copy and paste them?
• 19:16
  SPEAKER_02: Are we last passing them?
• 19:17
  SPEAKER_02: Are we slacking them and then deleting the message afterwards?
• 19:20
  SPEAKER_02: I know who you are, people, right?
• 19:22
  SPEAKER_02: Like, how do I get them?
• 19:23
  SPEAKER_02: I've never done that.
• 19:24
  SPEAKER_02: No, right?
• 19:25
  SPEAKER_02: So it's like, okay, I found this thing, I've got this data, I got this pipeline's gonna be great.
• 19:29
  SPEAKER_02: I got the credentials, this is gonna be fantastic.
• 19:32
  SPEAKER_02: And we'll connect this data source.
• 19:34
  SPEAKER_02: I slack you.
• 19:34
  SPEAKER_02: I'm like, are you sure this password's right?
• 19:36
  SPEAKER_02: Because I can't.
• 19:37
  SPEAKER_02: I'm getting rejected.
• 19:38
  SPEAKER_02: And it's a security group, right?
• 19:39
  SPEAKER_02: So like now I'm like, oh god, I'm okay.
• 19:41
  SPEAKER_02: I found like I found the thing I need, figure out how to get credentials, figure out how to get credentials here, but there's a security group.
• 19:47
  SPEAKER_02: So now I'm like off talking to a network team.
• 19:50
  SPEAKER_02: And so the thing that's really hard about many organizations is we have so many different work streams that have overlapping concerns where people are just like working in their little silos.
• 20:01
  SPEAKER_02: And so much of what we've done over the past 15 years of DevOps is really, really, and this isn't an opinion.
• 20:10
  SPEAKER_02: There's numbers to back this up.
• 20:12
  SPEAKER_02: Like we have settled uh into ticket ops being the new definition of DevOps.
• 20:17
  SPEAKER_02: You need something, you simply just ask the DevOps guy.
• 20:20
  SPEAKER_02: He's not gonna be mean like he was in 2004.
• 20:23
  SPEAKER_02: We all watched the Flickr video.
• 20:24
  SPEAKER_02: We're not mean anymore.
• 20:25
  SPEAKER_02: We've got a culture, we're all nice, but you still have to ask me.
• 20:29
  SPEAKER_02: I'm the gatekeeper for passwords.
• 20:31
  SPEAKER_02: That guy's the gatekeeper for YAML that you're gonna put in GitHub Actions, that guy's the gatekeeper for networks, right?
• 20:38
  SPEAKER_02: Like you still have to find all this stuff.
• 20:40
  SPEAKER_02: And there's so much coordination and glue that you have to put together.
• 20:43
  SPEAKER_02: And so, one of the concepts of mass driver that is fundamentally different than I'd venture to say anybody in the space, and something that we're working on open sourcing right now is we have this concept, the heart of Mass Driver called the bundle.
• 20:56
  SPEAKER_02: And so this was like this first like moment that I had a few years ago where it's like I've seen people stumble across this problem.
• 21:01
  SPEAKER_02: Somebody's like, hey, we're ready to provision this thing.
• 21:03
  SPEAKER_02: You just need yourself a little bit of Terraform and sprinkle a little Terraform on there, and you would not believe what the world is gonna be for you.
• 21:09
  SPEAKER_02: It's gonna be magic, reproducibility, auto trails, et cetera.
• 21:12
  SPEAKER_02: Just remember, after you put that little sprinkle of Terraform, to also put your sprinkle of your staging workflow and your production workflow and your preview environments workflow.
• 21:20
  SPEAKER_02: And don't forget to copy our sneak config and all the configurations for that database to make sure it's actually compliant.
• 21:26
  SPEAKER_02: But just a heads up, we also started moving to Wiz.
• 21:29
  SPEAKER_02: So you need to wiz rules as well.
• 21:31
  SPEAKER_02: Just as soon as you guys get all that, you're gonna get this database.
• 21:34
  SPEAKER_02: And so the fundamental difference with Mass Driver is from the point of view of the infrastructure as code person who's owning a module or owning what we refer to as like an abstraction, right?
• 21:43
  SPEAKER_02: It's not AWS Aurora RDS Postgres.
• 21:46
  SPEAKER_02: This is how my business runs Postgres.
• 21:48
  SPEAKER_02: And that's the fundamental difference that our customers see as they see that shift.
• 21:52
  SPEAKER_02: I don't want to support Infinity Cloud Services.
• 21:55
  SPEAKER_02: I want to say this is my golden path, not for running an app.
• 21:58
  SPEAKER_02: This is my golden path for how it Offer a service.
• 22:01
  SPEAKER_02: We offer Postgres to our developers.
• 22:03
  SPEAKER_02: Anybody can deploy it, anybody can connect to it using the diagram.
• 22:08
  SPEAKER_02: Now, when you draw these lines, what you're doing is you're making a request.
• 22:11
  SPEAKER_02: I'm saying my application would like access to this database.
• 22:15
  SPEAKER_02: And instead of it going through 10 approval processes in service now or whatever, from the point of view of the Terraform or Bicep or CloudFormation author, you put all of your infrastructure as code, your policy and your workflow into this concept we call a bundle.
• 22:29
  SPEAKER_02: We have a tool that takes all three of your tools, packages it into this artifact.
• 22:33
  SPEAKER_02: And now your Terraform is like a real shippable software asset.
• 22:38
  SPEAKER_02: So now a developer on the other side of this, so let's say you're at that stage of provisioning, getting the resource out there in the first place.
• 22:43
  SPEAKER_02: They say, ooh, I'm allowed to deploy Postgres.
• 22:46
  SPEAKER_02: This is awesome.
• 22:46
  SPEAKER_02: They drag Postgres onto their canvas.
• 22:48
  SPEAKER_02: They hit deploy.
• 22:49
  SPEAKER_02: They put in some configuration options.
• 22:50
  SPEAKER_02: And what they don't realize is they're filling out this form.
• 22:52
  SPEAKER_02: They're filling out the Terraform variables.
• 22:54
  SPEAKER_02: And when they hit deploy, instead of me copying walls of YAML in a hundred different workflows, the platform's familiar with your environments and how to reproduce them.
• 23:02
  SPEAKER_02: And says, oh, let me look at the let me look at the workflows that were packaged with this IAC, and I'm going to run that for you.
• 23:08
  SPEAKER_02: So now you're getting access to this stuff.
• 23:10
  SPEAKER_02: Now, what gets crazy is the lines, if you look at MassDriver, you look at Airflow, look at MassDriver, you'll both see lines.
• 23:15
  SPEAKER_02: The lines in Mass Driver are your security and compliance.
• 23:18
  SPEAKER_02: So it's saying, I've got this Postgres database over here.
• 23:21
  SPEAKER_02: It might be written with Terraform, OpenTOFU, whatever.
• 23:23
  SPEAKER_02: I've got a Lambda, a Helm chart, some notebook over here that needs access to it.
• 23:29
  SPEAKER_02: And so now when you draw that line, what's happening is that same author has codified, hey, this is how we access Postgres.
• 23:35
  SPEAKER_02: In here is the IAM policies for accessing Postgres.
• 23:39
  SPEAKER_02: It's up to you to have a role.
• 23:40
  SPEAKER_02: I'm going to push you the policies.
• 23:42
  SPEAKER_02: You bind your role to these policies, and you can now access it.
• 23:44
  SPEAKER_02: But now what we've done is we've made the entire diagramming process the request process, right?
• 23:51
  SPEAKER_02: So we're saying, hey, I want access to this.
• 23:53
  SPEAKER_02: You've already approved how we access things.
• 23:56
  SPEAKER_02: There is an audit trail of me accessing it.
• 23:58
  SPEAKER_02: And so now I've made compliance happy.
• 24:00
  SPEAKER_02: There's this audit trail there.
• 24:02
  SPEAKER_02: I've gotten access without any ability to get around any sort of security and compliance.
• 24:06
  SPEAKER_02: And that's the thing that really makes these teams move fast is I can look at something and say, hey, you know what?
• 24:11
  SPEAKER_02: I do want to know the cloud.
• 24:12
  SPEAKER_02: I want to be excited by my services.
• 24:14
  SPEAKER_02: I want to understand how to run Postgres.
• 24:16
  SPEAKER_02: And boom, Postgres 16, a couple of extensions.
• 24:18
  SPEAKER_02: I've got it.
• 24:19
  SPEAKER_02: You've taken out all that part that no one cares about.
• 24:22
  SPEAKER_02: No developer ML person, oh yeah, give me that GitHub actions, like give me some IAM policies and security groups.
• 24:28
  SPEAKER_02: Like, we don't care.
• 24:29
  SPEAKER_02: We want the top of the TCP stack, is what we're interested in from accessing Postgres.
• 24:35
  SPEAKER_02: But on the other side, these people that are excited about IAM and security and compliance and network, their work never feels like a work product.
• 24:42
  SPEAKER_02: It just constantly feels like a queue of to-dos for ticket ops.
• 24:45
  SPEAKER_02: And so now what you end up with is hey, I am the person that knows how we operate Postgres with the team that operates Postgres.
• 24:52
  SPEAKER_02: And like now we actually have a real work product.
• 24:54
  SPEAKER_02: I can give this to you.
• 24:55
  SPEAKER_02: And what I mean by that is a layman can now use it.
• 24:58
  SPEAKER_02: It's not me handing you a list of to-dos and ask me when you have a question.
• 25:02
  SPEAKER_02: I've done what SaaS has done for the past 20 years, and I've made something that is very hard for the layman to do trivial and giving them exactly what they wanted without them having to learn how to do my job.
• 25:12
  SPEAKER_02: And I think like that, that is going to be the crux of like how we scale ops over the next couple of years is figuring out like what are the right abstractions for teams.
• 25:19
  SPEAKER_02: Like how do we make it easy for them to access it?
• 25:21
  SPEAKER_02: Easy for them to get what they need to be able to do, but cut that layer off where it's like that doesn't matter to me.
• 25:26
  SPEAKER_02: That's a compliance or security team decision.
• 25:29
  SPEAKER_02: It shouldn't be in my wheelhouse.
• 25:30
  SPEAKER_02: It shouldn't be for me to see Sneak tell me 20 minutes later I did it wrong.
• 25:33
  SPEAKER_02: I just shouldn't be able to do it wrong in the first place.
• 25:35
  SPEAKER_03: That's great.
• 25:36
  SPEAKER_03: I think you're describing a very elegant solution to what we've spent 25 minutes talking about is a huge problem that every data team is encountering.
• 25:45
  SPEAKER_03: And I think there are so many parts of that that I would love to dig into a little more, but we are, you know, out of time.
• 25:49
  SPEAKER_03: We could continue talking about this for many hours.
• 25:52
  SPEAKER_03: I think the next time somebody asks me why we can't just create a dashboard for something tomorrow, I'm going to point them to this episode because I think it sums it up really nicely.
• 26:03
  SPEAKER_03: But yeah, thank you both for going through all of that.
• 26:06
  SPEAKER_03: I think there was a lot of really insightful stuff here.
• 26:09
  SPEAKER_03: I always like to end with a standard final question.
• 26:12
  SPEAKER_03: I'm curious your take from this perspective on, and that is what would you most like to see from the Airflow project in the near future?
• 26:18
  SPEAKER_02: I'm all on the hop side.
• 26:19
  SPEAKER_02: I've been touching code as a CEO for over the past few years, but I haven't actually deployed Airflow in a minute.
• 26:24
  SPEAKER_02: And I remember the last time, I don't know if it's still painful, but I remember the last time I had to do it.
• 26:28
  SPEAKER_02: I was like, wow.
• 26:30
  SPEAKER_01: Astro is in business as we uh very easy.
• 26:34
  SPEAKER_03: I think it's a little easier now than it used to be, certainly easier than when I was doing it back in like 2019, but it's still you might still get some wows out there.
• 26:44
  SPEAKER_01: I think more like first-class support for event-driven wordflows is that's a big part of what the community is interested in going forward.
• 26:54
  SPEAKER_01: But being able to hook a DAG into responding to an arbitrary webhook is really what I want to see out of the airflow balls.
• 27:03
  SPEAKER_01: I think we've seen really data centric things come up with the support of data sets as a way to trigger pipelines.
• 27:11
  SPEAKER_01: And I think that solves a very domain-specific problem.
• 27:14
  SPEAKER_01: But I'd love to see it get even more in general as we go forward.
• 27:18
  SPEAKER_02: Yeah.
• 27:19
  SPEAKER_02: I think one of the things that I've seen recently that I think would be very beneficial for most open source projects to kind of lean into is I saw this really interesting project recently, right?
• 27:29
  SPEAKER_02: You gotta think again, like what is the cause of most outages today?
• 27:32
  SPEAKER_02: It's configuration change, the opt level, right?
• 27:34
  SPEAKER_02: Like I've made a change to something and it broke, right?
• 27:36
  SPEAKER_02: It's DNS, or it is somebody changed some terraform someplace, right?
• 27:40
  SPEAKER_02: And I think one of the things that makes almost all open source projects a bit easier is there's this tool that's been released recently called Creek P R E Q.
• 27:49
  SPEAKER_02: And there's this new concept of something called a CRE.
• 27:51
  SPEAKER_02: So very similar to a C B for security.
• 27:53
  SPEAKER_02: A C R E is like the reliability equivalent.
• 27:56
  SPEAKER_02: It's common reliability enumerations.
• 27:58
  SPEAKER_02: And what they effectively are is from the point of view of this software, that they're very common ways of misconfiguring it.
• 28:04
  SPEAKER_02: It's going to lead to an outage.
• 28:06
  SPEAKER_02: And so there's this movement now around this preek tool in CRE for maintainers of projects to take some of the old GitHub issues of, hey, we found this combination of configs that makes this hard or causes an outage.
• 28:16
  SPEAKER_02: And so I would just love to see in general more open source projects like start to lean into this idea of, hey, let's document our CREs, include it with our repo.
• 28:25
  SPEAKER_02: So now if you're like, hey, man, that's something's really weird going on with this project, I don't know what it is.
• 28:29
  SPEAKER_02: I can run the preek tool and be like, oh, the reason you just had this outage is because you had this combination of configs.
• 28:35
  SPEAKER_02: And in a place like Airflow, where you have so much config where you're just getting lost and something's down in data, you're like, oh man, this is expensive for us right now.
• 28:46
  SPEAKER_02: And knowing that like you're gonna spend eight hours and find that config, like deep in a GitHub issue someplace, it's just it's the worst.
• 28:54
  SPEAKER_02: And it's a super, it's a super cool tool, and it pretty much works with any configuration language, so it's easy to bring in.
• 29:00
  SPEAKER_01: That's really neat.
• 29:01
  SPEAKER_01: I'd love to see more of that.
• 29:03
  SPEAKER_01: Airflow does have like you look at the config reference API and just like hundreds and hundreds of configs, and some of them do compete, and it's easily easy enough to have one config that turns off your scheduler or sets your like max parallelism way lower than is needed for your workflow.
• 29:23
  SPEAKER_03: This is such a big challenge with Airflow in general, is that again, with the like purposeful, like unopinionated stance of like you can have the answer to everything be throw a config in there.
• 29:36
  SPEAKER_03: And then as you said, it just starts to get impossible for people to manage or even know what's in it or fix something if it goes wrong.
• 29:44
  SPEAKER_03: So I love both of those answers.
• 29:46
  SPEAKER_03: I think those are great things for the project to think about going forward.
• 29:49
  SPEAKER_03: Well, thank you both so much for joining us today.
• 29:52
  SPEAKER_03: It's been really great having you on.
• 29:53
  SPEAKER_03: What is the best way for folks to get in touch with you?
• 29:55
  SPEAKER_02: I'm on LinkedIn.
• 29:56
  SPEAKER_02: I'm Corey, master.cloud.
• 29:58
  SPEAKER_02: You can send me an email.
• 29:58
  SPEAKER_02: I check that stuff every once in a while.
• 30:00
  SPEAKER_02: But uh LinkedIn is where I'm probably the most active, where I'm borderline the most professional.
• 30:06
  SPEAKER_01: Yeah, folks can find me at also on LinkedIn, Jake Ferrero or on Instagram, Jake T Ferrero.
• 30:14
  SPEAKER_03: Okay, great.
• 30:14
  SPEAKER_03: We'll throw all of that in the show notes.
• 30:16
  SPEAKER_03: And I would imagine folks will have questions again.
• 30:19
  SPEAKER_03: I think there's a lot more we could dig into here if we had more time.
• 30:23
  SPEAKER_03: And this is a really big topic.
• 30:24
  SPEAKER_03: So appreciate you both digging into it.
• 30:27
  SPEAKER_03: And we had, I think, no wildlife encounters during this recording.
• 30:32
  SPEAKER_02: No wildlife.
• 30:32
  SPEAKER_02: I didn't even see my dog climbing the tree back here.
• 30:34
  SPEAKER_02: She must have heard data and was like, no, not today.
• 30:37
  SPEAKER_03: I'm into the I'm not getting into like terraform anything.
• 30:42
  SPEAKER_03: Nothing.
• 30:42
  SPEAKER_02: She's not a fan.
• 30:43
  SPEAKER_02: I think if she could take the CKE, the Kubernetes cert, I think she could pass it if she could only like type.
• 30:49
  SPEAKER_02: She hears this stuff all day long.
• 30:50
  SPEAKER_02: She's like, I got this.
• 30:51
  SPEAKER_02: My odd.
• 30:53
  SPEAKER_03: Yeah, I feel that.
• 30:55
  SPEAKER_03: That's great.
• 30:55
  SPEAKER_03: All right.
• 30:56
  SPEAKER_03: Thank you both so much again.
• 30:57
  SPEAKER_03: It's been really great chatting with you.
• 30:59
  SPEAKER_01: Yeah, thanks for getting us together, Kevin.
• 31:01
  SPEAKER_01: Good to see you, Corey.
• 31:02
  SPEAKER_00: Good to see you.
• 31:05
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 31:12
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 31:16
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.