Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Mastering Data Orchestration with Airflow at M Science with Ben Tallman

Transcript

Mastering Data Orchestration with Airflow at M Science with Ben Tallman · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We run maybe eight to nine petabytes of data and hundreds of billions of records, changing on a pretty rate.
• 0:08
  SPEAKER_02: Basically, we run data at scale.
• 0:11
  SPEAKER_02: And without a tool like Airflow, we would be spending half of our energy managing and running workflows that we wouldn't have any centralized monitoring or guidance on.
• 0:26
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering and AI.
• 0:34
  SPEAKER_00: Hello, everyone.
• 0:36
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:37
  SPEAKER_00: I'm your host, Kenton Dannis.
• 0:39
  SPEAKER_00: And before we dive in today, I just want to share a friendly reminder that Airflow Summit is coming up.
• 0:44
  SPEAKER_00: It's taking place on September 10th through 12th this year at the Weston St.
• 0:49
  SPEAKER_00: Francis in San Francisco.
• 0:50
  SPEAKER_00: I will be there along with many other Airflow enthusiasts.
• 0:53
  SPEAKER_00: We'll have three great days of talks from companies like Ford Motors, NerdWallet, Uber, and many more.
• 0:59
  SPEAKER_00: So we'll have a link in our show notes for you to register and for you to get 30% off.
• 1:03
  SPEAKER_00: So now is a great time to get your tickets.
• 1:05
  SPEAKER_00: And all right, with that reminder, I'm super happy to be joined today by Ben Tallman, who is the CTO at M Science.
• 1:11
  SPEAKER_00: Ben, how are you doing today?
• 1:12
  SPEAKER_02: I'm doing very well.
• 1:13
  SPEAKER_02: Thank you for having me.
• 1:15
  SPEAKER_00: Okay.
• 1:15
  SPEAKER_00: Yeah, of course.
• 1:16
  SPEAKER_00: And remind me, Ben, are you still located in Portland?
• 1:19
  SPEAKER_02: Yeah, we're based or I'm based in Portland, Oregon, where give or take 20% of the businesses these days.
• 1:25
  SPEAKER_02: Okay.
• 1:25
  SPEAKER_02: Although we are a remote first.
• 1:28
  SPEAKER_00: Okay.
• 1:28
  SPEAKER_00: Us too.
• 1:29
  SPEAKER_00: I'm up in Seattle.
• 1:29
  SPEAKER_00: I mean, I'm from Seattle originally, but I did live in Portland for about a year and a half, relatively recently.
• 1:35
  SPEAKER_00: So I'm a PNW local and I really liked Portland, actually.
• 1:39
  SPEAKER_00: It was it's a great place.
• 1:40
  SPEAKER_02: I would love Seattle too, actually.
• 1:42
  SPEAKER_02: Seattle would be a great place to live if I didn't have to drive ever.
• 1:46
  SPEAKER_00: Yes, exactly.
• 1:47
  SPEAKER_00: That is the biggest downside.
• 1:48
  SPEAKER_00: It always made me laugh whenever folks in Portland would complain about either the traffic or the parking.
• 1:53
  SPEAKER_00: And I'm like, oh no.
• 1:55
  SPEAKER_02: Yeah, different worlds.
• 1:56
  SPEAKER_00: Yeah, different world for sure, but both great places.
• 1:59
  SPEAKER_00: Cool.
• 2:00
  SPEAKER_00: Let's go ahead and dive in.
• 2:01
  SPEAKER_00: So for our listeners in today's podcast, we're going to discuss Ben's history with the Airflow project and then how M Science powers their analytics products with Airflow orchestration.
• 2:10
  SPEAKER_00: So this year I mentioned the Airflow Summit in our intro.
• 2:13
  SPEAKER_00: We're super excited about this year's summit because it's celebrating Airflow's 10th anniversary, which is hard to believe.
• 2:19
  SPEAKER_00: For those not aware, it was originally created by Airbnb in 2014.
• 2:24
  SPEAKER_00: So 10 years ago, and then was open sourced back in 2016.
• 2:27
  SPEAKER_00: And Ben, you have quite a bit of history with the Airflow project yourself.
• 2:31
  SPEAKER_00: Can you maybe share for our listeners how you originally got involved with Airflow?
• 2:35
  SPEAKER_02: Yeah, actually back in 2016, thank you for popping that date out there.
• 2:39
  SPEAKER_02: It makes it easy to remember.
• 2:41
  SPEAKER_02: We had at a company called Apogee, where I was were looking really for a cron replacement.
• 2:46
  SPEAKER_02: And we fell upon something else.
• 2:49
  SPEAKER_02: I'll leave it nameless for now, but fairly quickly it became clear that was actually worse than cron.
• 2:55
  SPEAKER_02: Cron never failed itself to start scripts failed, but that would fail periodically and just they would disappear.
• 3:03
  SPEAKER_02: And so one of the things we noticed when looking in at the open source there was that no one was really contributing to that package.
• 3:11
  SPEAKER_02: It wasn't a dead package, but there were like five or six main contributors, and then they went away and periodically one contribution.
• 3:19
  SPEAKER_02: So we looked around at what else was out there, and Airflow was very nascent at this time.
• 3:25
  SPEAKER_02: And we thought it's got lots of one thing we want to make sure is that it doesn't disappear.
• 3:31
  SPEAKER_02: So we want a vibrant community.
• 3:33
  SPEAKER_02: And I, even though I was based in Portland, I visited the Bay Area every week for five years.
• 3:38
  SPEAKER_02: So there I was.
• 3:40
  SPEAKER_02: And on my way home from our San Jose office, I swung through indirectly the Airbnb office for an Airflow meetup.
• 3:49
  SPEAKER_02: And that happened to be the meetup where Airflow, where we all, me very new, but we all voted to join the Apache Foundation, I guess you would call it, as a I can't remember what you call it, but as a new member essentially, or a joining member.
• 4:08
  SPEAKER_02: And the thing that was immediately apparent was that there were, they had thought maybe 20 people would show up.
• 4:14
  SPEAKER_02: And I think there were probably 80 people there at that meeting.
• 4:16
  SPEAKER_02: In fact, I think we had to move out of the one room into a bigger room.
• 4:21
  SPEAKER_02: And Airbnb, that meant they brought two kegs with us because we wouldn't want to be far from the kegs, but there you go.
• 4:27
  SPEAKER_02: And we never really looked back, I would say.
• 4:30
  SPEAKER_02: Fairly quickly, I think we were processing about 40,000 test instances an hour.
• 4:36
  SPEAKER_02: And yeah, it was Airflow had its own issues, but they were very different than the previous issues.
• 4:42
  SPEAKER_02: Just instantly being able to find where the problem was.
• 4:46
  SPEAKER_02: In I think we probably had about 40 different DAGs that hold stats from around the business and drop them into a data warehouse.
• 4:55
  SPEAKER_00: Okay.
• 4:56
  SPEAKER_00: That's a very cool piece of Airflow history to have been a part of back in that original open sourcing.
• 5:02
  SPEAKER_00: Uh I'm sure lots of folks in the community will be excited to hear about that.
• 5:06
  SPEAKER_02: Max and Sid were both there that night.
• 5:08
  SPEAKER_02: It was very early, very early days and felt very community oriented.
• 5:14
  SPEAKER_02: It was very cool.
• 5:15
  SPEAKER_00: Yeah, even back then.
• 5:16
  SPEAKER_00: That's awesome.
• 5:17
  SPEAKER_00: There's certainly been quite a bit of change and advancement in Airflow over that time.
• 5:22
  SPEAKER_00: It seems crazy how much advancement there's been since 2019, which was when I first got involved in the project.
• 5:28
  SPEAKER_00: So several years after it was launched.
• 5:30
  SPEAKER_00: How has all of that change impacted how you and your teams work with Airflow?
• 5:35
  SPEAKER_02: Sure.
• 5:35
  SPEAKER_02: So at apogee, which then sold to Google, and I actually ran production grade Airflow instance at Google as well.
• 5:43
  SPEAKER_02: Pre-Google service, they offered as a SaaS service, and they were trying, starting to do that then.
• 5:50
  SPEAKER_02: At that time, I would say on my four-member sort of data engineering team, we spent probably 60% of our time modifying airflow and running Airflow, keeping the care and feeding of Airflow versus building and managing and running that.
• 6:09
  SPEAKER_02: And that's actually really fun and really cool and interesting, but it isn't actually what a data engineering team is put together to do, right?
• 6:19
  SPEAKER_02: Your mandate is actually to basically move the business forward, not build a tool called Airflow.
• 6:25
  SPEAKER_02: And where there's overlap, that's great.
• 6:27
  SPEAKER_02: I'm actually very proud of the fact that in M Science, we don't really, we have zero pull requests on Airflow.
• 6:37
  SPEAKER_02: And back in the day, I had a pull fairly regular pull requests.
• 6:41
  SPEAKER_02: And so we, you know, much larger team, and we focus 100% on building and running bags and not on building and running airflow.
• 6:52
  SPEAKER_02: I spend a fair amount of time thinking about airflow, but that's not the team, right?
• 7:00
  SPEAKER_02: We I don't employ a team to do that anymore.
• 7:02
  SPEAKER_02: It runs.
• 7:03
  SPEAKER_02: Part of that is from Astronomer, and part of that is just it's a much more mature pack.
• 7:09
  SPEAKER_02: And it to give you some idea at M Science, we use data to predict public company revenues.
• 7:17
  SPEAKER_02: And we we run maybe eight to nine petabytes of data and hundreds of billions of records changing on a pretty rate.
• 7:28
  SPEAKER_02: Very basically, we run data at scale.
• 7:31
  SPEAKER_02: And without a tool like Airflow, we would be spending half of our energy managing and and running workflows that we wouldn't have any centralized monitoring or guidance on.
• 7:46
  SPEAKER_00: Yeah, totally.
• 7:46
  SPEAKER_00: Certainly a lot more stability in the project and maturity, as you've said now, which I think everybody appreciates.
• 7:53
  SPEAKER_00: I remember myself back, I started with Airflow in 2019, helping a team transition to Airflow from a legacy scheduler back before I worked for Astronomer.
• 8:02
  SPEAKER_00: And I remember the days of, oh, you have to restart the scheduler every so often manually.
• 8:07
  SPEAKER_00: And like you would come in on Monday morning, something had gone down, and Airflow is just not running.
• 8:11
  SPEAKER_00: And the kind of management and just day-to-day of making sure that it would run and be stable was really hard for a small team.
• 8:20
  SPEAKER_00: And I think you just don't hear about that nearly as much anymore as the project has progressed, which is great.
• 8:24
  SPEAKER_02: Considering how much work Airflow was to run, it's pretty startling how bad cron must be.
• 8:32
  SPEAKER_00: Yeah.
• 8:32
  SPEAKER_02: Cron just runs.
• 8:33
  SPEAKER_02: That part doesn't fall over.
• 8:35
  SPEAKER_02: But when you're running at scale with a large team, you're turning out terabytes of log files.
• 8:41
  SPEAKER_02: Trying to figure out where something went sideways is a startlingly difficult project.
• 8:46
  SPEAKER_00: Yeah.
• 8:47
  SPEAKER_00: Certainly not a lot of observability in cron, for sure.
• 8:51
  SPEAKER_02: No.
• 8:51
  SPEAKER_02: And it seemed obvious to me that there would really, I thought I was going to be able to just go out to Git or somewhere and find someone who would have built an observability layer for Chron.
• 9:03
  SPEAKER_02: Sure.
• 9:03
  SPEAKER_02: Nope, nothing like that.
• 9:05
  SPEAKER_00: Yeah.
• 9:06
  SPEAKER_00: I do think there's still so much discussion of observability now, which is warranted for sure, and improvements to airflow and observability.
• 9:14
  SPEAKER_00: And astronomers investing heavily in observability in our own products.
• 9:17
  SPEAKER_00: And it is funny to think back of like where everything was back in the days when it was more popular.
• 9:24
  SPEAKER_00: Like we have come a long way for sure.
• 9:26
  SPEAKER_00: Yeah, well, that's great.
• 9:27
  SPEAKER_00: So you talked a little bit about how you use Airflow currently and your role at M Science.
• 9:33
  SPEAKER_00: Let's dig into that a little bit more.
• 9:34
  SPEAKER_00: So at M Science, you provide this analyst-curated research and insights to your customers.
• 9:39
  SPEAKER_00: So data is obviously very central to what you do.
• 9:42
  SPEAKER_00: Maybe talk a little bit more about the products you provide so folks can get a good understanding of how that works.
• 9:48
  SPEAKER_02: Sure.
• 9:48
  SPEAKER_02: So at its core, we look at data to predict company revenues and really company performance.
• 9:55
  SPEAKER_02: And then we sell them those insights to hedge funds and private equity firms and really even and corporates.
• 10:02
  SPEAKER_02: What that means, though, is that we are at any given time really in the weeds using data to look at 200 to 250 different corporations.
• 10:16
  SPEAKER_02: And keeping all that straight is a really serious challenge, right?
• 10:21
  SPEAKER_02: The thing about the financial markets is they basically reset quarterly when the companies, when these companies release their results, everything we've done up till then essentially resets and we start over again for the next 90 days.
• 10:37
  SPEAKER_02: What's not exactly 90 days, but in general, what that means though is that we're never ahead of ball.
• 10:45
  SPEAKER_02: We're always just a step behind.
• 10:47
  SPEAKER_02: And time is money.
• 10:48
  SPEAKER_02: When I think about what we do, what the technology team brings to M Science, we're one of our core mandates is to reduce the time to value for our customers, right?
• 10:59
  SPEAKER_02: And if you take even one step above, what we're trying to do there is reduce churn and increase upsell, right?
• 11:05
  SPEAKER_02: So what we care about is getting the data all the way into the processes that our customers are running internally, right?
• 11:15
  SPEAKER_02: And just like software, shelfware is something that happens to people all the time.
• 11:20
  SPEAKER_02: They buy the product, but they don't use it for the first year.
• 11:22
  SPEAKER_02: The subscription's up, or maybe they don't re-up.
• 11:26
  SPEAKER_02: And for us, the same thing.
• 11:28
  SPEAKER_02: People might buy our product, but then a year later, they're not going to reinvest in that subscription unless they've found value.
• 11:38
  SPEAKER_02: And if it just sat on the shelf, then they didn't find value.
• 11:41
  SPEAKER_02: That is an opportunity for us to lean in with them, but we'd prefer that when they buy it in the first place, they get it directly into their processes and they begin to find value and leverage our data to make money.
• 11:53
  SPEAKER_02: If they do that, they're not going to churn.
• 11:55
  SPEAKER_02: So we're always looking for ways to take any complexity out of the process, right?
• 12:01
  SPEAKER_02: One thing that Airflow lets us do, we have observability into our product from the time that the data arrives in mostly our S3 bucket, let's say, sometimes in a Snowflake share, sometimes a Delta share, but we actually have some SFTP deliveries still too.
• 12:21
  SPEAKER_02: But no matter where it comes in, it's going to come in, going to come in through a process managed by an airflow DAG.
• 12:27
  SPEAKER_02: It will then move from you can call it a bronze through a silver layer through a gold layer.
• 12:33
  SPEAKER_02: And along the way, you know, we take hundreds of data sources and produce hundreds of data products, but it's a matrix, right?
• 12:44
  SPEAKER_02: It isn't that we buy one data source and produce one data product and do that 200 times.
• 12:50
  SPEAKER_02: We merge those.
• 12:52
  SPEAKER_02: And so when we look at a ticker for or a company for the and and look at their data, we're not looking, and by the way, it's never first party data.
• 13:02
  SPEAKER_02: So we're always using a piece of data that's from outside a company to review that company.
• 13:09
  SPEAKER_02: And we're trying to see so that if a data source is late, for instance, we need to figure out what data products will be delayed by that particular data source.
• 13:20
  SPEAKER_02: And what Airflow lets us do is really observe the product across those different DAGs.
• 13:27
  SPEAKER_02: So when one, when we have a delay in one product in one data source, we know that it will affect these 15 data products.
• 13:35
  SPEAKER_02: And we can use that to, along the way, essentially reduce the time it takes us to move data in the front of the business and out to the customers.
• 13:46
  SPEAKER_02: And that that's money.
• 13:59
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 14:11
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 14:19
  SPEAKER_00: That makes sense.
• 14:20
  SPEAKER_00: I think data products is something that we hear more and more about, where Airflow kind of going way back historically, as we were talking about before in the kind of 2016 to 2020 days.
• 14:31
  SPEAKER_00: It was used a lot more internally at companies, powering dashboards and things like that, which it still is, obviously, but more and more we're hearing about it actually being used to provide data that is the product that companies are giving to their customers.
• 14:45
  SPEAKER_00: So that's pretty core to your business then.
• 14:48
  SPEAKER_02: I think we've been data products or something that M Science has been selling for six years.
• 14:54
  SPEAKER_02: And if you consider the PDFs and the written research as a product as well for 23 years, and this is a place where the content is key, right?
• 15:04
  SPEAKER_02: Most places struggle to develop content.
• 15:06
  SPEAKER_02: They want to build viral content and valuable value.
• 15:10
  SPEAKER_02: We have that content.
• 15:11
  SPEAKER_02: The hard part is orchestrating that in a way that's reliable and aligned.
• 15:18
  SPEAKER_02: So when our product at the tip of our product is that core written research that sits on top, as we said, things that are orchestrated, a set of dashboards.
• 15:29
  SPEAKER_02: And below that, a set of data that's just very curated, and below that, more raw data.
• 15:35
  SPEAKER_02: And we orchestrate that whole process in Airflow.
• 15:40
  SPEAKER_02: That doesn't mean that we don't leverage, we're a huge Databricks customer, we're a big Snowflake customer, we're a big AWS customer, smaller Azure, et cetera.
• 15:50
  SPEAKER_02: But in general, what that means, but what leveraging airflow means is that we're not locked into any of those vendors.
• 15:58
  SPEAKER_02: We're not locked in anywhere.
• 16:00
  SPEAKER_02: And we're able to see because we're not really looking just at the data, we're looking at the product across.
• 16:06
  SPEAKER_02: And one of the things that we've done at MScience that's really key is to instrument airflow.
• 16:12
  SPEAKER_02: And actually, we're working with Astronomer to bring some of that stuff back into airflow in a way because if you look at observing a product, it doesn't live in one DAG and it doesn't live, it lives in a matrix.
• 16:26
  SPEAKER_02: You know, DAGs, you end up wanting to trace a product through multiple DAGs and even deployments, right?
• 16:32
  SPEAKER_02: And so being able to observe those and do that.
• 16:36
  SPEAKER_02: I think I was talking to the team there the other day.
• 16:39
  SPEAKER_02: And one of the things that is become clear in 30 years in technology is that you never actually know what the important things to track are going to be when you build it.
• 16:50
  SPEAKER_02: You think you do.
• 16:51
  SPEAKER_02: You think it's very obvious.
• 16:53
  SPEAKER_02: But after the fact, it turns out actually that what you're really trying to monitor and observe turns out to be different than you thought it was going to be.
• 17:01
  SPEAKER_02: And really, the only thing you know it won't be is the thing you thought it was going to be.
• 17:06
  SPEAKER_02: That sounds circular, but it's just really true, right?
• 17:09
  SPEAKER_02: Anyone who thinks they're going to predict what the important pieces are just hasn't been there very long, is what I would say.
• 17:16
  SPEAKER_00: Yeah, of course.
• 17:18
  SPEAKER_00: You mentioned using Airflow to track these data products and they're touched by both different pipelines or different DAGs within your Airflow instance and also in different systems.
• 17:29
  SPEAKER_00: Like Airflow is almost never used alone, or at least I've never seen that.
• 17:34
  SPEAKER_00: Are there airflow features that you use to help with that visualization or connection of kind of the different pieces around a data set?
• 17:43
  SPEAKER_00: So, like data sets as the airflow feature comes to mind, or like a lineage integration or anything like that that you're leveraging to make that easier or more visible?
• 17:54
  SPEAKER_02: That's interesting.
• 17:55
  SPEAKER_02: You talk about lineage and I think open lineage is the astronomer package there.
• 17:58
  SPEAKER_02: But in general, our experience with those is that they they haven't actually been a huge win for us.
• 18:08
  SPEAKER_02: We think they're sort of on the cusp of it.
• 18:10
  SPEAKER_02: What we did, we actually leaned in on the Pythonic nature of Airflow, and we used some mix-ins to perform a little bit of magic.
• 18:20
  SPEAKER_02: And we output basically a huge amount of data about our instance and the run, the running of our instance into a data warehouse, and we report on top of that.
• 18:35
  SPEAKER_02: The thing that I'm really excited about when we look at step up from Airflow a little bit and think about Astronomer is that we're at they're starting to build tools that allow us to query the runtime metadata after the fact across the whole implementation.
• 18:53
  SPEAKER_02: And that's actually what I did at Apogee.
• 18:56
  SPEAKER_02: And one of the big differences between running your own Airflow and renting it on a SaaS basis is that we I actually only had one transactional database inside my data warehouse, and that was the Airflow metadata store.
• 19:13
  SPEAKER_02: And it allowed me to just build Tableau reports on top of a metadata store.
• 19:18
  SPEAKER_02: And then I didn't have to get it right the first time.
• 19:21
  SPEAKER_02: As long as you use really good convention-based naming, you could go query your runtime, your task instance store and find out a lot of information about what had happened.
• 19:33
  SPEAKER_02: And that's what we do here.
• 19:34
  SPEAKER_02: We just have to export it into an external system.
• 19:37
  SPEAKER_02: And that's what we're really starting to see at Astronomer is that stuff starting to happen.
• 19:44
  SPEAKER_02: You know, it goes back to you don't know what you're going to want to query until after it's in the database.
• 19:51
  SPEAKER_00: Yeah, that totally makes sense.
• 19:53
  SPEAKER_00: And I think also you didn't mention this, but I was just thinking as you were talking about the data sets feature in Airflow as it currently is, I think also works a little better if you know how everything is going to be architected ahead of time, which as you're saying is not.
• 20:07
  SPEAKER_00: So certainly lots of development in the pipeline for the open source project on that feature, especially as we look towards Airflow three.
• 20:14
  SPEAKER_00: But one of the limitations that we've talked about now is that you kind of have to tell Airflow, here's how it all connects.
• 20:21
  SPEAKER_00: It's not going to go the other way.
• 20:23
  SPEAKER_02: And that's where I think of that semantic layer where if you've named everything properly and you have access to that store after the fact, you don't have to depend so much upon how getting it right the first time, frankly.
• 20:37
  SPEAKER_02: Right.
• 20:37
  SPEAKER_02: That's the problem with the data sets.
• 20:39
  SPEAKER_02: Things like that always work great on version phase four of your project or phase three of the project, but no one ever gets to phase three or four of the project, right?
• 20:48
  SPEAKER_02: You do phase one, phase two, it's all running, and then the team deflates a little bit and it's in production.
• 20:54
  SPEAKER_02: And that next step is painful, right?
• 20:56
  SPEAKER_02: Like going back and redoing all of your DAX and to be sitting on top of the right data sets that are the actual right ones, not the ones you predicted, that's the piece that never happens, right?
• 21:08
  SPEAKER_02: And so being able to query that database after the fact and label things after the fact it's very similar to the way tools like Cloudability or Cloud Health Tech allowed you to basically apply tags to your AWS infrastructure after it had run because you didn't get them tagged properly to start with, right?
• 21:32
  SPEAKER_02: So you could use a sort of after-the-facts tagging language to do that.
• 21:37
  SPEAKER_02: And by being able to separate the observability from the code, you're able to do.
• 21:44
  SPEAKER_02: And that's what we we look at.
• 21:46
  SPEAKER_00: That makes sense.
• 21:46
  SPEAKER_00: I'm just looking at our time here.
• 21:48
  SPEAKER_00: I think that actually leads nicely into my last question, which I always have a standard one I ask everybody.
• 21:53
  SPEAKER_00: And that's what would you most like to see from development of the Airflow project in the near future?
• 21:58
  SPEAKER_02: That's interesting.
• 22:00
  SPEAKER_02: I would really like to see.
• 22:04
  SPEAKER_02: So we sort of have an audit trail for DAGs now.
• 22:09
  SPEAKER_02: And I would like to see an audit trail for variables, we configuration variables, because we end up using those.
• 22:18
  SPEAKER_02: We use them because it's Python to do clever things with our source code or with our DAGs, right?
• 22:25
  SPEAKER_02: But the problem is you don't really know when those changed over time.
• 22:28
  SPEAKER_02: And so being I like separating the variables from the code, but I also like to be able to leverage them for logic as well.
• 22:38
  SPEAKER_02: And what that means is that you've lost the get history of the actual DAG because it's really the intersection of the Python code and the variables.
• 22:50
  SPEAKER_02: So from that direction, and I haven't really, I think I've mentioned that a couple of times, but I haven't talked about that a whole lot.
• 22:57
  SPEAKER_02: I think the other piece is less about Airflow and more about Astronomer, but really being able to think about apply those data set labels across, not so much data sets, but apply labels across stags, across infrastructure and environments, and really be able to build notifications and build both rules and ML-based quality checks into the system.
• 23:23
  SPEAKER_00: Yeah, totally.
• 23:24
  SPEAKER_00: That's great.
• 23:24
  SPEAKER_00: I know cross-deployment kind of dependencies and things like that is certainly something that's on our radar.
• 23:30
  SPEAKER_00: But I think that yeah, just even the audit logging for variables is certainly a very fair request for the open source project.
• 23:36
  SPEAKER_00: I think that's a reasonable.
• 23:38
  SPEAKER_02: Yeah, it's interesting because there that is also where we tend to store uh it's not an easy one because it's where we tend to store secrets.
• 23:46
  SPEAKER_02: It's not quite secrets.
• 23:48
  SPEAKER_02: We use more security there, but even being able to have a change log of that would be really good.
• 23:53
  SPEAKER_00: That makes sense.
• 23:54
  SPEAKER_00: Awesome.
• 23:55
  SPEAKER_00: Ben, thanks so much for your time today.
• 23:57
  SPEAKER_00: What's the best way for folks to get in touch with you?
• 23:59
  SPEAKER_02: Oh, wow.
• 24:00
  SPEAKER_02: I get a lot of email.
• 24:02
  SPEAKER_00: You don't have to promise a prompt response or anything.
• 24:06
  SPEAKER_02: BeTolement at mscience.com.
• 24:08
  SPEAKER_02: That's probably the best way to get me.
• 24:10
  SPEAKER_00: Okay.
• 24:10
  SPEAKER_00: Great, Ben.
• 24:11
  SPEAKER_00: It's been really great talking to you.
• 24:13
  SPEAKER_00: Thanks again.
• 24:14
  SPEAKER_00: Thank you.
• 24:14
  SPEAKER_02: Have a good day.
• 24:16
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:23
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:27
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.