Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth

Transcript

Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: A lot of that is right after a game.
• 0:02
  SPEAKER_02: At the major league level, we can't supply reports during a game like you actually may see in the NBA or in the NFL due to some nefarious cheating scandals that happened a few years ago.
• 0:14
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:23
  SPEAKER_01: All right.
• 0:24
  SPEAKER_01: Hello, everyone.
• 0:25
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:27
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm joined by Alexander Booth, the assistant director of RD for the Texas Rangers Baseball Club.
• 0:35
  SPEAKER_01: And Alexander, how are you doing?
• 0:36
  SPEAKER_02: Not so bad.
• 0:37
  SPEAKER_02: Kind of had a tough game yesterday, but uh season still looks in good shape here.
• 0:41
  SPEAKER_02: And I'm very excited to see if we can go back to back with 2023 in terms of winning the World Series, that is.
• 0:46
  SPEAKER_01: Yeah, high bar to top for sure.
• 0:48
  SPEAKER_01: And so you guys are just getting started with the general season, is that right?
• 0:52
  SPEAKER_02: Yep, the season's about one month of the way through.
• 0:55
  SPEAKER_02: So we've still got the whole summer to go here.
• 0:57
  SPEAKER_02: It is really interesting.
• 0:58
  SPEAKER_02: These early games do count, and we've already played seven games against the Astros, which is a lot, and three games against Seattle.
• 1:04
  SPEAKER_02: So some of these in division rivals decided to have some high pressure games throughout the entire year.
• 1:09
  SPEAKER_02: Supposed to maybe just in September.
• 1:11
  SPEAKER_01: That's awesome.
• 1:11
  SPEAKER_01: Well, I'm admittedly not the biggest follower of baseball, but in my defense, I am from Seattle originally, and that's where I live now.
• 1:18
  SPEAKER_01: So I think growing up, you know, the Mariners don't have the greatest track record.
• 1:22
  SPEAKER_01: So it's not the most fun one to follow.
• 1:24
  SPEAKER_02: But yeah, no, definitely understand.
• 1:25
  SPEAKER_02: Seattle's a great team this year, though.
• 1:27
  SPEAKER_02: So it's a really fun division to be in with Seattle's great pitching staff, and you can never count out the Astros and the Angels are still, of course, with Mike Trout, they're always a threat.
• 1:36
  SPEAKER_02: So definitely an exciting year in baseball.
• 1:38
  SPEAKER_01: That's great.
• 1:38
  SPEAKER_01: Well, I'm sure we'll have lots of local people jumping on the bandwagon as it gets more exciting.
• 1:43
  SPEAKER_01: That's great.
• 1:44
  SPEAKER_01: Hope your future games go better than yesterday's, of course.
• 1:47
  SPEAKER_01: Well, let's go ahead and dive in.
• 1:49
  SPEAKER_01: So today we're gonna be talking about how the Texas Rangers use data to win games and how you got started with Airflow and some of the use cases that Airflow powers today for you.
• 1:58
  SPEAKER_01: Let's start to give everybody a lay of the land.
• 2:00
  SPEAKER_01: Maybe talk a little bit about your role, Texas Rangers Baseball Club.
• 2:04
  SPEAKER_01: How long have you been there and what do you do?
• 2:05
  SPEAKER_02: Yeah, definitely.
• 2:06
  SPEAKER_02: So my role is the assistant director of research and development within baseball operations for the Texas Rangers.
• 2:11
  SPEAKER_02: I've been at the club since 2018.
• 2:13
  SPEAKER_02: So this is going to be my seventh season.
• 2:15
  SPEAKER_02: And when I started, I was maybe the fourth or fifth person on their data analytics RD side.
• 2:20
  SPEAKER_02: So just for clarity, I do work on the baseball operation side.
• 2:23
  SPEAKER_02: So that is the money ball side of things, the where we can interact with players, recommend trades, look at contracts, then of course recommend strategy on and off the field across all leagues and levels of our minor league and major league development systems.
• 2:36
  SPEAKER_02: So my current role as assistant director of RD, I work with strategizing and architecting our modern kind of big data ecosystem.
• 2:43
  SPEAKER_02: You may not think big data in baseball, board where working on that data lies in baseball now.
• 2:48
  SPEAKER_02: And it's really fascinating to see that since 2021, there's been an explosion in technology in the space.
• 2:53
  SPEAKER_02: We now have multiple cameras at every single stadium.
• 2:57
  SPEAKER_02: We're doing pose tracking, we're looking at biomechanics, the movement of the body.
• 3:02
  SPEAKER_02: We're tracking the players and the balls at hundreds of frames per second for the entire game.
• 3:07
  SPEAKER_02: And that data is not only just coming to us from the major leagues, but also the minor league side, even the amateur side, some of the Division I schools, LSU, Vanderbilt, Florida, Wake Forest, they have these high-tech systems in place now as well.
• 3:20
  SPEAKER_02: So with this rapid rise of big data in baseball, how do we make that available?
• 3:27
  SPEAKER_02: How do we transform it?
• 3:28
  SPEAKER_02: How do we run AI and machine learning on it to help make better decisions at the end of the day?
• 3:33
  SPEAKER_02: Because that's what data is in baseball.
• 3:35
  SPEAKER_02: It's a decision-making tool.
• 3:37
  SPEAKER_02: How can we have confidence in the players that we put on the field?
• 3:40
  SPEAKER_02: How can we have confidence in the lineup?
• 3:42
  SPEAKER_02: How do we have confidence in the trades that we use and recommend?
• 3:45
  SPEAKER_02: And that's all going to be furnished with the data driving those decisions, in addition to the domain expertise, of course, of the coaches and people that have been in the game for many years.
• 3:54
  SPEAKER_01: That's a really big jump in just a short amount of time, three years, I think, at Astronomy.
• 3:59
  SPEAKER_01: You know, we're used to thinking of big data for everything.
• 4:02
  SPEAKER_01: But as you said, a lot of people probably don't associate that as much with baseball.
• 4:06
  SPEAKER_01: And that seems like it's even a pretty new development in terms of obviously the history of the sport.
• 4:11
  SPEAKER_02: Yeah, it's really interesting how Major League Baseball even advertises this.
• 4:15
  SPEAKER_02: Now, if you watch a Major League Baseball broadcast, you'll see highlights in the game of the ball being tracked and say, oh, this home run went 450 feet.
• 4:23
  SPEAKER_02: And you get a little animation behind it, and it says powered by Statcast.
• 4:27
  SPEAKER_02: So Statcast originally started in 2015, but there's been so many iterations on it.
• 4:32
  SPEAKER_02: First, it was only tracking baseballs, and it was only doing that at a fraction of the frame rate.
• 4:37
  SPEAKER_02: Then they started adding in the player tracking.
• 4:39
  SPEAKER_02: Then they started adding in kind of bat tracking and biomechanics tracking.
• 4:43
  SPEAKER_02: We now track the bat at 300 frames a second, which is just crazy.
• 4:47
  SPEAKER_02: And then also new last year was weather tracking.
• 4:50
  SPEAKER_02: So we now get weather data for every single batted ball event.
• 4:53
  SPEAKER_02: So we can look exactly at how the wind impacted your hit.
• 4:57
  SPEAKER_02: And that's important to us because we play in a closed roof stadium most of the year.
• 5:00
  SPEAKER_02: We do open the roof, maybe about 10% of our games.
• 5:03
  SPEAKER_02: But when we play with a closed roof, there's obviously no wind.
• 5:05
  SPEAKER_02: So if you play in a park like Wrigley's, that's in Chicago, the wind typically helps get a lot of extra home runs pushed out to the outfield.
• 5:13
  SPEAKER_02: So that player, if they played in our stadium, they probably wouldn't hit as many home runs because of that wind impact.
• 5:19
  SPEAKER_02: So it's just really interesting to see how this new data is shaping some data strategies as well as interpretations of finding that true talent of the player themselves.
• 5:27
  SPEAKER_01: That's really cool.
• 5:27
  SPEAKER_01: I would imagine in a game, a series that's so, so competitive, having even that minor advantage that the data can give you is really important.
• 5:36
  SPEAKER_02: Definitely.
• 5:37
  SPEAKER_01: So with that massive increase in both the amount of data that you're collecting for all of these different use cases that you've already mentioned and implementing that even just within the last couple of years.
• 5:47
  SPEAKER_01: Maybe talk a little bit about how you did that.
• 5:49
  SPEAKER_01: Obviously, we talk a lot about airflow on this podcast.
• 5:52
  SPEAKER_01: So maybe talk about how airflow fits into that as well.
• 5:55
  SPEAKER_02: So airflow has been instrumental in how we organize and manage our data orchestrations.
• 6:00
  SPEAKER_02: And this started just because the players wanted their data.
• 6:03
  SPEAKER_02: They wanted their data faster.
• 6:05
  SPEAKER_02: A lot of modern baseball players are growing up actually in little league and high school and college, surrounded by reports about how they're performing.
• 6:13
  SPEAKER_02: They did their own data about how they are as a player and their benchmarks that they need to hit.
• 6:18
  SPEAKER_02: So when they reach our system, whether through the draft or through signings, they're asking for these reports as fast as possible.
• 6:24
  SPEAKER_02: And not only do we have these big data sets that we need to transform, join together, organize, but then we also need to get them in front of our players as fast as possible with a rigorous schedule.
• 6:34
  SPEAKER_02: So when we needed a scalable solution that could scale with the amount of data vendors that we had, as well as allow for those interdependencies between data pipelines, which I'll probably start calling DAGs here.
• 6:45
  SPEAKER_02: This is the Airflow podcast, right?
• 6:47
  SPEAKER_02: So all of our different DAGs that kind of take data from the weather, from the player tracking, from the ball tracking, from the minor leagues, from college.
• 6:55
  SPEAKER_02: How do we start transforming those, organizing them, shaping them together, and then driving that forward and making sure that they run as quickly and as seamlessly as possible?
• 7:03
  SPEAKER_02: Well, there's been a lot of new innovations in Airflow in the last few years that we have really taken advantage of.
• 7:09
  SPEAKER_02: One of the first being that data set aware scheduling.
• 7:12
  SPEAKER_02: We just updated actually to Airflow 2.9, I think last week, maybe the week before.
• 7:16
  SPEAKER_02: So 2.9 has a really great update where you can actually have multiple data sets, have some dinner operators, as well as adding some time-based scheduling there.
• 7:25
  SPEAKER_02: But that's a big use case for us.
• 7:27
  SPEAKER_02: How do we kind of have these different data sets that still have dependencies on each other?
• 7:31
  SPEAKER_02: Again, I need both the ball tracking and the player tracking to be done before I then make a DAG to send out the reports or join that data together.
• 7:40
  SPEAKER_02: So having those be data set aware in our DAGs has been very helpful.
• 7:44
  SPEAKER_02: Plus, it's been very great to scale.
• 7:47
  SPEAKER_02: We get new agreements with new data vendors every single week, it feels.
• 7:51
  SPEAKER_02: And so being able to ingest those pipelines and then again schedule them and be able to combine them with our existing data sets and our existing DAG workflows, we needed that scalable solution and we love Airflow there because of that.
• 8:05
  SPEAKER_02: We're huge fans of the open source community.
• 8:07
  SPEAKER_02: When I moved into this role in 2021, 2022, I made a push to really invest and utilize open source solutions.
• 8:15
  SPEAKER_02: Of course, we've been burned in the past with vendor lock-in.
• 8:18
  SPEAKER_02: We've been burned in the past with specific vendors not iterating or including new features that maybe we specifically needed to want to help or that we wanted to use for our use cases.
• 8:29
  SPEAKER_02: Now, of course, Airflow being open source has been a huge driver of that.
• 8:32
  SPEAKER_02: We've actually submitted a couple of issues that got resolved by the community, which is really awesome to see and hear, specifically around some of the Databricks operators and the big query operators.
• 8:43
  SPEAKER_02: Those are two of our main data platforms that we have in baseball.
• 8:46
  SPEAKER_02: And the constant attribution to open source, the way that the community innovates and progressively makes new and awesome releases of the product made it, again, like a natural fit for us, especially with our partnership with Astronomer and Airflow too.
• 8:59
  SPEAKER_01: That's so awesome.
• 9:00
  SPEAKER_01: I work with the Airflow community a lot in my day-to-day.
• 9:02
  SPEAKER_01: And I have to say, this is like the ideal Airflow user that you're describing to me having an early upgrade story.
• 9:08
  SPEAKER_01: I mean, 2.9 was just released two weeks ago.
• 9:10
  SPEAKER_01: So the fact that you're already using it and making use of those new features is so great to hear.
• 9:14
  SPEAKER_01: And we obviously love stories about people actually giving that feedback back.
• 9:18
  SPEAKER_01: I know the folks developing Airflow really care about that and value it a lot.
• 9:21
  SPEAKER_01: So really cool to hear.
• 9:23
  SPEAKER_02: The timing was really serendipitous because we just had a problem maybe about a month ago where we needed two data sets to also run on a time.
• 9:31
  SPEAKER_02: And we had some really, really bad code that we're trying to get working in the existing Airflow.
• 9:36
  SPEAKER_02: And then I read some of the upcoming release notes and I was like, wait a second, let's just put a pin in this two weeks.
• 9:40
  SPEAKER_02: And then that's the other great thing about open source.
• 9:42
  SPEAKER_02: If you have a problem, you're typically not the only one that's had this problem or difficulty.
• 9:47
  SPEAKER_02: And just being able to, again, have that open collaboration and conversation around this and work as a community towards that solution is awesome.
• 9:55
  SPEAKER_02: Again, we've been submitting issues.
• 9:57
  SPEAKER_02: I'm hoping to be more impactful in that open source community moving forward.
• 10:01
  SPEAKER_02: But that is, again, one of the great reasons why we utilized Airflow for our modern orchestration strategy.
• 10:07
  SPEAKER_01: I think that particular feature that you're talking about that was just released being able to schedule on or have more advanced conditional data set scheduling was maybe one of the most frequently requested features we've seen from the community since the original data sets came out back in Airflow 2.4.
• 10:23
  SPEAKER_01: Obviously, just it changes the game with what you can do in terms of event-based triggering, which it sounds like for you when you have so many different data sets that your pipelines are dependent on can be really important.
• 10:33
  SPEAKER_02: And the event-based is huge as well.
• 10:35
  SPEAKER_02: Like I said, the players want their information quickly, and a lot of that is right after a game.
• 10:40
  SPEAKER_02: At the Major League level, we can't supply reports during a game, like you actually may see in the NBA or in the NFL due to some nefarious cheating scandals that happened a few years ago.
• 10:51
  SPEAKER_02: Their MLB is really restricted in what can be communicated during a game.
• 10:55
  SPEAKER_02: But as soon as the game is over, the game is fresh.
• 10:58
  SPEAKER_02: The baseball hitter whose home run was robbed at the wall, they're upset about that.
• 11:04
  SPEAKER_02: But being able to see the report that says, hey, that was a home run in 29 out of 30 parks.
• 11:09
  SPEAKER_02: Our metrics, our KPIs say you did a really, really good hit there.
• 11:12
  SPEAKER_02: Keep trying like that and you're gonna be great.
• 11:15
  SPEAKER_02: Or the pitcher whose strike three call was actually caught a ball by the umpire and he's mad about it and now he ended up walking the guy.
• 11:23
  SPEAKER_02: They want to look at the data immediately.
• 11:25
  SPEAKER_02: So as soon as we get the pitch, as soon as we know that the pitch has been processed, that's when we want to start triggering some of these DAGs, start triggering some of these pipelines to start getting that data already ready to go.
• 11:37
  SPEAKER_02: So as soon as the final pitch is thrown in the ninth, the DAGs have already run.
• 11:42
  SPEAKER_02: We already have those reports available, and the players can digest that immediately.
• 11:46
  SPEAKER_02: Baseball's a marathon, not a sprint.
• 11:48
  SPEAKER_02: It's 162 game season.
• 11:50
  SPEAKER_02: So we can't wait 24 hours.
• 11:52
  SPEAKER_02: By the time it's tomorrow, they're already focusing on that day's game, repairing with that day's data for that night's pitcher.
• 12:00
  SPEAKER_02: So it's really important to get that speed and velocity in terms of return time for those data reports.
• 12:05
  SPEAKER_02: So being able to use data sets, being able to have that event-based, that almost micro batch or to an extent, almost streaming pipelines through those DAGs have been more than valuable in our kind of revolutionized new ecosystem here.
• 12:20
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:29
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:41
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 12:48
  SPEAKER_01: It's great to hear about that use case where airflow is really impacting the success of the team and your players.
• 12:54
  SPEAKER_01: Maybe for that delivering reports directly after the game, anything specific you can talk about in terms of what is being tracked during the game and the data that you're using, then what way is that being given to the players afterwards so that they can improve?
• 13:08
  SPEAKER_02: So in our World Series run last year, a big conversation around us was of course winning on the road.
• 13:13
  SPEAKER_02: But beyond that, looking at a more micro view at specifically the games, we had a huge increase in our defensive ability.
• 13:21
  SPEAKER_02: And there was a lot of conversation around how we were able to be in the right spot at the right time.
• 13:25
  SPEAKER_02: Or of course, our players are really great at defending and they have a lot of innate talent in that.
• 13:30
  SPEAKER_02: But during the course of last year, we really wanted to focus on how we can optimize our defensive positioning recommendations.
• 13:36
  SPEAKER_02: So before the game, we have a whole set of machine learning models that permutate and figure out based on the count, based on the pitcher, based on if runners are on base, where should our second baseman stand?
• 13:49
  SPEAKER_02: Where should our shortstop stand?
• 13:51
  SPEAKER_02: And then that's going to give them, again, given their talent, the best likelihood of being able to make a play at the ball.
• 13:58
  SPEAKER_02: The players don't always remember where exactly they need to stand.
• 14:01
  SPEAKER_02: Of course, we do have that player tracking during the game.
• 14:04
  SPEAKER_02: So this is again one of those items where after the game is over, we're able to sit down and look at these reports.
• 14:09
  SPEAKER_02: We have our recommended positioning where we said, hey, this is where we thought you should stand given the situation.
• 14:15
  SPEAKER_02: And here's where you actually stood.
• 14:16
  SPEAKER_02: And then we can watch the video, see how the play was made, and then say, oh, it didn't really matter.
• 14:21
  SPEAKER_02: You made the play anyway, or great job.
• 14:23
  SPEAKER_02: You stood at the spot and it really helps you make the play.
• 14:25
  SPEAKER_02: And then of course, sometimes when they don't send anywhere near the data recommendation and they miss the play, we can use that as again motivation to stand in our preferred recommended spot.
• 14:35
  SPEAKER_02: So that's another one of these big data sources because the players are being tracked at 30 frames a second across a three-hour game.
• 14:42
  SPEAKER_02: That ends up being a lot of information to process and bring in.
• 14:46
  SPEAKER_02: But having that ready after the game is over to look at and connect to again the video as well as the players themselves and their own experiences really drives home that point.
• 14:56
  SPEAKER_02: So I thought that was a really interesting story.
• 14:58
  SPEAKER_02: We had a lot of great double plays.
• 14:59
  SPEAKER_02: We had a lot of great catches in the outfield as well as in the infield.
• 15:03
  SPEAKER_02: And of course, so much of that has to be attributed to the players.
• 15:06
  SPEAKER_02: But I like to think that the data helped a little bit in recommending that they stand in the right spot so that they're more able to make the play given their talent.
• 15:14
  SPEAKER_02: And I think my last story I'll mention there too, we talked a little bit about the weather.
• 15:17
  SPEAKER_02: It's actually had a huge impact in the ALDS last year.
• 15:21
  SPEAKER_02: In the ALDS, we played Baltimore, the Baltimore Orioles.
• 15:24
  SPEAKER_02: And the first two games were actually in Baltimore.
• 15:27
  SPEAKER_02: So Baltimore, they moved their left field fence further back.
• 15:30
  SPEAKER_02: So it's technically harder to hit home runs to left field.
• 15:34
  SPEAKER_02: But again, looking at the weather, we knew that the wind was blowing out to left field that day.
• 15:38
  SPEAKER_02: We stacked some guys in the lineup that maybe had a little bit more power to that side of the field, which people don't normally do in Camden Yards, and ended up with Mitch Garver hitting like a grand slam in game two, which was really great to see.
• 15:50
  SPEAKER_02: And we went on to, of course, win both of those games on the road.
• 15:54
  SPEAKER_02: I think it's really interesting to see how data is shaping the game.
• 15:57
  SPEAKER_02: And because of technologies like Airflow, that's able to organize, orchestrate, and connect various different DAGs and various different data pipelines and even across various cloud ecosystems.
• 16:09
  SPEAKER_02: I mentioned BigQuery a little bit.
• 16:11
  SPEAKER_02: We also have presence on AWS and Databricks.
• 16:14
  SPEAKER_02: There's a lot of moving parts to where this data is going.
• 16:17
  SPEAKER_02: And there's also a lot of moving parts to how it gets consumed through various BI tools, through various web applications.
• 16:23
  SPEAKER_02: Again, having that unified system to connect all of those disparate points and effectively has been just such a revolution to us.
• 16:31
  SPEAKER_02: And before we moved into this airflow space, it was a lot of cron jobs, a lot of task manager or task scheduler on local machines.
• 16:40
  SPEAKER_02: It's been a completely new paradigm in data orchestration and data movement and that commitment to open source and the community.
• 16:48
  SPEAKER_02: I can see why so many people are passionate about the product in this space.
• 16:52
  SPEAKER_01: Those are two really cool stories.
• 16:55
  SPEAKER_01: And I think ML applications and ML ops and using that Airflow is a really hot topic right now.
• 17:02
  SPEAKER_01: And so you're talking about using the data to model where your players should stand or how the weather is going to impact them ahead of time and using that to provide recommendations.
• 17:11
  SPEAKER_01: Where do you think Airflow stands out for those use cases?
• 17:15
  SPEAKER_01: I mean, you mentioned a little bit about it sitting at the middle and connecting with all of these other tools for those machine learning applications specifically.
• 17:23
  SPEAKER_01: Any kind of learning that you've had there after implementing Airflow?
• 17:27
  SPEAKER_02: I think it all depends on having that kind of modern AI architecture already defined.
• 17:33
  SPEAKER_02: So whether it's in SageMaker, that's the AWS machine learning hosting environment, whether it's in kind of BigQuery ML, whether it's in Azure ML or Databricks, or even what we've experimented with is actually having a physical model that we've containerized and then be able to interact with that container using Airflow too.
• 17:53
  SPEAKER_02: Of course, Airflow as that main orchestration piece can connect with all of that.
• 17:57
  SPEAKER_02: But I think that's a really important piece of the data pipeline is having some conditional logic in the DAG to say, all right, what data is coming in and where should it be routed to?
• 18:08
  SPEAKER_02: Which model should it be routed to get this prediction?
• 18:11
  SPEAKER_02: Or should it even be going through a model?
• 18:13
  SPEAKER_02: Is there enough confidence in the values?
• 18:15
  SPEAKER_02: We have a lot of missing values, null values, outliers.
• 18:18
  SPEAKER_02: And so we're doing a lot of work now to detect when a baseball is recorded to be thrown at 140 miles an hour, which is impossible.
• 18:26
  SPEAKER_02: Then maybe we shouldn't necessarily make a prediction on that pitch.
• 18:30
  SPEAKER_02: But being able to have that conditional logic, being able to have dynamic tasks, that's been another big use case we've had in an airflow, being able to take 300 pitches, or maybe that was only 295, and be able to blast them out as individual tasks to be run through in a machine learning model to get those individualized predictions back.
• 18:49
  SPEAKER_02: But I think just the way to continue being that connector, being able to host all these different ways that and that organizations are competing with AI again, whether it's containerized systems, whether it's with APIs, whether it's with cloud-based partners.
• 19:05
  SPEAKER_02: And I think just continued support there as well as continued flexibility and how those predictions should interact with existing data pipelines and DAGs is where the community is going to be going with that continued integration, at least in my opinion.
• 19:20
  SPEAKER_01: I think that's spot on from my experience as well.
• 19:23
  SPEAKER_01: And obviously, cool to hear about that use case.
• 19:25
  SPEAKER_01: I think Airflow has a lot of features of a generic orchestrator that help a lot when implementing that sort of work at scale that you obviously could not do with a cron job or something more rudimentary.
• 19:37
  SPEAKER_01: That's cool to hear.
• 19:38
  SPEAKER_02: I think the the dynamic task, that example in baseball is may not have been the best one.
• 19:42
  SPEAKER_02: I think maybe a better one is on games.
• 19:44
  SPEAKER_02: So not every day there's a different number of games played.
• 19:47
  SPEAKER_02: Sometimes every team plays, sometimes there's rain delays, sometimes teams have off days, then you incorporate minor league systems as well.
• 19:54
  SPEAKER_02: So typically the way our pipelines work is we need to figure out what games happened yesterday or today, and then for each game, go off and get all the pitches that were tracked for that game.
• 20:04
  SPEAKER_02: So having a dynamic way to figure out how many games were played or having that flexibility to just create those requests in parallel using dynamic tasks has been an awesome feature.
• 20:16
  SPEAKER_02: And we've been able to scale that really effectively using some of the scheduler manipulations and I can't remember what the other one was.
• 20:23
  SPEAKER_02: Gosh, it was like the cluster assignment behind the scenes to be able to optimize the slots so we didn't end up running into any rate limiting issues.
• 20:33
  SPEAKER_01: Since you've already upgraded to two nine and you are using dynamic task mapping, have you tried the new custom naming of mapped task instances?
• 20:42
  SPEAKER_02: I have not.
• 20:43
  SPEAKER_02: We've been all focused in on the data sets as well as some of the UI stuff.
• 20:47
  SPEAKER_02: Oh yeah, the display names for data.
• 20:49
  SPEAKER_01: Yeah.
• 20:49
  SPEAKER_01: I think a big one for especially in cases like yours where you've got like a massive, potentially massive number of tasks each time it's running.
• 20:56
  SPEAKER_01: It's just previously you would have an index number instance.
• 21:00
  SPEAKER_01: So if something failed, you would go, all right, well, instance 40 failed.
• 21:04
  SPEAKER_01: What is instance 40?
• 21:05
  SPEAKER_01: Which game is that referring to?
• 21:07
  SPEAKER_01: So now you can assign those names programmatically.
• 21:09
  SPEAKER_01: It just makes it meant a lot easier.
• 21:12
  SPEAKER_01: Another fun thing that came out just a couple of weeks ago.
• 21:14
  SPEAKER_02: I am actually super excited about this feature because yes, we do have zero to 40.
• 21:19
  SPEAKER_02: And I'm like, well, game number 37 failed, and I have no idea what game number 37 is.
• 21:24
  SPEAKER_02: So I'm either going deep into the logs because I'm having too many print statements out, but that is a very exciting update.
• 21:30
  SPEAKER_02: So yeah, as you said, it's a very fun feature.
• 21:32
  SPEAKER_02: And I'm sure now that we're already on two nine, that's something that I'll be putting into practice.
• 21:37
  SPEAKER_02: So maybe today, actually.
• 21:38
  SPEAKER_01: Yeah, that's awesome.
• 21:39
  SPEAKER_01: Like I said, my team does a lot of work with the community and talking about new airflow features.
• 21:43
  SPEAKER_01: And I think that this has been really fun.
• 21:45
  SPEAKER_01: Some of these I think are just very approachable.
• 21:47
  SPEAKER_01: You know, everybody's familiar with baseball and loves watching it.
• 21:50
  SPEAKER_01: So these are fun examples, I think, that we can take and say, hey, this is how somebody in the community is using this feature.
• 21:56
  SPEAKER_02: Yeah, definitely.
• 21:57
  SPEAKER_02: It's again, next time you're watching a game, every single ball that's thrown, every ball that's hit, every movement that somebody does, it's being tracked.
• 22:05
  SPEAKER_02: And we want to use that information to make better decisions about the strategy of the game, best way we can win the interpretation of the player performance.
• 22:13
  SPEAKER_02: And it's with Airflow that we're able to connect all of those different vendors that are able to connect to those different data points that are being extracted.
• 22:21
  SPEAKER_02: And then, of course, to the use of AI and ML, be able to make those better decisions at speed and at velocity and at scale.
• 22:28
  SPEAKER_02: And I think that's really the unsung hero part in our data orchestration and our big data ecosystem.
• 22:34
  SPEAKER_01: That's great to hear.
• 22:35
  SPEAKER_01: Well, I know we are about at our scheduled time and we really appreciate you taking the time to join us today.
• 22:40
  SPEAKER_01: What's the best way for folks to get in touch with you?
• 22:43
  SPEAKER_02: So you can always find me on LinkedIn at Alexander Booth.
• 22:47
  SPEAKER_02: That's probably the best way to get in touch.
• 22:49
  SPEAKER_02: Otherwise, yeah, I don't think any of my other social medias are public, so maybe I need to after this.
• 22:54
  SPEAKER_01: You and me both.
• 22:55
  SPEAKER_01: All right.
• 22:55
  SPEAKER_01: Well, thanks again, Alexander.
• 22:56
  SPEAKER_01: We're really appreciated hearing your stories.
• 22:58
  SPEAKER_01: Super interesting.
• 22:59
  SPEAKER_02: Yeah, no, thanks for the opportunity to talk and excited to continue seeing where the community goes.
• 23:04
  SPEAKER_02: And if you have any other kind of ways for us to continue to provide feedback or recommendations for new features, would love to continue being a part of that open source community.
• 23:14
  SPEAKER_02: So yeah, thanks for the opportunity here to speak.
• 23:16
  SPEAKER_01: Thanks again.
• 23:18
  SPEAKER_02: Thank you.
• 23:19
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 23:25
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 23:29
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star reveal.