Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot

Transcript

Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: We've used dynamic DAG generation in a number of places, like generating DAGs that move data to and from Airtable.
• 0:06
  SPEAKER_02: Since a common requirement for web applications that human generated or human-augmented data is to have that integration with Air Airtable, generating DAGs from web scraper scripts and configuration files, since we do a fair amount of web scraping and at a high level of abstraction, like all web scrapers are the same.
• 0:23
  SPEAKER_02: Or you can imagine a DAG that would treat them as the same.
• 0:42
  SPEAKER_01: Hi everyone.
• 0:43
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:44
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm joined by Jennifer Malot, who is a technical lead at the Center for Security and Emerging Technology at Georgetown University.
• 0:54
  SPEAKER_01: Jennifer, how are you?
• 0:57
  SPEAKER_02: I am doing okay.
• 0:58
  SPEAKER_02: Really glad to be here.
• 1:00
  SPEAKER_02: Okay.
• 1:00
  SPEAKER_01: Great.
• 1:01
  SPEAKER_01: Are you impacted by the cold snap that is currently hitting, it seems like a lot of the US?
• 1:08
  SPEAKER_02: Yes, very much.
• 1:09
  SPEAKER_02: And we've also been getting an unusual amount of snow, which in DC kind of means that everything grinds to a hull to some degree, although they're getting better at plowing and so on.
• 1:16
  SPEAKER_01: But yeah, we definitely, I'm over in Seattle and we it it is not snowing.
• 1:20
  SPEAKER_01: It's a little bit cold here, but not not too much uh compared to normal.
• 1:24
  SPEAKER_01: But we have the same sort of situation where it doesn't snow very frequently here.
• 1:27
  SPEAKER_01: And when it does, we have a lot of hills and no snow plows owned by the city.
• 1:32
  SPEAKER_01: So uh things are things are pretty shut down when that happens.
• 1:36
  SPEAKER_01: Ot times.
• 1:37
  SPEAKER_01: Yeah, well, I'm glad uh it's not too disruptive, at least uh for you right now.
• 1:42
  SPEAKER_01: Let's go ahead and dive in.
• 1:44
  SPEAKER_01: So for our listeners for today's podcast, we're going to discuss how the Center for Security and Emerging Technology does data orchestration for emerging technology analysis.
• 1:55
  SPEAKER_01: Uh so Jennifer, to start, can you tell me a bit about what the Center for Security and Emerging Technology or CSET does?
• 2:04
  SPEAKER_02: Yeah, so CSET is a think tank at Georgetown University.
• 2:07
  SPEAKER_02: And most of our employees are analysts, so they write reports on various security-related aspects of emerging technology like AI.
• 2:13
  SPEAKER_02: So that could be, we define security related maybe more broadly than people might often think.
• 2:18
  SPEAKER_02: So this could be analysis on workforce, implications, governance, peer watch, and so on.
• 2:22
  SPEAKER_02: And those reports are read by decision makers in government and elsewhere.
• 2:26
  SPEAKER_02: So listening to that, you might be like, okay, where does Airflow fit in with this?
• 2:29
  SPEAKER_02: But the analysis is actually often really data driven.
• 2:32
  SPEAKER_02: And to support that work, we have a 12-person data team.
• 2:35
  SPEAKER_02: So it includes people who work with the analysts, like run SQL queries to help aggregate data that's relevant to the reports those analysts are writing, data scientists who do data augmentation work, like training classifiers, clustering, and so on to enable new kinds of analysis.
• 2:50
  SPEAKER_02: And then also software engineers who work on our data infrastructure and writing airflow pipelines and building web applications to help enable more like self-service analysis.
• 2:59
  SPEAKER_01: Well, that's a really interesting use case that I haven't heard much about for Airflow.
• 3:04
  SPEAKER_01: I think this analysis of emerging technologies, especially AI, is one we hear a lot about, at least in my world.
• 3:11
  SPEAKER_01: I never hear a lot of specifics about how those things are taken into consideration by governments and universities that might do something about it.
• 3:19
  SPEAKER_01: So that's a really cool implementation of Airflow to hear about.
• 3:25
  SPEAKER_01: So tell me a little bit more about your Airflow usage.
• 3:28
  SPEAKER_01: What does that look like when you're, you know, supporting those pipelines that do that data-driven analysis?
• 3:34
  unknown: Yeah.
• 3:35
  SPEAKER_02: So when I first joined CSAT about five and a half years ago, the organization was really new, maybe six or seven months old, and we contracted with a number of vendors for data on everything from scholarly publication to patenting, investment workforce, and so on.
• 3:47
  SPEAKER_02: And we just needed the data to be automatically ingested into our data warehouse.
• 3:52
  SPEAKER_02: And we use BigQuery on Google Cloud.
• 3:53
  SPEAKER_02: So one of my early tasks was just writing ETL pipelines using Airflow to accomplish that.
• 3:59
  SPEAKER_02: And then along with tasks downstream of that, like merging different data sources together and deduplicating metadata and so on.
• 4:06
  SPEAKER_02: Later, our data scientists started doing tasks like training classifiers and clustering over our scholarly literature.
• 4:12
  SPEAKER_02: And we needed Airflow pipelines to deploy those classifiers, add new documents to existing clusters, and so on.
• 4:18
  SPEAKER_02: And then even later, we started this emerging technology observatory effort, which I co-lead, and that effort builds web applications that support more self-service data-driven exploration analysis of different aspects of emerging technologies.
• 4:31
  SPEAKER_02: And behind those web applications are even more Airflow pipelines that run other analytic queries and classifiers, move data to and from Airtable, where our analysts do manual data augmentation and keep databases behind our web applications up to date.
• 4:46
  SPEAKER_02: So I would say that Airflow is definitely really central to our data operations.
• 4:51
  SPEAKER_01: Okay, interesting.
• 4:52
  SPEAKER_01: And so one of the things you mentioned was, you know, using Airflow to run pipelines that ingest and augment scholarly literature, which I'd imagine is pretty dense.
• 5:02
  SPEAKER_01: How have you managed that from like an infrastructure and performance standpoint?
• 5:07
  SPEAKER_01: Or is that relevant?
• 5:08
  SPEAKER_02: Yeah, I would say it's at least somewhat relevant.
• 5:10
  SPEAKER_02: Like we're obviously not as large scale as many organizations, but some of our databases are like terabytes in size or over a billion records.
• 5:17
  SPEAKER_02: So we've definitely run into some scaling challenges now and then.
• 5:21
  SPEAKER_02: So we just use a lot of ordinarily, we'd use a lot of Kubernetes pod operators to run jobs on nodes of various sizes or data flow operators so we can leverage Apache Beam if we're working on a task that's really parallelizable, like classifier deployment and data normalization.
• 5:37
  SPEAKER_02: Occasionally we've run into stability issues with automating really long-running tasks that need to run on a custom feed, like reclustering.
• 5:43
  SPEAKER_02: So we've managed to work around that though, by just using sensors to detect when those tasks are finished and then trigger downstream tasks.
• 5:50
  SPEAKER_01: Okay, interesting.
• 5:52
  SPEAKER_01: So other than the long-running tasks, where it sounds like you've had to implement a little bit of a work around.
• 5:56
  SPEAKER_01: My next question was going to be are there any challenges associated with implementing those sort of methods for dealing with that larger data or long-running tasks?
• 6:05
  SPEAKER_01: I think one of the challenges that we often hear about people using Kubernetes pod operator, things like that, is just that they're they're pretty open-ended, and that's a pretty, you know, unique skill set to be working with those tools.
• 6:17
  SPEAKER_01: So it can be challenging to figure out what exactly you need to do to support your workloads.
• 6:23
  unknown: Yeah.
• 6:23
  SPEAKER_02: I can say more later about just how we make Airflow more accessible to the team in general.
• 6:29
  SPEAKER_02: But yeah, I think there weren't any.
• 6:32
  SPEAKER_02: It definitely took a little bit of doing to handle those long-running tasks.
• 6:37
  SPEAKER_02: Since that was Yeah, we we couldn't even use uh Kubernetes pod operators in those those cases, since we were like using custom VM.
• 6:44
  SPEAKER_02: So we had to do be a little bit hacky and do some workarounds.
• 6:47
  SPEAKER_02: But I would say there weren't any really big challenges.
• 6:50
  SPEAKER_02: So many people use Airflow, the Stack Overflow is usually pretty successful in helping us when um things aren't obvious.
• 6:56
  SPEAKER_01: Yeah, definitely one of the benefits to of the large Airflow community for sure.
• 7:02
  SPEAKER_01: Well, you just mentioned doing some work to make Airflow more accessible to your team.
• 7:07
  SPEAKER_01: It sounds like your team has a pretty varied background, which can sometimes make it challenging to standardize on technical tools.
• 7:13
  SPEAKER_01: So tell me a little bit more about what you've done to make Airflow more accessible to them.
• 7:19
  SPEAKER_02: Yeah, we've used a lot of dynamic DAG generation that's been super helpful in a number of different ways for this kind of issue.
• 7:25
  SPEAKER_02: Um, so one scenario we ran into early on was that people would want to write just a sequence of analytic SQL queries, run some checks on the results, and then if those pass, copy um the staging tables into a production data set.
• 7:40
  SPEAKER_02: So, like my first foray into using dynamic DAG generation to help people access Airflow was just writing a script that would generate a DAG that would do that task from like a configuration JSON and like a set of SQL queries.
• 7:53
  SPEAKER_02: And so other members of the team could just define that JSON and their directory of SQL queries, and then their own DAGs would just appear in the Airflow web interface without them actually needing to learn to use Airflow.
• 8:05
  SPEAKER_02: And since then we've used dynamic DAG generation in a number of places, like generating DAGs that move data to and from AirTable.
• 8:12
  SPEAKER_02: Since a common requirement for our web applications that human generated or human-augmented data is to have that integration with Air Airtable, generating DAGs from web scraper scripts and configuration files, since we do a fair amount of web scraping and at a high level of abstraction, like all web scrapers are the same.
• 8:29
  SPEAKER_02: Or you can imagine a DAG that would treat them as the same.
• 8:33
  SPEAKER_02: And then generating DAGs that copy data to Zenodo where we release our publicly available data sets.
• 8:38
  SPEAKER_02: But we found it that just really makes Airflow more accessible to people and produces boilerplate within our organization.
• 8:47
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 8:56
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 9:08
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 9:17
  SPEAKER_01: I think the idea of people with analytical backgrounds needing to operationalize their SQL queries is a really, really common one that we see in Airflow use all the time.
• 9:27
  SPEAKER_01: I think by the time this episode airs, the Airflow survey results from the year will have been out.
• 9:32
  SPEAKER_01: And one of the things that we ask the Airflow community in that is what other languages are you trying to run with Airflow and SQL?
• 9:39
  SPEAKER_01: It was by far the most so super common, which totally makes sense.
• 9:44
  SPEAKER_01: When you say you implemented, you know, dynamic DAG generation to take their queries and automatically turn them into a DAG in Airflow.
• 9:53
  SPEAKER_01: Are you using a script that you wrote yourself?
• 9:56
  SPEAKER_01: Have you leveraged a tool like DAG Factory that's out there in the community to help generate those DAGs, or how do you do that?
• 10:03
  SPEAKER_02: Yeah, I in this case it was a script I wrote myself.
• 10:07
  SPEAKER_02: This was several years ago.
• 10:08
  SPEAKER_02: I'm not sure if DAG Factory was around that it may have been, and I dismissed it, but the script is pretty simple to write.
• 10:14
  SPEAKER_02: Yeah, I have actually looked at DAG Factory and it definitely looks like a really helpful tool.
• 10:19
  SPEAKER_02: So we might look into that for future needs.
• 10:22
  SPEAKER_01: Yeah, sometimes I think a lot of teams can run into issues with dynamically generating DAGs when they get to higher scales, is typically where we've seen things start to go awry for people.
• 10:34
  SPEAKER_01: So sometimes using a tool like DAG Factory can help with something like that, or it just provides a little more structure that you don't have to make yourself.
• 10:41
  SPEAKER_01: But it sounds like what you've been, what you've implemented has been working pretty well.
• 10:47
  SPEAKER_02: I'll definitely look into it regardless.
• 10:49
  SPEAKER_01: Astronomer recently, well, recently it was maybe six months or so ago, took over the open source DAG Factory project.
• 10:56
  SPEAKER_01: So still obviously going to remain open source, but we're working a little bit on development on it.
• 11:00
  SPEAKER_01: So we're a little biased, I suppose, in promoting it.
• 11:04
  SPEAKER_01: But we see definitely a lot of interest in it within the Airflow community because people either don't have the background or even just don't want to be writing Python DAGs themselves.
• 11:14
  SPEAKER_01: I think you touched on this a little bit, but a lot of DAG code ends up being very, very similar across the DAGs in an organization.
• 11:21
  SPEAKER_01: And you just don't need to be dealing with that boilerplate yourself all the time.
• 11:25
  SPEAKER_02: Yeah, I like the idea of a project that abstracts this away across organizations as well as across projects within an organization.
• 11:31
  SPEAKER_02: So yeah, yeah.
• 11:33
  SPEAKER_01: And then one of the other automations that you mentioned was for the DAGs that publish the data that your team makes available publicly.
• 11:43
  SPEAKER_01: So I'm curious to learn a little bit more about what the dynamic aspect of that looks like.
• 11:49
  SPEAKER_02: Yeah, I think in that case, it's mostly just configuring the DAG based on the SQL queries that need to get run to prepare the data, and then also a little Python script that packages up the data, which can vary depending on the project, what would make the most sense.
• 12:04
  SPEAKER_02: So those are the two pieces that differ for DAG, but the rest of it just like hitting the Zenodo API and so on, is of course the same.
• 12:11
  SPEAKER_02: Okay.
• 12:12
  SPEAKER_01: Interesting.
• 12:12
  SPEAKER_01: Have you, for something like that, ever looked into using dynamic task mapping as a feature like within a DAG for creating the copies that you need to for the SQL scripts?
• 12:24
  SPEAKER_01: Or is that something you don't think would work for your use case?
• 12:28
  SPEAKER_02: I would say we we haven't run into issues with our current method.
• 12:33
  SPEAKER_02: I've been doing some reading about dynamic task mapping since it seems like it could definitely be useful for some of our work.
• 12:38
  SPEAKER_02: In this case, we would just we often just either read the contents of a directory that contains a number of SQL queries and then just like within our DAG, we'll just iterate over the contents of that directory and generate a task from that.
• 12:54
  SPEAKER_02: But it's entirely possible that dynamic task mapping would be a better or interesting alternate approach.
• 13:01
  SPEAKER_01: Yeah, it sounds like it could potentially work for something like that.
• 13:04
  SPEAKER_01: I think one of the biggest benefits of dynamic task mapping is that you retain all of the history for every DAG run that you've had and you have a little more control over, you know, retrying certain tasks or things.
• 13:17
  SPEAKER_01: So typically with some sort of dynamic flow, what we see is either people cram everything into one task, in which case, like if you have to rerun, it's it's not great, you don't have great visibility, or you have something like a script that generates multiple DAGs for the dynamic aspect, which typically works pretty well.
• 13:37
  SPEAKER_01: I think the only downside I have seen is again that if that script changes and you know one of the queries no longer exists, then that DAG no longer exists in Airflow.
• 13:47
  SPEAKER_01: So you don't have the history of what ran.
• 13:49
  SPEAKER_01: You can't see it anymore.
• 13:51
  SPEAKER_01: Versus with dynamic task mapping, you would just maintain all of that within your DAG or set of DAGs from one run to the other.
• 13:59
  SPEAKER_02: That definitely sounds helpful.
• 14:01
  SPEAKER_01: Yeah, pretty helpful.
• 14:03
  SPEAKER_01: Still, you know, I think we're still seeing a lot of additional adoption within dynamic task mapping, and some of the more recent versions of Airflow have built on top of the initial feature, which has been out for a couple of years now, but just adding more functionality to it.
• 14:18
  SPEAKER_01: I think people, you know, people always hit edge cases and need to be able to do something specific.
• 14:24
  SPEAKER_01: So now it's a little easier to map over tasks within various scenarios.
• 14:29
  SPEAKER_01: But yeah, I think there's also the aspect of whether it's worth it to migrate something that you already have that's working or not.
• 14:36
  SPEAKER_01: Usually that comes from some particular impetus of something going wrong.
• 14:41
  SPEAKER_02: Still, there's so much cool new stuff being released in Airflow all the time.
• 14:44
  SPEAKER_02: It seems to be worth the effort to try to keep things at least somewhat up to date.
• 14:48
  SPEAKER_01: Yeah, definitely.
• 14:49
  SPEAKER_01: I mean, I'm personally biased, like a big part of my job is you know, talking about new Airflow versions, but there is so much exciting stuff coming out within the project.
• 14:57
  SPEAKER_01: And I think we're looking at Airflow 3 coming out in a couple of months here, maybe less than a couple of months by the time this airs.
• 15:04
  SPEAKER_01: So even more great stuff to take advantage of where things like you're describing are just getting easier and easier in terms of implementation.
• 15:14
  SPEAKER_01: Yeah, cool.
• 15:16
  SPEAKER_01: Well, that was a great overview of your use case, really interesting to hear about.
• 15:20
  SPEAKER_01: I always like to end with a standard final question, and that is what is one thing you would like to see from the Airflow project in the near future.
• 15:28
  SPEAKER_02: Yeah, so I'm really excited about the efforts to make it easier to track data lineage.
• 15:33
  SPEAKER_02: Like I love seeing AIP 53, like getting open lineage integrated with Airflow completed a while back.
• 15:38
  SPEAKER_02: We've been experimenting with using this in cloud dataplex, which can consume that lineage information in GCP.
• 15:46
  SPEAKER_02: And I'm really excited for the upcoming AIP on extending that to getting lineage from hooks.
• 15:51
  SPEAKER_02: For us, just being able to track data lineage well is super helpful for our team since we're often querying tables that depend on like several layers of DAGs and analytic queries, and just the better data lineage tracking gets for us, the more the whole team can easily gain insight into where their tables come from, even without necessarily knowing airflow or reading through all our DAGs and documentation just makes it easier for everybody.
• 16:13
  SPEAKER_01: Yeah.
• 16:14
  SPEAKER_01: Tell me more about that.
• 16:15
  SPEAKER_01: Is it more like the visual aspect of being able to see kind of the entire graph of where everything's coming from that's helpful for you?
• 16:22
  SPEAKER_02: Yes, exactly.
• 16:23
  SPEAKER_02: Like I showed a couple of people, at least what we've been able to see within Cloud Dataplex.
• 16:27
  SPEAKER_02: And everybody was like, oh, this is so helpful to just be able to visualize it without having to dig through DAGs and documentation and so on.
• 16:33
  SPEAKER_01: Yeah, yeah, that's great.
• 16:35
  SPEAKER_01: We Astronomer is currently working on a product that we have out there now called Observe that, you know, does something very similar, just gives people more observability into their data pipelines, including how things are connected, you know, across DAGs, including across different environments, which makes debugging a lot easier and figuring out like if there's a problem where it actually originated.
• 16:55
  SPEAKER_01: But I think as you said, it's also really helpful just when you have a complex environment to be able to see how everything actually fits together, especially if you're working on separate teams that have separate responsibilities and aren't necessarily communicating closely, even though their data is used across them.
• 17:14
  SPEAKER_01: Sounds very clear.
• 17:15
  SPEAKER_01: Yeah, yeah, that's great to hear.
• 17:17
  SPEAKER_01: Okay, well, Jennifer, it's been really wonderful having you on.
• 17:20
  SPEAKER_01: To close out, what's the best way for folks to get in touch with you?
• 17:24
  SPEAKER_02: Probably my LinkedIn.
• 17:25
  SPEAKER_02: So I'm just on there as Jennifer Muller.
• 17:28
  SPEAKER_01: Okay, great.
• 17:29
  SPEAKER_01: Well, thank you again.
• 17:30
  SPEAKER_01: It was great having you on.
• 17:31
  SPEAKER_02: Yeah, it was great to be here.
• 17:32
  SPEAKER_02: Thanks for the opportunity.
• 17:36
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 17:43
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 17:46
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.