Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Season One Teaser
Transcript
- 0:00SPEAKER_00: Hi there.
- 0:00SPEAKER_00: Welcome to the Airflow podcast, produced by Astronomer.
- 0:04SPEAKER_00: The past few years have brought a dramatic shift in the data landscape.
- 0:07SPEAKER_00: As data gets bigger and faster, many organizations find themselves spending more time centralizing and preparing datasets for analytics than actually looking for insights.
- 0:16SPEAKER_00: In order to combat this effect, companies have invested heavily in extract, transform, load, or ETL, tools.
- 0:22SPEAKER_00: But ETL processes are changing and old school drag and drop tools no longer cut it for savvy data teens in search of a more flexible solution.
- 0:30SPEAKER_00: To satisfy this need for more customizable ETL pipelines, Airbnb data engineer Maxime Beauchemin opensourced Airflow, a workflow management system that defines tasks and their dependencies as code and then distributes and executes them on a defined schedule.
- 0:44SPEAKER_00: Built by developers for developers, Airflow is based on the principle that ETL is best expressed in code.
- 0:50SPEAKER_00: This eight-episode saga will cover the details of Airflow's origins, use cases, best practices, competitors, pain points, and future direction.
- 0:58SPEAKER_00: You'll hear from some extremely interesting people, including Maxime Boscherman, the creator of Airflow, Bolka De Bruin, head of advanced analytics technology at ING, Maxime Pekersky, CDO of the City of San Diego, Eric Bernardson, creator of Luigi and CTO of Better Mortgage, Patrick Atwater, Water Data Project Manager at Argo Labs, Scott Halgram, Data Engineer at Zapier, Chris Wirkomini, Principal Software Engineer at WePay, and Frank Sue, Data Engineer at Minds.io.
- 1:24SPEAKER_00: You'll notice that this group brings particularly diverse perspective.
- 1:27SPEAKER_00: From startup to corporate, data engineer to CTO, we've determined Airflow to be an extremely flexible and universal tool for ETL.
- 1:34SPEAKER_00: Furthermore, since it's completely open source, teams are building upon it to do some very different things.
- 1:38SPEAKER_00: Check out just a few of the responses we got to the question: what is Airflow?
- 1:42SPEAKER_03: Workflow automation and what that means to us is a very nice way to deal with all the little annoying things that inevitably pop up when you're dealing with public data.
- 1:54SPEAKER_01: For us, um I mean I just call it a data automation system.
- 1:58SPEAKER_01: I think that's probably the best way to describe it.
- 2:01SPEAKER_01: So it's a way to run scheduled jobs in sequence, um, you know, with dependencies and failovers and notifications and all of that good stuff.
- 2:11SPEAKER_01: So not necessarily just data jobs, but jobs of any kind.
- 2:16SPEAKER_02: Task orchestration machinery, I guess, uh, or workflow uh workflow orchestration uh software that allows us to connect uh the myriad of data sources that we have internally and externally, and move that to a platform and be able to manage that in a proper way without either having a lot of uh license costs involved or uh the inflexibility of proprietary connectors, which might not just work for that particular data source at that point in time.
- 2:48SPEAKER_00: And there you have it.
- 2:50SPEAKER_00: We're extremely excited to take this journey with you.
- 2:52SPEAKER_00: Tune in every two weeks for a new episode and visit www.astronomer.io to learn a little bit more about airflow.