Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Pain Points

Transcript

Pain Points · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Hi, and welcome to episode 5 of the Airflow podcast, produced by Astronomer.
• 0:05
  SPEAKER_02: In the first four episodes of this series, we've begun a deep investigation into Apache Airflow and have examined its origins, use cases, best practices, and competitors.
• 0:15
  SPEAKER_02: This week, we'll take a look at some of the pain points folks have experienced with Airflow.
• 0:21
  SPEAKER_02: It's been a while since our last episode, and I apologize for that.
• 0:24
  SPEAKER_02: We've done exciting things in the works.
• 0:26
  SPEAKER_02: Going forward, expect much more regular content.
• 0:29
  SPEAKER_02: We've taken the past month to synthesize the feedback we've received on the podcast and hone our approach to interviewing and editing, which will hopefully make for a better listening experience.
• 0:38
  SPEAKER_02: Thanks for bearing with us as we work to produce regular, high-quality content, and please keep the feedback coming as we've found it all extremely helpful.
• 0:47
  SPEAKER_02: This week, we'll examine conversations with both old guests and new to paint a comprehensive picture of Airflow's pain points.
• 0:54
  SPEAKER_02: While we still undoubtedly believe that Airflow is the future of ETL, it's important to acknowledge that any incubating project will have issues, and bringing those issues to the forefront of the community's attention will help shape the future of the project.
• 1:08
  SPEAKER_02: We'll talk with Thomas Lafiana, data engineer at Order My Gear, Frank Zhu, Data Engineer at Mines.io, and Alan Crookshank, Business Insights and Data Manager at Tails.com.
• 1:18
  SPEAKER_02: We'll begin with an excerpt from our discussion with Thomas.
• 1:21
  SPEAKER_02: Hope you enjoy.
• 1:36
  unknown: Cool.
• 1:36
  SPEAKER_04: So why don't we start with just like some intro on you, kind of uh what you do, where you are, what your company does, and kind of that whole shebang.
• 1:43
  SPEAKER_03: Yeah, absolutely.
• 1:44
  SPEAKER_03: So uh my name is Thomas Lapiana.
• 1:47
  SPEAKER_03: I'm a software engineer, uh, but really right now kind of more of a data engineer.
• 1:52
  SPEAKER_03: We all share the software engineer title here.
• 1:54
  SPEAKER_03: Um and I work at a company called Order My Gear, and what we do is um specialize in team sports um and and kind of giving them an online store.
• 2:05
  SPEAKER_03: Um so, say there's a high school team at you know, some say a basketball high school team, and they all want to order their gear and uh hence the name, order my gear.
• 2:16
  SPEAKER_03: And uh, you know, maybe the mom wants to order some you know hoodies and things like that.
• 2:20
  SPEAKER_03: Uh basically they they can set up this store and then the coach can email out a link to all the you know kids and their parents.
• 2:26
  SPEAKER_03: They can go on to the store and it's kind of like a one-stop shop for anything that that team might need.
• 2:30
  SPEAKER_03: Um, so could be, you know, if it's baseball, it could be ordering like gear and pads and stuff like that, uh, or it could be ordering kind of related merchandise, um, like I said, hoodies, you know, shirts, things like that.
• 2:44
  SPEAKER_04: Cool.
• 2:44
  SPEAKER_04: Sounds awesome.
• 2:45
  SPEAKER_04: Um, and then is your background in traditional software development, or like uh we've talked to some people who like, oh yeah, I used to like uh like much more on the AI side than move data engineering.
• 2:55
  SPEAKER_04: Uh so kind of like how do you find yourself doing what you're doing?
• 2:58
  SPEAKER_03: Yeah, um I'm not a computer scientist by education.
• 3:02
  SPEAKER_03: I was a poly sign major.
• 3:04
  SPEAKER_03: Um, but towards the end of that degree, kind of realized that that's not what I wanted to do, and that actually software was much more interesting.
• 3:11
  SPEAKER_03: Uh so right out of school, I got a job as a data intelligence consultant.
• 3:15
  SPEAKER_03: Uh, and that was actually my first time using Python and like a Postgres database and kind of building a pretty simple um data pipeline to get data out of an API and transform into database.
• 3:28
  SPEAKER_03: And so I came more from the business analyst side, um, but then the uh the technical aspect of that role is what really intrigued me.
• 3:35
  SPEAKER_03: So I just kept kind of moving further back end, uh and that's how I ended up as a data engineer.
• 3:41
  SPEAKER_04: Cool.
• 3:41
  SPEAKER_04: Um that's kind of the same thing that happened to me where first so astronomer is kind of my first job, and the first thing was like get data from this API and put it into this database, um, and it just kind of grows on from there.
• 3:52
  SPEAKER_04: Yep, absolutely.
• 3:54
  SPEAKER_04: Cool.
• 3:54
  SPEAKER_04: Um yeah, so why don't you want to tell a little bit about uh Order My Gears like data infrastructure?
• 4:01
  SPEAKER_04: Um like kind of what how what it was like when you started and kind of what it is now, and we'll like segue that into how you guys are using Airflow.
• 4:08
  SPEAKER_03: Yeah, absolutely.
• 4:10
  SPEAKER_03: So um we didn't have a data team really as of like kind of mid to late last year.
• 4:18
  SPEAKER_03: Uh sometime around halfway through the year, we hired a business analyst and started using Looker because before that we just kind of had our developers write some stuff in JavaScript and and kind of simple stuff like that.
• 4:30
  SPEAKER_03: Um we really decided like as the business is growing uh very quickly, you know, there's definitely something to be said for being able to access your data and analyze it and make good choices based off that.
• 4:40
  SPEAKER_03: And so we hired a business analyst uh who's really good at Looker and started kind of getting those dashboards up and running.
• 4:47
  SPEAKER_03: Um but what we quickly found was that kind of the the pretty basic infrastructure we had in place was not up to the task of the things we wanted to do with our data.
• 4:56
  SPEAKER_03: Um so we basically just had a very simple uh piece of software called metrics that was uh kind of listening over the network for you know API calls and things like that.
• 5:06
  SPEAKER_03: And it would you know grab that information that it saw and throw it into BigQuery.
• 5:10
  SPEAKER_03: Um it was very inflexible, uh, you know, not couldn't really backfill data.
• 5:17
  SPEAKER_03: Um that was a very manual process.
• 5:19
  SPEAKER_03: Um and so quickly, you know, as we we have this really powerful tool being Looker, and we want to do all these cool things with it, we very quickly ran into kind of the limits of what we could do with that.
• 5:28
  SPEAKER_03: Um and you know, it had served us well for about two or three years, it was written in Go.
• 5:32
  SPEAKER_03: Uh, but it we kind of realized it was time to build out something more full featured and kind of more industry standard.
• 5:39
  SPEAKER_03: And so uh that's kind of where I came in last fall, uh, about November, kind of joined the data team.
• 5:45
  SPEAKER_03: So it was you know, the business analyst and I and our manager, and started to just kind of do research into figuring out like what you know, what infrastructure is gonna kind of fix all the problems that we're having currently, and then also kind of give us the flexibility to grow into other areas we want to uh down the line.
• 6:05
  SPEAKER_03: And so we're we're a microservices architecture uh using Docker and Kubernetes for all the rest of our software.
• 6:11
  SPEAKER_03: So we figured, you know, might as well stick with that pattern and you know figure out how to get whatever our data infrastructure is going to be into that.
• 6:20
  SPEAKER_04: Cool.
• 6:20
  SPEAKER_04: Um so kind of going around along that same line, um if you could all if you could snap your fingers and have one like major feature that you want added to the project, uh, what would it be and why?
• 6:32
  SPEAKER_04: So you're talking about uh airflow, specifically like a new airflow feature?
• 6:36
  SPEAKER_03: Yeah, new airflow feature, or like some direction you want the project to go.
• 6:40
  SPEAKER_03: Yeah, that's uh that is a really fantastic question.
• 6:43
  SPEAKER_03: Um I mean I I guess you know for now, at least for our use case, uh it's doing everything we want it to.
• 6:52
  SPEAKER_03: Um can't think of anything.
• 6:56
  SPEAKER_03: I'd maybe would have had to prepare a little bit more, but no worries.
• 7:00
  SPEAKER_03: You know, you know, I do have one.
• 7:01
  SPEAKER_03: Um I think stability on uh lower resource machines, right?
• 7:07
  SPEAKER_03: Uh the biggest pain point for us, at least starting out, right, is we we don't have a lot of things running.
• 7:14
  SPEAKER_03: We don't need to have a huge pod for it.
• 7:17
  SPEAKER_03: But say there's like once a day when you know five jobs get run, right?
• 7:22
  SPEAKER_03: You know, we have we just have a few things to run daily and some stuff runs hourly.
• 7:25
  SPEAKER_03: So at that one time per day, you have a lot more jobs running than normal.
• 7:29
  SPEAKER_03: Uh, and sometimes what we would see if we didn't have enough resources is the scheduler would just kind of freeze.
• 7:36
  SPEAKER_03: It just didn't know what to do.
• 7:37
  SPEAKER_03: It would cue stuff, but it would never pick it up and run it.
• 7:40
  SPEAKER_03: Um, and it would never throw any errors.
• 7:43
  SPEAKER_03: Uh, you couldn't put a timeout on it because the job hadn't actually gotten picked up yet.
• 7:48
  SPEAKER_03: So it was just really weird silent failures like that um that we just had to deal with.
• 7:53
  SPEAKER_03: And and it took me a while to figure out, oh, it just needs more resources, right?
• 7:56
  SPEAKER_03: Because it was just pretty much purely silent failures.
• 7:58
  SPEAKER_03: I think a lot of the um kind of reliability around the scheduler uh in it, and I know it's come a long way, even from like 1.8 to 1.9, especially with backfilling, things like that.
• 8:11
  SPEAKER_03: But I think that's something I'd really like to see is just uh just maybe more reliability around that.
• 8:17
  SPEAKER_03: And you know, if you are maybe it has a better plan of how to deal with if it doesn't have enough resources, it'll you know throw some kind of error, let you know, instead of just kind of silently failing and hoping you notice that it's not doing anything.
• 8:28
  SPEAKER_04: Yeah.
• 8:29
  SPEAKER_04: Um one of our devs, Taylor, uh, he kind of has the same thing where sometimes tasks will just sit stuck in the cued state, and then like if you clear them out, they'll run again.
• 8:38
  SPEAKER_04: But like there's no clear indication as like why is it stuck in cued?
• 8:41
  SPEAKER_04: Um I guess there's like a race condition somewhere deep in the scheduler that like causes some weird behavior.
• 8:47
  SPEAKER_03: Yeah, yeah, exactly.
• 8:48
  SPEAKER_03: It's it's something that I dug deep into the issue at one point, but it was something with um, you know, the the scheduler was knowing to cue it, but somehow the workers weren't seeing it to be able to pick it up, and yeah, it was it was something weird, but um, I think really just kind of that, but you know, it seems like they're going in that direction anyway.
• 9:06
  SPEAKER_03: And and I do love kind of all the all the cloud support, you know.
• 9:10
  SPEAKER_03: It was pretty trivial to get um things like logging and Google Cloud Storage and things like that.
• 9:17
  SPEAKER_03: It was pretty trivial to get that stuff up and running.
• 9:19
  SPEAKER_03: Um so it seems like they they already have a lot of kind of those nice to have features in there, especially things like with the Slack operator.
• 9:27
  SPEAKER_03: It was like, you know, only an extra four lines of code to uh have an alert sent to Slack when something breaks, uh, which I thought was really nice.
• 9:35
  SPEAKER_04: Yeah.
• 9:35
  SPEAKER_04: Sure.
• 9:36
  SPEAKER_04: Um so kind of going along those same lines, uh, can you talk to me about some of uh like on a lower, like I guess higher level now, uh like some best dag writing practices that you guys you have internally?
• 9:46
  SPEAKER_04: Uh like you try to avoid using airflow variables or kind of like just any like subdags.
• 9:52
  SPEAKER_04: I know there's been a lot of like yes and no about that.
• 9:54
  SPEAKER_04: Yeah.
• 9:55
  SPEAKER_03: Um we don't have any subdags currently.
• 9:59
  SPEAKER_03: Um and basically the way we have it, and I think this is a pretty common setup, is um we write uh kind of a just a DAG template, and then we're basically just reading a YAML config file and then injecting values into uh into a template.
• 10:17
  SPEAKER_03: Um so basically I just have those templates completely locked down to make sure that there's nothing weird going on.
• 10:24
  SPEAKER_03: So, like I mean, like you said, um you know, we don't use any variables, don't use any sub tags.
• 10:29
  SPEAKER_03: Uh there might be a time when we do or or start looking at sensors or things like that, but um I think someone previously on your show uh said something that I totally agree with, which was that you know you can use Airflow with a certain set of features and it works flawlessly 100% of the time, or you can use that same airflow with a different set of features and it's a nightmare, right?
• 10:49
  SPEAKER_03: Uh seems like some stuff definitely causes more headaches than others.
• 10:53
  SPEAKER_03: So, you know, we use like bash operator, Python operator, Slack operator, those are the only operators we're currently using.
• 10:59
  SPEAKER_03: Um, we just kept it simple, kept it to our what we like actually need, not trying to do anything clever.
• 11:06
  SPEAKER_03: And um, that's kind of what we found to be best.
• 11:09
  SPEAKER_03: And and like things with XCOM, right?
• 11:10
  SPEAKER_03: When I decided that we needed to kind of add that in there, did a lot of testing and made sure that it wasn't going to cause any huge problems and and things like that.
• 11:18
  SPEAKER_03: Um our best practice are basically as long as we can test stuff and we're sure it's stable, we're okay with using it, but we try to avoid some stuff that gets a little more tricky.
• 11:28
  SPEAKER_04: Yeah, um, it's funny.
• 11:29
  SPEAKER_04: I think uh I read in a blog somewhere that's like exotic features are for the adventurous, yeah.
• 11:34
  SPEAKER_04: Kind of advising the same thing.
• 11:36
  SPEAKER_04: Yeah, absolutely.
• 11:37
  SPEAKER_04: Cool.
• 11:38
  SPEAKER_04: Um so uh kind of off topic, but um, just curious to know how's uh how's Caldecon AnacondaCon?
• 11:45
  SPEAKER_03: It was it was really fantastic.
• 11:46
  SPEAKER_03: Um, as someone who is definitely on the more data engineering side, it's always good to see what those crazy data scientists are up to.
• 11:54
  SPEAKER_03: Um and and it's kind of interesting, you know, even from a year ago, you know, even just a year later at this year's, the just the topics have shifted so much.
• 12:05
  SPEAKER_03: You know, a year ago, I don't think I I saw a single talk on like transfer learning and convolutional neural networks, and there were probably at least four or five talks this year about that.
• 12:15
  SPEAKER_03: Um, like that is just seeming to have exploded in the last year.
• 12:19
  SPEAKER_04: Yeah.
• 12:20
  SPEAKER_04: Um it's kind of funny, like what the data scientists are up to is gonna like determine what the data engineers need to do.
• 12:25
  SPEAKER_04: Yeah, very much so.
• 12:26
  SPEAKER_05: Yeah, very much so.
• 12:28
  SPEAKER_03: Yeah, I thought it was really interesting too.
• 12:31
  SPEAKER_03: There were there were a few talks on kind of like packaging and containerization and stuff in general.
• 12:36
  SPEAKER_03: Um, and that was that was like kind of a side talk last year, like a very small little discussion.
• 12:42
  SPEAKER_03: And this year it was huge and in multiple presentations.
• 12:45
  SPEAKER_03: Um and so I guess you know, you know, it wasn't just us, but like a lot of the industry seeing that kind of containerized data engineering and data science is is kind of the way to go.
• 12:54
  SPEAKER_03: Um, so I was kind of surprised to see that that was such a huge focus.
• 12:57
  SPEAKER_04: Yeah.
• 12:58
  SPEAKER_04: Um yeah, it's interesting, right?
• 13:01
  SPEAKER_04: Because I feel like the containerization parts happen a lot on the data engineering side before the data science side.
• 13:06
  SPEAKER_04: Yeah, absolutely.
• 13:07
  SPEAKER_04: It's almost like anticipating that need because I feel that's kind of how everything's gonna go, you know, like containerization.
• 13:12
  SPEAKER_04: One, like I'm ranting right now, but it's almost like a microservice if you kind of abstract it out enough, you know, like one thing that only does one thing.
• 13:19
  SPEAKER_03: Yep.
• 13:20
  SPEAKER_03: Yeah.
• 13:21
  SPEAKER_03: Um, so that was really great.
• 13:23
  SPEAKER_03: And there were a few tools that I saw.
• 13:26
  SPEAKER_03: Um, I'll I'll plug it here as well because it blew my mind.
• 13:29
  SPEAKER_03: But there's a uh something called Python Fire.
• 13:32
  SPEAKER_03: It's written by Google.
• 13:33
  SPEAKER_03: Uh and it basically, and this is gonna sound like a crazy oversimplification, but uh, in one line of code, you can give any script a command line interface.
• 13:43
  SPEAKER_03: Um, it's pretty impressive.
• 13:45
  SPEAKER_03: I know a lot of people use click.
• 13:46
  SPEAKER_03: I had previously used click, but uh Python Fire really does make it extremely easy to kind of give a CLI to anything you want to run.
• 13:53
  SPEAKER_03: Um so I definitely suggest looking at that if you haven't heard of it.
• 13:57
  SPEAKER_03: Python Fire, yeah.
• 13:58
  SPEAKER_03: I'll definitely look into that.
• 14:01
  SPEAKER_02: Thanks so much to Thomas for coming on.
• 14:02
  SPEAKER_02: I'd look to hear from him more in the coming weeks as we explore the setup of Airflow on Kubernetes.
• 14:07
  SPEAKER_02: Next, we have Frank Zoo from Minds.io.
• 14:10
  SPEAKER_02: We'll hear about Frank's use case and the troubles he's experienced getting Airflow set up for a three-person team.
• 14:15
  SPEAKER_02: Enjoy.
• 14:21
  SPEAKER_04: So start off the back, just tell me a little bit about Minds.io.
• 14:24
  SPEAKER_01: Uh so Minds, we are building um financial services for emerging markets.
• 14:29
  SPEAKER_01: Um starting out with um microfinance loans, we're moving into savings, they were moving into uh just building this uh uh full service uh financial platform for uh uh markets.
• 14:43
  SPEAKER_01: Um a lot of times you know um in emerging markets there's no access to credit.
• 14:48
  SPEAKER_01: Um there's no such thing as um uh like getting bank loans, right?
• 14:52
  SPEAKER_01: Because even that they require uh all this paperwork, all these red tape, and most of the people don't have access and get rejected uh for any type of credit.
• 14:59
  SPEAKER_01: Um so you know, getting apartments might take a two-year down payment or um interest rates are skyrocketing.
• 15:05
  SPEAKER_01: Um because just the uh financial you know infrastructure is just so unstable there.
• 15:09
  SPEAKER_01: So we're trying to really reconcile things.
• 15:11
  SPEAKER_01: We see there's just this uh huge opportunity, right?
• 15:14
  SPEAKER_01: Uh out in emerging markets.
• 15:16
  SPEAKER_01: Um, you know, if history will tell, you know, I think PayPal and Visa have done you know tremendous wonders here in the States.
• 15:21
  SPEAKER_01: And so we can only uh hope, you know, um we can only guess that that it'll it'll do wonders there as well.
• 15:27
  SPEAKER_01: Yeah, awesome.
• 15:28
  SPEAKER_04: Um and then can you say a little bit about your own background?
• 15:32
  SPEAKER_01: Yeah, uh I started out actually doing quantitative trading, quantitative trading in uh Citigroup.
• 15:36
  SPEAKER_01: Uh so I was doing that after a while.
• 15:38
  SPEAKER_01: Um this was um after right out of academia, I would did math in my undergrad and grad um mainly in probability and statistics, um, went straight into uh quantitative trading.
• 15:49
  SPEAKER_01: Um after a while, I really got interested in more of the data science y aspect of uh things and really exploring data and seeing what I could do with it uh beyond just trading in stocks and things like that.
• 16:00
  SPEAKER_01: Um I got actually got a call uh from a friend of a friend to join him to do a startup out in California.
• 16:07
  SPEAKER_01: Um this was Dottenbo's e-commerce uh furniture company, guessing stocks and guess, you know, furniture trends and uh e-commerce trends and things like that.
• 16:16
  SPEAKER_01: Uh so it's actually it was exciting to me because I I really wanted to see um uh what more could I do with data, right?
• 16:24
  SPEAKER_01: Um and so I jumped into the role as a data scientist, uh, eventually found out that that role really was more so data engineering 80% of the time.
• 16:32
  SPEAKER_01: Um that's where I really got my feet wet in all this um data engineering, data infrastructure uh role.
• 16:37
  SPEAKER_01: Uh from then I transitioned to become to lead the data engineering team over Augusto.
• 16:42
  SPEAKER_01: Um and there Augusto is just, you know, again, just uh refining sort of the tool set and skills, you know, finding out about uh moving from Luigi to Airflow and things like that, um, testing the waters with um with Spark and et cetera, uh all these things.
• 16:56
  SPEAKER_01: Um new technologies in here uh now I'm at Minds, um really being a more full stack engineer, but also, you know, as a team of three right now in San Francisco, uh highly contributing to also the BI data.
• 17:10
  SPEAKER_02: Very nice.
• 17:11
  SPEAKER_02: Well, we're pumped to have you on.
• 17:12
  SPEAKER_02: Just kind of going into that a little bit.
• 17:13
  SPEAKER_02: You mind telling us how you're using Airflow?
• 17:18
  SPEAKER_01: Yeah.
• 17:18
  SPEAKER_01: Um so I can talk a little bit uh previously how I've used it before and how I'm using it now.
• 17:22
  SPEAKER_01: Um previously I've gusta we've uh used it to ingest data from you know um dozens of sources, everything from you know uh Google ads to Facebook ads to you know pulling in data from all of our third party uh APIs from Salesforce and et cetera.
• 17:37
  SPEAKER_01: Uh we've done it pulling it to schedule pulls from our own internal databases, um, use it to schedule pulls and um back process from event data.
• 17:47
  SPEAKER_01: Uh we've soapplowed previously.
• 17:49
  SPEAKER_01: Um really just pulling all of that into S3, a giant fat data lake.
• 17:53
  SPEAKER_01: Um we use it to really sync up everything that's in the data lake and on you know a daily or hourly basis, uh munch that into uh a more readable format and dump it into Redshift, uh one of the data warehouses that we're using.
• 18:06
  SPEAKER_01: Um from there it even does all the scheduling from transform data to popping out CSV reports or uh things into liquor dashboards and et cetera.
• 18:14
  SPEAKER_01: I think we have just um a whole variety of things that we're doing with Airflow um schedules.
• 18:18
  SPEAKER_01: Um and every time it uh pop out Slack alerts as well, right?
• 18:22
  SPEAKER_01: So I think there's no uh limit to all the things that Airflow can do.
• 18:26
  SPEAKER_01: Like I said before, right?
• 18:27
  SPEAKER_01: It's a um plain and simple um scheduler where all the components are you can write yourself.
• 18:35
  SPEAKER_04: Yeah.
• 18:36
  SPEAKER_04: So last time we spoke, you said a little bit about how you're using Airflow to pull events, a schedule polling from the event stream.
• 18:43
  SPEAKER_04: Um can you dive into that a bit too?
• 18:47
  SPEAKER_04: Um sorry, could you remind me real quick?
• 18:49
  SPEAKER_04: Yeah, so I have this note from last time we spoke that uh one of the cool use cases you're doing is you're scheduling pulls from the event stream.
• 18:55
  SPEAKER_04: Uh is that just like your general API calls or uh using Airflow to pull from event stream.
• 19:02
  SPEAKER_01: So we have Snowplot that comes in.
• 19:04
  SPEAKER_01: Uh it pipes into uh Amazon Kinesis.
• 19:08
  SPEAKER_01: Um just one big fat Kafka.
• 19:10
  SPEAKER_01: Airflow that schedules and actually monitors um how many events, you know, whether per hour or per second, that they all match up.
• 19:17
  SPEAKER_01: Like there is something coming in.
• 19:18
  SPEAKER_01: Uh so we have airflow polling, uh Kafka polling, uh, kinesis, checking all that stuff is uh up to date.
• 19:26
  SPEAKER_01: Yeah, yeah.
• 19:27
  SPEAKER_01: So uh sound good and sound.
• 19:29
  SPEAKER_04: Um excuse me.
• 19:32
  SPEAKER_04: Yeah, go ahead.
• 19:33
  SPEAKER_04: Yeah.
• 19:33
  SPEAKER_04: Uh so you said before that uh you kind of bailed on snowplow in favor of airflow.
• 19:38
  SPEAKER_04: Um kind of how did that uh decision, how is that decision made?
• 19:44
  SPEAKER_01: Uh not so much bailed on uh snowplow.
• 19:46
  SPEAKER_01: Um I think Snowplow for us is uh really just um uh an an open source uh event tracking.
• 19:54
  SPEAKER_04: Okay.
• 19:54
  SPEAKER_01: We don't currently use Airflow or I built anything in Airflow for that.
• 19:58
  SPEAKER_01: Um Airflow really just um facilitates um everything afterwards uh to go into S3 and Data Warehouse and the rest of the infrastructure.
• 20:07
  SPEAKER_04: All righty, yeah.
• 20:08
  SPEAKER_04: Just wanted to make sure I had that right.
• 20:10
  SPEAKER_04: Awesome.
• 20:10
  SPEAKER_04: Um so then you said a little last time about how you have airflow jobs that put data in S3, run a Spark job, back to S3, and then to Redshift, um, kind of how you're handling the T of ETL.
• 20:23
  SPEAKER_04: Can you just dive into that a little bit?
• 20:25
  SPEAKER_04: Like how are you scaling that, how are you managing that, just anything along those lines?
• 20:29
  SPEAKER_01: So uh the way we've uh deployed uh Airflow in terms of uh DevOps and infrastructure standpoint is um we have our own Airflow uh Docker image, and that gets deployed on every single EC2 that we launch.
• 20:40
  SPEAKER_01: Um we scale things up with Terraform in terms of uh actual infrastructure.
• 20:44
  SPEAKER_01: Um so as needed, we can scale up different workers.
• 20:47
  SPEAKER_01: You know, um we just have this um one scheduler and all these workers all around it.
• 20:52
  SPEAKER_01: Um per DAG and subdag, right?
• 20:56
  SPEAKER_01: Uh the way we uh schedule each subdag actually is um a local executor to make sure every subdag runs on a single box um and on the rest, um it's all backed by Celery um with Redis.
• 21:09
  SPEAKER_04: Okay, cool.
• 21:10
  SPEAKER_04: Um and can you talk about how you're orchestrating those Docker containers?
• 21:15
  SPEAKER_01: Yeah, I think um in the meantime, uh or at least previously I was left off, it was uh really uh a bit manual in the sense that it's um scaling up the number of hosts uh or the number of boxes that we need.
• 21:27
  SPEAKER_01: Um but I definitely see the um uh pretty pretty uh near-term feature project to move over to Kubernetes where we don't have to manage any of that stuff.
• 21:37
  SPEAKER_01: Yeah, I could imagine that being a little bit more.
• 21:39
  SPEAKER_04: So uh we've kind of hit on this a little bit, but just kind of want you to expand more.
• 21:43
  SPEAKER_04: Um as you've kind of scaled up with Airflow and started using it for more and more parts of your business.
• 21:47
  SPEAKER_04: Kind of what challenges have you faced with how the project is when you just fork it off GitHub?
• 21:53
  SPEAKER_04: Sorry, could you read that?
• 21:54
  SPEAKER_04: You uh broke off a little bit.
• 21:55
  SPEAKER_04: Sorry about that.
• 21:56
  SPEAKER_04: Um so we kind of like dove into this a little bit before, but now just Full deep dive.
• 22:01
  SPEAKER_04: Can you talk about kind of the challenges you face with Airflow from based on like uh just forking it off of GitHub to how you're using it now?
• 22:09
  SPEAKER_04: Yeah.
• 22:10
  SPEAKER_01: Um I think um I think sometimes uh common challenges I see is uh there's definitely a lot of issues that people raise as an open source project.
• 22:21
  SPEAKER_01: Um and really I think having a different set of uh priorities on what gets fixed uh and what doesn't get fixed.
• 22:27
  SPEAKER_01: So I I think most of the time is any time any issue I'm seeing whether it's with the schedule or how things are running or um any types of customization I want to see or I I need for myself.
• 22:36
  SPEAKER_01: Um it's usually somewhere in the form of some issue uh on the project that uh I uh build up a fix for myself.
• 22:43
  SPEAKER_01: Um and then yeah, I I think that's or pretty much the main ones I think uh it's really around scheduling deployments where we've done a lot of customization, you know, a lot of on the retry logics, a lot in the making subdags work for us.
• 22:56
  SPEAKER_01: Um we've uh adopted the notion of as a call, right?
• 23:01
  SPEAKER_01: Um as uh really bringing subdags into the picture um to really you know promoting them to be more first class uh in airflow rather than you know just uh uh underneath uh the DAG.
• 23:11
  SPEAKER_01: So every single sub deck is you know self-contained, a full job that can run um you know end-to-end, whether it pulls the data, transforms the data, and loads it somewhere else.
• 23:20
  SPEAKER_01: Every subdag does that.
• 23:21
  SPEAKER_01: And they're really um we really built them up into their own little uh um uh blocks of work, right?
• 23:29
  SPEAKER_01: And so they're like sort of like skeleton copy and paste code for the entire company, whether you know whether whatever report you want to run, whether it's the annual you know, revenue report or um or uh you know ad streaming, you know, performance report and things like that, uh, all that is done you know with just you know six or seven lines of Python code.
• 23:47
  SPEAKER_01: Um that's how you know really reusable we've built up these sub decks.
• 23:51
  SPEAKER_01: Cool.
• 23:51
  SPEAKER_04: So kind of what what led you to that decision?
• 23:54
  SPEAKER_04: Was that something that you saw that doing that would kind of build a bridge for somebody who doesn't have a certain skill and give them a certain skill or capability?
• 24:04
  SPEAKER_01: Yeah, so we built it to be pretty flexible and um easy to use, especially for end users that are non-technical.
• 24:10
  SPEAKER_01: Um we um kept we we kept thinking, well, who would be the main users of Airflow, right?
• 24:15
  SPEAKER_01: Uh primarily it wouldn't uh be too much, too many engineers, right?
• 24:20
  SPEAKER_01: A lot of the product engineers would be focused on the product.
• 24:23
  SPEAKER_01: Um in the same way, they um they they probably wouldn't need to do so much analysis, right?
• 24:29
  SPEAKER_01: These these analysts and these data scientists primarily wouldn't uh really be coding uh at all, right?
• 24:34
  SPEAKER_01: I think that's the ideal.
• 24:35
  SPEAKER_01: You don't want a data scientist writing um your ETL code or stuff like that.
• 24:38
  SPEAKER_01: I think you really want to just have them uh for the most part focused on what they're good at.
• 24:42
  SPEAKER_01: And so what we wanted to do is really um empower them with Airflow to just write a couple lines of code and dump their SQL file of what they need to see, and it'll it'll be there at whatever time they schedule.
• 24:53
  SPEAKER_01: Um cool.
• 24:55
  SPEAKER_04: So uh so I have that right.
• 24:57
  SPEAKER_04: You've kind of rigged something so that you can just dump a SQL file into something and get it to translate it into a DAG.
• 25:04
  SPEAKER_04: Excuse me.
• 25:05
  SPEAKER_04: Uh so you we spoke of this a little bit last time, but uh you had you had some sort of template that um you just put a SQL file on there and it gets turned into a DAG.
• 25:16
  SPEAKER_01: Yeah, so uh the the way we've um the way we wired up sort of uh writing in Airflow or having uh everyone in the company internally contribute to Air Airflow.
• 25:26
  SPEAKER_01: Um the way they want to schedule their DAGs is just this uh really skeleton Python file.
• 25:30
  SPEAKER_01: Uh all it does is more or less just point to a SQL file because I think about you know 95% of the use case uh within the company um is to create uh reports out of SQL.
• 25:42
  SPEAKER_01: Um and so um it just points to a SQL file, the SQL file gets validated, everything gets you know run on a sandbox environment before it gets popped out to fraud.
• 25:50
  SPEAKER_01: But this way I think it helps uh both the end users, the our our analysts, and for you know the engineers maintaining everything to ensure that um one, that uh their jobs run you know correctly and properly, and it's super simple, all does run SQL.
• 26:07
  SPEAKER_01: And two, uh I think for my sanity that all dependencies actually work out.
• 26:11
  SPEAKER_01: Um so one of the cool things that we're doing is because it's all subdags, we can actually place it all over the place.
• 26:16
  SPEAKER_01: Um one of the uh another, again, going back to your previous question of things that uh I would probably like to see improved is uh sort of the UI aspect, right?
• 26:25
  SPEAKER_01: I think it's pretty easy to pop everything into uh its own DAG, but then your UI and searching for a DAG or seeing what job, et cetera, uh might blow up the UI and run out of memory on your um on your browser.
• 26:38
  SPEAKER_01: Um so it's pretty nice popping into everything into the sub DAGs.
• 26:42
  SPEAKER_01: Um there's not need to worry about where to schedule it downstream of what or upstream of what.
• 26:47
  SPEAKER_01: Um uh that's that's been pretty helpful.
• 26:49
  SPEAKER_01: Um and this way I can you know run my own uh end-to-end testing of CI running every single DAG.
• 26:55
  SPEAKER_01: Um and really uh we've automated the process of um figuring out where in the dependency tree your job needs to be.
• 27:02
  SPEAKER_01: Right?
• 27:02
  SPEAKER_01: You can just scan through the SQL file pretty easily, finding every table that it's related to, and uh put it somewhere in that tree.
• 27:13
  SPEAKER_02: Thanks so much to Frank for coming on.
• 27:15
  SPEAKER_02: We've spoken to a lot of folks using Airflow on larger teams, so it's really interesting to hear from someone working to get it stood up at a true startup.
• 27:22
  SPEAKER_02: Next in the last interview of this episode, we have Alan Krukshank, Business Insights and Data Manager at Tails.com.
• 27:28
  SPEAKER_02: You may recognize him as we had him on a few weeks ago to discuss best practices, but we enjoyed so much what he had to say about pain points that we felt it necessary to include him again here.
• 27:36
  SPEAKER_02: Hope you enjoy cool.
• 27:42
  SPEAKER_00: Uh so uh I'm Alan Kirkshank.
• 27:44
  SPEAKER_00: I look after data and analytics here at Tails.com.
• 27:47
  SPEAKER_00: We are an online pet food subscription business based in the UK.
• 27:51
  SPEAKER_00: Uh and we've been going for uh probably three or four years now.
• 27:55
  SPEAKER_00: Um I think I guess back background to the business.
• 27:58
  SPEAKER_00: Um, one of our one of our founders was a vet and realized that when you have a sick pet, a lot of the time the vet will put it on a prescription diet to help it get better.
• 28:08
  SPEAKER_00: Um, and there's lots of great research out there that good quality nutrition and the right nutrients going into a dog will help it live happier and longer.
• 28:16
  SPEAKER_00: And effectively we've taken that and we're applying it to dogs that aren't sick yet.
• 28:20
  SPEAKER_00: So the technology is there now for us to provide a custom blend of food for every pet that we supply food to.
• 28:27
  SPEAKER_00: Um and we do that, we ship the food every month direct to their door with a recipe that's unique to them.
• 28:34
  SPEAKER_04: Cool.
• 28:34
  SPEAKER_04: Um that sounds great.
• 28:36
  SPEAKER_04: And do you want to give some background on yourself?
• 28:39
  SPEAKER_00: Definitely.
• 28:39
  SPEAKER_00: So I actually I came into this in a slightly odd way.
• 28:44
  SPEAKER_00: Um, so I'm background as a as an engineer, not as a software engineer, as a as a I guess a physical engineer.
• 28:50
  SPEAKER_00: Um so trained in mechanical and production engineering and spent around three years helping factories around Europe uh either make more stuff or make the same amount of stuff with less money.
• 29:02
  SPEAKER_00: Um and I guess what that on the surface on the surface that's all about solving technical problems.
• 29:08
  SPEAKER_00: Now, what it actually is, and this is a has paralyzed all kinds of businesses, is that it's about helping groups of people get stuff done.
• 29:17
  SPEAKER_00: And sometimes what's in the way of that is a is a technical problem, and sometimes the technical problem is just a sign that there's actually something else going on.
• 29:24
  SPEAKER_00: Um, and after my time there, I was looking for something that involved a little bit less travel, something that I could really get my teeth into, and I guess something that's a bit more your own baby rather than being uh on the outside.
• 29:36
  SPEAKER_00: Uh and that's I came across the guys here at Tails.
• 29:39
  SPEAKER_00: Uh, you know, I'd worked with some of them before in various other things, um, and things lined up in the right way for me to join the team here.
• 29:47
  SPEAKER_00: And I guess it's a slightly odd parallel that now I'm I'm much more, I guess like you, Varaj, I'm I'm much more of a data engineer, data analyst kind of guy.
• 29:56
  SPEAKER_00: Um, but I think that the common aspect there is using data, using information, using technology to help solve real problems and help uh help people to do things that they couldn't do otherwise.
• 30:09
  SPEAKER_04: Um can you talk about some of like kind of the pain points you've had with the airflow as you've scaled up or started deploying more tags or anything along those lines, apart from what we just said?
• 30:18
  SPEAKER_00: I think let's think through the pain points.
• 30:22
  SPEAKER_00: Um I I guess there's there's an obvious pain point.
• 30:25
  SPEAKER_00: I think this is more of a pain point around Python than anything else of uh memory errors.
• 30:30
  SPEAKER_00: Uh if you're doing a lot of stuff in a task, it just it may just fall over, and you might not get a whole lot of warning around that.
• 30:37
  SPEAKER_00: Um and I think that's that's something partly that's one of the reasons that we that we used Dask so so extensively was so that if there was an error, we'd get a little bit more feedback from it.
• 30:47
  SPEAKER_00: Um but uh that's continually there, and I think that does that's Python, right?
• 30:54
  SPEAKER_00: Um I think the the other pain point that we'd had uh is actually around we've been we've now been running Airflow for 18 months, maybe two years.
• 31:08
  SPEAKER_00: Um and uh it's gone through a lot of changes in that that point.
• 31:13
  SPEAKER_00: Um and uh it it's become such a critical part of our infrastructure that that that means that upgrading it and keeping up with, you know, on one side you want to keep up with the changes that are being released upstream, at the same time, upgrading it and making sure that everything still works or past past the upgrade can sometimes be quite painful.
• 31:35
  SPEAKER_00: Um and I think there was the there was a relatively big upgrade.
• 31:39
  SPEAKER_00: Uh you guys will probably remember this much better than me, around six months, five, six months ago.
• 31:45
  SPEAKER_00: Um which I know it, you know, flushing out all of the changes, all of the the issues that that pushed through onto our like homegrown, homegrown uh operators.
• 31:57
  SPEAKER_00: Uh, took a good week or a good week or two, um, and it was was not the most fun thing to do.
• 32:02
  SPEAKER_00: Um, but I guess given the choice, right?
• 32:05
  SPEAKER_00: We either we we could have kept our old version of Airflow running, and I think it was because we wanted to get at some of the newer features that are out there, you know, it's worth going through the pain to keep up with the with the development.
• 32:18
  SPEAKER_00: And I think that that's only it's only been painful on a couple of the bigger upgrades.
• 32:21
  SPEAKER_00: I think the the minor version upgrades don't usually throw up anything and it'll be difficult.
• 32:26
  unknown: Yeah.
• 32:27
  SPEAKER_00: Actually, I think the other thing it might be worth talking about here on pain points is, and this might be more interesting to people who are using Airflow with Dask as well.
• 32:34
  SPEAKER_00: That's actually introduced one of our other our other pain points, is that is because effectively we use the Airflow worker as the the client for a DASK process, it's the one it is, it has a contact, it has a Dask context and it is kicking off tasks to run on a cluster.
• 32:51
  SPEAKER_00: Um, we've actually had to be really careful about version drift between our Dask cluster and our airflow cluster because we're deploying them separately.
• 32:59
  SPEAKER_00: They're two separate systems, but at the same time, it matters a lot that the versions of each of the Python libraries on either side are the same.
• 33:06
  SPEAKER_00: Um and you know, the I there have been at least four or five times over the last 12 months where an issue didn't there was some issue that came up and it looked like some really hairy issue.
• 33:20
  SPEAKER_00: I mean, for example, on one side it looked like S3 had stopped responding to us.
• 33:25
  SPEAKER_00: Um and what it turned out to be was just version drift.
• 33:28
  SPEAKER_00: Yeah, and there it was super deep down the stack between our Dask cluster and an Airflow cluster.
• 33:34
  SPEAKER_00: Um, and it meant we've got to get really pretty anal about version pinning various key libraries between the two.
• 33:42
  SPEAKER_00: Um that that's been that's been kind of tricky for sure.
• 33:46
  SPEAKER_04: Um so it's funny you mentioned the uh like upgrades, right?
• 33:49
  SPEAKER_04: So uh part of our offering is a Cloud Airflow offering.
• 33:52
  SPEAKER_04: Um and we're on one eight, and we're still trying to figure out the best way to move to one nine without uh without breaking a lot of those uh operator dependencies, right?
• 34:00
  SPEAKER_04: Because like it's just like little things where the 1.9 operators, like all the AWS stuff, has a base AWS hook, whereas before it didn't.
• 34:08
  SPEAKER_04: So like, and then the logging difference too.
• 34:11
  SPEAKER_04: So yeah, I definitely hear that with just little things that aren't breaking changes, but you want these new features and it takes a little while to move it over.
• 34:19
  SPEAKER_00: Yeah, I I think with it's I guess if this is gonna start sounding like the cult of Maxine Boschaman, but like one of the other things he one of the things he talked about in the article was around um how some of the DevOps stuff that's helped traditional software development hasn't really filtered through into uh the data engineering world yet.
• 34:37
  SPEAKER_05: Yeah.
• 34:37
  SPEAKER_00: Um and that means some of the stuff that would make this easier if we if we kept kept pace with what was going on around like comprehensive integration testing and you know, staging environments and all of this stuff there would make that a lot easier.
• 34:52
  SPEAKER_00: But because of the scale of the work that we're doing, uh you know, setting up a multi-terabyte staging environment is is A, not cheap, and B not trivial.
• 35:02
  SPEAKER_00: Um and that means that you know, in theory, yeah, sure.
• 35:07
  SPEAKER_00: I mean, you just spin up a replica of the whole stack on your laptop and just run it through some unit tests.
• 35:13
  SPEAKER_00: But when you're dealing with a distributed system where it doesn't fit on one laptop, and that's why it's a distributed system in the first place, it it makes a lot of these testing practices which would make upgrades like the one you're talking about easy, uh, it makes them harder.
• 35:27
  SPEAKER_00: And I think there's a lot of work for people like us to do over the next few years to bring some of bring data engineering back up to scratch.
• 35:36
  SPEAKER_04: Yeah, um, I think from like definitely what you said about the uh the best testing practices, we've had a lot of that too, because uh because it's just like a system agnostic task orchestrator and you're integrating with like all these different systems, like a client's S3 bucket, your own S3 bucket, someone else's S3, like you don't know kind of where the error lies.
• 35:56
  SPEAKER_04: And you're like, wait, is it with my code?
• 35:58
  SPEAKER_04: Is it with the code I'm referencing from this external system?
• 36:01
  SPEAKER_04: Like, is it something with the task isn't getting passed through through multiple nodes?
• 36:05
  SPEAKER_04: Like there's just so many layers that something can break, it just takes forever to debug anything.
• 36:11
  SPEAKER_00: I think actually one of the one of the nice features of Airflow, um, which I guess came from a different use case, is the is the connection abstraction.
• 36:21
  SPEAKER_00: So being able to define proper the connection properties to a given data source in the web UI and that be part of the backend data store, um is great from a security point of view, and I think that's where it came from.
• 36:34
  SPEAKER_00: It has the other added benefit that you can in different environments define those connections differently.
• 36:40
  SPEAKER_00: And so one of the one of the things that we've done to help testing is um have some conventions around what we call the data store names.
• 36:50
  SPEAKER_00: And on people's local dev environments, the same data store names exist that they're connected to either staging S3 buckets or they're connected to local versions of the same database.
• 37:02
  SPEAKER_00: So that you can in a in a small way we can run tests on some of this stuff, um effectively by running the same running the same DAGs that are pointing to the same connections, but those connections are set up differently to point to different places.
• 37:18
  SPEAKER_00: And that's a really neat way of doing some of this testing stuff within Airflow.
• 37:25
  SPEAKER_04: Yeah.
• 37:25
  SPEAKER_04: Um that's it, that's interesting because a lot of other people we talked to mention the connections panel as a pain point because it takes forever to like there's no automated way to like input connections into it.
• 37:36
  SPEAKER_00: Oh, I think I think it's one of the I think it's one of the best points about Airflow.
• 37:41
  SPEAKER_00: I think that the connections abstraction uh from those two perspectives.
• 37:45
  SPEAKER_00: One of having different environments where they're set up differently, and that be the not in the code.
• 37:52
  SPEAKER_00: Um, and also from a security aspect of the connection details not being in the code is also a is a good thing uh for data protection and and all that jazz.
• 38:04
  SPEAKER_02: Thanks to Alan for coming on, and thanks to you for tuning in this week.
• 38:08
  SPEAKER_02: If you haven't heard, Astronomer recently announced that we're putting our entire organizational focus on airflow.
• 38:14
  SPEAKER_02: That means that going forward, our primary goal will be to help other companies adopt airflow.
• 38:20
  SPEAKER_02: It also means that we'll be able to produce much more regular podcast episodes and other content, such as our open source library of Airflow plugins.
• 38:27
  SPEAKER_02: You can head over to github.com slash airflow dash plugins if you're interested in seeing what we've been working on.
• 38:33
  SPEAKER_02: And if you use Airflow regularly, we'd love to have you on the podcast.
• 38:36
  SPEAKER_02: Just shoot me an email at pete at astronomer.io to get the conversation going.
• 38:43
  SPEAKER_02: Thanks again for tuning in, and we'll see you soon.