Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Competitors: Luigi

Transcript

Competitors: Luigi · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: Hi there, and welcome to episode four of the Airflow podcast produced by Astronomer.
• 0:05
  SPEAKER_00: Over the past six months, we've taken a complete deep dive into the data engineering space and have heard from some extremely interesting people about their use of Apache Airflow.
• 0:14
  SPEAKER_00: We're extremely excited to keep the ball rolling in our mission to solidify airflow as the gold standard for ETL.
• 0:20
  SPEAKER_00: If you didn't catch the last episode, my name is Pete DeJoy and I'm a product specialist here at Astronomer.
• 0:25
  SPEAKER_00: I conducted these interviews with Viraj Perek, one of Astronomer's data engineers.
• 0:29
  SPEAKER_00: While you'll likely hear from me occasionally, you'll probably hear more of Viraj's voice asking interview questions, as he's the real Airflow expert.
• 0:37
  SPEAKER_00: For reference, he saw something like this.
• 0:40
  SPEAKER_02: Hi, I'm Viraj.
• 0:42
  SPEAKER_02: I like the New York Knicks, Long Walk on the Beach, and talking about data infrastructure.
• 0:48
  SPEAKER_00: This week, we have a bit of a different episode for you all, in that we strayed away from Airflow to discuss one of its main competitors in the open source workflow management space, Luigi.
• 0:58
  SPEAKER_00: Towards that objective, we interviewed Eric Bernardson, creator of Luigi and current CTO of Better Mortgage.
• 1:03
  SPEAKER_00: Eric made Luigi Wallet Spotify in an effort to improve their machine learning and music recommendation engines, and he hopped on with us to discuss the motivations behind its creations, the problems it was designed to solve, its limitations, and his current opinions on Airflow.
• 1:16
  SPEAKER_00: We had an awesome time chatting with him, and we really hope you enjoyed the interview.
• 1:25
  SPEAKER_01: Cool.
• 1:26
  SPEAKER_02: So kind of jumping right in here, do you want to talk to me a little bit about your background and uh kind of how you got to where you are now?
• 1:34
  SPEAKER_01: Sure.
• 1:35
  SPEAKER_01: Uh so I was at Spotify for almost six years.
• 1:39
  SPEAKER_01: Uh started out in Sweden, uh, where I'm from, uh, and um joined in 2008.
• 1:45
  SPEAKER_01: And um back then Spotify was I think it was like 20 engineers or something like that.
• 1:49
  SPEAKER_01: Uh so still pretty small.
• 1:51
  SPEAKER_01: And I originally joined to build a music recommendation system, uh, but sort of realized um uh that uh pretty early on that uh we had far bigger issues to solve around just like data in general and you know getting um reporting working and understanding your user's behavior and stuff like that.
• 2:11
  SPEAKER_01: So I started a team, a data team at Spotify, which was like me and two other people.
• 2:17
  SPEAKER_01: Now I think it's like 400 people, something ridiculous.
• 2:20
  SPEAKER_01: Uh so did that for a number of years at Spotify, uh, and then I moved to New York and uh started running the machine learning team at Spotify.
• 2:30
  SPEAKER_01: So I did that for another three years here in New York, uh, focusing more on the machine learning side.
• 2:36
  SPEAKER_01: And you know, we're at that point we had the data and um we wanted to to um realize it was like truly like really valuable and and you know we're at the point in the product maturity where it made sense to start investing in music recommendations, spent another three years building up a team, which ended up being about 20 people uh that did music recommendations, and then I left about three years ago to start over at a small company.
• 3:02
  SPEAKER_01: Now I'm here as a CTO at uh it's called Better, and so we're in the mortgage space.
• 3:08
  SPEAKER_01: So kind of a similar story, we're still like a little bit too focused, too much focused on like kind of core product, getting the data in the right place.
• 3:16
  SPEAKER_01: I'm still not at the point where starting to think about workflow tools and machine learning, uh, but I think we're gonna get there pretty soon.
• 3:24
  SPEAKER_02: Yeah, super interesting.
• 3:25
  SPEAKER_02: Um, it's funny you mentioned Spotify in 2008 because uh I was a big Google Reader guy back in the day.
• 3:30
  SPEAKER_02: I remember seeing a post about it and saying like you can download it, but you have to have a uh you can have a US IP address to do that.
• 3:36
  SPEAKER_02: So I like how to go through a proxy to get Spotify like really, really early on.
• 3:40
  SPEAKER_02: Um kind of yeah, so kind of just going right along that, uh, can you talk to me kind of about how the kind of the inspirations for the Luigi project and what kind of made you build it the way you did?
• 3:53
  SPEAKER_01: Yeah, so I used to run reporting at Spotify and and basically I mean I ran our entire Hadoop cluster and all kinds of stuff.
• 4:00
  SPEAKER_01: I mean, there was like three people doing everything, and you know, some days I would like we we had a f like a stack of machines in our um in one of our rooms.
• 4:09
  SPEAKER_01: Sometimes I would just like go in there and like reboot machines and all kinds of stuff.
• 4:13
  SPEAKER_01: Uh but but so I I did everything related to data, uh but one one kind of big part of that was uh reporting.
• 4:21
  SPEAKER_01: And um and reporting, we still like pretty early on we generated a ton of data, so it was a non-trivial task, and and the labels.
• 4:28
  SPEAKER_01: So when I'm saying reporting, I'm talking about royalty reports, right?
• 4:32
  SPEAKER_01: Which is it's pretty important that you get those right.
• 4:35
  SPEAKER_01: Yeah, for sure.
• 4:36
  SPEAKER_01: And and so there was like a pretty complicated machinery, you know, a lot of manual scripts that had to be run every day or every week uh or every month, uh, to you know, sync the data from all the different uh backend machines into Hadoop and then run a bunch of map reduced jobs.
• 4:53
  SPEAKER_01: Um, you know, first of all, to collect all the song and song uh data and figure out like what was the territory at that point, like what's the user premium or a free user at that point, like join with a bunch of user data and track level data, and then uh break it up by music label and stuff like that.
• 5:11
  SPEAKER_01: So, and then every music label had their own slightly different uh royalty model, and so and it wasn't just music labels, it's actually not to get into like music licensing, but it's a pretty complicated structure with like mechanical rights and whatever.
• 5:26
  SPEAKER_01: There's a lot of different parties involved, and so uh so that was like one of the biggest rationales for this.
• 5:32
  SPEAKER_01: At the same time, I was still working on my music recommendation stuff on the side, uh, and that was also kind of a similar thing, is like a lot of very complicated, it was like basically a big dependency graph of um data dependencies, like you know, once you get this data set, and then you know, once you have these three data sets, you can get this other data set, a lot of joining, a lot of splitting, and things like that.
• 5:54
  SPEAKER_01: And and and and everything was most of this stuff was powered through Hadoop, so you have to run MapReduce jobs, so most of these jobs could take you know 10 hours, 10 minutes up to like several hours.
• 6:06
  SPEAKER_01: Uh, and so yeah, so I I saw a lot of uh patterns and and started thinking about these like data dependencies, and um for various reasons.
• 6:16
  SPEAKER_01: Like previously in my life, I've done a lot of C and kind of like the idea of make files and have worked a lot with make.
• 6:24
  SPEAKER_01: Uh, although I don't love the tool itself, it's like super magic and kind of weird and has a lot of arcane things.
• 6:30
  SPEAKER_01: But um, I do like the idea of expressing data dependencies in this similar way as make does it.
• 6:37
  SPEAKER_01: Uh, and and so I started experimenting with a couple of different ways to do that and um built a few different iterations uh of this uh data dependency management thing.
• 6:48
  SPEAKER_01: Uh the first version was pretty crappy, it was some kind of XML format, enormous, and then I realized this doesn't really quite work because in a lot of cases I actually want to have a more expressive high-level language to build up data dependencies, and so I rewrote it to be more like Python-based and not based on configuration files.
• 7:08
  SPEAKER_01: Uh, and so I built a second version, I don't remember what it's called, maybe conductor or something like that.
• 7:14
  SPEAKER_01: Or builder, it was called builder.
• 7:16
  SPEAKER_01: Uh, and the builder thing was a lot better, but it was like super complicated, it had like way too many concepts.
• 7:22
  SPEAKER_01: Like you had constant the idea of a task and a target, and they had like dependencies on each other.
• 7:29
  SPEAKER_01: And then at some point I realized you don't really need all these things, and so I built leads, which was like a vast simplification.
• 7:35
  SPEAKER_01: Uh, and we open sourced it in 2012, really.
• 7:38
  SPEAKER_01: Anyway, so there was a lot of iteration that led to this.
• 7:41
  SPEAKER_01: Uh, I think it was really helpful to be in charge of reporting and music recommendations because actually, I mean, I used this for like a really use case, iterating it a number of times, kind of applied all this stuff right now.
• 7:53
  SPEAKER_02: Yeah, I was looking through the uh I found a slide share thing for your a deck you had in like 2015 or something about Luigi's presentation at Near Data Science.
• 8:02
  SPEAKER_02: I thought it was like the best explanation I've seen of a workflow scheduler that like from everything.
• 8:07
  SPEAKER_02: Like, so a little bit of background on us is we're probably we're pushing airflow pretty hard as like part of our main product.
• 8:13
  SPEAKER_02: So we're thinking of ways to how to illustrate the importance of why you should use a workflow manager rather than cron jobs.
• 8:19
  SPEAKER_02: And I think that you had like these three slides where you showed the code, the code with like some uh some tri-accepts and then how it looks in Luigi.
• 8:28
  SPEAKER_02: Yeah, I think that illustrated the concept really well.
• 8:30
  SPEAKER_01: Yeah, and and and yeah, I I actually agree with that.
• 8:33
  SPEAKER_01: I I think that was so that was the best illustration that I could come up with, like why you need this, is like kind of walking through all the the faces of what happens when you don't do it.
• 8:42
  SPEAKER_01: And I used to have like 500 scripts that did roughly all of those things uh without Luigi, which is like, you know, how do you deal with failures and like how do you deal with you know partial fails, like restarting from where you were, and you know, how you know, and and basically everything I wrote for the first year or two had the same sort of issues like making sure that data is atomically written, like either it writes the results to disk or it fails and it doesn't write results at all, right?
• 9:11
  SPEAKER_01: Like you should always make sure it doesn't it doesn't write like garbage data and you know bad data, you don't know if it's so that that was like a massive thing, and then you know, dealing with sort of resuming big workflows where some of the files were created kind of in the middle, but you know, you you you you don't you might have you know some dependency failed and now you fix it and you want to rerun.
• 9:38
  SPEAKER_01: And um and and and I think another critical thing is item potency, like making sure if you run the same thing multiple times, it doesn't sort of clutter uh or doesn't destroy all the data you have.
• 9:51
  SPEAKER_01: Like it I guess that one's kind of harder to explain, but yeah.
• 9:55
  SPEAKER_02: Well, you want to keep your integrity high level, right?
• 9:57
  SPEAKER_02: And item potency preserve integrity and some integrity in some way.
• 10:00
  SPEAKER_01: Yeah, I think that's a good way to phrase it, right?
• 10:02
  SPEAKER_01: Like making sure you can trust the data, you don't write like garbage data that you don't that you can't trust, or yeah, you can always like if something fails and you rerun it, um you should assuming you fix the issue that caused it to fail, like it you shouldn't have to go back and like clean a bunch of data, right?
• 10:18
  SPEAKER_02: Yeah, for sure.
• 10:20
  SPEAKER_02: Um you kind of hit on this a little earlier, but can you talk about how you chose Python to do this versus some other like you said, you had a lot of experience in C.
• 10:28
  SPEAKER_02: Um, did it make did it just go really nicely with the machine learning libraries you were using, or what was the motivation behind Python?
• 10:35
  SPEAKER_01: Spotify early on was very much a Python shop.
• 10:38
  SPEAKER_01: I like Python.
• 10:40
  SPEAKER_01: I mean C would have been a terrible choice, probably, right?
• 10:43
  SPEAKER_01: Like it's it's not dynamic enough to do this well.
• 10:46
  SPEAKER_01: Uh and um it's just not a productive language.
• 10:49
  SPEAKER_01: I I I I I love writing C when I have a good reason for it, but there's rarely a very good reason for using C ever.
• 10:56
  SPEAKER_01: Uh so I mean I think looking back though, I I actually sort of almost questioned the choice of Python for this in a way because so much of the the like big data ecosystem is based on G JVM now, I I think it's actually arguable if Python was the right choice.
• 11:16
  SPEAKER_01: I I think in a lot of ways building on building it in Java or something like Scala could have been better and would have integrated more seamlessly with Hadoop or Spark or um whatnot.
• 11:29
  SPEAKER_01: Um so I don't know.
• 11:31
  SPEAKER_01: I think if I ever rewrote Luigi now, uh I'd probably do it in Java actually.
• 11:37
  SPEAKER_02: Or maybe that's really interesting.
• 11:39
  SPEAKER_02: Um some of the other big we've been interviewing a lot of Airflow users, and one of the big things they say about Airflow using Python is that it ports really well with their other workflows, right?
• 11:50
  SPEAKER_02: Um with their other with the other parts of the stack they're using.
• 11:53
  SPEAKER_02: Um I think a large part of that is because not too many of them have been using Hadoop in the same way that sounds like you have.
• 11:59
  SPEAKER_02: So uh you're saying that writing in Scala would kind of fit that better like enterprise uh the enterprise big data ecosystem than what you'd find probably as a startup.
• 12:09
  SPEAKER_01: Yeah, I mean maybe that's fair to say.
• 12:11
  SPEAKER_01: Um yeah, I don't think Python, I think Python would be the second choice.
• 12:15
  SPEAKER_01: I think I'd probably rewrite it in Scala if I could do that today.
• 12:18
  SPEAKER_01: But Python is still a pretty good language.
• 12:20
  SPEAKER_01: But what we ended up having a lot at Spotify was we use Luigi to power MapReduce jobs, right?
• 12:27
  SPEAKER_01: And like that context switching was like really expensive.
• 12:30
  SPEAKER_01: We used to use Hadoop streaming to like pipe data between Python and Hadoop, and it was just like super slow.
• 12:35
  SPEAKER_01: Now I I don't even know.
• 12:36
  SPEAKER_01: I mean, I feel like just because like you know, computers are much faster anyway, like the use case for Hadoop has gone down, like you don't really or like the the value of Hadoop is like not as much right now.
• 12:49
  SPEAKER_01: I think a lot of our actual music recommendation pipeline wouldn't even need Hadoop right now, and so maybe that's like less of a big reason now.
• 12:58
  SPEAKER_02: Yeah, that's really interesting, especially over such a small time period.
• 13:02
  SPEAKER_02: Yeah.
• 13:03
  SPEAKER_02: Um so kind of just going right right around that.
• 13:04
  SPEAKER_02: Um, you talked through some of like the core features of Luigi that you think really add to its usability or just things that you're really proud of than the way it handles.
• 13:13
  SPEAKER_01: Yeah, I think I think the data dependency modeling is is um is probably the most useful part and and probably the biggest learning curve too.
• 13:25
  SPEAKER_01: Like as a user of Luigi, you kind of have to wrap your head around like thinking about data is almost like functionally dependent on other data.
• 13:33
  SPEAKER_01: Um, with the parameters and uh the tasks.
• 13:37
  SPEAKER_01: Um and but I also think so it's like it's a learning curve to figure that out, but I also think like once you're used to that way of thinking, that's a really powerful way to phrase your data workflow.
• 13:48
  SPEAKER_01: Um so I I think Luigi did a lot of things really well there with the parameters and tasks and the targets.
• 13:56
  SPEAKER_01: Um and you know, later we started thinking about how to add like dynamic dependencies and things like that, which are sort of useful in some cases.
• 14:04
  SPEAKER_01: I think the support could be a little bit better.
• 14:06
  SPEAKER_01: Yeah.
• 14:06
  SPEAKER_01: Um trying to think what what else?
• 14:09
  SPEAKER_01: I think a pretty important design consideration was that we didn't want to tie it to any particular job runner, uh, which I think also made Hadoop really useful.
• 14:19
  SPEAKER_01: Um sorry, which made Luigi really useful, is you know, Hadoop, which is like one out of many different things you can do with it.
• 14:27
  SPEAKER_01: And so, I mean, now there's like plenty of people who use Luigi for like things that have nothing to do with Hadoop, just you know, there's it's actually fairly there's some sub-community of people in the biotech uh world that just use it for like gene sequencing stuff.
• 14:40
  SPEAKER_01: Uh oh really together those those workflows.
• 14:44
  SPEAKER_01: Um and you know, supposedly works pretty well for them.
• 14:47
  SPEAKER_02: Yeah.
• 14:48
  SPEAKER_02: Um, and then on the flip side a little bit, uh can you talk about a little about where you think you fell short a little bit or scenarios where Luigi, you need to schedule a bunch of data workflows, but Luigi's core features aren't really designed for that?
• 15:00
  SPEAKER_01: Yeah, I think I think there's a number of things we didn't build out as well as we should have.
• 15:07
  SPEAKER_01: A couple of also arguable design decisions.
• 15:11
  SPEAKER_01: I think I think one of the design, one of the bad designs that we picked early on was to base everything around classes and uh inheritance or like subclassing.
• 15:24
  SPEAKER_01: And and it's sort of you know, later I've I've come to the conclusion that implementation inheritance is is generally bad.
• 15:32
  SPEAKER_01: And I also think you know, having this complicated structure where you define classes on a global level, maybe it would have been nicer to just you know think of it as like functions or something that's a little bit more lightweight.
• 15:43
  SPEAKER_01: Um, we did spend a lot of time trying to get rid of as much boilerplate as possible.
• 15:49
  SPEAKER_01: Uh, but I think some, you know, it's still like you know a few lines of overhead for every task you define in Luigi.
• 15:55
  SPEAKER_01: Uh you have to define like subclass to Luigi.task, you have to add a bunch of parameters in every task.
• 16:01
  SPEAKER_01: Uh maybe you should do these like functions instead.
• 16:03
  SPEAKER_01: Actually, in Python 3.6 or 3.7, I think now you can have type annotations for functions.
• 16:09
  SPEAKER_01: So that couldn't that could probably fit the bill a little bit better.
• 16:12
  SPEAKER_01: I think another issue with Luigi was like we never really built the central scheduler to be as useful as it could have.
• 16:20
  SPEAKER_01: Uh it doesn't even come with tech with persistence out of the box.
• 16:24
  SPEAKER_01: And I think that's probably the biggest issue was that the UI, the web interface was never as useful as it could have been.
• 16:32
  SPEAKER_01: Um, and uh I think you could have actually built a lot more into that one.
• 16:37
  SPEAKER_01: You could have built a you know task launcher into that, you could have you know built like worker pools, you could have built actually a cron interface directly in the UI, or like had like a you know centralized server that would like kick off things and let you monitor what's going on with persistence.
• 16:53
  SPEAKER_01: Uh that's probably the biggest on your investment.
• 16:56
  SPEAKER_01: Uh, where you know, if I had to go back and do this, it'd probably start there.
• 17:03
  SPEAKER_02: That's really interesting.
• 17:04
  SPEAKER_02: Yeah, yeah, that's really interesting.
• 17:05
  SPEAKER_02: Um, I want to hit on a little bit what you said.
• 17:07
  SPEAKER_02: You said that uh you wish that the web server operated separately from the scheduler.
• 17:13
  SPEAKER_02: Uh could you just add a little more?
• 17:16
  SPEAKER_01: I wouldn't say separately from the scheduler.
• 17:18
  SPEAKER_01: I mean, I guess you could separate them too, but but um what what I think is that the this this the web interface right now doesn't really do much, it just like shows a graph of what's going on.
• 17:31
  SPEAKER_01: Uh and so I think there's a lot more that it could have been.
• 17:35
  SPEAKER_01: Like it could have been actually a hub for like launching new tasks and crawling things up, and and not just something that displays what's going on, but also like triggers things and kicks it off for workers to pick up.
• 17:51
  SPEAKER_01: And um, it has a database of how things have been running in the past.
• 17:55
  SPEAKER_01: And you know, you could even like in theory, you know, analyze that data and provide ETAs for things.
• 18:01
  SPEAKER_01: Uh, there's a plenty of stuff that the scheduler could do that it's not doing, uh, where uh could be really valuable.
• 18:08
  SPEAKER_02: Yeah, we uh talked to someone a couple weeks ago that was uh he was using Airflow for his workflows, right?
• 18:14
  SPEAKER_02: And what he was doing is he's sending the events, all Airflow uh logs get sent to a Postgres.
• 18:19
  SPEAKER_02: He was sending the uh data from Postgres to Keen IO to analyze the events, to analyze how the service is running, which I thought was really interesting.
• 18:28
  SPEAKER_02: Useful.
• 18:28
  SPEAKER_01: Yes, I I think that's I never really spent a lot of time looking at Airflow, but uh I think that's like one of the biggest benefits of Airflow compared to Luigi's like they seem to have invested more in um having persistence in the scheduler and you know making the UI actually useful.
• 18:44
  SPEAKER_02: Yeah.
• 18:45
  SPEAKER_02: Um I think that you can definitely do a lot in the UI, but there's still some room for improvement there.
• 18:49
  SPEAKER_02: Like you can't you can restart things, but you can't define new things, which is almost a breaking point if you're gonna have a non-technical user versus a technical user.
• 18:57
  SPEAKER_01: Yeah, yeah.
• 18:58
  SPEAKER_01: I mean that's like another sort of that's another border to cross.
• 19:02
  SPEAKER_01: And if you want to start getting into like non-technical use, I don't know if that's what I would start with, but certainly there's like cool stuff you could do there too.
• 19:10
  SPEAKER_01: I almost built like an if this then that thing.
• 19:15
  SPEAKER_01: I don't know, maybe.
• 19:16
  unknown: Yeah.
• 19:17
  SPEAKER_02: Um so just like zooming out a little bit, uh, you've like seen a lot of how data workflows have changed from early on at Spotify to a couple years later and now at a new company.
• 19:27
  SPEAKER_02: Can you kind of talk about how just the big data landscape has changed from when you first started with it and where you are with it now, and kind of how different tools have changed along that path?
• 19:37
  SPEAKER_01: Yeah, I mean, like, you know, to be clear, like I'm not super involved in the sort of big data ecosystem anymore.
• 19:45
  SPEAKER_01: Uh but I think just like looking at how it changed when I said Spotify over those six years is like you know, it used to be super painful to deal with it.
• 19:53
  SPEAKER_01: You know, the only option it really had was Hadoop, and you know, Spotify had to build a 1200-node Hadoop cluster to deal with not like that crazy amounts of data.
• 20:02
  SPEAKER_01: I mean, it was like petabytes, but um you know it's not like super outrageous.
• 20:08
  SPEAKER_01: Uh I I guess like the biggest change is just um everything went into the cloud, right?
• 20:13
  SPEAKER_01: And I think you know, now people are using hosted services for almost everything, right?
• 20:18
  SPEAKER_01: Like, you know, whether it's AWS or Google Cloud Platform, which Autofac actually shifted to.
• 20:24
  SPEAKER_01: They have a lot of tools for this stuff.
• 20:26
  SPEAKER_01: And probably if I spent more time with Luigi now, it'd probably be to integrate with those things rather than integrating with Hadoop.
• 20:34
  SPEAKER_02: Yeah, it makes a ton of sense.
• 20:37
  SPEAKER_02: Um, so that was kind of I ran through most of my agenda there for questions I had, and you gave me a lot of very interesting stuff outside of that.
• 20:49
  SPEAKER_02: Um I'm wondering if is there anything in particular that you think you'd want to talk about as far as like workflow schedulers go that I haven't asked you yet?
• 20:56
  SPEAKER_02: It's kind of a vague question I know.
• 20:58
  SPEAKER_01: Yeah, no, I uh so there's actually I so actually when I around the same time as I left uh Spotify, I actually talked to some VCs about you know, so we're interested in like the workflow landscape.
• 21:12
  SPEAKER_01: And I briefly considered starting a startup around, you know, productionizing Luigi.
• 21:17
  SPEAKER_01: And and so so I actually thought a lot about like what would it how would it work to make money from or like you know, how would you make money from something like Luigi?
• 21:26
  SPEAKER_01: Uh and so I I think there's still like an interesting business opportunity there.
• 21:31
  SPEAKER_01: Uh first of all, it probably redesigned Luigi from scratch, like it wouldn't be like Luigi per se, uh, but it'd be something like reminds people of Luigi.
• 21:39
  SPEAKER_01: Um and so so what would that look like?
• 21:42
  SPEAKER_01: First of all, I I think there's an opportunity to build a um you know cloud based scheduler uh that you don't have to host yourself, uh something you can just spin up easily and um you know, and and basically host the scheduler on you know some send.
• 22:00
  SPEAKER_01: Centralized location and then host the workers, and then clients would use the workers.
• 22:05
  SPEAKER_01: I also think there's an opportunity to make it multi-language.
• 22:10
  SPEAKER_01: So there's actually nothing that forces the scheduler to have the same language as the workers, right?
• 22:15
  SPEAKER_01: Like Luigi workers right now are always in Python.
• 22:18
  SPEAKER_01: But in theory, like I mean this it's like a simple rest it call between the workers and the schedulers.
• 22:24
  SPEAKER_01: So you can easily have a scheduler that's written in some high performance language and then have multiple supported client libraries, like for both Python and Scala and whatever people want to use.
• 22:36
  SPEAKER_01: So I thought quite a lot about that.
• 22:38
  SPEAKER_01: I still think there's an opportunity there.
• 22:40
  SPEAKER_01: I don't know how like big the market landscape is, like I never, or like what the total addressable market is.
• 22:45
  SPEAKER_01: Um, but I think there's something you could in theory do there to make money.
• 22:49
  SPEAKER_01: I think one of the reasons I didn't end up doing this was actually I started thinking about like how you really make money from it, and I realized a lot of it probably will end up being just integrating with various systems.
• 23:01
  SPEAKER_01: Uh, I'd imagine like all the real money is in enterprise here.
• 23:05
  SPEAKER_01: Like you kind of look at Cloud Lero Horton works here, and and my understanding is most of the money they make is from like huge enterprise customers, and it almost becomes like a contracting firm at this point.
• 23:16
  SPEAKER_01: Like they go in and like work with like big enterprise customers.
• 23:19
  SPEAKER_01: And I'd imagine if you ever built a business around something like Luigi, a lot of work would be to like go into like big banks and just like figure out how to integrate Luigi with their COBOL systems, which you know doesn't strike me as like the most scalable or most fun uh product to build.
• 23:34
  SPEAKER_01: Um, but I but I think the sort of free side of it, like building that schedule that people can just use off the shelf and you know create an account and spin up the schedule or in the cloud.
• 23:43
  SPEAKER_01: I I think there's an interesting product there to be built that no one's really thought about.
• 23:47
  SPEAKER_02: Yeah, for sure.
• 23:48
  SPEAKER_02: Ironically enough, that's kind of what we're looking to do with our product.
• 23:51
  SPEAKER_02: Uh more of like a managed airflow instance, not really Luigi, but a lot of the same features that you were talking about.
• 23:57
  SPEAKER_02: Um, and very much a lot of the same little hiccups with working with older corporations having to integrate this new stuff with their old stuff.
• 24:05
  SPEAKER_01: Yeah, no, I think you're onto something.
• 24:07
  SPEAKER_01: I I think there is an opportunity there.
• 24:09
  SPEAKER_01: Again, like I don't know how big it is, but I think there's definitely money to be made.
• 24:13
  SPEAKER_01: And you know, I I can't say if it's like a million-dollar or billion-dollar opportunity, but there's definitely an opportunity there.
• 24:20
  SPEAKER_00: Herod, I'm kind of curious, how'd you guys come up with the name?
• 24:22
  SPEAKER_00: I'd love to know what that process looked like.
• 24:25
  SPEAKER_01: Cool.
• 24:26
  SPEAKER_01: Um it's like it's always like hard naming things, right?
• 24:31
  SPEAKER_01: It's like one of the hard problems in computer science.
• 24:33
  SPEAKER_01: But um we had a lot of, I remember me and Elias, who was the other author of Luigi, we uh we got together and we just wrote a lot of names on a whiteboard.
• 24:43
  SPEAKER_01: And I mean Luigi comes from from Mario, right?
• 24:47
  SPEAKER_01: Like because a couple of the themes we're looking into is like plumbing, and so we came up with a lot of like plumbing-related themes.
• 24:54
  SPEAKER_01: Uh we looked at like conducting, which I don't know, like there's a couple of themes we came up around around conducting and and like a few other themes.
• 25:03
  SPEAKER_01: And um, so Luigi was just like one of the plumbing-related ones because you know, because it's super Mario, but um uh I don't remember if there was already software called Mario, whatever.
• 25:13
  SPEAKER_01: I think we actually wasn't really much in Google search results, and we thought that that's probably a good good thing.
• 25:21
  SPEAKER_01: Um, so that's why we picked Luigi.
• 25:23
  SPEAKER_01: I I think it was like one of those things, like none of us was really happy with it, but at least it was like we were like none of us was like actively disagreeing with also doesn't hurt that the green matches the Spotify color scheme.
• 25:35
  SPEAKER_01: Yeah, that's actually a good point, and that's a good observation.
• 25:38
  SPEAKER_01: That's actually another side benefit that uh you know we like Luigi a little bit more than Mario also because it's green like Spotify.
• 25:47
  SPEAKER_00: Yeah, it makes sense.
• 25:48
  SPEAKER_00: So, yeah, so I got one more question for you before we hop off and call it a day.
• 25:51
  SPEAKER_00: Um, great hearing you talk about Luigi.
• 25:54
  SPEAKER_00: Do you have any opinions on the other workflow management systems in the space, like Airflow, Uzi, Askban, etc.?
• 26:00
  SPEAKER_01: I I don't know much about Airflow because it was launched like you know, kind of just before I left in that landscape for a little bit.
• 26:10
  SPEAKER_01: I mean, I used to be lovely old and then I left Spotify around 2015.
• 26:15
  SPEAKER_01: Umzi was around for a long time, but Uzi's just garbage, right?
• 26:19
  SPEAKER_01: Like it's just like you know, enormous XML madness, and I think so.
• 26:26
  SPEAKER_01: I don't know, it seems like everyone's like moving to Airflow now.
• 26:30
  SPEAKER_01: So and from what I heard, Airflow and Luigi are sort of similar.
• 26:33
  SPEAKER_01: Like Airflow has a little bit better UI.
• 26:38
  SPEAKER_01: Um there's like that Luigi has that Airflow doesn't have, but I think most of the time it's the other way around where like Airflow has a little bit more feature than Luigi.
• 26:48
  SPEAKER_01: So I don't know, I guess it's just evolution, right?
• 26:51
  SPEAKER_01: Like eventually it seems like people are moving more with Airflow and a little bit less Luigi, but uh I'm fine with that.
• 26:59
  SPEAKER_00: Awesome.
• 26:59
  SPEAKER_00: So that's all we got for you this week.
• 27:01
  SPEAKER_00: Thanks so much to Eric for coming on.
• 27:02
  SPEAKER_00: We'll link to his blog in the episode description.
• 27:04
  SPEAKER_00: Definitely check it out if you have a few minutes because he has some really, really awesome pieces in there.
• 27:08
  SPEAKER_00: As always, feel free to send us a message if you're interested in coming on the show to talk Airflow.
• 27:12
  SPEAKER_00: We've had a few people contact us since starting airing this podcast, and we have some awesome interviews coming up that we're really excited to release.
• 27:19
  SPEAKER_00: Aside from that, if you're interested in airflow or managed airflow or airflow support, check out our site at www.astronomer.io.
• 27:27
  SPEAKER_00: We have all the materials that you'll need to get fully up and running with Airflow, including a guided development program that we're calling Space Camp.
• 27:34
  SPEAKER_00: We hope you enjoyed this episode, and we'd hugely appreciate any feedback that you have.
• 27:38
  SPEAKER_00: Thanks so much.