Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building Scalable ML Infrastructure at Outerbounds with Savin Goyal

Transcript

Building Scalable ML Infrastructure at Outerbounds with Savin Goyal · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: One of the biggest challenges that people run into is that, hey, I may not have GPUs on my laptop.
• 0:04
  SPEAKER_03: I may not have sufficient memory on my laptop.
• 0:06
  SPEAKER_03: And you can drop additional decorators where we'll take just those functions and run it in the cloud or where your compute resources are located transparently for you.
• 0:14
  SPEAKER_03: And the experience would be as if let's say somebody just came in and attached a bunch of GPUs or extra RAM to your laptop.
• 0:20
  SPEAKER_03: But still everything is getting managed and controlled by your laptop.
• 0:25
  SPEAKER_03: So we run like a Python process that is basically doing this top sort for you and like managing everything for you.
• 0:30
  SPEAKER_03: There isn't any centralized Metaflow service or any such thing.
• 0:53
  SPEAKER_00: Join the Astronomer Roadshow for an exclusive deep dive into the latest features, hands-on demos, and expert insights, all in a city near you.
• 1:02
  SPEAKER_00: We'll be in London in May, New York City in June, Sydney in July, San Francisco in July, and Chicago in August.
• 1:10
  SPEAKER_00: Don't miss your chance to connect with the Airflow community and get the inside scoop on what's next.
• 1:15
  SPEAKER_00: Register now.
• 1:16
  SPEAKER_00: The link is in the show notes.
• 1:27
  SPEAKER_01: And today I'm joined by Savine Goyal, co-founder and CTO at Outer Bands.
• 1:33
  SPEAKER_01: And as you will see, he did a lot of things before creating his company.
• 1:38
  SPEAKER_01: So I'm very excited for this podcast episode.
• 1:40
  SPEAKER_01: How are you doing, Savine?
• 1:42
  SPEAKER_01: Pretty well.
• 1:43
  SPEAKER_01: Thanks for having me here.
• 1:45
  SPEAKER_01: Yeah, it's a pleasure.
• 1:46
  SPEAKER_01: So today we are going to discuss about machine learning.
• 1:50
  SPEAKER_01: So I'm very excited about that because honestly, I don't know that much.
• 1:54
  SPEAKER_01: So I will learn as you guys you are listening to the episode.
• 1:58
  SPEAKER_01: So we will discuss about machine learning, we will discuss about Metaflow as well, how to make your machine learning data pipelines more robust and easier to create and manage.
• 2:08
  SPEAKER_01: And also obviously, we will see how to connect Metaflow with Airflow.
• 2:13
  SPEAKER_01: And we will start by having a bit of the background of Savine because again, he has a really interesting background.
• 2:19
  SPEAKER_01: So without further ado, let's begin.
• 2:22
  SPEAKER_01: So Sabine, can you actually tell us more about yourself?
• 2:26
  SPEAKER_01: Like, what did you do before starting outer bounds?
• 2:32
  SPEAKER_03: Yeah, so I think in my career I've spent like a whole bunch of time building out infrastructure for data engineering, machine learning, and also AI.
• 2:41
  SPEAKER_03: And prior to Outer Bounds, I was at Netflix for close to six years, building out their capabilities around the infrastructure stack for, let's say, many of the things that you might enjoy watching at with Netflix, for example.
• 2:53
  SPEAKER_03: Let's say if you think about Netflix recommendations or how Netflix figures out what content it should be provisioning on its service, there's a significant amount of data engineering, machine learning, AI that sort of goes in.
• 3:05
  SPEAKER_03: And then the open question there always is that okay, what is that infrastructure stack?
• 3:08
  SPEAKER_03: So I spent like a good number of years building that out there.
• 3:12
  SPEAKER_03: Uh prior to that, I was at LinkedIn trying to solve similar problems, but more in a social network context or a professional network context.
• 3:20
  SPEAKER_01: Okay.
• 3:20
  SPEAKER_01: Any reason why you were on the infra side and not like a data engineer?
• 3:25
  SPEAKER_01: Like why did you decide to go on the infrastructure side?
• 3:28
  SPEAKER_03: Yeah, no, I think it's more of a happenstance, I would say.
• 3:32
  SPEAKER_03: So I remember my very first job, we had basically a team of data scientists and a team of software engineers.
• 3:41
  SPEAKER_03: And this company, they were basically building out ad tech click literate prediction models.
• 3:47
  SPEAKER_03: And when I started, I was quite curious to do machine learning, AI all by myself.
• 3:53
  SPEAKER_03: And I started on the data science team as it used to be known then.
• 3:58
  SPEAKER_03: And very quickly I realized that I just did not have the patience to do a whole bunch of experiments and heuristics that I needed to really, at the end of the day, gain, let's say, a few basis points of uh prediction accuracy.
• 4:12
  SPEAKER_03: Very impactful work, but it just requires a very different mindset.
• 4:16
  SPEAKER_03: And I ended up being that individual who knew a little bit of software engineering, a little bit of data science.
• 4:21
  SPEAKER_03: So I ended up becoming that go-between person that was helping out the data science team with all the engineering work, ensuring that all the models are productionized and kept up to date, all the infrastructure that would be needed.
• 4:33
  SPEAKER_03: And this is almost a decade ago.
• 4:35
  SPEAKER_03: And interestingly, the keyword ML engineer did not exist at that time.
• 4:41
  SPEAKER_03: So people had a really hard time figuring out that okay, what do they call me?
• 4:44
  SPEAKER_03: Am I a data scientist?
• 4:45
  SPEAKER_03: Am I an engineer?
• 4:46
  SPEAKER_03: So that's how I got started.
• 4:48
  SPEAKER_03: And there were very few people at that time who were actually truly focused on building infrastructure to enable data engineers, data scientists to be a lot more product.
• 4:59
  SPEAKER_03: And honestly, I very much liked that work because it also helped me to be very horizontal because at any given point in time, I could vicariously focus on many different problems, right?
• 5:10
  SPEAKER_03: Working with teams that were, let's say, doing fraud detection versus recommendation systems versus let's say you know optimizing marketing budgets or doing NLPM.
• 5:18
  SPEAKER_03: These are all very interesting problems.
• 5:20
  SPEAKER_03: I had the liberty of not really focusing myself too much on the experimentation side of the house, but still have my fingers dipped into all of these super exciting areas.
• 5:28
  SPEAKER_03: And yeah, it's been like now more than a decade, and I very much enjoy the role that I ended up choosing for myself by like sheer happenstance.
• 5:37
  SPEAKER_01: That's awesome because I share actually uh a bit of a similar story.
• 5:40
  SPEAKER_01: I started as a data scientist, and then it happened that I was better as a data engineer.
• 5:45
  SPEAKER_01: I just had this software engineering background as well, and I realized that data engineering for me was better than data scientists.
• 5:51
  SPEAKER_01: You never know exactly what happens, but at some point you just uh follow your guts and you feel that you are better in this position than the previous one that you thought would be better.
• 6:00
  SPEAKER_01: But so this is great.
• 6:02
  SPEAKER_01: I guess one question to you.
• 6:03
  SPEAKER_01: When you were at Netflix, can you tell us more a bit about like maybe the stack that you were using and also a bit of what exactly you were doing, what kind of use cases, like what were you managing in the uh on the entrance side?
• 6:16
  SPEAKER_03: Yeah, so of course, if you think about let's say a company like Netflix, they have done machine learning AI for like decades now.
• 6:23
  SPEAKER_03: Back in the day, Netflix Price was really popular, one of the very first publicly available corporate data set that many labs could like use to actually try out net new ideas in the machine learning space.
• 6:33
  SPEAKER_03: And then of course their recommendation systems have been also really famous and popular.
• 6:37
  SPEAKER_03: Now, the interesting thing is that if you were, let's say, doing ML in let's say late 2000s, early 2010s, then it's very likely that your machine learning stack was an extension of your big data stack, which in many ways was very much Hadoop or Spark oriented.
• 6:53
  SPEAKER_03: And in around like mid-2010s, the machine learning universe basically moved away from Java onto a more sort of like Pythonic ecosystem, like TensorFlow, PowerTorch, and like many other open source projects have started to gain popularity.
• 7:06
  SPEAKER_03: And at Netflix, we were at like a similar crossroads as well.
• 7:10
  SPEAKER_03: So this is circa like 2016-2017, where we had a very sophisticated recommendation system, but then Netflix's appetite to invest in ML and AI across like other parts of the business was also growing immensely.
• 7:24
  SPEAKER_03: You can imagine Netflix is spending tens of billions of dollars any given year on licensing or producing content.
• 7:30
  SPEAKER_03: So it needs to make a decision in terms of what content should be on its service, right?
• 7:35
  SPEAKER_03: There are like hundreds of millions of Netflix subscribers.
• 7:38
  SPEAKER_03: How do you basically more efficiently stream content onto the viewers' devices, whether they're televisions or cell phones or iPads?
• 7:46
  SPEAKER_03: How do you, let's say, experiment with pricing?
• 7:49
  SPEAKER_03: How do you fight fraud?
• 7:50
  SPEAKER_03: Many interesting data problems that come in.
• 7:54
  SPEAKER_03: And an interesting question is that okay, how do you optimize these?
• 7:57
  SPEAKER_03: And these are then like problems that are rooted in data.
• 8:00
  SPEAKER_03: These are problems rooted in an incredible amount of compute rooted in machine learning, statistics.
• 8:06
  SPEAKER_03: And then the question was that okay, can the stack that has served us so well or the recommendation universe carry us forward for many of these problems as well?
• 8:15
  SPEAKER_03: And the answer was unfortunately not a resounding yes.
• 8:20
  SPEAKER_03: It was a maybe or maybe a no, uh, because it just wasn't like a pythonic ecosystem that let's say a lot of net new talent that was coming into Netflix was exposed to.
• 8:30
  SPEAKER_03: So we then decided to take a fresh look at uh what that modern ML infrastructure would look like.
• 8:36
  SPEAKER_03: What are the key components, ingredients of for it, what already exists within an organization, what can we, let's say, adopt from the outside, either buy a commercial solution, adopt an open source product, and what can we build internally?
• 8:49
  SPEAKER_03: And I think that was a very pragmatic choice because many times when it comes to modernization, there's like a knee-jerk reaction that lets just replace everything that we have built so far and not really figured out what are some of the good things that can still be part of the new world.
• 9:03
  SPEAKER_03: And that's the exercise that we started to take.
• 9:07
  SPEAKER_03: And our understanding was that there are certain key components that are necessary both for your data engineering ecosystem, for your ML ecosystem, as well as for your AI ecosystem.
• 9:17
  SPEAKER_03: And the relative weight that you provide to each of those layers may vary somewhat, but at the end of the day, it is a good mental model to see the world through.
• 9:26
  SPEAKER_03: And for us, that was basically you need like a great data platform to manage, let's say, your data assets, let's say your machine learning outcomes, whether it's like models, artifacts, your embedding and whatnot.
• 9:39
  SPEAKER_03: You need a compute platform.
• 9:41
  SPEAKER_03: And now the shape of this compute platform can look very different depending on what you're doing.
• 9:45
  SPEAKER_03: So, for example, let's say if you're doing data engineering workload, then maybe having a homogeneous compute infrastructure might be great for you because you're really like optimizing for CPU cores and sort of like memory.
• 9:55
  SPEAKER_03: Then if you're doing anything related to machine learning, then heterogeneity, become a first-class citizen where you want to like actually experiment with a lot of exotic hardware to leak out maximum performance.
• 10:05
  SPEAKER_03: Again, you're like with AI, of course, we all know like how important GPUs have become.
• 10:10
  SPEAKER_03: Then there is this notion of orchestration that again becomes like really important that okay, now that you have all of these data assets in place, now you have access to, let's say, this compute ecosystem.
• 10:19
  SPEAKER_03: How do you actually author your compute?
• 10:20
  SPEAKER_03: And when I say orchestration, of course, there's the workflow orchestration universe.
• 10:23
  SPEAKER_03: But if you take a very conceptual worldview, then it's more about okay, I have a workload, I need to be able to author that workload very quickly, let's say, as a data engineer or a data scientist, be able to experiment and ideate through a whole bunch of different hypotheses.
• 10:38
  SPEAKER_03: And then the hypotheses that are uh more like leading to more confident results, how do I productionize those and make those part of my production system?
• 10:47
  SPEAKER_03: And then workflow orchestrators end up making a great sort of like progress here in this space.
• 10:52
  SPEAKER_03: And the interesting thing again is that with data engineering workloads, you have workloads of a specific nature with machine learning workloads.
• 10:59
  SPEAKER_03: The workloads are still like very similar as far as orchestration is concerned.
• 11:03
  SPEAKER_03: But then you can have workloads that may run for a really long duration, for example, it's spanning, let's say, weeks and months, or you can have workloads that are very fat in nature because you want to do hyperparameter tuning, so you want to run like multiple jobs in parallel at any given point in time and so on and so forth.
• 11:18
  SPEAKER_03: But still that basic run top sort on my dependency graph and run that reliably is very much pertinent.
• 11:26
  SPEAKER_03: And then as you start going up the value chain, up the tool stack, then there are certain ML-specific components that come in.
• 11:33
  SPEAKER_03: Um, one bigger difference between, let's say, data engineering and machine learning workloads that sort of come up, and of course, it's like also very debatable, is that ML workloads, by their very nature, are very experimental in nature.
• 11:44
  SPEAKER_03: So you have multiple versions of it and you want to be able to move from one version to the other version very quickly.
• 11:50
  SPEAKER_03: Then another concern that sort of comes up is that how do you do fault detection for your machine learning workloads?
• 11:56
  SPEAKER_03: Because many times those faults could be statistical in nature, right?
• 12:00
  SPEAKER_03: So then those things come into picture.
• 12:02
  SPEAKER_03: And then there's this question of okay, what does the business logic actually looks like?
• 12:06
  SPEAKER_03: Because of course the business logic is also very different in many ways.
• 12:10
  SPEAKER_03: Few things that are common, which are let's say on dependency hell.
• 12:13
  SPEAKER_03: Everybody is familiar with that.
• 12:15
  SPEAKER_03: That if you are rely not only on your code, but code written by somebody else, how do you experiment through that?
• 12:20
  SPEAKER_03: How do you make that reliable?
• 12:21
  SPEAKER_03: That problem becomes a little bit more non-trivial in, let's say, machine learning universe, because not only you have Pythonic dependencies, but you also have like other non-pythonic dependencies that are also as critical, like let's say your GPU drivers and like other packages that you might be relying on.
• 12:38
  SPEAKER_03: And now, if you look at, let's say, this entire thick stack, right?
• 12:42
  SPEAKER_03: Like each of these problems are really massive problems that no one single team can solve by themselves, right?
• 12:48
  SPEAKER_03: And we did have the luxury at Netflix that we had a data platform team, we had a compute platform team, we had an orchestration team.
• 12:54
  SPEAKER_03: So different names solving each of these problems.
• 12:57
  SPEAKER_03: But now, if you think about, let's say, a data scientist, an ML engineer, they are coming more from a universe of, let's say, quantitative sciences, right?
• 13:05
  SPEAKER_03: They have, let's say, Gordon School to study statistics, to study math, to study econometrics.
• 13:11
  SPEAKER_03: And this is a good amount of infrastructure that anybody has to contend with.
• 13:15
  SPEAKER_03: And now the big sort of like productivity, the thing that sort of happening productivity in your organization is that how do you actually provision the stack in the most easy and straightforward manner for your data science team, as well as how do you provide a user experience such that they don't have to worry about the nitty-gritty of, let's say, how do you spin up GPUs, how do you optimize your GPU instances, how do you move, let's say, large volume of data from your data warehouse to these compute instances, or from one compute instance to the other?
• 13:46
  SPEAKER_03: How do you make sure that people don't have to think about experimentation day in and day out all by themselves, that the things that I'm doing are like reproducible by default, so that I have a chance of, let's say, recovering from failures or building on top of my TRSS work, or do I have to now manage Docker files by hand?
• 14:03
  SPEAKER_03: These are a thousand different paper cuts that basically are the difference between a data science team being able to ship or experiment with thousands of models in a week versus, let's say, a team of data scientists who are experimenting with a handful of models in a quarter.
• 14:18
  SPEAKER_03: And you can imagine going from a few models in a quarter to thousands of models in a week.
• 14:23
  SPEAKER_03: That's a step change improvement in productivity.
• 14:26
  SPEAKER_03: And especially on the data science side of the house, what usually ends up happening is that your first model is usually a crappy model.
• 14:32
  SPEAKER_03: Your second model, second version is going to be slightly better, and then the third version is going to be even more better, and so on and so forth.
• 14:39
  SPEAKER_03: So the actual gains with ML comes from the ability to be able to experiment effectively, and that's basically what we were optimizing for.
• 14:47
  SPEAKER_03: And that was basically what my team was tasked with.
• 14:50
  SPEAKER_03: And we decided that okay, like a whole bunch of tooling that Netflix has built over the past very many years, that's an asset.
• 14:57
  SPEAKER_03: And can we basically build a layer around it that basically then provides an opinionated view around how data scientists should be thinking about, let's say, data platform, compute platform, orchestration, versioning, and so on and so forth.
• 15:10
  SPEAKER_03: And that's basically your C Metaflow project.
• 15:13
  SPEAKER_01: I I must say that I didn't realize how much like how many challenges you can have when you actually want to run the machine learning pipelines.
• 15:23
  SPEAKER_01: So before before this conversation, that's actually many challenges that maybe we don't realize.
• 15:27
  SPEAKER_01: And to your point, like obviously at Netflix you had this luxury of having a team for computing stuff, uh, a team for the orchestra orchestrating stuff, which is definitely not a luxury that many companies have.
• 15:38
  SPEAKER_01: So I guess one quick question before talking about Metaflow because you just introduced it.
• 15:43
  SPEAKER_01: What was the orchestration system that Netflix was using?
• 15:46
  SPEAKER_03: Was it Airflow or was it a built-in uh in the I think it's like very common in like large companies that you'll have like scores of orchestration systems?
• 15:57
  SPEAKER_03: So I left like like four years ago, and even back then we had at least like five major orchestration systems.
• 16:08
  SPEAKER_03: So data engineering universe was using one, then uh many of the software engineering teams were using yet another one for video encoding.
• 16:16
  SPEAKER_03: We had like yet another workflow orchestration system.
• 16:18
  SPEAKER_03: And usually what ends up happening is that depending on the problem that you are trying to solve, workflow orchestration systems, just like databases, right?
• 16:26
  SPEAKER_03: Like each optimizes for a very different outcome.
• 16:29
  SPEAKER_03: So depending on what your perspective is, what your philosophy is, you may end up choosing one or the other.
• 16:35
  SPEAKER_03: The orchestration system that Netflix uses today is called Maestro, and I think very recently they've done a blog post around the desktop.
• 16:47
  SPEAKER_02: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 16:56
  SPEAKER_02: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 17:08
  SPEAKER_02: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 17:16
  SPEAKER_01: You mentioned something very interesting.
• 17:18
  SPEAKER_01: So you don't think that we can have one orchestrator for everything?
• 17:21
  SPEAKER_01: Do you think that you should have one orchestrator for some specific?
• 17:25
  SPEAKER_03: I do feel that you can have one orchestrator for many things.
• 17:29
  SPEAKER_01: Okay.
• 17:29
  SPEAKER_03: Right?
• 17:30
  SPEAKER_03: Like maybe not all, because again, the shape of the problem is like very different, but it's also not supposed to be a cop-out that for every single small little detail that's different from let's say your current use case, spin up a brand new sort of like orchestrator out there.
• 17:47
  SPEAKER_03: And we very strongly like feel and believe in that philosophy because at the end of the day, you can also think about let's say an orchestrator as a mechanism or as an interface where multiple different teams collaborate, right?
• 17:59
  SPEAKER_03: Because if you are, let's say, in any kind of medium-sized team, then what's happening is that a lot of work or a lot of effort is spent around change management.
• 18:09
  SPEAKER_03: That let's say one team is producing something, it's a model, it's a data artifact, it's some library or something, and then another team has to consume that, right?
• 18:18
  SPEAKER_03: So now, depending on what you're doing, it's very likely that let's say your orchestrator can be that like the change management mechanism for you.
• 18:27
  SPEAKER_03: So imagine there's a team of data engineers that are writing some workload, and that workload generates some data set.
• 18:34
  SPEAKER_03: Let's say it generates data set for everything that customers have viewed, let's say, in the last 24 hours.
• 18:41
  SPEAKER_03: And then that data set is picked up by some ML team to generate, let's say, LTV predictions.
• 18:48
  SPEAKER_03: And now there has to be some signaling mechanism that data set has been generated, and now that data set is fit for consumption, and then a further dock stream process to start.
• 18:57
  SPEAKER_03: And orchestration systems can be like a good mechanism to actually provide that interface barrier between like different teams.
• 19:04
  SPEAKER_03: And now the interesting thing that at times comes in is that these two teams could be like coming from a variety of different universes as well, right?
• 19:10
  SPEAKER_03: Like it could be a data engineering team and a software engineering team, or a software engineering team and an ML team, or a data engineering team and a security team, and so on and so forth.
• 19:18
  SPEAKER_03: And that's the perspective that we ended up taking.
• 19:20
  SPEAKER_03: And also, there was also one other thing that we saw where most companies they also have an evolution around how they think about workflow orchestration.
• 19:30
  SPEAKER_03: And we also wanted to make sure that our customers, let's say, our data scientists were in many ways insulated from any of those like movements of the underlying infrastructure, right?
• 19:40
  SPEAKER_03: Because you can imagine it, and it doesn't necessarily just apply to your orchestration, right?
• 19:46
  SPEAKER_03: The world is constantly moving.
• 19:48
  SPEAKER_03: So your orchestration capabilities are moving, your compute capabilities are moving, the data warehouses, databases you're introducing, everything is moving.
• 19:55
  SPEAKER_03: And the one thing that we really wanted to provision for our data scientists was that if they're happy with the workload that they have authored, can we provide, let's say, stability as a service such that we are on the hook for providing stable APIs, stable interfaces, even though let's say behind the scenes, if let's say you're using Airflow and Airflow moves from 1.x to 2.x, then our compilation logic changes.
• 20:19
  SPEAKER_03: But then as a data scientist, you don't have to come in and be like, oh my gosh, now I need to re-implement all of my work, which can be really expensive and risky for any organization, right?
• 20:30
  SPEAKER_03: And or let's say you're using Kubernetes as your compute system, and for whatever reason your Kubernetes APIs change, or you're using some sort of like data warehouse, and now something has to change.
• 20:41
  SPEAKER_03: If we can minimize the odds that a data science team or a data engineering team has to come back and re-implement their workloads, that's a big productivity boost for us as well.
• 20:50
  SPEAKER_03: Because then you are constantly just like moving forward rather than, hey, here's a project that I worked two years ago, and now I need to come back.
• 20:56
  SPEAKER_03: And so they implement the thing and avoid the risk that something like breaks in the process.
• 21:00
  SPEAKER_03: No organization should ever be in that situation right there.
• 21:04
  SPEAKER_01: Yeah, it would be a nightmare, for sure.
• 21:06
  SPEAKER_03: Exactly.
• 21:07
  SPEAKER_01: So you mentioned that obviously now we know that there are definitely many challenges for running uh machine learning workloads.
• 21:14
  SPEAKER_01: And you also mentioned that this additional layer on top of the stack, to your point, to abstract all the complexities uh that you might have when you want to run uh machine learning workloads.
• 21:24
  SPEAKER_01: And you talked about uh Metaflow.
• 21:27
  SPEAKER_01: So can you tell us more about this?
• 21:29
  SPEAKER_01: What exactly is Metaflow?
• 21:31
  SPEAKER_01: Why should I use it?
• 21:32
  SPEAKER_01: And also what's the point between Metaflow and Airflow?
• 21:35
  SPEAKER_03: Yeah, excellent question.
• 21:37
  SPEAKER_03: So basically, Metaflow's prime focus is to solve all the problems that I just called out in all the stack.
• 21:43
  SPEAKER_03: And then, of course, like with a key focus on the fact that much of the complexity, of course, each individual layers by themselves are complex, but then the interaction between these two layers is also incredibly important, right?
• 21:53
  SPEAKER_03: If let's say you're dealing with, let's say, massive data sets, how do you make sure that data transport is optimized between your data where?
• 22:00
  SPEAKER_03: To your compute instances, or like one of the big issues that sort of ends up biting people anytime they have to build any kind of production workloads is how do they iterate their way?
• 22:09
  SPEAKER_03: What is the development story for authoring these workloads?
• 22:12
  SPEAKER_03: So those are all the things that we are basically like trying to solve.
• 22:15
  SPEAKER_03: Uh, people are curious, it's an open source project.
• 22:17
  SPEAKER_03: So if you go to github.com slash netfix slash metaflow, you can find like all the code and examples there.
• 22:23
  SPEAKER_03: Now, on the orchestration uh side of the house, specifically when we let's say think about airflow versus metaflow, the naming was like a bit unfortunate with like flow in both the places.
• 22:35
  SPEAKER_03: So a lot of people think about it more in terms of hey, should I be considering or should I even be entertaining this question of Metaflow versus Airflow?
• 22:42
  SPEAKER_03: I guess the right question always is Metaflow and Airflow.
• 22:45
  SPEAKER_03: And we have like multiple companies, I think RAMP and many others have follow a very similar sort of like pattern as well, where Airflow is a great tool where all your data engineers are operating and they're writing their data engineering pipelines to manifest these data products in many ways.
• 23:01
  SPEAKER_03: And now, if you think, let's say, take the perspective of a data scientist, what the data scientist really wants to do is that, let's say, from within their Jupyter notebook or whatever their favorite ID is, they want to write some code to be able to access these data sets, perform some compute on top of this.
• 23:17
  SPEAKER_03: And this compute in many ways can also look like a workflow in many ways, and iterate through multiple versions of it very quickly.
• 23:25
  SPEAKER_03: And then once they are happy, then they want to basically take this workflow and make this part of, let's say, a workflow orchestrator.
• 23:31
  SPEAKER_03: And when I say a workflow orchestrator, you can basically think about a workflow orchestrator in a very crude terms as I have my dependency graph of jobs, and can something very reliably, based on some signal, execute that dependency graph?
• 23:47
  SPEAKER_03: But then what is happening within each of the jobs is totally up to me.
• 23:51
  SPEAKER_03: Right.
• 23:51
  SPEAKER_03: Okay.
• 23:52
  SPEAKER_03: And now, of course, so you cannot imagine, let's say, a data scientist coming in and very willingly wanting to spend a whole lot of time, let's say, figuring out like what should my Kubernetes pod look like?
• 24:02
  SPEAKER_03: How do I pass data from let's say one step to the other?
• 24:05
  SPEAKER_03: How do I think about, let's say, getting the right version of GPU drivers?
• 24:09
  SPEAKER_03: How do I do things in the most like cost-effective way while having like access to all sorts of visualization charts, data science-related metrics that I need?
• 24:17
  SPEAKER_03: But once you are able to do everything, then one question that sort of is always open is that hey, can we somehow run this entire new workload that we have generated simply on top of Airflow?
• 24:28
  SPEAKER_03: Because it already exists.
• 24:30
  SPEAKER_03: There's a team at an organization that is on the hook for maintaining and managing the uptime of that product.
• 24:37
  SPEAKER_03: And if you can just like piggyback on top of it, it's more leverage that you're like gaining.
• 24:41
  SPEAKER_03: And that's basically one of the features that Sort of Metaflow provides, where you can author your workload.
• 24:46
  SPEAKER_03: Once you're happy with your workload, then you can compile that workload into an Airflow DAG, submit it to Airflow, and now this workload will run as a bona fide Airflow DAG, all the features that everything that you have access to in Airflow, you have access to that.
• 25:01
  SPEAKER_03: But then at the same time, you have access to all the Metaflow functionality that is all geared toward running high-scale ML AI workloads in like the most simplistic manner as possible.
• 25:10
  SPEAKER_03: So that's basically the primary mechanism with which people like actually end up using Metaflow in open source.
• 25:17
  SPEAKER_01: Okay, so I guess like just to better understand how it works.
• 25:19
  SPEAKER_01: So I can use Metaflow to create my AML workflow.
• 25:23
  SPEAKER_01: Am I gonna create the tasks directly in Metaflow and then I will compile those tasks into a Nairflow DAG or like how does it work exactly?
• 25:34
  SPEAKER_03: So basically, as of today, you can just do pip install Metaflow.
• 25:37
  SPEAKER_03: Okay, not deploy any infrastructure related to Metaflow.
• 25:41
  SPEAKER_03: Of course, if you do that, then you get like additional functionalities as well.
• 25:44
  SPEAKER_03: And then you can start writing pure Python.
• 25:47
  SPEAKER_03: And you can have functions.
• 25:49
  SPEAKER_03: And then if you let's say annotate your function with a decorator called step, then it becomes a node in your graph.
• 25:55
  SPEAKER_03: And then we have this self-dotnext functionality that defines edges in your graph.
• 26:00
  SPEAKER_03: And then Metaflow will basically you can take this workload and you can run this locally without any sort of like external dependencies or anything, right?
• 26:09
  SPEAKER_03: And Metaflow will basically provide you all the bells and whistles around versioning, dependency management, compute management, everything that you would like need from a data science ML perspective.
• 26:18
  SPEAKER_03: Now, what happens is that as you're running, this workload is running locally on your laptop.
• 26:23
  SPEAKER_03: And one of the biggest challenges that people run into is that hey, I may not have GPUs on my laptop, I may not have sufficient memory on my laptop.
• 26:29
  SPEAKER_03: And you can drop additional decorators where we'll take just those functions and run it in the cloud or where your compute resources are located transparently for you.
• 26:37
  SPEAKER_03: And the experience would be as if let's say somebody just came in and attached a bunch of GPUs or extra RAM to your laptop.
• 26:43
  SPEAKER_03: But still everything is getting managed and controlled by your laptop.
• 26:48
  SPEAKER_03: So we run like a Python process that is basically doing this top sort for you and like managing everything for you.
• 26:53
  SPEAKER_03: There isn't any centralized Metaflow service or any such thing.
• 26:57
  SPEAKER_03: And what that means is that if you shut your laptop, then your workload will get killed, right?
• 27:02
  SPEAKER_03: Okay.
• 27:02
  SPEAKER_03: And then the big question that comes up is that okay, now if I need durable execution, ROS get there.
• 27:08
  SPEAKER_03: And one idea or one option that we could have had is that okay, we could have gone out and implemented our own workflow orchestrator.
• 27:15
  SPEAKER_03: But then the reality is that most companies already have a workflow orchestrator, even before the first data scientist is hired, because we have data engineers there already.
• 27:24
  SPEAKER_03: And the goal was that okay, given that there's already a workflow orchestrator, rather than reinventing the world, we can just compile everything down into a language that the workflow orchestrator can understand.
• 27:33
  SPEAKER_03: Right.
• 27:34
  SPEAKER_03: In this case, we basically, then if you decide that, okay, like my workload is working as I expect, now I need to run this on an external server.
• 27:42
  SPEAKER_03: You can then, with just one single command, you can generate an Airflow DAG out of it and you can deploy it to your Airflow server.
• 27:48
  SPEAKER_03: And now it's a bona fide airflow DAG.
• 27:50
  SPEAKER_03: You get all the benefits of Airflow, but then you also get all the benefits of Metaflow at the same time.
• 27:56
  SPEAKER_01: That's awesome.
• 27:57
  SPEAKER_01: And I guess because I'm not gonna lie, I also checked the documentation.
• 28:01
  SPEAKER_01: Uh, there is a Metaflow UI.
• 28:03
  SPEAKER_01: Am I gonna get access to the Metaflow UI as well?
• 28:05
  SPEAKER_01: Like exactly.
• 28:07
  SPEAKER_01: I use the Airflow UI and the Metaflow UI, they serve different purposes.
• 28:11
  SPEAKER_01: Like, how do they work together?
• 28:12
  SPEAKER_03: Exactly.
• 28:13
  SPEAKER_03: So you can imagine the Airflow UI will capture all your workloads that are deployed to Airflow.
• 28:18
  SPEAKER_03: The Metaflow UI is basically capturing all the workloads that are running through Metaflow, whether you're running it locally or through Airflow.
• 28:24
  SPEAKER_03: Now, Airflow is good at, let's say, providing you visibility into the pipeline and all the operational monitoring and everything.
• 28:31
  SPEAKER_03: But then on the open source Metaflow UI will basically then help you understand in depth what is happening inside the workload through taking like more of an ML-first approach versus the Airflow UI will take like more of a pipeline first approach.
• 28:45
  SPEAKER_03: So that's the differentiation.
• 28:46
  SPEAKER_03: So the Metaflow UI is not supposed to serve as a replacement for the Airflow UI, but it's more okay.
• 28:52
  SPEAKER_03: You get like your top-level pipeline view.
• 28:55
  SPEAKER_03: But now if you want to go deeper and get access to, let's say, more Metaflow bits, then you get access to the Metaflow UI.
• 29:02
  SPEAKER_03: And the way we stretch up the Metaflow UI is that you can also move between these two UIs.
• 29:06
  SPEAKER_03: So within the Metaflow UI, you'll have all the links to your Airflow UI.
• 29:10
  SPEAKER_03: And then within the Airflow UI will attach all the metadata so that you can get to your Metaflow UI as well.
• 29:15
  SPEAKER_03: Because many times you can imagine maybe you've set up Page-A-Duty alerts on your Airflow UI.
• 29:20
  SPEAKER_03: So if a pipeline failed, it generates a page for you.
• 29:23
  SPEAKER_03: So you get to the Airflow UI from the Airflow UI.
• 29:25
  SPEAKER_03: Now you want to get a deeper view into what's happening in the Metaflow land.
• 29:29
  SPEAKER_03: So you can get into the Metaflow UI as well if you want, as well as like vice versa.
• 29:33
  SPEAKER_03: And then the other thing that we also look for here is that you can imagine failures can happen quite often.
• 29:40
  SPEAKER_03: Many times these failures are statistical in nature.
• 29:43
  SPEAKER_03: And these failures can oftentimes result in cascading failures too, because your workload fails, your pipeline fails, and then other downstream teams are blocked.
• 29:51
  SPEAKER_03: So how quickly you're able to recuperate is really important.
• 29:54
  SPEAKER_03: And within Metaflow, what we end up doing is that we capture your entire state.
• 29:58
  SPEAKER_03: We capture all the intermediate variables, your third-party dependencies, everything, so that you get like a very strong guarantee around reproducibility, where with one single command, you're able to replay that compute step by step so that you can actually try to reproduce that exact same failures.
• 30:14
  SPEAKER_03: Because one of the big struggles that we saw happen quite often with machine learning workloads was that a failure will happen, that a failure has happened.
• 30:20
  SPEAKER_03: But replicating that failure is going to take so much time and it's going to be so painful that many times figures give up and they're like, okay, let's figure out some other sort of like strategy.
• 30:31
  SPEAKER_03: And during that entire duration, all the downstream teams are still waiting.
• 30:36
  SPEAKER_03: So for us, the goal was that okay, how can we also reduce the time to recovery?
• 30:39
  SPEAKER_03: And to reduce the time to recovery, the time to introducing the bug is really extremely important.
• 30:44
  SPEAKER_03: And that's the mechanism with which sort of Flatflow is able to capture everything that is happening within the workload becomes really useful for us.
• 30:53
  SPEAKER_01: The more we are talking about Metaflow, the more I want to try it.
• 30:56
  SPEAKER_01: Honestly, this conversation is super interesting.
• 30:58
  SPEAKER_01: I definitely want to get my hands on it.
• 31:00
  SPEAKER_01: I guess actually, speaking of getting my hands on it, obviously it is open source, so anybody can can install and contribute uh to Metaflow, which is great.
• 31:08
  SPEAKER_01: But let's say I just want to try it on the cloud.
• 31:11
  SPEAKER_01: Can I do that?
• 31:12
  SPEAKER_01: Is there any way to do it?
• 31:13
  SPEAKER_03: Okay.
• 31:14
  SPEAKER_03: Yes, I mean, of course, you can imagine Metaflow is a cloud native solution.
• 31:17
  SPEAKER_03: And many times when people are, let's say, placing a bet on, let's say, a workflow orchestrator like Airflow or a data science solution like Metaflow, then you do need to build significant confidence in the solution before going forward with it.
• 31:32
  SPEAKER_03: So we offer Metaflow sandboxes.
• 31:34
  SPEAKER_03: So if you go to our open source website, metaflow.org, you'll find a link to sandboxes, which will basically provide you a pre-provisioned environment with compute and an S3 bucket and everything.
• 31:44
  SPEAKER_03: So if you can actually try out our entire open source examples and bring in your own examples to basically figure out is this something that really truly works for you, works at the scale that you're looking for.
• 31:56
  SPEAKER_03: And if it does, then we do have all the deployment templates available in open source as well, then you can like use to deploy Metaflow internally in your organization.
• 32:06
  SPEAKER_01: Okay.
• 32:06
  SPEAKER_01: And I guess so you've created your company out of bounds.
• 32:09
  SPEAKER_01: I guess it is using Metaflow, right?
• 32:12
  SPEAKER_03: Yes, yes.
• 32:12
  SPEAKER_03: So we basically provide a managed service around Metaflow because you can imagine here, like one of the big advantages that we had at Netflix was that we were standing on the shoulders of the giants.
• 32:21
  SPEAKER_03: So a compute platform, there's a data platform and many other things, right?
• 32:25
  SPEAKER_03: And as it so happens, you know, many companies may not, should not invest all by themselves in these solutions because this at the end of the day is undifferentiated heavy lifting.
• 32:35
  SPEAKER_03: Just because you build a great internal compute platform doesn't necessarily mean that the business that the company is involved in, that sort of like the odds of succeeding in that sort of like also goes up.
• 32:45
  SPEAKER_03: An example is Netflix doesn't run its own data centers, trusts AWS to do the right thing.
• 32:50
  SPEAKER_03: And many other companies also make the similar decision that hey, we need ML capabilities, and we want to adopt a product uh that basically provides these ML capabilities on top of our existing stack, as well as filling in the existing gaps that we may have, if any, and that's where Outablounce comes in.
• 33:07
  SPEAKER_01: Okay, I'm definitely gonna try that out.
• 33:09
  SPEAKER_01: Okay, I guess like one last question, which is a bit different, but I would love to have your uh perspective on that.
• 33:18
  SPEAKER_01: What would you say to uh someone that wants to become a data engineer or a ML engineer maybe with this world of AI?
• 33:27
  SPEAKER_01: What would be your recommendations?
• 33:28
  SPEAKER_01: If I want to become an ML engineer or a data engineer, and when I'm reading everything online, AI is gonna replace my job and so on.
• 33:38
  SPEAKER_01: Like, what would be your recommendations?
• 33:40
  SPEAKER_01: What would you say to someone that wants to be in my career?
• 33:43
  SPEAKER_03: The most successful data scientists, ML engineers, AI researchers, data engineers weren't the ones who were exceptionally skilled at technology.
• 33:55
  SPEAKER_03: They weren't the most successful ones were the ones who really internalized what it took to make the business win.
• 34:04
  SPEAKER_03: Right?
• 34:04
  SPEAKER_03: What is the business problem that they are tasked to solve?
• 34:07
  SPEAKER_03: I've seen plenty of scenarios where people would get caught up on, let's say, the latest real-time streaming technology and whatnot, only to realize that let's say the dashboard or let's say the data that's needed, it gets looked at it very important, dashboard, very important data sets, but it gets looked at let's say once a week or something.
• 34:27
  SPEAKER_03: And it's like the accuracy and being able to figure out how to get to the outcome quite as fast as possible.
• 34:33
  SPEAKER_03: That's the most important thing that sort of makes a difference.
• 34:36
  SPEAKER_03: And trust me, AI is nowhere close to doing any of that.
• 34:40
  SPEAKER_03: So I would say that the key to success is really understanding what is the business problem at hand, really internalizing what is really needed, because most organizations they do not do well with internal communications.
• 34:54
  SPEAKER_03: So you have to really truly understand what is it that's really needed, dig much, much deeper, and that's where like the real solutions lie.
• 35:04
  SPEAKER_03: And no AI is going to be capable of doing that.
• 35:07
  SPEAKER_03: If let's say we get to that universe, we'll we'll be happy in many other regards as well.
• 35:12
  SPEAKER_03: So I'm not too worried about AI coming in.
• 35:16
  SPEAKER_03: Of course, it has definitely, if you think about AI as a very smart autocomplete, it has definitely made everybody's lives easier.
• 35:24
  SPEAKER_03: Uh, but then uh it also provides you a lot more time and energy to focus on what are the true expectations of the organization for me.
• 35:34
  SPEAKER_03: And that's basically what is going to really uh take you high out.
• 35:38
  SPEAKER_01: I love that.
• 35:39
  SPEAKER_01: I love that.
• 35:40
  SPEAKER_01: Thank you so much.
• 35:41
  SPEAKER_01: Super, super interesting.
• 35:42
  SPEAKER_01: Okay, that's it.
• 35:43
  SPEAKER_01: That's it for today.
• 35:44
  SPEAKER_01: Thank you so much for joining.
• 35:46
  SPEAKER_01: I guess one last question what is the best way for folks to get in touch with you if they want to learn more about Metaflow or your company, or like what is the best way?
• 35:55
  SPEAKER_03: Yeah, please uh feel free to connect with me on LinkedIn.
• 35:58
  SPEAKER_03: That's always the easiest way to reach out.
• 36:01
  SPEAKER_03: And we have an open source Slack community as well, chat.metaflow.org.
• 36:06
  SPEAKER_03: And if you're interested in talking about data orchestration and other things, happy to have a chat.
• 36:11
  SPEAKER_03: And of course, I'm also available on the Airflow Slack.
• 36:13
  SPEAKER_03: If you're already on the Airflow Slack yet, just ping me and happy to engage there as well.
• 36:18
  SPEAKER_01: Awesome.
• 36:19
  SPEAKER_01: Thank you so much.
• 36:20
  SPEAKER_01: Have a wonderful day and take care.
• 36:21
  SPEAKER_01: See you for another episode.
• 36:23
  SPEAKER_01: Yeah, thank you.
• 36:24
  SPEAKER_01: Thanks for having me.
• 36:27
  SPEAKER_02: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 36:34
  SPEAKER_02: You'll find links in the show notes to any resources mentioned in today's show.
• 36:37
  SPEAKER_02: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.