Latest / Agentic Conversations (formally mlops.community) / Human-centric ML Infrastructure: A Netflix Original // Savin Goyal // MLOps Meetup #44

Transcript

Human-centric ML Infrastructure: A Netflix Original // Savin Goyal // MLOps Meetup #44 · Agentic Conversations (formally mlops.community)

0:16
It looks like we've got some some people in here.
0:19
I think we've got a critical mass and you all know hopefully
0:24
that today I am talking to seven glial.
0:28
He is one of the main takeaways Tanner's of Netflix's meta flow
0:35
and the first thing that I'm going to ask him that mean when
0:41
is stranger things for coming out?
0:45
That's what I want to know. Well, yeah, you mean even I'm
0:51
curious to know and it's a difference.
0:52
Even for coming out. I knew it.
0:55
As soon as I get to, I had a feeling you were going to play
0:59
that off smooth. So that is not what we came here
1:03
to talk about, is it, we came here to dive into meta flow.
1:07
This is a to be continued version 2 of this talk.
1:13
If anyone was around on version one, we had Ravi your coworker,
1:17
come on here, and he gave us the breakdown of what meta flow is.
1:22
But I would love to know before we start with any of that and
1:26
get into meta flow, you've had a pretty prolific career already.
1:30
I can tell you're young still so you've been able to accomplish a
1:33
lot up until now? Can you just give us a bit of
1:36
background on how you got to be where you are right now, doing
1:41
this with Netflix and working on meta flow.
1:46
Yeah, great question. I think I was just incredibly
1:50
lucky to be at the right point at the right time and I think
1:57
when I was graduating it was definitely a desire to work in
2:02
Data Systems. So early on, in my career, I
2:05
worked on Big Data Systems and then as the community, won't
2:10
order Bank adopting, machine learning more and more.
2:12
It was only a natural segue and that's how I found myself
2:16
working on machine learning systems and Netflix, you know,
2:21
it happens to be a great place to do.
2:23
Engineering the company runs, very deep In and still has
2:29
Global impact. So yeah, it's nothing.
2:32
Nothing specific in the sense that is, you know, like I don't
2:35
really have a specific action item or a plan that I can share,
2:41
but I guess it was more about so many things that happen to me
2:46
very naturally where you are chemically.
2:49
Yeah. Awesome.
2:51
So for those who missed, The first installment of this chat,
2:56
can you just give us a break down real quick of what meta
3:00
flow is and how it came about sure.
3:04
So Meta fluid is Netflix's machine learning framework and
3:10
so it's used internally for on spots of use cases that we have
3:15
ranging from, you know, analyzing our contents late
3:20
figuring out, you know, how to fight fraud on ourselves better.
3:23
You're building it, legit infrastructure and so forth.
3:26
So it was built to cater to a diverse set of use cases, that
3:31
we found ourselves getting more and more into and also the the
3:38
data scientist that we have internally at Netflix employed
3:41
on our rolls, essentially experts in machine learning,
3:46
experts and statistics, and other quantitative fields, and
3:51
they essentially desired, you have Machine learning framework
3:55
that would essentially allow them to exercise their freedom
3:58
of choice and tooling what sort of algorithms they want to use,
4:02
what sort of other monitoring tooling.
4:04
They want to use how they want to actually go about building.
4:07
Their machine learning models without sort of like the tool,
4:11
the framework by itself imposing any sort of constraints and that
4:15
was kind of like the unique need that we had when we set about
4:18
building metal floor and most of the design decisions that you
4:22
see around, Many of you are mostly around fostering that
4:27
sense of Freedom that yes, you know, I'm not tied to a specific
4:30
framework or I'm not tied to doing machine learning or data
4:33
science in a particular way. And but then at the same time
4:38
for any machine learning problem, there are certain
4:41
concerns that cut across all the problems, right?
4:44
Like you need to make sure that you have good lineage of all the
4:48
work that you are doing experiment, tracking, you need
4:52
to make sure that you are able to sort of like, Train and
4:54
deploy your models at scale. And the definition of that scale
4:58
is very maybe you are processing.
4:59
A lot of data mean, you're not really processing.
5:02
A lot of data, but you're training a lot of models, or
5:05
maybe your model is pretty heavy and huge.
5:10
And we wanted to make sure that aren't willing to take care of
5:12
all of those concerns. And the user, they wouldn't be
5:16
any sort of like necessary. Expectation on their behalf to
5:22
sunlight dive deep into systems engineering and figure out like,
5:25
okay, how do I launch containers?
5:27
How do we think about reproducibility, how do I sort
5:30
of light move terabytes, and petabytes of data across
5:35
different systems? So we just like, looked at all
5:38
the different small little pain points that are data scientists
5:41
were facing. And we decided to okay, you
5:43
know, like let's just like start building tools that can take
5:48
care of each of those pain points.
5:49
One at a time. And if you do that for a
5:52
sufficiently long duration, do you end up with something that
5:55
looks like a free movie? That's, that's how we could.
5:59
That's awesome. So in the stack where does meta
6:04
flow stand like, what do you feel like our other tools that
6:09
are similar to what you're doing, or what?
6:13
And we're like, I'm just trying to get an idea of the workflow
6:19
and I know that Netflix has quite a few tools that the data
6:24
scientists are using. So can you break down?
6:28
What that looks like and how where it goes?
6:31
Sure. Yeah.
6:32
So now when you look at the stack of machine learning, right
6:37
there are different components involved and those components,
6:39
they started like stay true for any sort of machine learning
6:41
problem. Like the very first that is that
6:45
of a data platform. So you need access to data and
6:49
for Netflix, our data exist in a variety of different formats.
6:53
So S3 is a data link. So most of the metadata most of
6:58
the data sets are present in a string and then at the same time
7:04
we are a video streaming company.
7:06
So we have like petabytes of video data the actual raw files
7:11
and given, you know, the increasing interest in computer
7:14
vision. And so the benefits that it can
7:17
have for us. We also do a lot of Sitting on
7:20
those file as well and those files live.
7:21
One is 39 are exposed using a variety of different mechanisms
7:25
to an internal users. And now once you start like go
7:30
beyond the data stack then next time that comes in is your
7:34
compute lie. Now, you need, you need to
7:36
process this data somehow, right?
7:39
And again, you know, there's, there's a spectrum of compute
7:42
layer for us. Internally the most simple one,
7:45
is that of a laptop, like if we can just do your work on a
7:49
laptop. Up top if everything fits on
7:50
your laptop that just of chords a very high degree of
7:55
flexibility. And a lot of problems that we
7:57
solve internally things can just put it on a laptop, and we want
8:00
to make sure that we are providing some sort of support
8:02
for that, then you have Cloud workstations.
8:06
So, say an easy to instance, or Docker container running on one
8:11
of these ec2 instances, where people can log in and they can
8:13
do their work. And usually, these will provide
8:16
them are not many more resources than what Jennifer Lopez laptop
8:21
or Mac can provide you and then at the same time, there are
8:26
certain. Bespoke means that people have
8:28
that hair out. I need to run some conclude that
8:31
might need say a terabyte of RAM and that point in time, you can
8:35
very easily found out that compute to ec2 instances which
8:39
provide very exotic combinations of these sources these days.
8:44
And those combinations are only growing day by day and for a lot
8:49
of fun. Organizations.
8:50
This container orchestration system essentially looks like
8:54
some Sorry. It essentially looks like some
8:59
sort of like maintainer orchestration system
9:01
predominantly say, kubernetes is like database batch and
9:05
internally at Netflix. Our container orchestration
9:09
system start three days. So it's it's an open source
9:14
project, it predates, kubernetes and provides like very similar
9:18
to guarantees as well. Didn't know if you go a level
9:24
Beyond this right now, these compute instances, they are
9:27
running containers. So then you need to have some
9:30
infrastructure to either build and bake, these Docker
9:35
containers and deploy to these instances.
9:39
And that's again in like some sort of infrastructure that
9:41
needs to exist. We have some homegrown see, ICD
9:45
around Docker images. There's a lot of tooling that
9:47
excessive side in the world as well.
9:49
And then Then also you need some sort of orchestration
9:53
infrastructure as well. So, you know, these machine
9:57
learning models like many times, people want to run these that
10:00
particular Cadence. So at Netflix, you know, we are
10:03
logging a lot of data. So anytime we have, say, a fresh
10:07
data set, you might want to read trigger model training or maybe,
10:10
you know, we might want to re trigger model training, every
10:13
week, every day, every few hours.
10:15
So you need to have some sort of workflow scheduler that can take
10:17
care of that. Airflow is an open source,
10:19
alternative. The clouds.
10:22
They provide some tools as well internally as it happens with
10:27
many other big companies. We have like now tens of
10:32
internal workflow orchestrators that will have built for
10:34
specific. Use cases to work with like most
10:36
of those. And then if you didn't start to
10:41
sort of like go beyond this, then you sort of like get into
10:43
the realm of, in the knobs, which is around like a king.
10:47
Now all the layers of Stack that I spoke about they don't
10:51
necessarily have anything to do with machine learning.
10:53
But now if you're say running on workload which is a machine
10:57
learning workload, then let me certain extra things that you
10:59
need to do, right? I repeatability becomes a big
11:03
concern that whatever I have whatever works when I'm
11:07
executing and any point of time in the future, I should be
11:10
guaranteed that I can execute the same workload and notionally
11:15
get similar results, right? Like that would be like
11:17
something and then given that. So very Iterative process as
11:21
well. Like billing machine models, you
11:23
need like experimentation as a first class citizen.
11:27
Also performance becomes really really important for a lot of
11:32
people like performance usually means like machine performance
11:36
but now as it happens machines are actually rather important
11:40
data scientist. So you have to focus on equal
11:43
measures on how productive your data scientists are be if let's
11:48
invalid as opposed to spending More than half a day just like
11:52
building baking a Docker image. Like so like fight dependencies,
11:56
that's lost productivity for business.
11:59
So that's that's the stack and I'm pretty sure you like for a
12:02
lot of people who are attending this call in their organizations
12:05
as well. The stack look very much similar
12:07
in some sense. But now what sort of like tricky
12:11
is, how do you select move between these layers of the
12:14
stack, right? So now let's say, if I need to
12:19
train again. So Model.
12:21
It's the I need to train hundreds of principle models in
12:23
parallel. How do I go about launching
12:26
those hundred instances in parallel?
12:29
Making that Docker image moving my data from my data warehouse
12:32
to those doctors instances training that model getting that
12:36
result back still maintaining some sort of expecting the edge
12:40
and it's been tracking and that becomes harder.
12:43
There are of course best practices and given that members
12:49
of my team and myself. If we have worked in this area
12:51
for the years and years, we have those practices that we saw like
12:54
come across but then ideally to provide better leverage, it's
12:59
nice. If we can be all of those best
13:01
practices into a software system and that's what at the end of
13:05
the demon of Lewis. So I just inserted.
13:09
Absolutely. Yeah, so much, I mean and so
13:13
what I kind of got from that you went very vertically on the
13:17
stack and just a lot of people in here, probably already knew
13:21
that Netflix recreated its own kubernetes, but that's news to
13:24
me and that's pretty amazing. It doesn't surprise me but it's
13:28
a it's definitely interesting to hear about.
13:31
I I have heard about Runway, that's another project.
13:36
Can you break down? Like where Meta flow starts and
13:39
Runway and what those two like why they're separate and what
13:44
they do. Sure.
13:45
Yeah. So so bailiff little historical
13:48
context here. So Netflix has had like really
13:53
big investments in the recommendations infrastructure
13:55
space and if you look at recommendation systems, you
14:00
know, like 10 years ago, 15 years ago, most people who are
14:04
doing machine learning. They were more or less in the
14:06
gvm scholars. Ack.
14:09
So our recommendations infrastructure stack is pretty
14:12
much the GM scholarly house. And now we are sort of, like, in
14:16
the process of like, figuring out like what, you know, what
14:20
does the future entail? And what are the pieces that we
14:24
can continue to use from that part of the universe?
14:27
And what are the latest and greatest in the python and our
14:30
ecosystem that can again sort of like get us more leverage and in
14:36
that scheme Runway is Lead for us Model Management System where
14:43
the end of the day you have users who are building these
14:46
really complicated, recommendation models and now
14:50
these recommendation models, they are surveying inferences to
14:55
all the members who log into Netflix and we need a mechanism
14:58
to track Which models are actually being used, how they
15:01
are performing. We also bundling the like a
15:05
smoke Monitoring Solutions that we have internally and that's
15:08
nuts. Good kind of tooling that Runway
15:09
provides which is more focused on just the Model Management.
15:14
Lifecycle for member facing are like, Netflix subscriber facing
15:18
inferencing. While the use cases that made a
15:22
flow is currently solving at. The moment are more around
15:25
training models in Python and our, as well as function as a
15:29
service platform for posting some of these models for
15:34
internal use cases. Okay.
15:37
So we've got A question in the chat from Ron.
15:40
Thanks for asking this wrong. He's wondering what tools were
15:45
storage. Does Netflix use for data Ops,
15:49
IE the front and management of data sets.
15:52
And how does that integrate with meta flow?
15:56
So Ron, I think you may have misheard us.
15:59
It's met to flow. Not ml flow that we're talking
16:02
about here. Yeah, that's a great question.
16:04
Now, you know, my team we stand on the shoulders of the Giants.
16:08
So Netflix has a very Advanced Data platform and there are like
16:14
numerous presentations on that on the internet so I would
16:18
highly recommend checking those out but I'll totally give you a
16:23
brief tldr on what we have available.
16:26
So as I said all of our data is stored in is 3 and we were sort
16:31
of like one of the very first companies to use S3 as our data
16:35
link to decouple storage from compute.
16:38
Look at Cena, like some of the Big Data Systems that were being
16:41
built 10 years ago like the Hadoop ecosystem.
16:45
That would couple your storage with compute and at that point
16:48
in time we decided that you know, for us that decoupling is
16:52
rather important and we would essentially prefer to pay the
16:56
cost of moving data over the network.
17:00
Just so that we can have this isolation and then we can have a
17:02
variety of different query engines to run on top of S3.
17:06
So so this data, you can Is this data through, you know, Presto
17:10
spark. So we have all of those query
17:13
engines available internally to our end users and also very
17:18
specifically for data access for machine learning.
17:22
Usually, what ends up happening is that the data scientists.
17:26
They are usually interested in just like fetching, one single
17:29
partition or a bunch of partitions from a table that's
17:33
available in the data warehouse. And what we have seen in
17:36
practice is that going through a query engine For those sort of
17:39
request is usually not really that efficient because you're
17:44
overloading the query engine. Then you know, if anybody else
17:48
is running a really heavy query on, the query engine that it
17:51
will also penalize the amount of time, it takes you to get your
17:54
data back. So we have invested in tooling
17:58
within our team so that you can directly go to our data
18:01
warehouse. And we can, like bypass
18:04
short-circuit these query engines can get you access to
18:08
this. Data as quickly as possible to
18:10
the queue lock, like you like to do for gbps, depending on the
18:13
instance type that you're using. And so that's so we are made
18:18
like big big Investments around the taxes and then of course,
18:23
there are other concerns that cut across machine learning,
18:26
which is like, how are you cataloging your data systems?
18:30
How do you users even figure out what data sets, actually exist?
18:35
How do you are? How are you sort of like
18:37
managing the governor? It's policies and how you taking
18:40
care of key details and all of that.
18:42
And so I need a platform T which provides us all of that ruling
18:46
and just piggyback on that. Awesome.
18:50
And I know I was watching a video with you.
18:53
I think it was up ML or something like that, some
18:56
festival or conference hard to have festivals these days.
19:01
But the in that talk, you mentioned, how when we think
19:08
about Netflix and machine learning, we think about
19:11
recommender systems. But now it's it's kind of
19:14
bloated and grown into such a big part of Of Netflix, just
19:20
business model. Can you explain the other areas
19:22
that you're using machine learning?
19:27
Yeah, I mean as you can imagine regulate Netflix, it's it's a
19:32
major company in the entertainment business and in
19:38
every companies like these are like hit a point where you start
19:42
liking any further progress, you have to select start investing
19:46
in data science because everything is running.
19:48
Such a high school, right? And like just to articulate like
19:52
some of the problems that we run into where we use data science
19:55
is whenever we have to roll out, say any changes to the service
20:01
of that audio experience. Now these changes we have to be
20:05
very mindful, right? We are not in any way, hampering
20:10
your viewing experience so we'll roll out a small canary will
20:15
observe how it's performing and then if it's me Forming as per
20:19
expectation. Then we'll roll out.
20:21
Roll it out like bigger audience, right?
20:24
And this needs to be done with care, but then, okay, how do you
20:27
figure out given that, you know, if it fixes available on
20:31
hundreds of different devices and people are using Netflix
20:34
with line different kind of internet connections and all any
20:37
change that gets rolled out. How do people select build
20:40
confidence on those changes? So so we observe, you know, like
20:45
some of the quality metrics, the quality of streaming that you're
20:48
getting. And then we smell like roll it
20:50
back and it's like some analysis that's done to sort of like
20:54
qualify a chance and that, you know, like there's a green and
20:57
you saw like roll it out to many more.
20:59
People are not, and some of this work is actually open source as
21:02
well. So our Canary analysis tool,
21:05
it's called kinda and also another Netflix Open Source,
21:10
tool is Pinnacle, that's RC IC D tool.
21:13
So it works hand-in-hand with that and that's like one of the
21:16
areas where you know, we have significant Investments and
21:18
machines Learning then, you know, we are processing like a
21:24
lot of payments. So we need to be very crafty
21:27
about, like, just like detecting fraud that's happening on our
21:31
platform. So, so we need to invest in
21:33
those areas content. Analysis is okay in one area
21:39
that try to sort of like use data science to look at like,
21:43
you know, what's working for us, what's not?
21:46
Yeah, I mean, as I can imagine things like just just think of a
21:50
big company like Netflix is and at a sufficient skill each of
21:55
the problems that could be solved by a heuristic sort of
21:58
like start to reach the limits of those heuristics.
22:01
And that's when you have to introduce machine learning and
22:05
that was indeed like sort of like the motivating factor for
22:09
us building better flow because now each of these different
22:13
problems they are. They require different
22:16
solutions. They require different ruling
22:19
The data scientists were working on each of these problems.
22:21
They have very different skill sets as well very different
22:25
areas than the specialized it and we essentially wanted to
22:30
like build something that could just like take care of their
22:33
daily pains and yeah that that was the entire exercise and that
22:39
leads perfectly into the next question that I was going to ask
22:42
you about like what do you feel the hardest part of building and
22:47
operating? Machine learning workflows is.
22:51
Honestly, everything is hot and at the same time, everything
22:54
should be rather easy and straightforward.
22:57
So unfortunately, I don't I don't really have like one
23:00
single thing that I can talk about that here.
23:03
Like this is, do you think that is the hardest?
23:05
Like I can give you a lot. A list of English like tens and
23:08
hundreds of things that I find are like equal equally important
23:12
because it's like you know if you have to build a
23:15
well-thought-out system that works really well in practice.
23:21
Then you have to focus on all the components that go in, you
23:24
know, if you miss out on even one then that will just likely
23:28
to more grief down the line. So as I said before, I'd like
23:32
one of the most basic things for any sort of like machine
23:36
learning project is around data. Right?
23:40
How do you make sure that you know your data is of appropriate
23:43
quality? And not just like appropriate
23:46
quality when you looked at it for the first time, but as you
23:48
are getting more and more data, Energy bar is maintained and
23:52
that's, that's a big problem that you know, a lot of very
23:55
smart, people are trying to solve, people are making good
23:57
progress on that front, then how do you get access to that data?
24:02
So, you know, like most training algorithms, the take input data,
24:10
as data frames versus the data, that's persisted in data, stores
24:13
is usually often in the form of cables.
24:16
So this this significant amount of work being done that.
24:20
Okay, how do we sort of like get access to this data as
24:23
efficiently as possible in the correct format?
24:27
So that you're not spending State minutes are ours just like
24:30
moving data around then cost becomes again one big concern
24:36
that? Yes, of course.
24:37
Now I like a lot of these things.
24:38
If you don't care about the cost, then it is like throw more
24:41
and more resources. But then for, for a lot of
24:45
organizations it just isn't prudent.
24:49
Like we live in a Little world. So everything has a particular
24:53
cost and the cost is not just in terms of like you like what
24:57
you're a duck. Just bend is going to be.
24:58
But also what is your total ownership cost of your machine
25:02
Learning Systems, how much no data scientist cost and like
25:08
what would be the operation about in going forward?
25:11
How do we sort of make minimize that to the best of our
25:13
abilities? Then reproducibility something
25:17
that sounded like very near and dear to my heart.
25:20
And they the academic definition of reproducibility.
25:24
Is that? Yes.
25:24
Now we have a data set and we have the code and that usually
25:29
be good for reproducibility, but in practice that snow really the
25:33
case. So yes, of course, you know, you
25:36
need to realize that you need the code but they're on the same
25:38
time. The world is moving, right?
25:41
Like say, if you installed a specific version of high trash
25:44
today, it will pull in some underlying dependencies.
25:48
If you install the same version of height of tomorrow.
25:50
They put in a different set of dependency that your behavior of
25:54
your trainer can sort of like change and then at the same
25:59
time, like, one thing that's incredibly hard to do today is
26:04
collaboration. So it's it's really, really
26:07
tricky for data scientists to meaningfully collaborate with
26:11
one another. And that's what I could beat
26:13
ability, becomes a big concern, right?
26:14
Like if let's say you are working on a Model, but I can't
26:18
really take your code and run it for life.
26:20
Billy on my own laptop or whatever system I have available
26:24
then I can't even think about building on top of it and so so
26:29
I think the need of the are of like the entire envelopes
26:32
Community is to just like now think about how can we sort like
26:36
move from a world where say you have like five data scientists
26:41
on it in each of those work on an individual Problem by
26:43
themselves to say five data scientists meaningfully
26:48
collaborating on a single project.
26:50
It because I think very soon we are going to do some like also
26:54
hit the bounds of like how much context one person in keeping
26:59
their head and deliver on a project.
27:02
So yeah that's that's another so many good points there with with
27:06
that. I mean you made this point
27:08
earlier, too and the cost like optimizing not only spend or
27:13
optimizing, not only the data science workflow, but optimizing
27:17
what the data scientist is doing is such a great A point and
27:22
yeah, along with that. I mean this reproducibility I
27:26
know it's part of like the three pillars that you all are using
27:29
and we'll get into that in a second but it it's such a great
27:32
point that you were talking about this because I remember
27:36
about a year ago I was talking to a friend and he was saying
27:39
how he had to work on a colleagues?
27:44
I can't remember. It was like he had to figure out
27:47
the model that a colleague had made and he said, Like the first
27:52
week and a half, he was just figuring out what was going on
27:56
and it was like, he was going around this Labyrinth and trying
28:00
to pick up these breadcrumbs right?
28:02
And trying to see. Okay, what's happening here?
28:04
And I spoke about this just last week with Neil latia who's the
28:09
head of ml at Monza bank and he was saying how, how difficult
28:14
this problem is, because it's very much in our heads, right?
28:17
Like machine learning and very especially when It's
28:20
exploratory. We're maybe we're trying to
28:23
research and so we know what's going on, very clearly but when
28:26
we try to go into what someone else is doing and we try to pick
28:32
up the whatever they were working on then it gets very
28:36
difficult so I love that. You're trying to attack that
28:39
problem and maybe we can just talk about these three pillars
28:43
and why you chose these three pillars.
28:45
I think it's pretty clear that the reproducibility one is huge.
28:48
It's really important. Then there's scalability and
28:52
then forgive me because I forgot the third one.
28:55
Yeah, I mean, I guess this was like a presentation that I gave
28:59
a long while back where we had like reproducibility scalability
29:02
and usability. And so in for that presentation,
29:08
we essentially wanted to focus on like three ideas.
29:10
So we like yes like these three ideas aren't really relevant for
29:13
us in practice and that's that's why we saw like mentioned that.
29:18
But yes, of course you know, they are like a lot of Other
29:21
solid Dimensions to machine Learning Systems as well, that
29:24
are equally important. Now on the reproducibility side
29:28
of things, I mean the reality is that the problem is difficult,
29:32
if it wasn't then it would have already been solved.
29:35
But so so and I mean I don't clean that you know like we have
29:41
solved the problem and can make at the end of the day there's no
29:45
like one single definition of reproducibility as well like,
29:49
you know, if Trying to make things reproducible.
29:52
Then theoretically, you know, like you need to have like an
29:55
audit log of everything that has happened and this audit log by
29:58
itself. You should, you should be able
30:00
to sound like walk back in time and Replay that called It Rock.
30:04
So it's not just like I could capture like what all changes
30:07
happened, but I should be able to roll back to a particular
30:09
State and being able to meaningfully execute out that
30:12
and the components that I spoke about, right?
30:14
Like data happens to be one. So you need to have a data
30:17
platform story that your data platform, underlying data
30:19
assets, actually, Allow you to do that roll back.
30:21
Many times, people don't have that capability user code as
30:25
another one, and, you know that you have like Version Control.
30:28
You can just like copy code for every execution and that's all I
30:32
gives you some roll back at increase.
30:34
Then you have like, dependencies like library dependencies and
30:38
all that. You can also snap shot those and
30:40
also for their like Solutions around that as well.
30:43
Then the fourth one is that, you know, your actual grunting, your
30:47
infrastructure is also a rapidly evolving.
30:50
So You need to make sure that there is some way to go like
30:52
that as well. And then m/l models by
30:56
themselves. They have like sufficient degree
30:58
of entropies of degree of Randomness.
31:00
We need to have a story of that which is like very, very tricky.
31:03
And now, all of these, like, managing all of these entropies
31:07
and rollbacks an audit, these all incur some cost.
31:12
And you know, many times on the basis of, like, whatever problem
31:15
you are in, it might be okay to forego some Notions of
31:19
reproducibility. Because you know, fact, you just
31:21
don't care about that as much, right?
31:23
And so it's really important to sort of like keep that Viewpoint
31:28
in mind and then sort of like Design Systems that yes, you
31:31
know, I d-- end of the day. It's all about trade-offs,
31:33
right? Like there's no, like 12 system
31:35
that is like pharmaceutical promises.
31:37
You the Holy Grail of reproducibility?
31:39
It's all about like, okay. All right, what are the things
31:41
that you care about and what are the things that you're okay?
31:43
Sorta, like not really worrying about as much.
31:46
And so to for us, you know? When we looked internally like
31:53
if you're running any sort of machine learning workload in
31:55
production, even if you know, for a moment we don't think
31:59
about the collaboration and the shareability aspect.
32:03
One thing that's really important is that how do you
32:05
recover from failures, right? Say say you have a training
32:08
workflow and that training workflow failed for whatever
32:11
reason in production, right now, as an end user, I want to be
32:16
able to replicate that exact same.
32:18
Whether I want to be able able to run that Pipeline and
32:21
recreate that failure. So that then I can figure out
32:24
how to fix that, right? Otherwise then I'm shooting
32:26
arrows in the dark hoping that something will stick and that
32:29
again you know like leads to a lot of heartburn and leads to a
32:33
lot of loss of productivity Cycles because you don't really
32:35
know how to fix the problem if you can't even reproduce it in
32:38
the first set. So for us this notion of
32:41
repeatability, this notion of reproducibility was more from
32:45
the debug ability, anything that you're building, they need to be
32:49
debuggable, right? Name, things things are going to
32:52
bring things are going to fail. Your data distributions are
32:54
going to change some of the assumptions that you had Big Ten
32:57
might not hold true say you know, three months from now, six
33:00
months from now and at that point, when you need to be able
33:02
to go back in time. Look at like okay why were so
33:05
many things working? Why are certain things not
33:07
working but still Delta and then sort of like work your way
33:10
around that and so that was really important.
33:14
Now on the scalability side of things, we We also do understand
33:21
that, you know, like for us when we started building medical, the
33:25
notion of squeal was that of like diverse set of problems.
33:29
But then when you are looking at the diverse set of problems,
33:32
like one thing to note is that the iteration cycle of putida
33:37
scientists like they usually start off on a small data set,
33:40
right? Like say something that fits on
33:42
their laptop, something that fits with the cloud workstation.
33:45
But then very soon, they'll be like, okay, you know, like this
33:48
thing works. Now, I want to treat my model on
33:50
the entirety of the data set, but now this data does not fit
33:58
in the instance that I have. What do I do, right.
34:02
And we want to make sure that you know, like the many
34:04
Solutions that you can do at that point in time.
34:06
Like but the simplest one that also has the lowest amount of
34:11
operational overhead is if somehow we can sort of like,
34:14
give you access to a much bigger distance so that you can just
34:17
like, whatever code you have. You just execute that code on a
34:20
much bigger instance, without making any changes to your code.
34:23
Like that now would be like far more simpler and
34:25
straightforward. So we wanted to make sure that
34:27
we were providing those Primitives so that if let's say,
34:30
you know, I'm feeling a model and run into an out of memory
34:32
exception and then I realized that You know, like I only have
34:35
16 gigs of RAM it on my laptop but what I need is like the keys
34:39
to a hundred gigs of RAM. Then I can just like change like
34:43
one a mutation in then the code automatically runs on an
34:46
instance, which has 100 gigs of RAM and then all the results are
34:50
by back to me while I don't have to think about that.
34:52
Oh, you know, this instance was actually in the cloud, and if
34:55
you have an instance in the cloud, you need to First create
34:57
a Docker container. You need to put your code and
35:00
you need to get the data somewhere, then you need to get
35:02
the results back. As a data scientist, I do really
35:05
want to know all of that, but I just want to capability to just
35:09
just my code, the system should know everything else and the
35:13
computer Engineers should just execute and give me back the
35:16
results. And then, you know, like if you
35:18
sound like stretch this idea further than okay go, like, can
35:20
I run multiple copies of my code with different parameters on
35:25
multiple different instances? And that also, again, you like
35:27
gets you a little bit farther ahead.
35:30
And of course, there comes a point where, you know, like your
35:32
model size becomes very We can still date that you have to go
35:36
through like some distributed model training approach, but
35:40
there's this sort of like this entire world and of like, you
35:43
know, just like scaling your distance vertically or then just
35:46
like screening instances horizontally and just by doing
35:49
some rudimentary shouting of data that gets you much further
35:53
along and that's that's, you know, another recommendation
35:55
that we have for our users internally that.
35:57
Yes, you know, it's like think about like some of these basic
36:00
Primitives before thinking about anything super complex.
36:04
But the yes, you know, if they can't really do without
36:07
distributed model training than, yes, of course we'll go in and
36:09
we'll support that as well. So I'm wondering sorry to cut
36:13
you off. Just you all along that point
36:15
and I see there's a question from Kenny.
36:17
I'll get to that in just a second Kenny, but I'm wondering,
36:20
like you're giving people basically so much power, right?
36:27
That they could do a lot of harm with it too, because they may
36:31
just be, like, well, I have unlimited resources.
36:34
Sources. Let me just train, this insane
36:37
thing or let me just have these instances that are insanely
36:40
large, how do you keep them from going overboard?
36:44
Sure. Yeah, that's that's an excellent
36:46
question and you know, something that we have given in to a
36:49
value, a lot of thought about. So when you, when you look at
36:51
like these containers orchestration systems, right?
36:53
And that's that's really like beauty of like some of these
36:56
systems like you know, titles that have built internally and
36:59
ews batch with which we have an integration of the open-source
37:02
come in handy where They have this notion of queuing and
37:06
priorities. So you can have like certain
37:09
cues which only have access to a certain fixed amount of compute.
37:14
And that means that no matter how many jobs you launched, it
37:18
can at any point in time, only support a set number of V CPUs
37:23
or gpus and all the other jobs, they'll just like, wait in the
37:26
queue and you know, if you are being creative with these cues
37:30
and the priorities amongst your cues, then you can very easily.
37:34
Provide flexibility to your data scientists where, you know, if
37:39
they just want to sort of like do some sort of exploration,
37:45
then they have access to a limited number of instances, in
37:47
their compute environment, and once everything is approved.
37:50
And okay for production, then they should be able to move
37:53
their compute into a queue, which has access to fun figuring
37:57
sources. Okay, that makes sense.
38:01
So Ken he's asking can you explain this pillar of
38:06
usability? He's curious as to what the
38:09
usability pain points you had to resolve were?
38:13
Yes. Yes, the so for us medical was
38:16
you know, inherently of a big exercise of usability singer.
38:19
So like my fundamental belief is that tools should work for
38:25
people and people should not work for tools and So so for us,
38:30
it was rather important that, you know, we don't place an
38:32
expectation on our data scientists to be systems
38:34
engineers and also you know, like meta fluids.
38:39
It's a python Library, it's not a DSL and that was a very
38:43
conscious rise because as soon as you introduce an ESL then
38:46
people have to sort like flip through manuals and understand
38:49
like okay like how disorder like work through that DSL and but if
38:54
you are just like using a programming language that you're
38:56
already used to and the reason why we went with python And our
39:00
was because these people they are working in these languages
39:03
day in and day out that's more like makes it rather simple and
39:06
straightforward for them. And then at the end of the day,
39:11
you know, as I said at the beginning of this conversation
39:14
as well, Is this like a machine time and then there's people
39:19
type right? And we ideally, I would want my
39:24
data scientists to like, spend a lot of time.
39:27
They can Docker containers flipping through the manuals for
39:30
tigers and kubernetes Ne WS passion to think about like,
39:33
hey, you know, what does this kubernetes error code mean?
39:37
Also, if you are running a anything on the cloud, right?
39:40
Like instance of Spain all the time, they're like a lot of
39:42
errors that happen. So so you need to have a good
39:45
amount of like, you know, no resiliency baked into the system
39:50
and that resiliency should also be exposed to this user in a
39:54
very easy to consume Manner. And that's again, something that
40:00
aids in usability at the end of the day.
40:03
Yeah, I mean, if you look at the like everything that I spoke
40:07
about like, you know, walking between like different layers of
40:10
the stack there, like a whole bunch of things that people need
40:12
to do and our goal was to abstract that away, but Even
40:17
then, not really hide that in all of these activities were
40:20
happening behind the scenes because when failures happen,
40:22
they should be able to go ahead and keep up.
40:23
Like one example here, would be that every single time.
40:26
You execute your code. We actually snapshot the code
40:29
and we store that in perpetuity and that's sort of like the
40:32
executable runtime which we try to make as you producible as
40:36
possible. And now if a user had to do it
40:40
by themselves, then they would so like find themselves in some
40:43
sort of like version. Also a lot get off school and
40:46
that A their usability, right? Then what it means is that every
40:49
single time I execute my code, I need to take a few more actions
40:54
to make sure that I'm logging what the behavior of my code is,
40:56
what my ex will get hash was in terms of and we would want to
40:59
take care of that out of the box whether there should not be any
41:02
extracted that they should be taking.
41:05
So. So for us it's more of an
41:07
exercise in design that can be sort of like remove as many
41:11
actions that a user needs to take for any of these
41:14
operational concerns and that's that's Early usability on you.
41:19
Sweet. So I don't know if you can
41:23
answer this one but you know, role-based Access Control has
41:27
been something that has been coming up a lot recently and I'm
41:31
wondering how you are doing that especially when it comes to data
41:36
because data is so sensitive, right?
41:38
And so get how, how does I don't know if meta?
41:42
Flow, does anything specific for that?
41:45
Or if just inside of Netflix, you're doing it some way, but
41:49
But can you speak to that at all?
41:50
Yeah, definitely you don't like as you can imagine.
41:53
Even in sight Netflix, we do have some data sets that are
41:58
like far more Kosher for need to have like a higher degree of
42:03
security and access control compared to some other data
42:06
sets. So we do have systems internally
42:10
that you know will authenticate certain workflows and only allow
42:13
access to certain individuals and now as it happens with men
42:18
of your right leg. We don't want to bake any of
42:21
these controls within matter of you, but we want to make sure
42:23
that made of flu obeys to any of these constraints that are
42:28
placed. So, for example, for our
42:31
integration with AWS and open source, we obey all the I am
42:36
restrictions that exist, right? So, like, another example, is
42:43
that right now, you can go to rhema.org slash sandbox and will
42:48
essentially Me give you a cloud instance which has like certain
42:52
number of e CPUs has S3 buckets as like step functions and
42:56
everything. So that you can very easily try
42:58
out meta flow on the cloud and see, you know, like this thing
43:01
actually works and behind the scenes, we have to make sure
43:05
that, you know, there's like strict isolation that people are
43:07
not able to read anybody else's data, and the data is cleaned
43:11
out as soon as their sandbox expires.
43:15
And all of that Machinery, essentially happens why Mm
43:19
policies, I am Rose. There's like nothing specific
43:21
that we have baked in metal flew.
43:23
So if let's say, you know, there are many organizations who have
43:26
a specific posture towards security axis and you want to
43:29
make sure that, you know, they can just keep installment of you
43:32
and just works out of the box with them without us needing to
43:37
do any sort of customization. And then that gives all of these
43:40
organizations, much greater degree of flexibility as well as
43:44
control over how they want to manage their resources, who can
43:48
launch. A compute instance.
43:52
So getting a question in the chat here and I believe I know
43:56
the answer to it but is meta flow based on kubernetes.
44:00
It is not right, it or is it? So it's it's a good question.
44:05
So Titus internally is has moved on top of kubernetes and we have
44:11
an integration with tigers, so as far as like a computer
44:15
platform is concerned. So, so there is a component test
44:17
integration. Metal flew by itself.
44:22
It follows a plug-in based architecture support as it's
44:24
more about like, okay. This is a unit of compute and
44:27
now we need to execute that unit of compute somewhere.
44:29
It could be your laptop, it could be a cloud instance.
44:32
And there are sort of like PR's open, and it's also an active
44:38
media of work for us in the open source Community as well to have
44:41
like, more kubernetes Integrations.
44:43
So that's where we are at right now, but the official open
44:47
source integration that we have right now, okay.
44:49
With a. Wh so let's talk a little bit
44:51
about open source because I mean, it's one thing to be doing
44:55
this for Netflix specifically, but then you go and you open
44:59
source it, and I'm sure that opens up a whole new can of
45:03
worms, what have you seen since you open source?
45:07
That what are some learnings that came across that you didn't
45:11
have when you were working at Netflix.
45:14
So now one thing about Netflix is that, you know, we have this
45:19
Sure, of freedom and responsibility, which means that
45:22
our custom with his ideas and schemes, they have the freedom
45:26
to utilize, whatever tooling works best with, which includes
45:31
any open source Alternatives or any other commercial Solutions.
45:34
Even when we were building metal flew at that point in time, we
45:37
have to make sure that case you don't like.
45:39
Like we would only build something if we felt that there
45:43
were certain tools that were lacking in the ecosystem at that
45:46
point of time. So like three years ago when we
45:48
started building that, I flew the Emma loves will look very
45:51
different but then at the same time we wanted to make sure that
45:54
case of course we don't want to build something that already
45:58
exists. Something already exist.
46:00
We would much rather adopt that and build different things case
46:03
in point. Being argues of the aw Scout,
46:05
right? My face.
46:06
But I've gone and built their own data centers.
46:08
Amazon is always going to excel at that.
46:10
So we're like, yes, I will just like Leverage that and build to
46:12
live on top. So, in that sense, when we open
46:16
source made of low, we had some what Well, fear idea of like
46:21
what other tooling existed and that was sort of like the big
46:24
rationale around open-sourcing, that after we felt that.
46:27
Yes, there is a gap in terms of the tooling that exists and we
46:30
can certainly make a meaningful contribution to the community
46:34
and ever since open source, you know, like we have spoken to a
46:38
lot of companies, a lot of individuals who are using my
46:41
referral who are Keen to adopt night after as well.
46:44
And that has also owned our thinking around how the
46:48
community is a Evolving how people are sort of looking at
46:51
things and has allowed us to sort of like either validate
46:55
some of our hypotheses that we held that.
46:57
Yes, you know, like these are some of the areas that hold much
47:00
more stronger promise for us as well as, you know, some of the
47:02
things that we were thinking about that.
47:04
Okay. You know, maybe this might be
47:05
useful turns out that if, you know, people are not really too
47:08
soon on those things and data access happens to be, you know,
47:12
like one of those areas where it indeed is.
47:15
A real thin point for a lot of people.
47:17
Like what does the data ecosystem?
47:19
Hi, how do I get access to data? And like, what are some of the
47:23
good patterns around just like, arranging denied my data?
47:25
Whereas, yeah, I remember talking to a guy, almost a year
47:32
ago, and he was telling me how so he worked in health Tech and
47:39
it was a global company and they were only allowed to use the
47:46
data that came from Canada in Canada.
47:49
That data could never leave Canada.
47:51
So his problem in his head was how can I make sure that only
47:56
people in Canada have access to that data?
47:58
But then they also have access to all the other data, right?
48:02
So there's a lot of things that are happening right now and I
48:05
know that's why this keeps coming back up is like how can
48:08
we access? What are the rules?
48:10
What are the best practices around this axis and that along
48:14
with security is starting to crop up more and more.
48:16
And I'm wondering I guess this may We've got just a little bit
48:21
of time left. This may or may not be our last
48:24
question, but I'm wondering what kind of Trends you have been
48:27
seeing and what are these things that you thought would be
48:32
interesting or what people are asking for?
48:35
Where do you feel like the market is going?
48:38
That kind of big question that you can you can answer in two
48:44
minutes or in two hours? Yeah I mean this is The very
48:49
loaded question like that. Yeah, I mean, we are definitely
48:53
seeing like a lot of activity happening around like all layers
48:56
of the stack. I mean, just recently, tecton
48:59
raised to be crowned, a lot of activities are happening on the
49:02
feature store side of things as well.
49:05
And then model inferencing is becoming a big Hot Topic as well
49:12
and on the topic of influencing, you know, like there's this like
49:15
a strong push around. What does like micro service
49:19
based inferencing look like but At the end of the day, the
49:23
problem is more around. Like, okay, you have these
49:25
models and all these models need to interact with the real world.
49:28
And what are those integration points?
49:30
Like, and those integration points, could be a micro
49:33
service. They could be some sort of,
49:34
like, in process communication, some sidecar could be some bad
49:38
scoring. Similarly, with feature stores
49:41
as well, right? Like, feature stores are kind of
49:44
like, you know, what databases are to software engineering.
49:47
And ultimately, at the end of the day, you don't have like one
49:49
single database, you have like a lot of Different databases
49:53
suited for like different applications.
49:55
So, so I think now the industry is evolving towards the point
49:59
where, you know, like they're moving from like one definition,
50:01
like hey, we have a feature store to like, hey, you know,
50:04
this is a feature store suited for this particular use case.
50:06
And these are the constraints and these are the trade-offs
50:08
that we are making. So, yeah, I mean, you know,
50:11
we're sort of like, going through the normal cycle of
50:15
maturity in this ecosystem where early on even well, I guess, you
50:19
know, we have these problems. Let's build something.
50:22
Build something and you're like, yeah, I know, maybe like some of
50:24
the things that people might not really have hit the target, but
50:28
then, as you learn more, you start like hone your
50:31
understanding better about the problem.
50:35
And I think that's that's the cycle that we have a great
50:39
answer. So I guess I could sneak one
50:44
more in before we finish. And I'm wondering how you seen
50:49
companies on when you're when we're talking about the open
50:51
source. Horse Market.
50:53
How have you been seeing companies using this?
50:56
I instantly think because reproducibility is such a big
50:59
deal with meta fo I would imagine that it is very well
51:04
liked by the fintech or bank because it for them, it's huge.
51:09
And when you said oh yeah reproducibility for us was more
51:12
about trying to go through and do these bug bashes?
51:17
I didn't even think about that. I was more thinking about, okay?
51:19
When you're having to deal with laws, Regulations you need
51:23
reproducibility, right? So what are some of the
51:26
companies that you've been seeing really interested in meta
51:29
flow and then also, how have you been seeing them combined it
51:33
with other tools that are out there?
51:35
Yeah, great question. So essentially you know like
51:41
because our Focus has been to reduce The Accidental,
51:46
complexity machine Learning Systems as much as possible.
51:49
They've seen a significant update From organizations who
51:54
have major investments in machine learning, but we don't
51:57
want to sort of like, you know, spin up yet.
51:58
Another team to manage a machine learning platform per se and
52:05
they already have some deployment on AWS Some Cloud
52:07
presents. Real existing like this tool for
52:10
like helps us get more productive without actually
52:13
having some posted recently carrying period Ute or anything
52:16
of that sort. So so like small companies
52:20
startups there, It's been good at options so far.
52:26
And then, on the point of like verticals, yes.
52:29
And I've come across like, a lot of banks, a lot of healthcare
52:32
companies, use Metal Flow, bunch of hedge funds Automotive Majors
52:40
retail foam. So there has been adoption
52:44
across the band. Now, the problem with an open
52:46
source project is that you don't necessarily get to know who all
52:50
are using my love you. Unless, you know, somebody
52:53
reaches out to us and they're like, hey, here's some feedback
52:55
or hey, here's something that's broken and you need to fix that.
52:58
So so my understanding of like all the adopters of Metal Flow
53:02
is unfortunately kind of broken. So, yeah, that's that's there
53:07
and then I guess the next question was that how do we play
53:10
with other open source tooling now?
53:14
So we are an open protocol in the sense that you know within
53:18
my love. You're just like running
53:19
arbitrary, python code arbitrary.
53:22
Code which means that you can import anything and everything
53:25
that you want. So I know that a lot of people
53:28
like ml flows model tracking like metric tracking API, and we
53:34
do have people who just use ml flow within a few steps and then
53:39
that gets them, all the benefits of metal flew in.
53:42
Then they also have the benefit of going into the ml flow UI and
53:47
looking at all the metrics that they were locking and tracking.
53:50
And I'll get people get all called to use, metal floor for
53:54
training, their models and then once they're models have been
53:57
trained and they will post those as cortex model.
54:02
So that's something we have food.
54:05
Yeah, I mean like plenty of these sort of instances, I've
54:08
come across so far. Yeah, I imagine there's a ton of
54:11
use cases that we do not have time to get into today.
54:15
So 7, thank you so much. We're talking to us and giving
54:22
us this breakdown on meta flow. I know it is a for me.
54:25
I'm love the project. I love what you're doing for the
54:29
exact reason that you said like you are basically at Netflix and
54:35
you have to because of the culture of Netflix, you are
54:39
allowing people to say yeah. We want to use this or no, we
54:42
don't write. So you have to bring the a game
54:46
when you're creating this and you have to make sure that it is
54:50
Able to plug and play with a lot of different things and so I
54:54
really appreciate you doing this deep dive with us and I want to
54:58
say thanks, even though you didn't tell me when stranger
55:02
things is coming out if they're even recording, nobody's on
55:05
anymore. So it's all good.
55:06
If you can tell me if they're even recording season for yet,
55:09
or what is going on, all right. I won't pregnancy.
55:12
I have been doing ballet job that they don't tell you, huh?
55:16
So, thanks everybody for sticking around if you're still
55:19
with us. And join us in the MLS Community
55:24
slack. If you are not there already,
55:26
I'm throwing the link to that in the chat again, for the last
55:31
little piece and other than that, yes, a beans in there.
55:36
So if you want to continue this conversation with him he is
55:39
there. I'm sure he would not mind if
55:41
you reach out to him. And the the last thing I guess
55:47
would be if they don't want to If they don't want to come into
55:52
the community chat, you're also on LinkedIn, you can find them
55:55
there, are you on Twitter, you are on Twitter, he leave.
55:59
So he's all over the place, you can find him wherever the
56:04
internet is found. All right, Savion, I really
56:06
appreciate it, man. Thanks for having me.
56:10
Take care everybody.