Latest / Agentic Conversations (formally mlops.community) / MLX: Opinionated ML Pipelines in MLflow // Xiangrui Meng // Coffee Sessions #112

Transcript

MLX: Opinionated ML Pipelines in MLflow // Xiangrui Meng // Coffee Sessions #112 · Agentic Conversations (formally mlops.community)

0:00
Yo why is this guy such a big deal?
0:03
Just break it down? I mean it's hard to hot
0:09
question, you just came out with but I can actually answer that
0:12
xiangru remaing who joined us from data bricks where he has
0:16
with the first machine learning engineer, has been for about
0:19
eight years is actually one of the lead maintainers for both ml
0:24
flow and ml live. And by the way, is also an
0:27
Apache, PMC member for spark. Yeah, he's a big deal.
0:31
This guy has built a lot of what you guys are out.
0:33
There are all using and he came on to tell us what he does and
0:37
how he thinks about it. All did my favorite part was at
0:40
the end when he just walked us through, basically how he looked
0:44
at the problem and how to tackle the problem, it was like seeing
0:49
the internal workings of someone's mind.
0:53
Like he was so clear at, okay, well, I thought about this and
0:57
then we thought about this and then I tried to go and find
0:59
where It was already being done but then I realized well you
1:02
know with tf-x it's not really working because they're built
1:06
for Google scale and so then I went to other things and at the
1:09
end of the day we did it this way and that's why we were
1:11
opinionated. I almost wanted to keep riffing
1:14
on that too bad that time was kind of up and what about you
1:19
over some highlights takeaways on your end.
1:22
I actually really liked hearing about his experience.
1:24
Getting started data works and sticking around like it's so
1:27
cool when somebody. Has been actually at the same
1:31
place, working on machine learning.
1:34
At the same time that so much, in machine learning has changed
1:37
and I thought that was a pretty remarkable just part of the
1:40
podcast to just listen to reflect on.
1:43
And I'm really grateful that he was able to come on and share
1:45
all that with us. So true so true.
1:49
So let's get into it. This was an awesome episode,
1:52
wish angry and he has affected, he has touched so many lives,
1:57
inadvertently through spark through.
2:00
Ml flow through MLA by mean it's awesome to get to talk to this
2:04
guy. Let's jump right into it.
2:09
Why don't you give us a little bit of a download on who you are
2:13
and what you do? Yep.
2:16
Hi this is John Ray from data Brooks, so I'm currently Uber
2:21
Tech lead on the machine learning engineering team and
2:23
just working with individual engineering teams of our self.
2:26
Our ammo product features. I joined Did he breaks about
2:32
eight half years ago? So I was the first motion linear
2:35
engineer here and initially, just working on the machine
2:38
learning library in spark called a mullet.
2:41
And basically just the design, the apis and implement the major
2:45
algorithms and working with the community to skill.
2:49
The project development also has spent a few years our internal
2:53
data team just as being a data engineer and data scientists
2:58
using our own product. To handle our lock locks and ETL
3:03
data and data warehouse and data analytics.
3:07
Yep, that's me if you like there's so much that I could
3:09
just ask you. And in that short bio that you
3:12
just shared the fact that you join data bricks eight and a
3:15
half years ago, the fact that you were the first ml engineer.
3:18
There you worked on a metal live early on like each of those
3:22
things, we could go into to meet you as.
3:24
I'm going to take some creative Liberty here Amaura.
3:27
You got something to say first vitro's yeah.
3:29
No, no. Where are you gonna go with
3:30
this? Because I want to know, what
3:32
kept you at a two bricks for eight years and what has enticed
3:35
you enough. But Vishnu like, where do you
3:37
want to go with this? Let's roll with it.
3:39
Let's throw that what's kept you here at database for eight and a
3:42
half years now hungry. So maybe to answer that, it's me
3:48
talk about the work I did before data breaks.
3:51
It's what Seth Lincoln, for one year, it's I started there as a
3:55
machine learning research engineer, and the building some
3:58
machine learning platforms on top of Of mapreduce.
4:02
But I, when I started, it's a very clear to me.
4:06
Mapreduce wasn't really designed for machine learning, right?
4:09
So big people, do it. Much learning our mapreduce,
4:12
because mapreduce, is the production, cluster compute and,
4:17
but in order to do machine learning, it's a little bit
4:19
awkward is because mapreduce pedophile is just a reader batch
4:23
and the same web page and which learning you need many
4:25
iterations and people just end up with holding the mappers.
4:30
And that just let them do more iterations.
4:32
Throw away the mapreduce Paradigm racial, but at the end
4:37
you just feel you're doing something on the wrong platform.
4:41
You're doing the correct thing. People need a machine learning
4:43
platform. That's very clear but you are
4:46
doing it on the, on the, on the wrong base ratio.
4:50
But over time, I feel wow, this is not really the direction to
4:52
go and then I saw the spark, it's a there is an intent in
4:58
Sister team and the Try to setting up spark.
5:02
I took a read the spark paper. I feel, oh, this is nice.
5:06
This is a data caching and you can handle many iterations and
5:10
that's good for machine learning.
5:12
So that's how why I happen to know the the co-founders but I
5:16
didn't know their work. It's just a small word.
5:19
I just call them and say well you started this new company
5:22
called Data works, but it seems, you guys are all system people.
5:26
You need some people who can handle this and a little X-ray
5:30
so much learning so maybe I can I can I can do that.
5:34
So that's how I get into Data breaks so but during the
5:38
interview is the the co-founders and show me the product the
5:43
earlier version of the product. So that surprised me a lot
5:47
because I got in touch with data breaks because of spark.
5:51
But at that time is I think the co-founders already has a clear
5:55
prediction of the future and the future is cloud computing,
5:59
right? Do they know that within couple
6:03
years and everyone every company will be in the cloud and then
6:07
they start designing the product for that future rather than
6:12
design the product folder for the, for the year 2013.
6:17
Rachel, and that surprised me, and their product vision is
6:20
really this a unified analytics platform.
6:23
And if you saw the early demo version of the product, it
6:28
doesn't look very different from the Product null.
6:30
We just added a lot of features, but essentially, the direction,
6:34
never never changed ratio. That's really why I want to just
6:41
stay with data brakes and keep improving this product.
6:45
Go towards the origin of the product vision is I think the,
6:48
the co-founders and has very accurate prediction of the
6:53
future. And they also have a really
6:56
strong education. It's just as they align the
6:59
teams that And execute towards the same direction and an owl
7:03
show. Definitely, I also believe that
7:05
it's having this a unified engine platform can make this
7:09
that people like a data, analyst data scientist, and the data
7:14
Engineers who you are. They don't speak the same
7:16
language and the leave under the same roof and the try to use the
7:20
same tool to collaborate. That's amazing but we're not
7:24
there yet so that's why while I think there are still lots of
7:27
things to do and that's That's kind of why I stay here for more
7:32
than eight years. And why do you think you're not
7:35
there yet? What still needs to be done.
7:37
There's so many things. That's why we have this Emma off
7:40
spark test, right? So it, so that's why I'm an
7:42
abscess do a mess. So this, it's not simple.
7:47
For example, it's a remember there in the mapreduce stage?
7:52
There are, there is the statistics Professor from
7:54
Stanford and talk about. Oh it's a why data scientist?
7:59
I need to know. Mapreduce ratio is survived by
8:03
producer. So popular and if you look at
8:05
them at the white elephant book and then figure out.
8:08
Okay? So because instead of saying you
8:12
just compute me and virus. Now you need to understand how
8:15
you compile it down to Matt, reduce what's inside, the mapper
8:19
was inside a reducer and that's very hard.
8:22
That's so hard. Right show.
8:24
And he doesn't believe that data scientists need to know,
8:27
mapreduce. Rachel did the center only need
8:30
No, they want to compute me and the virus.
8:32
The engine should be able to take care of that.
8:35
So to make a super simple and so data centers, don't need to get
8:40
to the low level details and that's how we came up with spark
8:44
a spark. You don't need to understand
8:46
mapreduce anymore, right? Show you just a use spark to do
8:49
the things you need. And I think the same applies to
8:52
General data science and machine learning is still so hard.
8:55
People still need to learn many things.
8:59
They are not. The nervous and they eventually,
9:02
you shouldn't be their skill set.
9:06
So you have, that's why I think it says there's still so many
9:09
things to do because the product is not that simple to use.
9:14
Let's let that sit for a second. No, I think I, I totally see
9:19
what you're saying there around the fact that there's still so
9:23
much that can be done to empower people to do their best work.
9:27
I think a lot of people on our podcast in the community in
9:31
general or like I wish I had to I spent less time figuring out.
9:36
Tools of the trade and more time using them to drive value in,
9:41
whatever that might be right. In terms of actually applying
9:44
the tools. And I'm kind of wondering, you
9:47
know, you have had this incredible experience of being
9:52
at data bricks for, you know, quite some time.
9:54
Being able to be the first machine learning engineer.
9:57
What is it that made you passionate about?
9:59
The fact that people should be able to apply machine learning?
10:02
What does he what got you into machine learning in the first
10:04
place? Yeah, so my major at school
10:09
wasn't bushel any my major at school was competition of math.
10:14
Why do is really just a numerical optimization is and
10:17
the simulation linear algebra and but this is actually a very
10:22
familiar with implementing those machine learning algorithms
10:25
because about the building blocks for the solvers really
10:28
linear. Algebra ratio, do Matrix Vector,
10:31
multiplication, how to do this scalar in a scalable and the
10:36
That way. So that was my major.
10:38
That's why I started as a machine-learning engineer and,
10:40
but on the back end implementing those algorithms but over time,
10:44
I see that. Okay, so, you know, other to
10:49
utilize the your data really many, many companies, and they
10:53
collect, so many data petabytes terabytes ratio, but when you
10:58
talk to them is, They will tell you oh events really they want
11:03
to they want to get value from it, the not just a pain, the
11:06
storage cost and what's the best way to get value?
11:09
While they can have some data scientists to do this, say eta
11:15
to do some dashboards and to tell them what happened in the
11:18
past Rachel but the current more about what will happen in the
11:21
future, right? So they want to get into the sap
11:24
predictive business and the which requires machine learning
11:28
and the biggest our way through base, but at the end too.
11:31
Still, they need machine learning models and to power
11:33
their business. So that's why I feel what it
11:36
would be a huge impact, right? Show to work on machine learning
11:40
to make money easy to, to end users.
11:43
So many companies there with data setting in their, in their
11:47
storage account, right? So they can really just make
11:50
value out of it. One of the questions that popped
11:53
up to me. Immediately, there is from the
11:56
time that you started both at data Brooks and in machine
11:59
learning to today there. Have been so many changes in the
12:02
ways that people think about ml the models that they're able to
12:05
build the tools that they use, right?
12:08
Like it's remaining ten years. It's hard to imagine like how
12:12
far we've come and how much further we have to go right with
12:15
things like large language models, you know, models like
12:19
Dolly and stuff like that. There's there's there's it's
12:22
like we've come so far and they're still so far to go and
12:25
I'm kind of curious from your standpoint as a not just like a
12:29
tool builder for the machine learning and Here.
12:31
But if somebody, that works on like, very fundamental, I guess
12:36
you could say like almost conceptual like infrastructure
12:40
that supports the entire machine learning ecosystem.
12:43
How do you think about changes the changes that have happened
12:49
and, you know, in terms of keeping what you build fresh for
12:53
the future? I hope that kind of made sense.
12:56
I'm thinking this out with you right now.
12:59
Yeah, I think I the very beginning.
13:01
It's a when I work on a mullet I think it's mostly our how to
13:05
make an algorithm scalable. So people can show over large
13:08
data sets. And but over time it says see
13:11
the strands shifted to help from how to how to use, those are
13:16
pretty fit models. Those model modules created by
13:20
others, and to to train our own model right show together a good
13:24
accurate just as a predictive model for your own business
13:27
problems. And then, Another Trend I see is
13:30
people care more about production.
13:33
So back in the days and a lot of people just doing the batch
13:37
scoring ratio after the trail model, they do best score in the
13:40
upload, the scores and the to a database and to serve.
13:44
But over time, people see that. Oh, that doesn't address the
13:46
business need. They start doing a lot of real
13:49
times inference and but how to get models into production,
13:54
Rachel, how to coordinate the work amount is a multiple teams
13:58
and becomes a really big challenge.
14:03
So that's why we see a lot of focuses on Emma Ops rather than
14:07
the core algorithm at this time. And there are more ammo to say I
14:12
endpoints ratio, you can use and rather than thinking about
14:15
what's behind the endpoint to get your job step.
14:19
It's a it's a great point about how the challenge has gone from.
14:23
You know, first it was just about scale and being able to do
14:27
more. And the way that people But want
14:29
to use it now has continuously changed I know Demetrios has a
14:33
question that he wants to ask you some going to kick it to
14:35
him. I was just thinking about like
14:38
when you talk about that Evolution and I hear it being
14:42
mentioned quite a bit and they'll like the problems or
14:46
this snags that you can get into when you're going to real time
14:49
and real time is very difficult and I was wondering what you see
14:54
as some core challenges. When you're trying to make that
14:59
transition to badge and real-time and what are use cases
15:03
that lend themselves to real time.
15:06
Because I imagine there's certain use cases, like, I
15:10
believe. Vishnu we had on here, fairly
15:13
recently, a guest who was saying, yeah, like real time
15:18
isn't really a thing because they're in the healthcare
15:22
sector. They don't need to necessarily
15:24
have all that data right away and We be streaming and making
15:28
predictions, right? But then there are other use
15:30
cases like a recommender system. That it's much more interesting
15:34
if it's real time, right? Yep.
15:35
So two questions really, what are some challenges that you've
15:39
seen with getting to real time and then what are some use cases
15:43
that lend themselves to real time?
15:45
Yeah. So the first one is 0, depends
15:47
out requirements, right? So it's not necessarily put
15:50
every model into real-time serving and but as mentioned
15:54
that there's more and more real time, it's not because while
15:57
this Thermo use case is demanding real time.
16:00
It was because back in the days, there wasn't good infrastructure
16:03
to support real time and people have to do badge and the do
16:06
this. Periodically, upload of the
16:08
predictions? Yes.
16:11
So but now, if you think about the challenges of real-time,
16:14
it's really about how to automate the entire lifecycle of
16:19
your models in production and the especially around the data
16:23
set of things. For example, is a how you make
16:25
sure your model. Is doing a good job ratio.
16:30
And the because if you put a motor in real time, that means
16:33
you really care about the time sensitivity of your data, right?
16:38
So you want your model to give most just predictions based on
16:42
the most recent data and that also means your date, you think
16:45
your data change over time, that means you have to monitor your
16:50
model closely and to see if your model has degraded performance.
16:54
And but it's your it is not just the model itself, right?
16:57
So you can see, oh, there is a feature Drift, But maybe I'll
17:00
model performance is fine, but maybe your model called just
17:03
model quality, stay the same, but Downstream business becomes
17:07
bad ratio. You want to monitor all of them
17:09
together and then, well, if there is a just say adrift
17:14
significant drift, you have to trigger some model, retraining,
17:18
or your do this on. Updates of your model that make
17:21
his hard. Because if you go real-time that
17:25
means your data changed frequently.
17:28
If your data changed frequently, that means your model.
17:31
Also need to change frequently. One of the things I kind of want
17:34
to ask here is I was taking a look at your LinkedIn profile
17:40
because I was aggressively stocking you in preparation for
17:43
this for this podcast. And one of the things that you
17:48
wrote and And is about it's a nice little equation for a my
17:53
laps. I'm going to read it here.
17:55
And it's basically a devops Plus data ops+ model Ops equals ml
18:00
Ops. And I was like, okay, that's an
18:03
interesting summary right there. And I'm curious like, what led
18:07
you to that formulation and why is that accurate?
18:13
Yeah, this is because we saw you're ready.
18:16
Emma. Ops is a team sport.
18:18
This is observation from our customers and they have this
18:21
data science teams, who creates the output models of training
18:26
code ratio. There is this a production
18:29
engineering team and who are not necessarily motion and Engineers
18:33
because not every company has a much learning engineering team,
18:37
so they are traditional, kind of a devops team and the apply this
18:41
standard stuff. Our engineering practices, and
18:45
there's always this data engineering team and supplying
18:48
those training data, and the features and the to the data
18:51
scientist ratio, you really well, those three teams need to
18:55
collaborate while to get some models trip to production.
19:00
And that's why I already called this data, Ops photo ops and the
19:03
devops and putting them together and it's a mobs, but it just a
19:09
betray definition and it just don't want to call.
19:12
It. It's a Mattie Mattie team spot
19:15
and you, you have to have different personas collaborating
19:18
to make it all, Emma off successful.
19:21
I feel like I could ask you about your data bricks
19:23
experience and the past and your opinions until the end of time.
19:28
But I really would love to talk a little bit more about some of
19:31
the projects that you're working on.
19:32
I know that you worked on ML lib and that you're also currently
19:35
involved very much an ml flow, and let's start with ML flow,
19:38
because we've actually had a couple different episodes about
19:41
that recently. And I know Demetria, A couple
19:42
questions he went out, he wants to ask there too can you tell us
19:45
about your work and what you do to support ml flow right now?
19:49
Yeah. Yeah.
19:51
So I start looking at em off low, as you are, will be the
19:57
next major feature to it and then while it's funny, I always
20:02
tell people is there so many flow products.
20:05
Also hear that from your previous podcast.
20:07
It's there's tested flow. There is a meta flow Well, scoop
20:13
flow Rachel and my flow, but if look at other airflow yet, if
20:18
you look at other flow products, they have flow components yet,
20:22
right? So but if you look at a more
20:24
flow, it's a tracking and is, is really good, right?
20:28
So model flavors and the model registry, but it's kind of
20:32
lacking the Flow Part. Rachel.
20:35
And yeah, this is why while we introduced this ml flow
20:39
pipelines and 2ml flow is actually just as ad is flow
20:43
part. But if this is just a just
20:46
joking, it's now really because I'm a flow is lacking, the flow
20:49
part and that's what. So it's really.
20:51
We I'm working on this amount of flow pipelines and the based on
20:54
some pinpoints. We heard from data breaks
20:58
customers and especially around production izing machine
21:03
learning model and What we called the production hand off?
21:07
It's actually a mansion is really a collaboration between
21:10
the datasets team and the production Ingenuity and once
21:14
the data scientist figure out, oh this is a good way to train a
21:17
model, Rachel, let's ship it to production and then the
21:21
production team, look at the artifacts ratio the models, the
21:25
code and say, hey well this is not really the production
21:28
quality code and you cannot really ship a notebook to
21:32
production. And that's also a good debate
21:34
about Whether no books can be in production or not.
21:37
We are as you can, imagine. We've heard quite a few
21:39
conversations about that and slack.
21:41
It is one of those debatable things.
21:44
What I'm wondering though, is what have you seen that actually
21:47
works. Yes, it's then while I look at
21:50
this problem and see how we can solve the problem, just there to
21:56
kind of a Solutions with heard from customers.
21:59
And it's not really a solution. You just do what they do in
22:03
practice the first Version is, they will have their production
22:08
engineering team fully take over the project at the first time
22:13
this trip to production. Let me State took over the
22:15
notebook, they look at the code and they do the refactoring and
22:19
they make it motorized and from now on their own.
22:24
Ratio data science. Data centers are really
22:26
responsible for the experimentation part.
22:29
Once it's done once they gathered into a good model, they
22:33
hand it off. And the problem is you will run
22:37
into this machine learning engineering team to your days.
22:41
The machine in engineering team is not the traditional
22:43
production to software engineering team.
22:45
So we become the bottleneck because this not only more and
22:49
more kind of a machine learning project in production, they have
22:52
limited capacity. You're early within our
22:56
customers. I see they're usually more data
22:58
scientist their machine and Engineers, right?
23:01
So that means they cannot scale. Yeah and in this in this version
23:08
and then another version I I saw is the data scientist kind of
23:14
okay? Take the advice from the
23:15
production engineering team and the do the code refactoring of
23:19
their notebooks and make them regular motorized and using Some
23:23
IDs, they are not familiar with and then finally they can ship
23:26
it to production and there happened.
23:29
But production models never is just stay there forever, right
23:32
show. They get degraded performance
23:34
over time and once there is some issues, the production Engineers
23:38
come back to Data Center is here.
23:40
Your mother has a problem and try to try to improve it and now
23:44
wow, the data center look at. Should I go with the IDS and not
23:48
familiar is also that go back to the notebook and where I produce
23:52
the original model. And the ureter.
23:54
This was The Notebook because that's the truth I feel nervous
23:57
and then they do some iteration found out though, it was when
24:01
wrongs, for example, some feature need updates and then
24:05
they have to refactor this notebook again into motorized
24:09
called and she paid to production again.
24:12
So I heard complaints in this version is really the
24:15
productivity on the data. Center site is really bad and
24:19
the people complain about even just modify a single line of
24:23
code. And I have to spend a day or two
24:25
days, it's ratio to get into the production.
24:28
So that's the initial. The biggest pinpoint with heard
24:33
from customers and Emma flow. Pipelines is a punitive approach
24:38
to solve that problem and the solution here is really we try
24:44
to design workflow and like the data scientists to develop
24:49
models and the in this opinionated workflow. is that
24:54
whenever they find out the good model and we promise while this
24:59
project is ready to ship to production, they don't need a
25:02
refactoring and on the other production engineer side they
25:07
can just say ignore the acceleration notebooks and from
25:11
data centers, the only book characters and motorized code
25:14
and show I try to see Design This opinionated workflow to
25:18
combine the best of notebooks and ID is and in F02 to simplify
25:25
this collaboration, and the standardized on this production
25:30
hand of. So that's the the first pin
25:35
points. I'm a flop.
25:37
Has tried to solve is really required production.
25:40
Handoff in are a misogynist and ever, and I have a product, a
25:44
teachable hub, and if you are into a Hobson's, want to hear
25:47
more than just buzz words. Subscribe to the envelopes
25:50
Community podcast and listen to the best practitioners in the
25:53
field. Sharing their wisdom is
25:56
something that, of course, we hear all the time in the
26:00
community. It is something that is a story
26:03
that has been played over and over and probably a nightmare
26:07
for some, and I feel like there are a few companies that are
26:11
trying to attack it that way, they're really recognizing that
26:17
data scientists like that exploratory phase, and they like
26:21
to use their jupyter notebooks, but When you have the hand off,
26:26
you cannot bring that jupyter notebook into production because
26:30
it's just not, it's not nice. And I remember when about a
26:34
month ago or a month and a half ago, I was in Toronto for the mo
26:38
UPS world's conference, and I was talking to a data scientist
26:41
and he was saying, oh, my ideal scenario is just to be able to
26:46
have my notebook and throw it over the fence.
26:49
And then I don't want to know anything about it.
26:51
I don't want to know, I just want to know if If my model is
26:55
producing, what it said, it like the accuracy of the model and
26:59
that's it. And I was like, but what, like,
27:02
what are you gonna do? Like, how are the machine
27:05
learning Engineers, or the people who are production, izing
27:07
it? How are they supposed to
27:09
understand this model? He was, and he was like, well,
27:14
they have the jupyter notebook. So, it should be fine.
27:19
That's where I, yeah, you need to spend eternity.
27:22
Yeah. Yeah.
27:23
You To have that. So it's really it's cool that
27:26
you all are doing it this way and I like to see that and I
27:29
like to see the trend and the more companies are trying to
27:34
tackle this problem, right? And more teams, see the need for
27:40
it because of this like it's almost this inflexibility where
27:45
data scientists don't want to not use jupyter notebooks but
27:48
machine learning Engineers. Don't want to use jupyter
27:50
notebooks. And so you have to have a little
27:53
bit of both. 'The, I want to get into this
27:54
next section, which I know you've probably got a ton to
27:58
talk about while building ml flow while building.
28:02
Just while working at data bricks and going from spark,
28:06
like, 00 to spark. You know what it is now.
28:11
Do you have War Stories or as we're trying to Rebrand them and
28:16
they'll oops stories. What can you tell us about?
28:20
What are some huge ones that you have?
28:24
Mmm. Yeah, let me let me think.
28:30
Yeah, I think it's We built many good features ratio is a
28:38
remember a couple years ago. It's there are teams building
28:42
training features and building tracking features and the
28:44
building this hosted hosted products.
28:49
But I did some experiment is basically I say, just try to be
28:55
a user and now I want to get a model into production and then
28:59
let's see how hard it is, right show.
29:02
And I try to wear this step, I'm out engineer hat, right?
29:07
So, try to think about how I want to put a production of
29:11
security. And as my top priority rather
29:14
than just as a Data Center, and data, scientists have just a
29:17
productivity. So then I try to see, oh, wow,
29:22
where is the isolation story, right?
29:25
Show how I can make my production environment secure.
29:28
And that means I have to lock the data centers out, right?
29:33
So don't give them. Right access.
29:36
But after a lock them out, is how can they get models into
29:43
into this production, right? So do they just have through the
29:45
model artifact over the fence? And at that time is I only have
29:49
the model artifact, I cannot relate.
29:51
I don't know how to automatically update this model
29:54
ratio, whenever this model has an issue, I have to always go
29:57
back to the data centers. They say hey could you throw a
30:00
second version over Rachel? And then while I Realizing even
30:05
we do those individual components, which is great, but
30:09
things don't connect together, right?
30:10
So, it says, there are so many gaps in the in between and the
30:14
we have to also just as a build features and to simplifies those
30:21
those connections as well, right?
30:23
So at that time I do feel though, it's a before I did this
30:27
experiment. I feel what our product features
30:29
are good ratio as that's to improving, but after I did this,
30:34
And they said, oh, there's so many things to do and we should
30:37
start it now. So we did the internal kind of
30:42
pivot have team and the true just have focused on I'm Ops.
30:46
So after this experiment. Yeah, it's always, it's always
30:50
has to do something with access of some sort or some kind of,
30:54
you know, compliance thing where you kind of learn a little bit
30:57
about, you know, o is the system structure set up, right?
31:02
Or are we thinking about building this?
31:04
The right way, I actually have a question with respect to how you
31:09
design ml flow in an opinionated or muffled pipelines.
31:13
And opinionated way as you said, right?
31:15
And I think what I want to hone into here is like When we say
31:21
opinionated, it usually means that there's a disagreement,
31:25
right? There's one people some side
31:26
that wants to do one thing, one way and sometimes someone that
31:28
wants to get another right and I'm kind of curious like apart
31:31
from the jupyter notebooks question.
31:33
Can you give us another example where you guys had to decide?
31:36
Look, there's one way that we believe ml engineering should be
31:40
done in this context with respect to operationalizing and
31:42
there's a totally different way to do it, but we decide to go
31:45
come by. Yes.
31:47
Oh wow. When I see this problem, Is
31:50
start looking for existing Solutions because I don't
31:54
believe that I'm the first one to solve this problem, Rachel.
31:57
And then I start looking at tfx meta flow and those products
32:02
exist, many years ago, right? So especially tf-x is from
32:05
Google, is kind of bulletproof ratio for large-scale
32:10
production, use cases. And then I want to their
32:13
tutorial. It's a look at this.
32:17
Well, if I'm wearing the data centers hat, Is this is not
32:20
really something I want to use and because I have to construct
32:25
the dag myself, right? So a lot of boilerplate code if
32:29
you look at their their tutorial and then I try to try to
32:34
understand why. So why while just that this has
32:38
scientists have to do this. Oh, why does the product was
32:41
designed this way? And my answer is this is my
32:46
guest is tf-x was designed inside.
32:48
Google. And to solve Google scale of
32:51
problems ratio. They're targeting users are
32:54
Google's ml Engineers or we call full stack data centers and who
32:58
can do things and to it who follow the software engineering
33:01
practices. And then while tf-x makes sense,
33:05
ratio is a flexible tool, you can compose and the differently
33:09
and it can skill. But for the data scientist, we
33:14
see among our customers, and I don't feel that's the product
33:18
design for them. And similarly might have flow.
33:21
It's very similar is meta flow. You can easily construct the
33:25
steps and then construct the deck.
33:27
And, but at the end, is, you are asking a data scientist to
33:33
construct the entire pipeline ratio.
33:36
But we'll look at this and I look at Solutions in other space
33:41
and which one is the most interesting.
33:43
One is the the build tools. imagine that maybe many, many
33:47
years ago, how people compile software project, they may have
33:51
some best scripts ratio, which has GCC it, and then just you
33:56
run the best scripts, it will compile it, and the from
34:00
beginning and then give you the battery out and then it came
34:05
make Ratio. Because best groups is really
34:08
kind of a hard and everyone has their own best scripts it.
34:11
When you do exchange, they need to look at the best script to
34:14
understand what's inside and what's the logic but the make
34:17
standardized is you have the Target and you have see how you
34:22
produce that Target ratio and you can use make and then you
34:25
can use making style to install the software.
34:29
And is that good enough? It's a main case, something like
34:33
tf-x and Method flow ratio. You have those You can't Define
34:37
this the the dependencies and then it will help you to execute
34:42
efficiently. I look at, there's a stack
34:45
Overflow question about. So why just know Java developers
34:49
use make? And the answer is it because
34:54
there is an Apache Maven ratio. What Maven does is mavin.
34:58
Standardized are a couple of things.
35:01
It's one is the pom file ratio, the metadata about your projects
35:06
and it's portable exchangeable. Everyone understand this and you
35:09
can have dependencies declared inside this pom file and then it
35:14
has this standard build life cycle.
35:18
You have compiled, you have test, you have package, you have
35:21
published. Rachel.
35:24
If everyone follows it they don't really need to go to the
35:27
lower level make ratio and then it has this the standards
35:31
directory layout, right show cells means cells test where you
35:36
put your documentation and it's all standardized.
35:40
If you follow this opinionated way and you can get your project
35:45
build easily with maven. and then I'm thinking about how this
35:52
maps to machine learning, Rachel, is there any to that's
35:57
the best part of the machine learning Bill to and I don't see
36:02
any, I don't see any so then I look at the reason is maybe
36:06
because machine learning problems are unique are all
36:10
unique and you cannot really have a single build life cycle
36:13
to Define to capture all the machine learning projects.
36:16
But then I think it's adding or unique is it just says say there
36:20
is an 80/20 rule applied here racial make it K % of the
36:25
problems are just a simple problems.
36:28
There are classification regression forecasting, are they
36:32
fall naturally? Formed clusters ratio.
36:36
If there are clusters, can we apply a standard solution to
36:41
solve them? And then we can keep the data
36:45
scientist and in their comfortable ratio.
36:48
Think about feature engineering, and think about the modeling and
36:51
the reasoning about the output, the prediction accuracy.
36:54
Of the model. And but let me take care of the
36:58
rest, right? So maybe we can define those
37:01
about life cycle for typical machine learning problem, test
37:08
ratio in this emotionally, I'm a flow pipeline will now release.
37:12
The first, predefined, we call predefined pipeline templates,
37:16
and the first one is for regression, that means if you
37:21
have a simple regression problem, you can just say it,
37:24
Over this template. And we already have a lot of
37:28
things free implemented for you. For example, how to do the
37:31
splits, is it always a random split and is it always good?
37:36
It's not right. So if you have a new data coming
37:39
in and you always turn it over the data over the past one
37:44
month, You don't always want to do a random speeds because now
37:49
you will get the training data from the previous batch becoming
37:53
the test data in the current batch, right?
37:55
So you want to do a split based on the content for your records.
37:59
And but do you want data scientists to implement this
38:02
part that they don't, they don't really care, right?
38:05
So we can just have this built-in as a split step and the
38:09
user, don't need to write any code ratio, then they can fuck
38:11
himself. What features has formations.
38:14
They want to do what estimator is about to use.
38:17
Or even if they don't care about the estimators, they want to
38:20
use, they can just give it to all technical wrestling
38:23
Alzheimer automatically, shouldn't they just provide,
38:25
hey, well, this is my budget for different environments ratio
38:30
local development. I want a small budget able to
38:32
quickly test this into it but when I get into staging
38:35
production, I want to increase my budget and the to get the
38:39
better deterrent model and then we can offer Because we
38:44
predefined this tag, we know what's happening.
38:46
At even individual steps, we can offer those say, we call stamp
38:50
cards, right? So, we'll assume that the user
38:52
user notebook to execute those steps.
38:55
And if they do, for example, if they finish this, please step,
38:59
we're going to show this that distribution comparison between
39:03
the training validation test and make sure well, you don't have
39:07
huge outliers and that will rule, you know, just as I say
39:12
magic, Mm. Other evaluation in validation
39:14
test ratio after you run trillion and we're going to show
39:19
you a model Matrix and then we're also going to show for
39:22
example, what's the worst examples in this transaction?
39:27
And you can see them and you can see oh, what's missing.
39:31
You're modeling ratio. After you do evaluate, we're
39:34
going to show the featuring Partners automatically for you
39:38
and so basically their money things are barter plate code.
39:42
Yes. And but they're useful.
39:45
I don't want data scientist to rewrite them for all their
39:49
projects and we have them built in and we try to add more best
39:52
practices and by the it. So that's opinionated.
39:56
Yes, Rachel that's very opinionated.
39:59
I am thrilled that you took us through that because I think We
40:05
have on periodically open source contributors maintainers
40:10
creators of projects but this is one of the few times where I
40:13
really feel like I got a very deep understanding of how you
40:16
thought about not just the solution to the problem but how
40:20
you even understood that? This was a problem, right?
40:23
Going back to the tf-x comparison thinking about the
40:26
make File versus project Maven in a different coding context
40:31
entirely and what that means for machine learning engineering.
40:35
Thank you so much for sharing that.
40:37
What I want to do with our remaining time here very
40:41
quickly, this kind of get a sense of, you know, I want to go
40:44
back to where you started this conversation.
40:46
You mentioned that the data bricks Founders that you
40:50
interviewed with had a really Clear Vision of the future and
40:54
you're sitting here as a engineer that's been helping the
40:59
machine learning ecosystem, build for the last eight years.
41:01
I want understand like what's your vision for the future,
41:03
right? Like 10 years from now with
41:06
where we're at? No, with the adoption of cloud
41:08
computing. This is the real time.
41:10
All this is, where are we as data scientist?
41:12
And machine learning integers where the time elapsed Yet.
41:16
So, I would say I'm not laughs will fit a way to data
41:22
scientist. It's, it's like, like a
41:24
mentioned, is that how you compute me and the virus, our
41:29
mapreduce, right? So they used to learn, and then,
41:32
with the new kind of a product coming out, they don't have to
41:35
learn. So with I think it's a weight
41:39
with discuss a lot about whether data, scientists can own
41:43
entrance. That's from development to
41:45
Blackstone and to meet my preference, is that future will
41:50
never come. Is it will be the software
41:53
Engineers omission and Engineers owning production ratio.
41:56
But how, how much we can do is to enable more data, to just
42:02
enable data scientists, to do, more ratio rather than yourself
42:06
owning the entire production is contagious, allowed them to own
42:10
the Modal iterations, right show.
42:13
And can they just simply push your code change and get
42:16
everything just as a report back to them and the can we get
42:21
automatically model deployment and the through this offline and
42:25
online, evaluation Rachel, I think that's the future is data
42:30
sent his don't need to worry about what's inside the
42:33
production. Because the production system,
42:35
the MRP system was already provides all the guardrails and
42:41
the Help them to ship model into production.
42:46
However, I think is in order to get that future in, its I really
42:51
think it's data breaks. That's why while and thinking
42:54
about is the red platform to build that future.
42:58
Because, as I mentioned, is Emma, Ops is really the devops
43:02
mobile apps and data Ops. You need a platform that can
43:05
handle all of them. And for example, if you have a
43:09
modal deployed to production ratio and the production system,
43:13
if you don't want to bother page, S&S Whenever there is some
43:16
issue happening, right? So it won't the production
43:19
system needs to know how this model performs and how do we
43:24
know that it's, you need the labels, at least you need the
43:29
labels, right? So the labels may arrive late
43:31
than the the request ratio. That means you need to add some
43:36
data engine, to join the labels. And with the, with the original
43:41
feature and predictions and to monitor your model quality,
43:45
Leti. And then you also need to know
43:48
what's the downstream business impact ratio elect you have a
43:52
city, our model, but you actually care about the the
43:56
final conversion rate ratio damage, your system also need to
44:00
know the conversion rate, that means your ml system need to
44:03
know the conversion rate in order to keep your model
44:07
up-to-date. Yeah, that would require modal
44:11
machine learning and data together.
44:13
Yep, to get Sky things. Calling.
44:16
So yeah, I think that's how I think about our future is
44:19
basically product become very simple or em, Ops becomes much,
44:23
much simpler and data centers, can they don't need to learn
44:28
extra tools, right? So they just say stay with
44:33
within their expertise and the focusing on the modeling and how
44:37
to, how to use machine learning to power the business.
44:42
So I think that's the future we're building.
44:45
The fun part of the data scientists.
44:47
Get to do the fun part. Yes.
44:49
And what they enjoy doing and then everybody just in an ideal
44:54
world, everybody just gets to and do what they enjoy doing and
44:57
then they leave the other parts for the other people.
45:00
So, do you remember what we finish?
45:03
Yeah, exactly. I want to know what is a piece
45:06
of technology that you're bullish on.
45:08
That would surprise people. That would surprise people.
45:14
I think, what, how do we Define surprise?
45:17
It's a urinate. My observation is if I have a
45:21
technology, right? Show that able people to live up
45:27
what people can do, right? So, for example, if you can able
45:32
citizen data centers to do production engine machine
45:35
learning, so that would be a big surprise because it just says a
45:39
make the Product reachable to money ratio if you can make data
45:45
analyst to do the engineering ratio and that's a big surprise.
45:51
So but we hope well with the to Cersei the, our product of the
45:57
future version of product. We do, we can enable citizen
46:00
data centers. A lot of people who are new to
46:03
machine learning and the who don't even know machine learning
46:06
can can get models into production, So that's that's
46:12
that would be a big surprise to everyone and but that's also a
46:16
challenge. It's, we had a product similar
46:20
edges aren't along this direction.
46:22
It's a last year, I mainly work on this all Time Out product.
46:25
The card data works with ML and the key feature we have is
46:30
rather than just a return opaque modal and the from this
46:36
hypervisor tuning, back to the users will actually return.
46:40
A coat to reproduce that model training ratio because we
46:45
believe while it's not always, the ultimate can do the best for
46:50
you. Rachel, sometimes you have to
46:52
invite your own kind of domain knowledge of the business
46:56
problem and to further improve this model and get it into
46:59
production. So yeah, they're in this way
47:03
while we do have customers and who are data engineers and the
47:07
who use all time L and then while we With a few
47:11
modifications in the generated code and then they can already
47:15
get really good quality. Production models.
47:21
So that's along the way. But I don't think we are there
47:23
yet. There's there's too many, many
47:25
things to do. Nice.
47:29
I meant surprise people as in if someone were to ask you about a
47:34
piece of technology that you're bullish on and then you said, oh
47:39
I'm really interested in home automation.
47:42
They would go. What?
47:43
That is crazy. That's a surprise.
47:44
I would have never guessed that you are interested in home
47:47
automation. I think data you betting on what
47:51
you're doing at data bricks is probably an easy one that might
47:54
not surprise people, so, That you have any others that are
48:00
surprising like you're like, oh well this new very very small
48:05
project that is open sourced out of Australia is very exciting to
48:10
me or something like that. Mmm.
48:14
It Kearney just that, nothing comes out of my mind.
48:19
Okay, so that's right. When you read books, what do you
48:24
read? What's the last book, you read?
48:28
The last one is really this card.
48:30
A good strategy by strategy book is recommended by our CEO is.
48:35
It's about how we think about how we Define the strategy and
48:40
thinking about our strengths and weaknesses.
48:43
Rachel, how you get a good diagnosis of the market and then
48:49
see how you position your product.
48:51
So that that was the last book. I read, that's awesome.
48:56
That sounds like a really cool book.
48:57
I want to Into that right man. We're going to wrap it up here.
49:01
Thank you for coming on. It has been so incredible to get
49:04
to chat with you. I mean this has been a long time
49:06
coming. I know that we've wanted to do
49:08
this for a while. I've been a fan of your work and
49:10
everything that you've been doing it day or brick.
49:12
So it's almost a little bit like a dream to be able to have you
49:16
here, and I'm very thankful that you've given us the time.
49:20
Yeah, thank you for inviting me.