Latest / Agentic Conversations (formally mlops.community) / The Godfather Of MLOps // D. Sculley // MLOps Coffee Sessions #32

Transcript

The Godfather Of MLOps // D. Sculley // MLOps Coffee Sessions #32 · Agentic Conversations (formally mlops.community)

0:16
I've been excited for this one all week to tell you the truth.
0:21
I was just telling D that we jumped on here and before the
0:28
actual recording. Happened.
0:31
I was talking to my wife and I was like, wow, I get to
0:34
interview The Godfather of ml Ops himself today.
0:39
This is like the biggest let me just put this into a little bit
0:44
of perspective. But for most of us that are on
0:48
this call even realize that we wanted to be doing things around
0:53
machine learning or ml Ops D had already written the biggest
0:58
papers in the space. Ace so that tells you a little
1:02
bit about who we have on today D Scully.
1:06
Thank you for joining us. I really appreciate you taking
1:08
the time especially because you don't do this much.
1:13
And so that just makes it 10 times more and honor for us to
1:17
have the pleasure to of interviewing you.
1:20
Well, that's that's very kind and little bit embarrassing but
1:23
I'm really happy for the conversation.
1:26
Well, I think we can start it off with one thing that Is top
1:31
of mind for me and that is you've written these papers
1:35
which are very instrumental in the ml Ops World.
1:38
And for those who don't know the papers that I'm talking about
1:42
and you probably have even if you don't know the name dese
1:46
Kelly you've read his papers because these are the papers
1:49
that everyone gets told to go read when you start learning
1:52
about ml Ops. One of them is the high interest
1:58
technical debt of machine learning.
2:00
And then you have the other one which is the ml test score and
2:05
both of these have aged really nicely being that it's already
2:10
2021 you wrote them in 2015 and 2017 and they're still the top
2:18
papers that you need to go out and read.
2:20
So my question to start out with is what's changed since you
2:26
wrote those papers. Yeah, so first of all, thanks
2:30
for the kind words that you know, if the papers are still
2:33
relevant. That's that's really nice to
2:34
hear or maybe maybe that's good news.
2:38
Maybe it's bad news. I'm not sure.
2:42
So yeah, I mean, obviously the world has changed a lot in the
2:46
last four five six years in terms of machine learning the
2:50
importance of machine learning across the world.
2:52
The number of people who are doing machine learning the
2:54
number of folks who are working on MLS.
2:58
Send Em on infrastructure and things like that.
3:00
Like it's it's night and day, you know, in terms of the the
3:05
broader world compared to what we saw then I think some of the
3:10
things that are different in a number one.
3:13
There is an ml Ops Community. That's really good, you know in
3:18
many ways the papers that we put out, you know at that time.
3:23
We're sort of CO2 fighing as best.
3:25
We could the experiences that we've had, you know working.
3:28
As teams and groups within Google for many years about you
3:31
know, sort of what we were finding were, you know the
3:36
issues with machine learning systems that didn't appear in
3:38
traditional applications and it was a little bit hard to get
3:43
those noticed and attended to and really in the last, you know
3:47
several years, you know, whether it's because these papers are
3:49
because you know, folks have hit those problems themselves and
3:52
and are solving them in good ways.
3:55
I think there's much much more awareness now.
3:58
That these are fundamental issues that you know, having
4:04
system-level reliability is different than having model
4:10
accuracy that you know model accuracy is important and good
4:14
but the model accuracy by itself does not guarantee system-level
4:19
reliability. And so looking at the unique
4:22
role that that machine learning based systems have within
4:26
larger. Well larger production
4:29
pipelines. Yeah, that's something that
4:32
really is now I think and it doesn't need doesn't require me
4:35
to go around saying it a whole lot anymore like those folks
4:38
know they're working on it. There are lots of folks who are
4:41
now much more expert than I am and it's really neat to see
4:45
Jesse question on that. You mentioned that it like now
4:49
it seems like we like that's more obvious but actually feel
4:52
that people are still need to hear.
4:54
These old best practices things that Maybe we've been repeating
4:59
over and over. Why do you think that I guess
5:02
recently that's become more obvious.
5:04
Is it in light of more tools or more communities growing around
5:09
these sorts of problems, or is it just the research as well
5:11
that has changed and shifted the direction of things.
5:15
So I think first and foremost, they're just more people doing
5:17
it. And so, you know, when you have
5:18
the lived experience everyday becomes more a part of the
5:24
conversation whether it's folks in industry or wherever else And
5:28
I mean the fact that you know, there are jobs titled.
5:31
Ml Ops like this is this is different than it was about six
5:35
years ago. The fact that there are you know
5:38
podcast about envelopes like yeah that's different than it
5:41
was several years ago. So so, you know, these are
5:46
obviously important issues. They're obviously not something
5:48
where you're going to you know have the up Sapphire that will
5:52
you know be the one little thing that you had to system and solve
5:54
everything right. These are deep, you know
5:57
complex. Choose the many ways, you know,
6:00
you're never going to flat out solve.
6:02
You're really going to manage different ways.
6:03
And so that requires, you know judgment it requires making an
6:07
important trade offs. And so you need a community
6:10
around that one question I have is as as Demetrius set of the
6:17
top these papers are still classic just reading the high
6:20
interest credit card or paper this morning as a refresher and
6:24
it felt like you were talking to my team for some of the issues.
6:28
Is that you know, we still need to kind of think through in
6:30
terms of specifically, I think in that mention that paper or
6:33
the test score paper, you mentioned tight integration
6:35
between engineers and researchers on a team level is
6:38
really is really useful. I guess my kind of question to
6:41
you Demetrius Crassus what's changed?
6:46
Do you feel that machine learning systems are more easily
6:49
capable of being system-level reliable a system-level now than
6:52
before or still are we still in from your perspective?
6:55
So grappling with some of the same sort of educational just
6:58
that people need to understand how they design these systems is
7:02
a to I'm tools Advanced we feel like yes, like we absolutely
7:06
it's also Advanced, you know, I mean, you know back in the day,
7:12
you know, every pipeline was a one-off adventure and some type
7:16
lines were really well engineered, you know, or been
7:19
around for you know, some number of years, but but there wasn't
7:25
the level of General infrastructure that was looking
7:27
to To offer these sort of, you know, large-scale Integrated
7:30
Solutions that are maybe you know, I mean TurnKey is probably
7:34
too strong a word in my opinion, but there are you know, many
7:37
different platforms that are attempting this right now and
7:40
with the think reasonable success in a lot of cases, you
7:44
know, the idea that you could say go to the cloud and have a
7:46
production pipeline up and running that at least, you know
7:51
has reasonable ways of addressing many of the issues in
7:54
the detected paper or on the on the test scores.
7:58
I think the greatest house for sure.
8:00
So I'm gonna throw David under the bus right now because And
8:08
all I'll preface this with David has a huge fan of the ml test
8:13
score and I know he's like an advocate at his company for
8:17
using it but there have been some people that said they don't
8:24
quite agree with it and W can give more color around it, but
8:29
I'm wondering If you feel like what you were writing was for
8:35
the general public or was it your experience and you just
8:41
want to make sure that others knew that this is a thing and
8:47
David maybe you can jump in and tell us where people were given
8:50
you some slack on what the test score was and I think it was
8:56
around automating stuff, right? Yeah.
8:58
Yeah. Yeah, and it was it was Was well
9:01
received by the way just to preface that too.
9:03
But yeah, there's definitely some not concerns but some
9:07
caution because you know, we work in a drug Discovery Space.
9:10
So it's a little bit high stakes and when you're in that when
9:13
you're in that situation automating everything isn't
9:15
always the best. Sometimes it could actually be
9:17
it can actually be problematic and so for when I was what I did
9:22
was I decided to do a research project on one particular model,
9:26
which I looked at as like a mini system and I went through the
9:28
criteria and evaluated it used it.
9:30
Come up with an actionable recommendations and a lot of
9:33
them were focused towards let's say people that were in a
9:38
production system where maybe they needed updates in
9:40
real-time. Maybe they needed to have
9:43
automated triggers or events that trigger a retraining job.
9:47
So they were looking for you know an environment that is as
9:50
automated as possible. And I think that's actually a
9:53
good thing right because it reduces error in some cases.
9:56
It can actually catch it again and windows manual intervention.
9:59
Sometimes things get messy. But in a very, you know, I guess
10:03
high stakes in you know domain sometimes you need those manual
10:07
interventions to part like to actually curate the results.
10:10
Does this make sense from a scientific point of view and in
10:14
those situations, they found it a little bit more like more it's
10:16
like overhead to try to implement some of these things
10:19
and I think I've heard that elsewhere like trying to
10:20
automate things and make everything continuous
10:23
continuously integrated and continuously to put can actually
10:26
add more overhead to your system and make things a little bit
10:29
trickier, so you At need all that but that was just to enter
10:32
you know, what Demetrius was talking about that you was just
10:35
a little bit of push back on, you know, all the over
10:39
automating things and and trying to make everything just like
10:43
this in a fully automated system.
10:45
Yeah. I think that those whole you
10:47
know, really totally reasonable right?
10:48
I mean if you're in a drug Discovery domain, you know or
10:53
something like that, you know, it is unreasonable to have a
10:57
model that's just going to go off and discover drugs without
10:59
human intervention it right. Like that's not how the world
11:01
works. And what's what's useful in that
11:06
situation is that you'll have you know in the end the
11:10
downstream validation is going to be looking at one particular
11:12
drug and it's going to go through, you know, all kinds of
11:15
validation, you know, whether it's the FDA or whatever else
11:18
that's going to show whether that specific output.
11:21
You know, one one prediction essentially is good or not.
11:25
Right? The test score was really
11:27
written for situations where you don't care about one prediction
11:29
you care about Millions or billions of predictions per day.
11:33
And in that case, you're probably not going to go full
11:35
FDA on each one of those individual predictions.
11:38
I think that makes a lot of sense and that was one thing.
11:41
I actually really like though that we got from that is we
11:44
realized we needed to improve our coverage over at basic
11:47
things like the data Transformations the except what
11:50
what do you have in a schema on the inputs that go into our
11:53
model and that seems kind of obvious but when we went through
11:56
that I realized we're actually missing a lot of these things
11:59
and we could do better and that's That's what I really
12:01
loved about it. Is it actually by evaluating it?
12:04
I was actually able to see areas of improvement and come up with
12:08
some action items what I can actually do to pay that debt
12:11
down and improve the system to make it more robust, which what
12:14
I think the goal of it is right to measure the production
12:18
Readiness of the system, but I do agree makes if you have like
12:21
a system that has like millions of outputs in that there's all
12:23
these things going on. It's so much more important to
12:25
do things in an automated fashion, right because manually
12:28
keeping track of all those things just Get super messy now
12:32
that that's great to hear that. You know it just using the
12:34
checklist, you know, like in the end.
12:36
It's not so much about you know, if you don't score 17 out of 20
12:40
or whatever it is then the ml police are going to come and
12:42
take you somewhere. It's really about the process of
12:44
asking the questions making sure that each of the important
12:47
questions that you're asking yourself that you have a good
12:49
answer to and the good answer might be.
12:51
Oh, you know, we need to fix something or the good answer
12:53
might be actually we've done a great job here.
12:55
The good answer might be we thought about this careful and
12:57
it's not needed here all very small, but I think it's
13:00
important to us. Each of those questions and I'm
13:03
wondering since we are talking about the ml a test score a
13:06
little bit. Now.
13:09
Is there anything that you would add on to it that you feel like?
13:12
Okay, we've we've seen Parts significantly evolved in the
13:18
ecosystem. What else need is needed or do
13:21
you feel like it's still good? Let's see.
13:25
So I don't think there's anything that I would take away
13:28
from from the test. But yeah, the world has
13:34
certainly evolved. Yeah, there's some some
13:37
interesting pieces there one is if I remember right and I could
13:44
be wrong but I believe this is true that the an old test score
13:47
paper was one of the first papers that made a key point of
13:52
looking for inclusion. You know what we might think of
13:55
as fairness issues or bias issues and that you know models.
14:00
Should be explicitly tested for things of that sort.
14:04
And that was you know, like two paragraphs and an eight-page
14:07
paper. I think that it's fair to say
14:09
there's an awful lot more that you could say about that now and
14:13
many people have and they're they're much better, you know
14:16
sources for that kind of work than this paper, but certainly
14:20
expanding on issues of bias and representation, especially for
14:26
models that interact with data that deal with you.
14:30
There is are humans are people is super super important.
14:36
It's the other stuff. If I remember right and you
14:42
know, like you'll forgive me but like the I had to go back and
14:47
remember what was in these papers myself there.
14:49
So if I remember right we talked about stratified testing in that
14:54
paper where or sliced analysis where it's really useful to make
15:03
sure that you're not just looking at one Aggregates a
15:06
accuracy metric or goodness metric 98% Oh good.
15:11
But breaking that down into subcategories of the data and
15:16
what those subcategories are is going to depend problem to
15:18
problem. Maybe it's looking at
15:19
performance per country or you know background color or
15:23
whatever. It might be.
15:27
I still believe that that's a really important thing.
15:29
It can help you identify areas where maybe your model isn't
15:33
going to be performing as well. But we've also found that there
15:38
are other forms of domain shift that are useful to think about
15:43
this is what I mean by domain shift is you know, what happens
15:45
when the data that your model is deployed on out in the world is
15:50
different in some way than the model that was is trained on.
15:54
You know, this is the sort of gotcha.
15:57
Scenario that machine learning papers tend to sort of look past
16:00
a little let's assume that they're all IID and you know as
16:03
we all know in practice this sort of problem.
16:05
It happens every single day and every single model we've ever
16:07
heard of so putting together useful tests for these domains
16:14
shift problems. I think is really really
16:17
important and so two additional kinds of tests that I can think
16:20
of one is, you know, explicitly looking at shifted data where
16:28
You're finding some source of data that is a little bit
16:34
different than the data that was trained on.
16:36
You know, I'm okay data was trained on you know a stream
16:38
from North America. All right, let's let's try it on
16:40
data from South America or images came all from iPhones.
16:45
Okay, let's make sure that it we tested on Androids or in these
16:51
images, you know, all of the key, you know, objects of
16:55
Interest are nicely centered. You know, what If they're
16:57
slightly rotated, whatever it might be not again problems are
17:00
problems Pacific but assessing data on that especially models
17:05
on that Shifty data, I think is gives a much better sense of
17:09
robustness. That's super important for
17:12
deployed models. The second form of shift is what
17:18
we have called contrastive what might better be thought of as
17:21
like counterfactual testing. So this is done by men.
17:26
Keep you waiting little bits of individual samples to seeing how
17:32
performance changes. So this is done a lot in the in
17:34
all fairness literature. So things like you know what
17:37
happens if you have a sentence where the doctor did his thing
17:42
and switch it to her thing, you know, so it's a small
17:45
counterfactual intervention with the words in that example.
17:50
Hopefully, the model doesn't go crazy.
17:52
And if it does it'd be really good to know and you seen some
17:57
Things with there's a neat papers where you know folks will
18:00
do like Photoshop copy and paste it images onto different
18:03
backgrounds or things like that. So anything that we can do to
18:07
intervene on our data to create specific counterfactual examples
18:13
these allowance for like a nice example by example way of
18:17
analyzing where the edge cases for our models might be lets see
18:23
other stuff. I would add and I guess Dave is
18:26
going to tell me that the more stuff I add the more stuff you
18:28
can get Flack for so sorry about that.
18:31
I think well one last thing maybe two last things so I think
18:38
that it's it's often the case that we we don't push our models
18:43
to the limits of their capabilities when we're
18:45
verifying them and and finding, you know, like what are the edge
18:50
cases? What are the real boundaries,
18:51
you know, if a model gives good performance when predicting on
18:54
an image, that's 32 by 32 In terms of resolution what happens
18:58
if the resolution gets halved but happens if it gets cut to a
19:01
quarter what happens if it's just one pixel, you know for an
19:07
audio model what happens, you know, as we degrade the sound
19:10
more and more or lower the volume or whatever.
19:12
It might be, you know, really understanding what are the
19:15
boundaries of our models and you know where the performance
19:19
limits I guess the other major category to think about is You
19:27
know something that's happened between 2015 or you know, as we
19:31
call it now the Dark Ages and now is there's a lot more
19:36
deployment platforms. Then there were then and this is
19:42
true, you know, whether you're training in the cloud and
19:46
deploying on mobile devices or whether you know, like things
19:48
are getting shipped off to little stuff in the web or
19:52
whether you're training on CPUs and serving on fancy special.
19:57
Hardware whatever it might be and each of those paths is a
20:03
unique path that needs to be tested.
20:05
And so, you know as we get more and more options there that you
20:08
know, there comes a combinatorial explosion that
20:11
really does need need full integration testing.
20:13
I think that that you post Harvest regret you go first.
20:17
Yeah that piece of being across platforms across the plumbing
20:20
platforms, you know machine learning in the browser is
20:25
something that appears to be something that's popular.
20:27
ER now on device that's certainly a use case that I
20:30
actually work on in the cloud. You know one question.
20:33
I kind of have jumping off that point in particular is As an
20:39
engineering lead that's very familiar with building machine
20:41
learning systems for reliability.
20:43
How do you try and triage the technical debt that could occur
20:46
from all these sort of like cross-platform interactions?
20:49
What advice would you give to other people?
20:50
That might be in your shoes thinking about that kind of
20:52
cross-platform work? Yeah.
20:55
Let's just say there's a wonderful phrase.
20:57
I didn't come up with this phrase.
20:58
But it's Google internal around. When when should you test and
21:02
you the phrase was something like test until fear turns to
21:06
boredom and That's a good one in general.
21:11
We probably need to be testing more than we'd like to you know,
21:15
in terms of triage like yes absolutely start with the you
21:17
know, the quote unquote most important ones.
21:21
But you know, I think one thing that the last several years has
21:24
shown is that reliability is really really important.
21:28
There are no small mistakes and if there are errors they're
21:35
going to get spotted and they're going to get publicized.
21:38
And and they're going to you know, impact users lives.
21:42
So, you know, I think that the bar is really really high and
21:45
that you know, it's worth spending the effort to to ensure
21:49
really strong reliability. I think that I think that makes
21:52
a ton of sense. I think the key and doing any of
21:55
this triage or thinking about reliability understanding the
21:59
risks of what what it is you're trying to do and I think early
22:03
on your your papers about, you know, the test score and the
22:07
high interest credit card. We're really about pointing out.
22:09
Okay, these are the risks when you try to put machine learning
22:12
into production in it in a way that doesn't pay attention to
22:15
all the details and I think you're right in particular that
22:19
there's a risk now that your models failures are publicly
22:22
spotlighted a lot more or that they have more impact on a
22:25
larger scale because more people are interacting with machine
22:27
Learning Systems David. What do you think?
22:31
Yeah. I just I read your most recent
22:34
paper and under specification and I see that you have at you.
22:37
It's like you extended some Concepts in the ml test score
22:41
and I just wanted to talk about the stress test a little bit
22:44
more of which I thought was really cool because I did that
22:47
first time. I think I saw something along
22:49
those lines as there was this paper on NLP and they were
22:51
showing like counterfactuals and like the impact that that could
22:54
happen a model and I thought that was really good because I
22:57
haven't seen many tests that go that deep into what's actually
23:00
going on or what the model was learning or what the outputs of
23:03
the model are. How do you how did you go about,
23:06
you know coming up with stressed?
23:08
Because I find that that can be a bit difficult.
23:10
Like right like you said if I need to consider all these risks
23:13
there's they can be so many especially when your models to
23:16
put out in production, right you there's so many is even the risk
23:19
of people messing with your model right getting it to like
23:21
learn something and I shouldn't learn so there's like all these
23:24
issues. Like, how do you how do you how
23:27
do you get how do you build that into your stress test without
23:29
you know coming up with every combination?
23:31
That's possible. Yeah.
23:36
Let's see so that paper. The under specification paper,
23:40
you know, I think it had 37 39 authors something like that.
23:44
Right? There was it was a lot of others
23:47
that there was some mean that got put out like seven seven
23:50
thousand authors, you know, but one of these weather were so
23:54
many authors on that paper is because you know, the approach
23:58
that we took in that paper was to ask domain experts, you know,
24:01
we looked at, you know, whatever it was six or seven different
24:04
domain areas to sort of see if we saw the same phenomenon
24:07
occurring Each of those two main areas, but in each one, we
24:10
needed the domain experts to guide us in terms of like what
24:14
are the most Salient stress tests?
24:16
You know, what's what are the sort of you know, what you might
24:20
think of as a sort of causal structures that you'd want to
24:23
see preserved. So I don't think that we have a
24:27
full answer, you know, like, you know, as good of an expert is
24:30
really, you know, I think it's true, but it's a little bit
24:33
flip. I would like there to be a more
24:40
automated system. Sorry Zoom keeps close my
24:45
screen. Okay, I'd like there to be a
24:47
more automated system for coming up with stress tests.
24:54
I believe that this is going to look like a useful way to encode
25:00
expert knowledge about domain areas and there are probably
25:04
some you know. Some people probably have really
25:08
good ideas about, you know, the right way to take like a
25:10
knowledge graph or something like that and turn that into
25:12
into stress tests. I haven't done that yet.
25:16
But I think that that's probably the right direction to be
25:18
pushing. But yeah, you know like in the
25:22
end each of these issues as a problem specific issue and so,
25:26
you know, like it's difficult to sort of sit back and say, you
25:29
know always test for this kind of counterfactual that kind of
25:31
counterfactual but I do think you should least always ask the
25:33
question of like, what are the the right counterfactual
25:36
thinking about Yeah, I was just going to say even if there is
25:40
always some element of domain expertise that's required to
25:43
really evaluate this. There's still some general
25:45
things that apply like I looked at I was reading this book and
25:48
resilience engineering or KS engineering.
25:51
I've seen someone and I think they're really cool because this
25:53
is the idea of how do I bring robustness or how do I make my
25:55
system robust is something that's not just specific to
25:58
software like a lot of systems are trying to built in build in
26:01
robustness. And I think in machine learning
26:03
it's not you know, it's a girl so trying to figure out how do
26:06
we do that? Yeah, I still think it's a very
26:09
hard problem. Even when I'm just trying I was
26:11
trying to write tests for like a model.
26:12
I was just saying what do I do? Like it's like I always default
26:15
to the basic stuff like data types or the range of value
26:18
something that that's not really, you know, evaluating it
26:21
as it should be, you know, and that's where I find there's not
26:25
much help. I've seen like online when I try
26:27
to find it. How do I really make this model
26:29
robust? I'm it's something that I'm
26:30
learning right now and it sounds like we're there is definitely
26:33
some room for improvement in that area.
26:35
Yeah. I mean in the End the reason why
26:39
it's a hard problem isn't because you know people aren't
26:41
really smart people are really smart.
26:43
But the issue is that you know, we use machine learning to solve
26:49
problems that we don't know how to solve with traditional code.
26:54
You know, that's why we have them it's it's because we don't
26:59
know how to specify the behavior in advance and because we don't
27:03
know how to specify the behavior of advanced testing that we have
27:06
the Here that we wanted to you know have is a you know, a
27:12
fundamentally hard problem. It's not just one where people
27:14
haven't thought about enough, you know for for our listeners.
27:17
I think a lot of people have had the chance to read the credit
27:21
card debt and and read the test score papers under specification
27:26
paper, you know, we read it and it was great.
27:28
I would love to know when your words, you know, maybe how you
27:31
would you know, what the lesson you took away from the
27:35
experience of writing that paper is and and And you know, I need
27:38
them to specification paper. Yeah, the under specification
27:40
tape is like how has it impacted your thinking about engineering
27:43
and all systems now. Yeah, so for folks who haven't
27:47
read that paper, you know for me the key takeaway was let's see,
27:53
so For a long time people, you know worried about you and
27:58
especially being back in the Dark Ages when we went from sort
28:01
of, you know models that were trained with convex, you know
28:05
loss functions to you know, all bets are off deep learning, you
28:10
know, non-convex blah blah blah. There's a lot of worry about
28:14
like well wait a minute are the results going to be reproducible
28:17
things like that and empirically what's happened.
28:21
Is that Folks have found by and large the results on I I detest
28:28
train splits for deep learning models tend to be really pretty
28:31
repeatable. I mean not exactly not bite for
28:34
bite, you know not to six decimal places for each
28:38
prediction, but really pretty pretty repeatable and what this
28:44
paper looked at was what happens when you do an evaluation of
28:48
repeatability in training deep Networks.
28:53
And you look not at an IID test range.
28:55
But but you look at one of these stress tests that's explicitly
28:59
from a slightly different distribution.
29:01
And their what we found was that actually a lot of that
29:05
repeatability went away when we look at out of distribution
29:09
performance. And so this was just with like
29:12
the the minimal possible changes that you could have from model
29:16
train of only training. It was just, you know, varying
29:17
the random seed that was used to start the models everything else
29:20
remained constant. We saw you know, pretty large
29:24
variation, you know amongst different versions of the model
29:28
where each version was trained with a different random seed on
29:32
things like, you know, perturbed versions of imagenet or
29:36
interesting genomics use cases or stuff like that.
29:41
And the the range of that variation was was often larger
29:45
than the range of variation between say a good model and a
29:48
not-so-good model and like some ml competition on that task so
29:53
So, you know qualitatively quite large.
29:56
Now for me that this is, you know, a really interesting and
30:00
be a little bit worrying because if a given random seed changes
30:06
your performance on out of distribution data.
30:11
And we believe that out of the distribution data is something
30:13
that we encounter every single day in the real world.
30:17
Exactly. It's so it's so messy.
30:21
Yeah, then then we maybe do have some issues on repeatability
30:25
there. So I think that there are some
30:28
reasonable Solutions, you know in one of our other recent
30:31
papers, you know, we found that model ensembles like just taking
30:38
and copies of the same model but maybe with just with different
30:41
random seeds and installing them together gives a lot of
30:44
robustness in these dimensional shift settings.
30:47
Like obviously there's a Additional cost to be paid in
30:52
terms of both inference time and in terms of system complexity,
30:56
but there is there does seem to be some really nice robustness
31:00
benefits that are consistent across domains and across
31:06
application areas. So that's one takeaway, but my
31:10
main takeaway from this is just that you know, well, it really
31:14
is important to be evaluating models on these out of domain
31:18
stress tests and to make sure that we understand the oceans of
31:21
of what we're thinking about when we're in deployment lent.
31:26
I was just going to say yeah, that's that's definitely the
31:28
case right that real-life data is a lot messier than we train
31:30
them. But I one thing I just wanted to
31:32
ask a little bit deeper you mentioned if I recall correctly
31:34
in the paper that you're not just referring to like
31:39
overfitting right? Like you're you're you're you're
31:41
you're getting data that's like your model is it's not it can't
31:45
generalize to this is something a little bit different right
31:47
you're referring to if I just change like the random seed like
31:51
It's from 1 2 3 2 1 2 for my results vary so much when
31:55
they're really shouldn't make much of a difference right like
31:57
the random seed should be something that I don't know.
32:00
Maybe it's like a parameter to but often that's not something I
32:02
would like to in a hyper parameter tuning super like
32:07
that. But you know, but I guess the
32:09
paper shows that that had an impact and like you said that's
32:12
worrying because the real life scenario is your data is not ID
32:16
independent and identically distributed right?
32:18
It's messy it's full of all these other things.
32:21
And also now this is other element that I've been thinking
32:23
about more is like someone can mess with it.
32:25
Right like someone could actually like mess up my model.
32:27
So it starts learning all these bad things.
32:30
So now I don't know I just feel like there's so many interesting
32:32
things to talk about but I want to I wanted to kind of shift
32:35
towards I guess the future I want to just hear your thoughts
32:38
and what you think about ml Ops and the space and where that's
32:41
heading. So one question I have is what
32:43
do you see as being some of the principal challenges that we
32:46
have right now in production eyes in machine learning and in
32:49
particular, what are those challenges?
32:51
In comparison to 2015 when you first started writing some of
32:54
these papers. Are you mentioned a little bit
32:56
of this earlier? But yeah, just what are your
32:58
thoughts on that? Where is this where we headed?
33:00
Yeah, super interesting stuff. So, you know, I will say, you
33:03
know, I like I used to work in this field a lot and you know,
33:09
I'm certainly happy to give my thoughts but there are likely
33:12
people who are more in the day to day to day of it right now
33:15
than I am who will have better thoughts than I do.
33:18
But so with that said looking forward to its one of the very
33:25
obvious trends of the last couple of years has been that
33:28
models keep getting bigger and open AI has these really nice
33:32
papers on like, you know, scaling laws of large language
33:34
models and stuff like that. But basically, you know show a
33:40
exponentially increasing model size over time and that carries
33:48
with it a couple of interesting challenges, you know, I think
33:50
it's very reasonable to assume that models are going to keep
33:52
getting bigger until they just Can't for you know, whatever
33:56
reason we're like, you know just resources become inconceivable
34:02
for larger models. So they're going to keep getting
34:04
bigger for a while in my in my opinion.
34:08
What's interesting about scale and scaling up is sort of like
34:12
what why is this happening and the reason and in my read of the
34:19
situation, is that basically you know as we get more and more
34:24
data and more and more parameters for the most of the
34:26
bottles, it's not that we're doing a better and better job of
34:30
learning on the common data, right like the common data, you
34:33
know, like the typical data, you know getting more and more of
34:36
that isn't that much more useful than just taking many more
34:40
gradient steps on the same basis that they already have instead
34:44
as we get larger and larger datasets what we get is more and
34:46
more exploration of the fringes and sort of rare edge cases and
34:53
A typical data, you know, so like you can think of gathering
34:56
a huge data set as a slightly inefficient way to gather, you
35:01
know, smaller data sets that are sprinkled all around the
35:04
interesting areas of the day space that you haven't already
35:06
covered. What is interesting from an MLs
35:10
perspective. Well, it's because as we expose
35:14
our models to more and more Educators or try and you know,
35:17
cover more and more specifics, I think that this means that there
35:21
are more and more potential places Has where our models
35:24
might be exhibiting unexpected or undesirable behavior.
35:28
And so our need for stress testing.
35:30
I think also grows with the the scale of the data that we're
35:35
talking about. And if you look at those graphs
35:42
and you know, look at that log linear scale over or whatever it
35:45
is, like, you know, it's certainly something to think
35:50
about, you know for MLS perspective.
35:52
Maybe that's great. Maybe it's job security.
35:54
I don't know but it's certainly something that's that's going to
35:58
require a lot of attention as models keep getting larger and
36:00
larger. See other things that are
36:03
interesting I think Federated learning is super interesting
36:06
from now in the labs perspective.
36:08
So Federated learning, you know, this is the area where it's a
36:13
great privacy-preserving mechanism, you know, you could
36:15
imagine data is stored on, you know, lots of devices, you know,
36:19
maybe people's phones or whatever.
36:22
We don't want to take the data off the device, but we're able
36:25
to sort of do this very very distributed Federated training
36:29
where model training is done sort of little bits at a time.
36:32
I'm on the individual devices the data never leaves each
36:34
device and sort of anonymized very highly private highly
36:38
secure updates get sent back back and forth.
36:42
It's fascinating. It's fantastic.
36:43
I have no idea how to make that operational from a nmls
36:49
perspective. What is what does
36:51
reproducibility in that situation look like what does
36:56
data testing look like in that regime?
36:59
I'm not working on Federated learning.
37:00
There are people who are really smart and Are working on that
37:02
they probably have great answers, but just in terms of
37:05
like thinking of the future, I think that you know, the ml Ops
37:08
Community would do well to focus on that a little bit at least
37:13
we've talked a little bit about the proliferation of devices.
37:17
I think that there's just going to be more and Yeah, I really
37:24
hope that we get more and better abstractions that help us deal
37:29
with that over time. But but for the time being
37:31
there's does seem to be a lot of the classic, you know, right
37:34
once debug everywhere kind of framework that we've often seen
37:39
and similar situations like, you know web pages in the 1990s or
37:43
whatever. They might have been.
37:46
Let's see. How about you were with what are
37:50
your thoughts you guys doing this every day?
37:55
You've definitely given us a difference is going to be honest
37:57
a lot to talk about. You know, I think at least from
37:58
my perspective my experience as a machine learning engineering a
38:04
startup company has really been realizing that a lot of the
38:07
problem is defining the problem more so than it is writing the
38:11
models even writing software right?
38:13
It's like, okay, we work on retinal images, which is
38:16
actually one of the interesting things that you guys talked
38:18
about in your understood. Under specification paper and a
38:21
lot of my time is spent. Well, what are the ways an image
38:23
can be bad? How do I go gather and
38:25
understanding of that and think about what that means
38:27
algorithmically and also just even from The annotation
38:30
standpoint. How do I make sure that my
38:32
annotations reliable? It's actually it is for the
38:36
machine learning engineer. It's almost like a lot more
38:39
meaningful to spend time thinking about your labels and
38:40
making sure your labels are have that sanctity than it is to
38:43
actually spend writing tons and tons of lines of code.
38:45
Oh that is so true. I mean, you know like the amount
38:48
of bang for your buck that you Get you know, either a spending
38:51
your time like trying to think of a slightly different
38:53
mathematical formulation for your model or be spending time
38:58
making your Raider guidelines more specific like I mean being
39:02
wins every day, right? Yeah, that's exactly right.
39:06
And I think the nice thing to see is that there's interesting
39:08
tooling that's coming out for that, you know companies like
39:10
Label Box and you know, I think there's one called a superb that
39:14
AI they're proposing new ways of basically making the whole
39:17
process more efficient based on some Stretch that's coming out
39:21
and I think that could be really promising from animal of stamp
39:23
because really if it's a lot easier to make if it makes my
39:25
life easier when it comes to annotations and you know getting
39:28
fewer and fishes with being able to have a higher level of
39:30
confidence in them. That's mean that's that's the
39:33
Holy Grail right for any kind of assembling any kind of data set.
39:37
What about you David? So I'm like my interests are
39:41
more in the in for side of things thinking about like you
39:43
mentioned like the models are getting larger and larger.
39:45
How do we have sufficient, you know inference and I think that
39:50
problem still needs some work. Like I look at them as some of
39:52
the things that I would love to work on like a better scheduler
39:57
the ability to share resources more efficiently and hopefully
40:01
abstractions that make it a less like I don't know like using a
40:05
spit like specialized Hardware isn't always trivial.
40:08
Sometimes you have to make sure you have all of these drivers.
40:10
They're just there's a lot of things that make it tricky.
40:13
I would hope that that gets better and I do think that that
40:17
that's that's so important. You know, it's Infrastructure is
40:20
what everything runs on and when that's not working.
40:21
Well, everything gets slowed down.
40:23
So I think that that's something that we all realize it's just
40:26
not talked about much but there is definitely some cool work
40:29
being done on improving that and I would love to see more of
40:32
that. I guess I envisioned that being
40:34
also pretty but another thing you kind of are talking about
40:37
was like security. There's this I think it's called
40:40
multi-party Computing or secure multi-party Computing which
40:47
seems similar to Federated. Ting but like you said, how do
40:50
we actually do that at scale like how it how do we actually
40:53
make that you know doable and I think that you know spending
40:56
more time understanding that from an Mo perspective is going
40:59
to be what we're going to invest more in more time again, also
41:02
the I gay I'm saying like the same answers but the data it's
41:05
always about the data labels in particular is also really
41:08
interesting area finding ways to do that in an automated way.
41:11
So I definitely see more more focus on you know, actually in
41:16
like making some of these these jobs feasible from a Including
41:19
point of view that we can do it where our resources are not
41:22
spending so much money that's yet.
41:24
So I'm thinking that the GPU sharing a way to make the most
41:26
out of what we have and then definitely security making some
41:29
of these secure methods more mainstream or more popular
41:34
because I see them as kind of small little areas of research
41:38
rather than something that everyone's doing in production.
41:40
Yeah. I'm so maybe there's ways to
41:41
make that simpler make that more available.
41:44
These are things that I can and as an m/l s/m and m/l engineer.
41:47
I didn't really worry about too much but now now security is
41:49
becoming so much more important fairness and bias and addressing
41:52
some of these issues that it's less model-specific less about
41:55
the algorithm and more about everything else that's going in
41:58
this environment. It's going into the group.
42:01
That's something I'm glad we're on the same page David because I
42:07
was going to mention security to and talking about also
42:12
everything that you mentioned, but then you said it beforehand
42:15
like when people start playing with your models and people Are
42:20
trying to train them in a nefarious way.
42:23
Like, how do you prepare for that?
42:25
What are you doing in those cases?
42:27
And and is this the job of the ml Ops or the machine learning
42:33
engineer the infrastructure team?
42:36
Like the only it's not my job to find.
42:38
Yeah. Exactly.
42:40
Exactly. I don't think I buy that one.
42:44
That's that's exactly it like well, is it is it is there a new
42:50
I mean Vishnu and I have talked about this quite at length of
42:53
the like splintering of the ml engineer.
42:56
And how is there going to be like you've seen devops go into
43:00
Dev set cops, right? And is there going to be a ml?
43:04
I don't even know how we could put it.
43:06
Ml SEC cops come about in the future too.
43:10
So that's something that I just wanted to tack on to the end of
43:14
what you were saying, but D. I'd love to hear your thoughts
43:18
on all of Hey, I think all of these things are you know, very
43:26
very plausible very likely. I was looking back.
43:33
I got to read the section that paper this morning and try to
43:37
remember what was in it. So, you know the old figure one
43:41
there that's like the figure with the little you know little
43:45
bit of machine learning code in the middle and then like the
43:47
different system pieces on the outside and that's a paper that
43:50
I've seen that's a figure that I've seen crop up and allegedly,
43:53
you know a wide variety of corners of the internet.
43:55
So that's been pretty fun. But looking at that figure
44:01
again. what I noticed in terms like what's missing from that
44:07
there's likely, you know, some pieces that are missing in terms
44:12
of there's a I think there was a box for data validation, but the
44:17
sort of like the full-on security and other words spam
44:19
filtering or like, you know, kicking Bad actors out of your
44:22
feedback pipeline or whatever don't like that's a really big
44:25
deal in a lot of cases and then on the output side You know,
44:31
it's often the case that you know, for example, like for a
44:36
lot of models are used in different production settings.
44:38
Like you don't necessarily want to take the raw output of a
44:41
model and feed it into your production setting unfiltered.
44:47
And we've all seen you know, high-profile cases of you know,
44:50
like chat Bots or whatever else like maybe needing some
44:52
filtering or stuff like that. Right?
44:56
And you know, I think that those kinds of pre and post, you know,
45:04
parts of the system are really important.
45:07
They're also places where you could absolutely a crew, you
45:10
know, a fair bit of technical debt and your system or like
45:13
something that gets like really hard to change your gets sort of
45:15
like In two particular version of a model or things like that
45:20
and looking forward to think that those are areas that could
45:22
have, you know, like whole communities around themselves in
45:25
terms of like best practices and infrastructure development all
45:28
the rest of it when Dad won't question also is a father that
45:31
like sort of like how in your opinion you've seen a bunch of
45:35
communities come up, you know, go away.
45:37
How does that affect this like the space like the community is
45:40
that you know, I don't know me. I haven't been around long
45:43
enough to think of other communities that have Like
45:46
accelerated the development of things or maybe I don't know
45:49
maybe not a made it worse. But I know I feel like
45:52
communities do have they play an important role like a look at a
45:55
project that cute flow or maybe one of these other ones that
45:57
open source where if there is a lot of people actively working
46:00
on it actually makes the tool more likely to be adopted by
46:03
another team. I know when we were evaluating
46:05
our infrastructure in which willing to you is that was a big
46:07
big thing. Like is there a community around
46:09
this that if we have a question we can ask is there are there
46:12
people that are actively working on this and it started going
46:15
back to Mike? Question is how does that shape
46:18
the development of the MLF sphere I guess in this like and
46:21
maybe if it doesn't have to be able out some other community
46:24
that you seen and maybe how that affects that on a research or
46:27
whatever area. You've seen.
46:29
Yeah, I so just like, you know having dramatically more
46:33
training data gives you more information at the edges.
46:38
I think that having a larger Community, you know, it's it's
46:41
one way of, you know, having better faith. if that you are
46:46
getting stress tests in a wide variety of important education
46:49
has you know where those stress tests are probably, you know
46:52
from different, you know, domains or different companies
46:56
whatever dance but it's much better to have one large
47:00
community than three smaller communities because of those
47:02
Edge effects and and sort of the ability to learn from people who
47:07
are doing something that's a little bit, you know different
47:10
than what you might be doing or what might be standard practice
47:14
at the current time that A ton of sense you're very talented at
47:18
analogies. You should have heard his was
47:22
sorry to butt in Vishnu but guys it recording with software
47:26
engineer daily and he did a spamming one in the north.
47:30
What was it Norway Norway and Sweden about spam as in and you
47:36
probably don't even remember doing this.
47:37
You probably came up with it off the cuff, but it was for me.
47:41
I was laughing. Like how did he just come up
47:44
with that analogy? and this example, so anyway, go
47:48
ahead Vishnu start over to know I'll have to have to go back and
47:51
listen to that one specifically then for that analogy and my
47:57
question kind of is it's almost about idea communication and how
48:00
you how you think about that right this concept of technical
48:05
debt in machine learning, you know, I see that the 2015 paper
48:10
was you know, one of the first time I think really really
48:12
pushing that term really helping people understand that this is
48:15
the concept You need to keep in mind.
48:18
As you sort of hear about perhaps this new idea like under
48:22
specification and stress tests. How do you think about or how do
48:25
you go about the process of like maybe formalizing this idea and
48:29
then communicating it to the people that you work with?
48:31
I'm just curious how you think about it.
48:32
Maybe it's organic. Maybe it's a little bit more,
48:36
you know structured as the way I've maybe I'm applying too much
48:38
structure to it. But I'm curious how you think
48:39
about communicating some of the ideas that you have about how to
48:42
engineer systems properly. Yeah.
48:44
I mean, I do think it's one of the reasons to write papers and
48:47
publish papers, you know, I I think it's a really good
48:49
practice, you know the for myself the best thing about
48:53
writing papers is to like actually try and figure out what
48:58
I think or what what is true to the best the best that I can or
49:03
we can I think that to First approximation science is what
49:09
happens when you write stuff down and without writing it's
49:15
very very difficult to really have something that's so so that
49:18
they can be put up for debate or for inspection can be disagreed
49:22
with can be critiqued. So that's step one.
49:27
You know. Well once Things are Written
49:32
then you know, there's a little bit of publication and things
49:34
like that. You know, I imagine that you
49:39
guys are probably better at that than I am but You know, I think
49:44
that good ideas tend to make their way into the community if
49:48
they're in a form that people can can digest and share.
49:51
So, you know, I don't think that you know, we need to like Rangel
49:56
our way into 60 minutes to talk about Emma laughs or technical
49:59
credit. I think that you know, making
50:01
sure that your number one there are good papers out there number
50:04
two where we can create useful pieces of infrastructure whether
50:10
it's an open source or other shareable or usable things.
50:14
That can help a lot for adoption.
50:15
If we have some best practices that can be adopted that way and
50:19
then things like this if just having conversations and talk
50:22
with people I think is really really useful.
50:24
Are we going to see a book anytime soon now from you know
50:28
book I don't think that's very likely for me.
50:35
Well amazing sitting down or Zoom or whatever.
50:40
We're doing here across the internet.
50:43
Set and talking with you. I really thank you for taking
50:46
the time to do this because I know that it is very valuable
50:53
for everyone that is out there listening.
50:55
And if anyone wants to get in touch with you, I think we can
51:00
direct them to your website which is incredible website to
51:05
will leave it in the links below.
51:07
There is an FAQ section and this is how I knew it was going to be
51:11
great chat because I think the FAQ one of the Students in the
51:14
FAQ section was is your name really D and the answer was yes
51:18
it is. That might be the only affect
51:22
you. Thank you.
51:24
I really appreciate the chat and it's great to hear about your
51:28
Visa work. So thanks so much.
51:29
Yeah. Thanks again D and we will see
51:32
everyone later. Yeah, these are guys.