Latest / Agentic Conversations (formally mlops.community) / Operationalizing Machine Learning at Scale // Christopher Bergh // MLOps Meetup #64

Transcript

Operationalizing Machine Learning at Scale // Christopher Bergh // MLOps Meetup #64 · Agentic Conversations (formally mlops.community)

0:01
Chris Berg is the CEO and head deta chef, at the dated kitchen,
0:13
Beatrice. I'm not singing with you.
0:15
All right, he's got more than 25 years of really 3030 and
0:27
software engineer. Experience at various points in
0:40
his career. He's been a CEO.
0:45
A CTO, VPN director of engineering.
0:53
Yeah, Chris Berg's gonna die. To us today about ml arms and
1:07
the day tops Manifesto. Oh, welcome everyone.
1:15
We've got another Christopher. The see what they do.
1:31
And there's your intro. Alright.
1:34
Alright there we go. That's what that's why people
1:38
got to show up on time. If they want the good intros,
1:43
there it is. Okay, so now that I've
1:46
officially embarrassed myself. No.
1:49
You're like at towards the end it was really you're really in a
1:52
Groove. I was jamming man Jim and
1:57
they're all set up and everything again.
2:03
Welcome, everyone that is with us hopefully you caught that
2:06
intro. If you did not, we'll put it on
2:09
YouTube for later. So now this is a few quick
2:15
announcements for those that are just jumping in.
2:17
If you're not in our ml, Ops Community slack, I encourage you
2:21
to jump in it. I will leave a link to that in
2:24
the chat or description if you're watching in the future.
2:29
Also I realized that people that are watching in the future.
2:33
I know this doesn't apply to anyone that is with us today,
2:35
but people that are watching in the future, YouTube told me that
2:38
half of you are not subscribed. So subscribe and hit that like
2:43
button so we can blow up and make this video go viral,
2:47
especially the intro part there and the last announcement that I
2:53
will make before. Or we jump into the conversation
2:55
with Chris is that we are partnering.
2:58
The MLS Community is partnering with the ml Ops World
3:02
Conference, that is coming up in June.
3:05
They've got some really cool stuff going on and I would love
3:09
for you to be a part of it. So again, I'm going to throw a
3:12
link to that conference in the chat and or description and I
3:16
encourage you all to check it out.
3:18
Join the conference, it's going to be really exciting and
3:21
there's all kinds of cool stuff going on.
3:23
On I may or may not play the guitar in that 12 to introduce a
3:28
few people. I will be emceeing for a few of
3:31
the talks so you never know. I mean the guitar is just right
3:34
here, it's at arm's length and if somebody says that you I
3:38
don't need a lot of provoking as you can tell earlier if I think
3:41
it would be awesome. Everyone deserves a guitar
3:43
introduction once in their life. That's it.
3:48
Excellent. So now let's start this off,
3:51
Chris, as I mentioned you are Of data kitchen and what you like
3:58
to call the head, chef. I know you all play with the
4:01
kitchen metaphors, quite a bit, and the chef, and you're the one
4:06
in the kitchen cooking up, the good stuff.
4:08
I would love to know the journey to data kitchen.
4:11
I know it's been around for a while.
4:14
I also know that we have a pending conversation for another
4:17
time about how you've bootstrapped, your own ml Ops
4:22
company. But maybe you could, Tell us
4:24
about this journey that you've taken.
4:28
I mean, you have over 30 years of experience.
4:30
You said so give us the cliff notes version of that, you don't
4:34
want that, I could talk for an hour straight.
4:36
You don't my life story. So you know, I'm an American
4:43
right? Grew up in central part of the
4:45
u.s. Wisconsin working-class kid work
4:47
my way through college. I after college, I taught in the
4:51
Peace Corps in Africa for two years and Went to graduate
4:54
school and like fell in love with AI.
4:56
And this was like 88 and 89. This was sort of deep and AI
5:00
winter was like, AI early January.
5:02
And the winds are blowing and was told, and here's an example,
5:05
my machine learning class had six people and, and if you go to
5:09
like a machine learning class, now that there's like 600 people
5:12
in and let, yeah, everyone wants to take it and and so I was like
5:16
super, in love with AI and I was like, this is this is awesome
5:19
and I wanted to work in. And so I worked for five years
5:21
at Nasa to automate Air traffic control to like do intelligence
5:26
sequencing and spacing advice for landing aircraft in.
5:29
So like really cool but I kind of fell in love out of love,
5:34
with AI, for a walk because it's like really hard to actually
5:36
make it work and like a lot of parameter tweaking and I started
5:40
to do software and startups and one got acquired by Microsoft.
5:44
And then, you know, I got married had kids and then about
5:46
2005, I thought, well, I'm a big software guy, right?
5:49
I've done some me, I done some software.
5:51
I'm going to do this data and analytics thing and they'll
5:53
Easy, like, I'll be home by 5:00, no problem, and like, my
5:57
life really sucked. Like it was bad because like,
6:01
yeah, I'd really nice people working for me.
6:02
You know what, we call now data, engineers and data, scientists
6:06
people who did this, and we had thousands of people using our
6:10
models and analytics and like, man, my life sucked.
6:13
People were calling me up saying you're late, but date is wrong.
6:17
The bottles off Chris and I'd have to placate them and talk to
6:20
them and I just, yeah, I'm I'm an engineering.
6:23
Nerd. I don't like conflict, right?
6:25
And you get like a good-looking sales.
6:27
VP from one of the top Pharma companies, in the world saying,
6:31
you deliver something wrong to my 5000 sales reps.
6:34
It's not fun, like it just like that, a fun experience.
6:39
And then with you and I were right, just interrupt, was this.
6:42
You being like a consultant? No, I worked at.
6:45
So I worked at a company that did sort of data science and
6:48
analytics for healthcare and it specifically Pharma companies
6:51
and I was the CEO of so under I people who were, you know, kind
6:57
of doing the work, but I was kind of the guy who is Going to
7:01
make the trains, run on time, right?
7:02
And so, and I worked for a doctor, I was really smart guy.
7:07
I knew a lot about health care, but didn't sort of know a lot
7:10
about the details of how things worked the technical side, and
7:14
he go off and say, okay, we're gonna do this cool thing with
7:16
data and he'd bring it back to me.
7:18
And I'd sit in a room with a couple people and we think of
7:21
some things and I go, hey, David, this is going to take us
7:25
two weeks to do and he looked at me like I killed a patient on
7:29
the table going. Yes, I thought that should take
7:32
two hours and not two weeks to do.
7:35
I'm like, so my life just generally kind of sucked for
7:37
years. Like things were breaking, we
7:41
couldn't go fast enough and we had hired a bunch of smart
7:43
people and everyone like they're two different tools.
7:46
You know, we had the, our people and the python people and the
7:48
Tableau people in The Click people in the visual data visual
7:54
data engineering people, and the people like SQL.
7:56
And so, you know, when we started the company, Sort of 8
8:01
years ago. I realize that that problem is
8:05
the problem I had was sort of generalizable to everyone and,
8:09
you know, I guess I'm getting up there in years and I wanted to,
8:11
like, remove the amount of pain that others people have.
8:15
Because I do think when you get involved in the day-to-day work
8:18
and sort of resting in sight out of data that you end up with
8:23
these kind of problems of you run a factory and how do you do
8:27
that? And you're also kind of a soft
8:30
Port team, and how do you deal with that?
8:32
That and you've got all these great new tools coming out and
8:36
really interested in data. And so how do you innovate?
8:39
And so that was really the Genesis and in the journey like
8:44
we're still profitable, we're still group bootstrapping, we've
8:48
been growing 30 to 50 percent a year and like five six years
8:51
ago. No one really knew what we
8:52
talked about and like we tried to call it date devops for data
8:56
science for a while and then we had to call it.
9:01
Now that didn't stick. We try Hello, dajjal analytic
9:03
operations that did that was. I thought that Benedict at all
9:07
and then we tried to call it. Yeah, analytic Ops but then if
9:10
you short it down, it's not it's not a good sounding word at all.
9:16
And so we settled on calling a day tops.
9:17
But like now there's model Ops and ml Ops and you know, Ops
9:21
everywhere and there's like a lot of it's actually really nice
9:23
to see the journey from like people not knowing what we're
9:28
talking about to having meetups and communities and conferences
9:31
because Because I think people have realized that you've got to
9:36
sort of build a system in order to do Innovative work and you
9:42
know the model or the data transformation is really the
9:45
cool part. And in order to get that done
9:48
you sort of got to build the factory around it and that's
9:51
sort of what I'm interested in. So I look at data Ops and that
9:56
term as something that is its own, its own thing.
10:02
It's Some discipline and it doesn't necessarily Encompass ml
10:07
Ops, but ml Ops does encompass data Ops.
10:12
Would you agree with that statement?
10:15
You know, I don't really honestly care, what Ops term,
10:18
you use right call it, you know your favorite Ops because like
10:23
the terms are so first of all, as an engineer, like I want
10:28
precise definitions but like, I run a company in Eating and you
10:32
want your term to be known and so I look at it from like a
10:37
completely Oddball way. So you could call it whatever
10:41
Ops term you want. But in general the way I touch
10:45
the elephant is there's a team of people who are working on a
10:49
shared technical thing right in that fat and how you run that
10:55
team and how you manage that team for success and how you
10:58
build a system for them to work in is really important and So I
11:02
think people have figured this out and not just in building
11:06
data science models. They figured it out in building
11:09
software and 50 60 years ago, they figured out and building
11:13
cars. And so those same set of
11:16
principles on how you run a team who are all sharing this thing.
11:22
You know, I think this chair technically complicated thing.
11:26
I think General they flow through devops and agile and
11:29
model apps, and data Ops and lean manufacturing in the Toyota
11:32
production system, they're all kind of the same sort of
11:34
conceptual ideas because one of the most important things is
11:39
because everyone's working on a shared technically complicated
11:41
thing. First of all honesty is really
11:43
important, right? And then second of all the
11:47
system at which you work in is an artifact that you need to
11:50
manage. And so for cars is the assembly
11:52
line, you know, for software, it's the infrastructure as code
11:56
and your CI and CD Pipeline and testing and for data on small
12:01
upset, Opposite. So it's a system that helps you
12:06
deploy helps. You monitor helps you test helps
12:09
you manage environments that you work in.
12:12
So yeah let's let's dive into that a little bit or let's like
12:16
zoom in. I should say because there is a
12:19
trend that I have seen and I was just talking to somebody about
12:22
this earlier today in the different companies that you see
12:26
that are doing ml Ops and the marketing side of things.
12:32
Things is saying that we have the best practices baked in to
12:36
our Solutions. So you don't have to know these
12:40
best practices or you won't get caught with your pants down.
12:44
Now, I always wonder and I really like, Ponder.
12:49
What are these best practices that these companies are talking
12:53
about. And so, maybe you have a few
12:55
that you've baked Into Your solution that you can talk
12:59
about. I mean, because there's this One
13:02
part of me that says, although that's just marketing, and then
13:05
there's another part of me that says, well, actually Version
13:07
Control is the best practice. And so, if you have that baked
13:10
into a solution or like being able to collaborate easily is a
13:16
really nice. Is it a feature best practice?
13:19
I don't know, but it is something that you need.
13:21
And there are, if you look at the, the old, well, not old.
13:27
But if you look at software development, there are best
13:29
practices that have arisen. And so, we're still in that
13:34
phase. I think right now we're I still
13:38
haven't figured out what are the best practices for ML Ops and
13:40
then how do they get baked into tools?
13:45
Yeah. Yeah, I guess one of the things
13:48
I sort of think about is I separate, you know there's
13:51
marketing and what it really does and then there's the the
13:55
tool that you use to create your model or your Insight, right,
14:01
that and what at the end of the day, what is that?
14:04
That's some code and some parameters, right?
14:07
That, and so you're right that number one.
14:10
Best practice is sticking and get inversion it right.
14:14
A number two, best practice is Is when that code runs in
14:19
production, right? Monitor it, and check to see if
14:25
it's right. Observe it, you know, monitor it
14:29
because hey, the model could go out of tune, the data going
14:33
into, it could be wrong. The data transformation could
14:36
break like, you know, shit happens, and you don't trust
14:40
your data providers, right? That's number of number two, and
14:43
then number three. Like when you're in development
14:46
with it, When you're playing around with it, you also need to
14:50
test it, right? You need to check to see if it's
14:52
accurate. You need to check to see if if
14:55
someone else has developed some data that's feeding into, it has
14:58
a broken it. So you need to regression and
15:01
functional test it. So source code, make sure you
15:04
test and development, but test the whole thing, the whole
15:07
system and then in production Monitor and check, its right,
15:11
that makes sense. And then the third part, which
15:13
is comes from software, is that that little basket of code that
15:17
you've written in parameters that runs in some engine python,
15:22
you know, whatever you know, python or some Auto email tool
15:26
or SAS that has to fit into some a piece of Hardware
15:30
infrastructure, right infrastructure as code and that
15:35
should be you should that in itself should be scriptable and
15:39
testable and Deployable so and so if you follow those
15:44
principles of and then The third principle or the fifth
15:48
principle. I would think is if you're doing
15:50
something by clicking buttons, automated, write a script, the
15:55
movement of work from Deb the production, the checking of
15:58
something and testing should be all automated the, you should
16:02
script up your infrastructure and not press buttons and like
16:05
everything should be kind of driven off of scripts as opposed
16:09
to you doing anything manually. And so the stance that you want
16:14
like psychologically, I think is Whatever you do with data, you
16:18
want to put into production and you want to sit back and have a
16:20
tell you if it's working, right? Until you if it's working wrong.
16:24
So number one, you don't get yelled at.
16:26
So you don't have that good-looking VP of sales from a
16:29
Pharma company call you up and yell at you.
16:31
Like you didn't want to do that. Like it's bad, right?
16:34
And and the other part is just don't trust, anybody, don't
16:37
trust your data providers, don't trust your, you got to check
16:41
stuff and you know it's work but it's worth it because One of the
16:48
challenges that a lot of teams have is they just end up kind of
16:52
focusing on their world? They're like, oh I did, I did
16:56
the data transformation. I did the model or I did the
16:58
visualization, my parts, good, you know.
17:01
And so there's a stance that you own the result and not your just
17:06
your peace. Right, you own you get a focus
17:10
on the value, you're delivering to a customer the whole part and
17:14
parcel of the pieces and not just like, hey I did the model
17:17
I'm done, you know, it's someone else's fault, it it breaks it
17:21
was very tempting to do but that's so those principles of
17:26
source code, testing infrastructure, as code
17:29
automation. Owning the owning the result.
17:35
I think our what you need to do and it doesn't really honestly,
17:40
it doesn't have a lot to do with if the model is in python or an
17:44
R. If it's a tensor flow model, if
17:46
it runs in data IQ or runs in data robot those the engines
17:52
that run the model and the tool that used to develop the model,
17:56
you know? Those to me those very and like
17:58
people people like their tools and like I just never want to go
18:03
through. Barbara's as python, or like you
18:05
like this IDE versus another ID, it's just not important.
18:08
In my mind, the system that you work in the factories more
18:12
important than the individual workstation that someone works
18:14
at. Yes, that is so important to
18:18
remember and keep in mind and there are two things that you
18:22
said there that I just want to reiterate because I find them.
18:25
So insightful, one is don't trust anyone.
18:28
Don't trust your data providers especially and we may have to
18:32
make that as Quote, for you. And the other one is the idea of
18:37
owning the outcome owning the whole thing, not just this small
18:42
piece and I've heard it time and time again.
18:45
Somebody else put it, I think it was last week or two weeks ago,
18:49
they were putting it as like the sum is greater than the parts,
18:53
right? Or what, whatever that saying
18:56
is, that stays greater than the whole.
18:57
Yeah, yeah, the whole is greater, the whole is greater
19:00
than the sum of its parts. But whatever, you know what I'm
19:02
saying. But that is the key idea here is
19:06
really looking at zooming out and not focusing just on this
19:09
little piece and then forgetting about it and giving up ownership
19:13
and I know that we've had great questions in here about how to
19:17
do that and how we can get data scientists more involved in the
19:23
whole process as opposed to just their specific piece.
19:28
And someone I remember mentioned should we put them on rotations
19:32
or like on? Handoffs.
19:34
And I think the the answer was yes, that one speaker talked
19:39
about, but this is for me, this is really an interesting piece
19:43
and it comes back to, these are more cultural aspects or these
19:49
are bigger pieces than just like what you were talking about.
19:53
It's not like if you run in python or are you're looking at
19:58
something that's bigger than that and making sure that it
20:01
runs no matter what? What?
20:03
And making sure that it works. Yeah, and tells you if it
20:07
doesn't and making sure that you and look up things, you said
20:12
that, that sort of some people call it project a product.
20:15
Some people call it customer value, Focus as opposed to the
20:18
task focus and you know being results-oriented instead of
20:23
process-oriented there's lots of different terms, right?
20:25
But it is actually harder to focus on the results than just
20:31
under piece of the task. And it's because you run the
20:36
risk of of your of your customer.
20:38
Saying, if it doesn't work or it's, this is a 1i r on it and
20:43
and then you can it's hard, right?
20:45
It's hard to hear negative feedback and like the idea here
20:48
is, can you push the negative feedback forward and quicker and
20:54
find out if things aren't working faster?
20:57
Because that way, you don't spend so much time on doing the
21:01
wrong thing. Yes.
21:03
And the real idea here is that, you know, we all are
21:08
introverted. Well, maybe you could you play
21:11
music, you're not introvert. But me, I you know, I thought of
21:14
like spending a month in front of a screen building something
21:17
like I'd actually like to do that.
21:18
I talk to people too much now and like I've written, I spent,
21:21
you know, most of my career has been long periods of thought in
21:25
front of a screen. And like I like it.
21:28
You know, it's fun and the thought like oh I could think of
21:31
something and then I have to talk to someone in six weeks
21:34
when I'm done that's really cool, right?
21:37
I can, and I can build my crystal Castle.
21:39
And it's really neat and I think about code and models when I
21:43
sleep and I wake up and the problem is, is that six weeks
21:46
later you may have thought, okay?
21:49
I had to do these 10 things and in that six weeks, you do the
21:52
those ten things and then we finally get feedback from the
21:55
customer. They say, yeah.
21:57
Great. You know what, I only wanted
21:59
these five things and I don't want the other five and here's
22:04
two more that you didn't think of.
22:06
Yeah, and so you ended up wasting your time, right?
22:10
You went wasted basically half your time because you got it
22:13
wrong. And so there's a humility that
22:16
needs to be in. Like, you just don't know what
22:17
the world wants. And if you can deliver real
22:21
value quicker, you can learn more.
22:24
And iteration cycle time, is increases the amount of learning
22:29
that you do. And if you're humble about what,
22:31
you know, and you're cognizant of your own psychological
22:34
Tendencies to want to sort of build For castles if you can
22:37
find ways to deliver, kind of little smaller chunks and so
22:41
maybe the best model in the world isn't what you need to do.
22:45
Maybe get a crappy model, a crappy regression that you spent
22:48
four hours doing but getting it in front of a customer quicker
22:52
maybe better than six weeks of building, you know, the best
22:56
Ensemble model you've ever heard and that's hard for people
23:00
because like it's more fun, they're like spend six weeks
23:03
playing around the data, but It's really is I think that's
23:08
the value. Focus is, is not just focus on
23:11
customer value, but focus on rapid feedback of customer
23:14
value. And that's a, it's hard to do,
23:17
but it really actually makes you, it makes you much more
23:21
effective and you get, you end up getting much more done in the
23:27
in the world that having much more to affect on the world.
23:32
So there's I want to get into Data Ops.
23:36
I also want to be cognizant of what you're saying, and I really
23:39
wanted to dive in on a few of those that you were talking
23:42
about maybe last thing before we jump into data apis and and
23:47
really it's again what you're talking about is so much of
23:51
this, these devops principles this agile, that can be
23:55
generalized enough to it's useful for ML Ops or data Ops or
24:00
whatever Ops you're calling it as you mentioned and I'm
24:05
wondering Because this is something that we've heard so
24:09
much of just get something working end to end, and then
24:13
build on top of it, but make sure that you can get something
24:16
working, and it is actually solving the solution of the
24:21
problem that you're setting out to solve, not like getting
24:24
caught up and then you go on some tangent.
24:27
And some Charles Martin, who was on here, like one of the first
24:31
guests on here, he talked about the quote from, I think it was
24:35
vapnik. Who said like one does not
24:38
impose a man, is it? I don't know if I can even
24:41
remember this quote, but it is something along the lines of.
24:44
You don't solve a problem by posing a harder problem in
24:50
between it and then having to solve that problem, right?
24:53
Like yeah, yeah, it's a horrible.
24:54
I'll figure out the real quote, and then put it in the chat or
24:57
something later, but, but that's the idea that sometimes when
25:01
you're sitting in these six weeks and you think well, you
25:05
know, I'm going to get 2X. And then you reach a certain
25:08
point and you go well, actually, what about if I create this or
25:11
that, and then it gets a little bit little bit hairy and you
25:15
lose track, but anyway. Yeah, I mean it's not.
25:20
This stuff is like I started my career in 1990 or 1991 working
25:25
at NASA, Ames, and my branch, chief would say, build a little
25:29
test, a little, learn a lot. And he was like, and I was just
25:33
like, oh, he's, you know, he's like my age.
25:35
He's like an old guy at the Time I was like, oh, that's just
25:37
engineering wisdom, and he set it up.
25:39
So, like every week we'd have a simulation run and we invited
25:42
in. We had paid what we called
25:45
pseudo pilots and paid their traffic controllers, come in.
25:48
And every week, we tweaked our algorithms and they go, this is
25:51
wrong, this is right. And then we go off and fix it.
25:54
And actually, that was like, a super fun with it'll work
25:57
because like every week, we find out how wrong we were right or
26:01
how, right? We were and we try something new
26:05
and like That sort of cycle of Rapid feedback is just it's it.
26:09
You can you grow and learn so much and it's, it's harder.
26:16
But after a while, I really, really like that.
26:19
And I think the any good engineering, wisdom is true, no
26:23
matter what language you, put it in and eat, you know, the agile
26:26
Manifesto was written in ten years later in 2001 right there.
26:29
Bob started 2009, right and so this is just good common sense
26:34
like Bill a little A little learn a lot that's like that
26:40
makes total sense to me and that's all we're really saying,
26:44
but and I'm just the Tweet that's onto it is build from a
26:48
manager's perspective. You have to build a system that
26:50
enables your people to do that and not kill themselves, and not
26:55
create a lot of debt and Technical debt and crap that
26:59
they got to fix up later. So, it's not, you know, it's not
27:03
quite as easy as like You know throw some stuff up and see what
27:07
sticks. It's like you got to continually
27:09
build a system that would enable that you to do that.
27:13
All right, so let's talk a little bit about data Ops and
27:17
let's get into it. I know you wrote the data Ops
27:19
Manifesto and I've also heard this was one that a friend of
27:25
mine wanted me to ask you. They were saying some say that
27:30
data Ops is Dan, a data management reborn.
27:34
Do you have thoughts on that? I certainly think the word like
27:42
ml Ops people who, you know, the Ops term is ending up in
27:48
compassing. The work that you do, in
27:51
addition to the system that you build to do the work and so that
27:56
happened in software with devops, you know.
27:59
And like, I don't know why an IDE is a devops tool.
28:03
I'm on mine, because it's just an IDE, but they call it that,
28:06
right? And so these, the word inflation
28:10
is happening in Tech all the time.
28:12
And so, some companies have taken their 10-year old.
28:18
Change data capture tool or their ETL tool and called it
28:22
data Ops. And I don't think that that's
28:24
true on no more than, you know, at a jupyter notebook is a model
28:32
Ops tool, right? So I mean, but that's, that's
28:35
what companies do, right? And so, like companies from a
28:39
marketing perspective when I grab on a term and, and we did
28:42
it too, right? We want to be known as data.
28:44
Ops we wrote, we wrote the manifesto number one because
28:47
Going to a conference. No one knew what the hell were
28:50
talking about? And we were trying like, we
28:54
debated should it be model Ops? I mean, you know, should it be
28:57
ml Ops? Should it be?
28:59
And so, like we just picked, we had to pick something.
29:01
So we pick data Ops. Yeah, and ironically, it's
29:03
actually even a bad name because it's not even about data.
29:06
It's actually about the code that acts upon the data and so,
29:13
all these, so I'm I'm a little, you know, I'm a run a company,
29:17
so I I have to mark it and you know, we chosen data offices or
29:20
term and it's fine and that's why we're with a Manifesto and
29:23
wrote a book. But from my standpoint, I just
29:26
want people to give Some attention to it and think it's
29:32
worthy of men and not just data Ops.
29:34
But any of these testing deployment automation
29:38
monitoring, I'm certain, you know, observe observations and
29:42
production just give it some love because if you do your
29:44
life's going to be better and you're not going to, there's
29:48
still a lot of companies I talked with where people are
29:51
frustrated, where there they got their masters and data science,
29:57
and they can't get any models into production.
30:00
They've been working in the field for a while and it takes
30:03
them, you know, six weeks to get anything done.
30:07
And what happens is organizations will build get hit
30:12
with the reality of crappy data, demanding customers and lack of
30:18
attention and building these fragile brittle systems, and
30:22
after a few months, that creates this Log Jam of meetings, and,
30:27
and tasks and process, and Everything slows down and that
30:34
just ends up being like a totally not fun environment to
30:37
work in. And so I find people are very
30:40
frustrated, right? Because they get caught in sort
30:43
of fear mode. Well one thing that has come up
30:46
and there's some great people in the community that are talking
30:50
about this. And I would love to hear your
30:52
opinion on the standardization that needs to happen.
30:56
And right now, there's A great way that someone put it is that
31:03
the standardization is trying to happen inside of companies.
31:07
As opposed to like oh this is going to be the one
31:10
standardization to rule them all and then you just get more
31:13
standardization on top of standardization.
31:15
And everyone thinks they're one way of standardizing, the whole
31:20
Market is going to flood to, but what I feel like I've seen is so
31:25
many different companies are trying to standardize within
31:28
their company but Having trouble doing that, you feel like that
31:32
is a correct reflection of what you did and what do you mean by
31:36
standardizing? Like standardizing on a tool or
31:38
what do you mean? I mean no like more the way that
31:41
you're doing things inside of that and it can be, it can be
31:47
fairly low level, or it can be more.
31:50
It can be higher, like just the processes and the way that you
31:57
go about like what you're doing with with those.
32:00
Processes or like the actual technology side it of saying,
32:04
hey we're going to use this, whatever it is like this is our
32:08
API way of defining things or or like the tech that's involved in
32:13
in it. Yeah.
32:20
I look at it like a different angle.
32:23
Like I think that there's a lack of perception of the need to
32:28
spend time on doing the operations part of the equation
32:34
on building the factory as opposed to building the car that
32:36
comes out of the factory. And I think like I don't
32:40
particularly like there's a lot of talk about what is the latest
32:44
cool model and the latest prediction and that's fantastic,
32:48
really cool stuff. But you got to devote think of
32:52
it, 15 percent of your time on building the factory and what is
32:56
in their testing monitoring deployment automation.
33:00
Environment management, building nice.
33:03
Little a nice whole system that you can drop your code into and
33:06
you're going to it'll in developmental tell you.
33:07
It's working, right? And it hasn't broken anything.
33:10
And when it's running in production, it'll tell you that
33:13
hey it's you know, you haven't been screwed by your data
33:15
providers or your models out of whack and like that.
33:19
But that takes time. An effort to build that and it's
33:22
not the cool stuff that you can look at you get from the media.
33:26
And like oh I want to use the new tensorflow Processing Unit
33:31
from Google. That's super cool.
33:33
I saw the rack and the Google I/O like, oh man, it's exabytes.
33:37
And flats is like that. That's the cool stuff and I'm
33:40
like, I don't deny that, that's cool, right?
33:43
I just say that the real work is you got to do it, 15 percent of
33:47
your time on making that doing the factory stuff.
33:49
Tough and that if you put some time into it you're going to be
33:53
better off and to me that's what I want people to standardize on
33:58
and I don't care what word you use.
34:00
I just wanted to get some attention and love because if
34:04
you don't you end up In the world of hurt.
34:09
I was in 10 years ago and I just want people to get out of that
34:13
world of hurt. So I'm thinking also like people
34:17
that want to use tools as these Lego blocks.
34:21
And right now, it feels like there is enough non
34:26
standardization that they can't easily plug into each other.
34:30
Right. Do you do you feel the same way
34:32
on that? um, To a certain extent.
34:39
I would agree. Yeah, I think the the, there's a
34:42
lack of a standard like another bit of engineering wisdom.
34:48
I learned at Nasa was good interfaces.
34:51
Make good neighbors. And, you know, we have a very
34:55
complex ecosystem of tools and servers in the data and analytic
35:01
value chain. Like, if you walk the journey
35:04
from where it comes from a transactional system.
35:07
Goes through. Streaming buckets data Lake
35:12
data, warehouse model, visualization governance that
35:16
could be half a dozen tools that run on a half a dozen, different
35:18
servers and and so in any point along the line you want to
35:23
change one of those things. You could mess things Downstream
35:26
up. And so defining your interfaces
35:29
and thinking in a system, systematic way is good.
35:32
And I'm not sure I'm less of a believer in the fact that you
35:37
could abstract all that. Away into one giant mega
35:42
abstraction of like here's the Uber abstraction that sits
35:47
across data transformation and visualization and model like
35:50
there has been standards of PML. There was this one of the
35:54
universe Chicago that was like called TFA.
35:57
It was another abstraction of the model and I haven't I tried
36:01
to build one back in the late 2000s and like I'm not a big
36:05
believer in abstracting it. I really think these are code
36:08
driven. And you got to sort of apply
36:10
some good engineering principles and like it's complicated and
36:15
software engineering, his is so you got to think in complexity
36:19
how to break things in the components, how to have good
36:21
interfaces and you know, standards can help but they're
36:25
not the Panacea. Yeah.
36:29
So talking about the data Ops Manifesto in almost going back
36:35
to what we were talking about in the beginning on You you got to
36:41
own the results and I've heard it put another way of like if
36:45
you build it you what is it? Like you build it, you ship it,
36:51
you own it kind of thing. Yeah yeah and I'm wondering like
36:57
it's got to be everyone's responsibility from the data
37:00
engineer to the data scientist. How can her, how have you seen
37:04
that? Cohesion and those two teams and
37:08
the greater. Teams, the machine learning
37:10
Engineers working well together, is there, anything along these
37:14
cultural side of things like what has worked when you see
37:19
those those different people and teams were?
37:21
Yeah, not more meetings. Like, I just, you know, let's
37:26
not think him buy and hold hands and have more meetings and
37:31
stand-ups like, you know, I just don't think I think you need to
37:37
have a shared Action between all those people that they can
37:42
understand in a technical way, their relationship to each
37:46
other. And so, because it's not just
37:49
data engineers and ml Ops engineers and data scientists.
37:53
And it's also people do visualization and governance,
37:57
right? And as people who do it, who
37:59
provide raw data like if it's a value chain, is a series of
38:03
steps, and if you don't have that shared abstraction, then
38:06
you're just going to have more meetings.
38:09
And so, you know, there's this other phrase that they that
38:13
people use is called pull the pain forward.
38:16
And so what that means is I'm going to change a piece of that
38:21
system. How can I then tell at my
38:24
desktop by myself without any meetings?
38:27
What the what the Ripple of that effective?
38:30
And that's super important because if you don't do that,
38:33
then you're really tempted to put your blinders on.
38:36
I changed my piece. Okay, let's have a meeting,
38:40
let's have a check list, you know, and then like, if you're
38:44
any good technically, then you get pulled into more meetings
38:47
because you're the person who has the whole system in their
38:49
head and then you're like your whole life's full of meetings
38:52
and you're not doing anything fun.
38:54
And so devote, 15% of your time to building that building,
38:59
something that shared abstraction that everyone can
39:02
see the end-to-end system in. And I think that is a way to
39:05
help. Help you help teams get away
39:09
from because I mean Communications fine, right?
39:13
I mean I'm not against people talk to each other.
39:15
I'm just saying it's much easier to build build an artifact that
39:20
that represents the factory, the flow of the value chain of data,
39:24
and give people visibility into it.
39:26
Then just have more meetings. So if I understand this
39:29
correctly, it, it's something that you.
39:33
So you would have each person having their own understanding
39:40
of what they're doing, what their responsibilities are, but
39:43
how it affects each other person in the value chain.
39:47
Yeah, exactly. So, like, let's say, let's say
39:51
I'm a data engineer and my data is going into a feature that's
39:55
in a model, and my date is also going into a A dashboard that
39:58
someone looks at and lo and behold, I like I want to change
40:01
something. But I get a new request, so I
40:05
change it. Now, one option is, I change it,
40:09
I do my work and then a week later, I have a meeting with the
40:13
data scientist and the people do data visualization, and they
40:15
say, what the f, you broke everything, that's a form of
40:20
collaboration, right? I just think why not pull the
40:23
pain forward and have that data engineer at their desktop, make
40:27
that change, and then run some tests against the model run.
40:30
Some tests against the VIS and see if it's right.
40:35
And you know what, that's the same.
40:37
You tilt it a different way in software, it's the same idea of
40:41
sort of back-end, front-end Engineers, because right now, A
40:44
back-end engineer changes, a rest API on an application.
40:48
They can run a test and see the effect of that in the UI.
40:53
And so the pains pulled forward for them and then they're like,
40:56
oh man, I shouldn't go change in the rest API because like I just
40:59
broke all my UI and and so is that a standard?
41:04
Well yeah. The rest API is a rest is a
41:07
standard but like, you know. But is it the semantics of the
41:11
rest differs on every application?
41:13
It's really. I've built a system that allows
41:16
me as an individual to get that look, see a local change and
41:21
judges Global effects. Yes.
41:24
And so I want the same thing and data and analytics.
41:27
I change because the models goes anywhere along the line.
41:33
They go before the data after the data and like anywhere along
41:37
the line, you can make a change in something, would break.
41:39
And even like the poor neighbor of data, governance Reich you
41:43
trying to keep track of what date is there or what what
41:46
happens there that you know, that should be, you should be
41:49
able to do that as code as well. And so that's all I want is like
41:53
Then just pull the pain forward. So you see your effect and like
41:56
life is so much better. Honestly, if you do that because
41:59
the other part is when you have a system like that, you as an
42:04
individual, contributor become replaceable and you know what,
42:08
you want yourself to be replaceable because that means
42:11
you can move on to different things and so on a good way to
42:16
judge a highly functioning data, and analytic team is, can you
42:19
have a 22 year-old recent CS graduate tweak them?
42:23
Model and deploy into production in a day or a week without an
42:29
end and having a really really high probability that it's all
42:33
going to work. And so, like, I don't know if
42:37
there's not a lot of organizations that are like
42:38
that. I was just going to say you've
42:40
seen those organizations. Yeah.
42:44
But you know what? In software engineering the
42:47
leadership says, this is a goal for my company.
42:50
I want to hire someone have them fix a bug and get it into
42:53
production. Their first week of work, they
42:55
measure themselves on that because they built a system.
42:59
They devoted not 15% of their engineering effort, 25-30
43:03
percent of their engineering. For this building that system
43:07
that's called depth of devops engineer.
43:08
Those people are paid well, and so ml Ops Engineers data office
43:13
Engineers. Whatever the Hell in column,
43:15
right? I don't care like, put some time
43:17
on it because man, I matters and if you do that your your life is
43:23
going to be better, you're not going to be stuck owning the
43:26
model or the data work. You did three years ago and
43:29
having to fix it. You're not going to be like
43:31
making doing something and then. You throw it over the wall to
43:35
someone else and you're onto something new and then you got a
43:38
context switch two weeks later when it finally gets into into
43:41
Q&A. And like there's a lot of things
43:43
about just your enjoyment personally of work, because
43:47
you're more in the flow of being able to work and get things done
43:51
and not being stuck owning stuff that you built two years ago and
43:57
I don't know. I thought I find there's a lot
43:59
of the average time of a CDO, is like two and a half years.
44:02
There's a lot of, I find a lot of Young data engineers and data
44:07
scientists are not happy with their jobs because they're
44:10
feeling like they don't have any effect.
44:13
Yeah. Yeah we've seen that quite a bit
44:15
too. And you see all kinds of blog
44:17
articles about it. Now let's talk about this
44:20
testing that you were talking about because I know that one of
44:24
the co-hosts for the podcast that we do was just talking to
44:29
me a week or two ago about how to better create tests in the ml
44:37
Ops environment. And you mentioned that like
44:40
being able to see the Ripple effects, how have you seen that?
44:43
Done. And have you seen it, done?
44:44
Well, well, first of all there, I have a, I don't think you
44:52
should those those tests are some code artifact, they're
44:55
automatically written, right? And I, I'm more of a fan of the
45:02
sort of functional programming model, right?
45:04
Where you in a functional programming model you you think
45:10
of everything like a big tube. You pour something.
45:12
In it runs you get the same result every time, you know, the
45:15
function machine from like third grade.
45:18
And if you do that and data and analytics, what happens is you
45:22
can just pour the same data in and you get, you should be able
45:25
to get the same result. And, well, what if you change
45:28
the code that's in the middle of that tube that you're pouring
45:31
data in? Well, you should get nearly the
45:33
same result or something close to it.
45:36
And so what I find is like, the one is automate the test, that's
45:41
code. Number two, is Better if you can
45:44
to work in a more functional way, right?
45:46
Where you sort of take data, pour it in and then run the test
45:49
all the way around the value chain.
45:51
And then have a big red green light sang, it works and
45:54
development or hey, something's weird.
45:58
And there's a lot of techniques to testing and, you know, from
46:03
simple line counts row counts to statistical controls to but and
46:11
there's a whole art to testing because you can't test
46:13
everything. But think of the amount of work
46:16
you do as a, as a data engineer data scientists think of at
46:21
least, 10% of your time, you should be writing some automated
46:23
test. And then this functional
46:26
programming test everywhere 6:50 your time running automated
46:30
tasks, but the other trick is low and behold, those tests
46:34
themselves can be run in production and they can, you
46:38
know, maybe not all of them. But the difference between a
46:41
test and development and a monitor and production is kind
46:45
of small in my mind. Because like, you know, if
46:49
you're running the system, it should work.
46:51
And so you can, then that's a reason why, you know, that sort
46:58
of double duty. A lot of the tests in
47:00
development have can be run in production.
47:04
And the real difference is in production, your data is
47:06
varying, but your code is fixed in development.
47:11
Your date is fixed, but your code is varying.
47:15
But between those the tests are fixed.
47:19
Yes, you're like, I'm like I'm a big, I'm more a fan of, like,
47:24
thinking in these functional functional, functional systems
47:27
because it makes your life easier because there's a lot of,
47:29
like, State Management and a lot of these systems and like
47:32
intermediate development, QA databases where you got a patch
47:35
and like, it's just, just start start over again.
47:39
You know, it's the cloud, you know, retrain your model from
47:43
the beginning, rerun your system.
47:45
It's yeah. It's pennies to do that.
47:47
Yeah. So you're giving me a lot of
47:50
great quotes here for later. I appreciate that.
47:54
I'm wondering how you feel considering Google just has been
47:58
announcing all of their fun stuff and I know you mentioned
48:01
that you've watched it. There is this common debate that
48:05
we get in the community about one tool to rule them all or the
48:09
best of breed pieces. And it feels like Google just
48:15
made their statement on. We're To have the one tool to
48:19
rule them all. Do you have any thoughts on
48:22
that? You know, I'm in here in the
48:24
same damn thing for 20 years, right?
48:27
The ones who'll and like, first of all, I was like everything's
48:30
going to be in the Oracle database and there's Oracle
48:33
cartridges and Java plugins and like You know, and then it was
48:39
going to be the Hadoop and bark and everything is going to be in
48:44
there. And and so yeah I think the
48:48
cloud vendors are really trying to have every tool that you need
48:52
to do data science and engineering.
48:54
Just like they're having to every tool that you need to do
48:56
software engineering and you know what, some other tools are
48:59
good and some of them aren't and maybe at some point in the
49:01
future, they'll be that everything you'll need in the
49:04
cloud. Vendor that right now.
49:06
It's There's pockets of good things in pockets of challenging
49:10
things. And most people I deal with have
49:13
a whole host of different tools and so and they have Legacy that
49:19
they're never going to want to convert.
49:21
And so widely Divergent toolchain is the norm and maybe
49:26
at some point they'll be wanting to hold a rule them all but like
49:29
it has happened in software. I mean, there's not like why,
49:32
what's the one tool that will keep people keep creating and
49:35
inventing in the level goes up? Up.
49:37
So yeah, I'm a little cynical on it just because number one, I
49:41
heard it before. And number two, man, people keep
49:43
inventing and creating and like it does just doesn't go away.
49:47
And so and number three in Legacy happens if you're going
49:53
to live in the real world and not start a Greenfield projects,
49:55
you're going to have old stuff and new stuff.
49:58
And there's so many companies that they're like, you can see
50:01
the the terms of one tool to rule them all.
50:08
You've got a teradata and then they got a Hadoop cluster and
50:12
then they got spark and like everyone, I everyone was telling
50:15
them, it was going to be one thing to rule them all and like,
50:17
okay. Now they got three things.
50:20
So another like I'm son and I are moving to the cloud.
50:23
So, I don't know. Yeah, well, I would say, I hope
50:29
it's not one to rule them all. I've said this before, it seems
50:32
like a pretty bleak future where we just use.
50:37
The cloud providers in everything that they have their
50:40
whole sweet. And then they monopolize, the
50:43
essentially, there's the three that you have as the try ah,
50:47
Polly or whatever you call it and yeah.
50:50
And then there's not I'm a fan of the little guys, so
50:54
hopefully, hopefully it's not like that and these arguments
50:57
that you're saying it does seem like, yeah, it's not going to
51:00
happen, they're going to try as hard as they can but like you
51:05
said, there's pockets of good and there's pockets.
51:07
Bad and people are going to try and for years and haven't done
51:10
it. And so, you know, I'm not, I
51:14
just don't because I also think the other part is, there's a lot
51:17
of people who want to, they're sort of from Revit resume stuff
51:21
in that happens, for people there, like this is, I read the
51:24
article, this is the new thing, and I need to have some
51:26
experience on it. And, like, you know, I find that
51:30
people who've got a 20 terabyte database in spark, like, I laugh
51:35
because it's like, you don't need 20 terabytes in size, Park.
51:38
You know, I can put that like on my Zip drive like like and so
51:44
there's people are trying to or you know I'm doing a complicated
51:49
Ensemble model and I should just be doing a Russian, right?
51:54
There's people want to get resume stacking as opposed to
51:57
focus on what's the fastest, and most, the least increment of
52:03
work. I can do to get some value to
52:04
the customer and then improve and And I think that's it's not
52:09
good because it actually doesn't, I don't know, maybe it,
52:12
maybe it helps people get jobs but like I don't even have to do
52:18
a lot with employers who are kind of doing resume filters and
52:20
searching because I mean you know, at some point just because
52:24
you happen to have experienced an ex tool doesn't mean that
52:27
you're any good at it. Well actually I will say that we
52:33
had Flavio on here a while ago to almost a year ago and he
52:37
mentioned that one of the biggest mess ups in his company
52:43
with machine learning was because somebody was trying to
52:46
do that resume driven development.
52:47
And so they said, let's try this new tool, which just went all
52:53
haywire. And he was the only one that
52:55
actually knew how to do that. So then when he was an on-call
52:58
and things went wrong, you got into a big, big problem.
53:02
And I remember, he said that it was like a recommender system
53:07
that When did the same thing to every single person that visited
53:11
the website for 18 days straight?
53:14
And so he was like, you can imagine how much money that lost
53:18
and I tell the story again and again because it is the perfect
53:21
example of that. It is, you know, these models
53:24
gone Rogue or the example of the resume driven development that
53:29
it doesn't need to happen. And so, so last question for you
53:34
before I let you run and I want to thank You again for coming on
53:38
here and opening my mind, you've given so many great quotes,
53:41
whether it's your own or your NASA mentors, that used to tell
53:46
you these things. If we are in these Legacy
53:49
systems, we're at an Enterprise and were fighting through the
53:53
Legacy. Swamp, how do I get started with
53:56
ML Ops? I think, well, that the two
54:01
things I tell people are number one is to every time you have
54:07
problem big or small, write it down and then once a month, sit
54:12
down with some people and look at the list and prioritize one
54:16
thing that you're going to fix and when you fix it, do it in an
54:20
automated script ruin way, so it doesn't happen again.
54:25
And then the second thing is to started with, Ops is write some
54:29
tests and have those tests run them in your development cycle
54:33
every time and then run them in production.
54:37
And you don't need any tools to do that.
54:40
And but but that's on the line that give you want to give 15%
54:44
of your time to these Ops type activities and a quality Circle
54:49
where you review problems and fix them.
54:51
You write some automated test is on that line because the real
54:54
challenge is most people are spending one percent of their
54:56
time on this stuff and 99% of other time on, you know, pulling
55:03
the what they feel is the heavy backpack, full of tasks out.
55:06
I got to get I'm going to get my data going and they are very
55:10
burdened with that backpack because it's full of requests
55:13
from the customers and they don't feel right about doing
55:17
these things and so if you just start doing it Wonder baby step
55:22
or two, you're going to get on, you're going to see the effects
55:24
and you're going to start saying oh wow, I should do this more.
55:28
Yeah, comes back to the idea of being reactive or proactive and
55:35
if you dedicate this 15% of your time, I really like that idea
55:39
like dedicate. 15% of your time to these Ops tasks and you'll be
55:45
in a position where you can be proactive as opposed to reactive
55:49
when when things are just happening.
55:51
And like you said, you're trying to lift the heavy weights but
55:55
you're encountering roadblocks along the way.
55:58
And, you know, you don't have to end up with being a gray-haired
56:01
old. Fart like me, who's telling
56:03
stories about working the weekend and leaving the kids
56:05
soccer match. It's like, you know, it's not,
56:08
there's a lot of if you start, once you get in the field,
56:11
you're going to start to have that.
56:12
And it's not, you're not successful when you're working
56:16
nights and weekends, you know, you just haven't done the work
56:20
to build a system to do that. And, you know, yeah, there's
56:22
some time like hope and heroism should only happen a little bit.
56:28
And if you're a hero all the time, if you're hoping stuff can
56:31
work. So like I mean, I think that's
56:33
really, that's what it comes down to is you're on the road to
56:36
get burnt out. Yeah, and writing canister.
56:39
Yeah. Well, thank you so much.
56:42
This has been enlightening to say the least I.
56:45
And I was the best intro ever man.
56:47
Thank you. I really appreciate it.
56:49
Yes, any time any time, it was my pleasure and I really
56:53
appreciate you coming on here. Sharing your wisdom with us.
56:57
You've been through a lot. And so, hopefully, we don't have
56:59
to go through most of it man. That's the idea.
57:03
Yeah. And so, and then, when you guys,
57:06
you all young, guys, get my age, you'll realize that there's
57:08
something else system, empty wrong, and you'll fix it.
57:10
And so that's, that's, that's what it's all about, exactly.
57:15
So, thanks again, Chris and everybody out there listening.
57:18
I appreciate you sticking around.
57:20
Go check out what Chris is doing if you liked it and best way to
57:24
reach out to you, Chris if they want to continue the
57:26
conversation first date, Kitchen.i out.
57:29
Just come to our website and you can get me there.
57:32
There we go. All right.
57:34
See you later. Thank you.
57:35
Bye.