Latest / Agentic Conversations (formally mlops.community) / Challenges Operationalizing ML (And Some Solutions) // Nathan Ryan Frank // #199

Transcript

Challenges Operationalizing ML (And Some Solutions) // Nathan Ryan Frank // #199 · Agentic Conversations (formally mlops.community)

0:00
Hi everybody, Nathan Frank. I'm currently the Director of
0:04
Machine Learning platform and Operations at Granger, and how I
0:08
like to drink my coffee is is really any way I can get it.
0:11
But you know, I have a machine that does like a simulated pore
0:16
over experience for me, and that that makes pretty decent comedy,
0:18
I think. But I'll take diner coffee any
0:22
day as well. What is up, good people of
0:25
Earth? This is another edition of the
0:27
ML OPS Community Podcast. I'm your host, Dimitri OS, and
0:32
today we're going to be going on a journey with none other than
0:36
my friend Nathan, Ryan Frank. This conversation, this
0:41
conversation, we get into it, We talk all about ML OPS and
0:45
software engineering, what Nathan's approach is to building
0:50
tools and how he looks at if there is a tool that's already
0:54
on the market, should he incorporate it into his stack?
0:57
What the trade-offs are there? What's the same in ML versus
1:02
software engineering? What's different and how he
1:05
approaches that? How he approaches ML OPS from
1:08
the DevOps perspective? And then we get into teams and
1:13
communications, what the common language gaps that he's seen
1:18
have been when it comes to different teams speaking
1:22
different languages. Maybe that's a product team or a
1:26
product individual that's trying to talk to a machine learning
1:29
engineer or a data scientist. What common language can they
1:34
use that will bridge these gaps and how he sets up how he's seen
1:40
the most success for teams and what their constitution looks
1:47
like? Last but not least, we talked a
1:49
little bit about his personal life, did some rapid fire on
1:53
some of the good stuff that he does in his free time.
1:56
Hope you enjoyed this conversation with my man Nathan
1:58
Frank. And as always, we would love to
2:03
get a review from you to help us just boost them rankings and the
2:06
Spotify algorithm so that whole world can go and listen to this
2:12
episode about Know OPS. I know there's probably a few
2:16
people in Kansas right now, probably middle-aged women that
2:22
are thinking, you know what I need right now?
2:24
I need, I need some ML OPS in my life.
2:27
And you hitting the review or giving us a incredible 5 stars
2:34
or whatever your app you're listening in on right now that
2:37
will make sure that this app sends it out to those
2:42
middle-aged women that are missing ML OPS in their life.
2:47
Just I can view it. So hit that review, give us the
2:52
like, and I'll see you have to start.
3:02
I feel like it is not fair to not start with the idea of this
3:08
telescope that you helped build. And what is the story there,
3:12
dude? You've got astronomers, students
3:15
and people all over the world that are leveraging this.
3:19
Give me the breakdown. Yeah, so Once Upon a time I was
3:25
an astrophysicist. I guess you could say that's why
3:28
studying undergrad physics with a constant, you know, for
3:31
concentration and astrophysics. And then I went to grad school
3:34
for physics and astronomy and for the research group that I
3:38
align myself, led by Doctor Dan Reichert at University of North
3:41
Carolina at Chapel Hill. The the research goal was around
3:46
these objects called gamma ray bursts.
3:48
So these these massive explosions, they emit the amount
3:51
of energy that our star will emit over the over the the sun
3:55
will meet over the course of its lifetime in seconds.
3:57
So they're just incredibly massive.
4:00
Super energetic explosions usually happen when like a
4:03
massive * either collapses into a black hole or two former
4:07
massive stars collide with each other anyway.
4:10
So that was the research goal of the group, and that's ostensibly
4:12
what I was trying to study and trying to model and understand
4:15
the behaviour and how do we, you know, characterise them and all
4:17
this kind of stuff. But in order to do that, he
4:19
built a number of telescopes, a handful about 8 in the Chilean
4:25
Andes. And then while I was there, we
4:27
built another installed, built as maybe nowhere installed
4:31
number, another 4IN in Australia.
4:35
I say built as in this number because these are largely like
4:37
what you would describe as high end damaged components.
4:39
So if you had the proper funding, you could go into
4:42
magazines and order this stuff and put it together.
4:46
And all of these were part of a network called the Skynet
4:48
Robotic telescope network. And so they had, and again a lot
4:52
of this predates my joining the team, but they built, you know,
4:56
all the software, sort of local software to control the
4:59
hardware, move the move the mount, move the telescope,
5:02
change between different philtres and operate the camera
5:05
and all this kind of stuff. And that was accessible by by a
5:10
web interface. So as a user you could log on to
5:12
the, you know, skynet.unc.edu, you could have a username and
5:15
password. You could say, I want to go take
5:16
a picture of whatever Jupiter, Saturn, this random object
5:19
could, these coordinates and it will go into a queue.
5:23
And when it got dark and there was a location that your object
5:28
was visible with the settings that you desired, it would go
5:31
and snap off an image or images of whatever you requested and it
5:36
would go back into and return to you and your interface and now
5:39
in the web interface. And so I learned a lot from
5:42
that, just about like one. It was incredibly fun to go to
5:46
the toilet and Andy and be up on top of the mountain, to go to
5:48
the Australian sort of not, you know, in town in Australia and
5:53
like, yeah, kind of be out there.
5:55
And definitely some fun stories with some of the truckers that
5:59
brought some of our stuff. Truckers in Australia are
6:01
unique. They're pretty, I don't know you
6:05
can kind of imagine they're pretty like kind of seeing it
6:06
and like you know pretty rough and tumble people.
6:11
And so yeah, it was just a really cool learn a lot about
6:12
how do we, how do we get all these interacting components to
6:15
talk to each other. How do we understand how to like
6:18
move the data through It was my exposure first exposure to an
6:21
actual software system that's like running as users.
6:24
And you have all these parts talking to each other and you
6:26
got like move the data from one place to another and then you
6:29
got to service it and like an actual analysis application,
6:32
processing, all this kind of stuff.
6:34
So I think it really sparked in me an interest in or or sparked
6:40
what like an innate interest in software, in building software
6:43
and building technologies and like that interface of theory or
6:48
research and actual end user applications.
6:51
And that's kind of, I think was like one of many turns in my
6:54
career. But that was one where I was
6:56
like, well, I'm in grad school, ostensibly to go get a PhD in
7:00
physics and maybe like spend a career in academia, but maybe
7:04
that's not really what I want to do.
7:05
It's like kind of came to this realisation that I'm either
7:07
gonna go teach like high school math and physics because I
7:10
really like working with students and want to just, you
7:12
know, spend the time with the students.
7:15
Or I'm going to go to industry and I ended up going to industry
7:18
leading, leaving my programme absolutely, and going to work
7:22
with sports and data and doing some interesting stuff there.
7:24
But but yeah, it was very fun to work on.
7:29
Very cool experience. Is that how the transition
7:32
happened into mill? You said, alright, I'm going to
7:36
go work in industry and turns out unless you're working for
7:40
SpaceX or NASA, or even if you are, I guess they're not really
7:45
hiring a lot of astrophysicists. So you said what's the next best
7:48
thing? Sports and betting?
7:50
Was there betting involved? And so there was not.
7:55
I should say I I was not. I've never been super interested
7:58
in in betting on sports for whatever reason.
8:01
I'm into math, I'm into, you know, statistics, I'm into
8:04
machine learning and I'm into sports.
8:06
But I never really think combined the two in any way that
8:09
was lucrative. I guess now I think the way it
8:12
happened is that like, I was kind of looking around and I
8:16
just had, I had never like again, my wife said this to me
8:18
after the fact. Like, how did you never have
8:20
this realisation? I never thought about like, oh,
8:21
I really love math. I really love basketball.
8:25
Is there a world where I can combine these things?
8:26
Look, it just never. And she makes fun of me to this
8:29
day because I said, well, I never like, if I thought about
8:31
working in the NBA, I thought I'd be playing.
8:33
And now like, I thought I was going to play in the NBA and I'm
8:36
a realistic human, human being. I I'm not even 6 feet.
8:40
Like, I was never going to be a professional baseball.
8:42
But what I meant was I didn't really think about, like these
8:44
other careers that might, you know, I just didn't have the
8:47
awareness that there were other things you could do to be around
8:49
4:00. So part of it was like this sort
8:52
of mental transition of like, oh, what are they like?
8:54
What does industry look like? What are the things about
8:56
studying physics that I'm really enjoying?
8:59
What are the things about, you know, the data analysis or you
9:03
know the problem solving that I like and what kind of career
9:06
paths are there? And then sort of simultaneously
9:08
at at UNC at the time there was some groups working on
9:14
basketball analytics stuff and so I just kind of got involved
9:17
as like ohh, that seems really interesting.
9:18
We like kind of tap into that and ended up sort of being this
9:22
career path and I was able to to find and so.
9:25
So, yeah, that was kind of the the transition.
9:28
And it started as analytics, but it went into and evolved into
9:32
machine learning. Yeah, sort of.
9:35
So it started from a very much sort of like traditional data
9:38
science or sort of statistics and analytics place.
9:42
There is a professor that that started a conference, a
9:45
basketball and it's concrete specific to kind of that
9:50
analytics piece that you know there's there's a big, the Sloan
9:53
Sports Analytics Conference, there's a big conference in this
9:55
space. And now is kind of runs the
9:57
gamut from sort of just very best to analytics to like very
10:00
advanced machine learning and like tracking and like you know
10:05
computer vision. Like there's a whole bunch of
10:06
things that that play into this now.
10:08
But this was a very small like very tailored, centred around
10:11
students a lot and a lot of these open conversations.
10:14
So that was my first first foray and I started kind of just doing
10:17
some side projects, looking at simple things, higher level
10:22
numbers, very analytic like more political than predictive I
10:28
guess. And one of the people that came
10:31
to the conference ended up being and it would be my future Bob.
10:35
So I just started talking with him about he had showed some
10:38
things they were doing on basically using tracking data,
10:41
so you know data about the motions of players on the ball
10:44
and trying to do that to do essentially like play search,
10:48
right. So you kind of have this visual
10:49
representation of of like a sequence and then you try and
10:52
find show me everything like that display or you know that
10:55
sequence and they have been applying it both to soccer or
10:59
football and non American football and basketball and I
11:04
saw that's that's like I've never seen anything like that
11:06
that's really fascinating. I just got to talking to him
11:08
about it and eventually ended up getting my first sort of post
11:12
grad school career there, a company called STATS LLC at the
11:15
time. Another called staff performed.
11:17
So I got to ask. And I think that's, yeah, that's
11:20
when I started doing more actual machine learning.
11:23
One of the things that I think the American announcers get a
11:28
lot of shit for abroad is that they don't really announce or
11:32
commentate on the games. They just basically run through
11:37
a bunch of stats and you always see in, especially in American
11:41
football games or in an NBA game, at some point in the game
11:44
you'll get the announcer saying this is the first time that this
11:49
has ever happened. And they go through like they
11:52
they slice and dice the data in a way that it's like the the
11:57
amount of free throws that have been hit by a guy from Slovakia
12:02
with his left hand. It's the first ever you know.
12:06
So remember was that you could do we know.
12:10
I think that was. OK.
12:11
So me specifically, maybe not. I mean, this is the type of
12:16
thing that that the company staff and all staff perform that
12:20
did. And they did it in a few
12:21
different ways. A lot of it was individuals who
12:25
were just really knowledgeable and passionate about sports.
12:28
And so they would actually like when things would happen in a
12:30
game, they'd be like, huh, I wonder like how many times
12:32
that's happened before. So those types of things that
12:34
you're calling out sometimes that's just a human who's
12:38
sitting there watching the game and is providing basically this
12:40
research support to broadcast one of the one of the things
12:43
that stats did. And so they'd be there watching
12:46
the games, I think sometimes even at the ballpark or or
12:50
wherever. And they're like, oh, that was
12:51
really interesting And they, you know, they queried it.
12:54
We have access to the data. So they they construct a query,
12:56
they figure it out. And and as you highlight,
12:59
sometimes they're so esoteric that it they almost, you know,
13:03
become meaningless. Like, I remember 1 there was a
13:06
Chicago White Sox game and it was like, this is the first time
13:08
that three outfielders named Rodriguez have all been on the
13:11
field at the same time. And it's like, OK, I guess
13:13
that's interesting. But like, is it really like,
13:16
what does that really mean about, you know, is that really
13:18
interesting about the game? Like it is an interesting
13:20
factoid I guess, but. Yeah, exactly.
13:23
It's exactly that. Yeah.
13:24
But there were some attempts, some attempts to try and
13:27
automate that or try to try and build models around predicting
13:31
or defining what is interesting and like a few different
13:34
approaches. There's sort of the, like,
13:35
anomaly detection approach, right, of just like how do we,
13:38
how do we know everything that's happened in this game or
13:40
everything that's ongoing? How do we find things that are
13:43
unusual, right. Statistically unusual, right.
13:45
So that's one way you can go. And there's another approach
13:48
that like, what do people actually find interesting,
13:50
right. And so you kind of, you can get
13:52
some explicit or implicit feedback on these insights or
13:56
these facts, right. You can kind of maybe you tweet
13:58
them out and you use like engagement as some proxy for
14:02
their interest, right? Or maybe you actually ask people
14:04
on some like 5 point scale, let's say.
14:06
This is not at all interesting, but this is very interesting and
14:09
you could try and have you know sort of a two-part system,
14:12
generate these snippets maybe or like highlight these things and
14:15
then try and predict whether or not people will find it
14:17
interesting. So, so there's something sort of
14:21
data-driven to that. That's kind of an interesting
14:23
problem. So this is a perfect segue into
14:29
one of the other things that I wanted to talk to you about and
14:31
how you approach building machine learning systems and
14:35
specifically like the tools and platforms that go into that,
14:39
right. Because you're talking about
14:41
basically like getting human feedback there from someone on,
14:45
hey, is this interesting? Is this not, That's one piece of
14:48
the puzzle, but there's a lot that goes into that platform and
14:54
that system before it even gets to where a human can say like,
14:58
yeah, cool or no, that was boring for me.
15:01
You've designed many A systems. I know that was like your
15:05
beginning of your career. Since then you've designed many
15:09
other ones. How do you approach it now when
15:11
you're looking at, OK, we need a tool or we need a platform that
15:16
is going to help us do XYZ to get this machine learning
15:19
product or this AI product out into the world and creating
15:23
value for our company. What does that look like?
15:27
Yeah. So I think there's like a
15:28
practical approach that I try and by try and think about,
15:33
which is like how much pain is it causing us to do this or like
15:38
how much churn is associated with with some aspect of this
15:41
process, right. Like that there is an approach
15:45
of like ohh, there are these tools and there's all these
15:48
there's all these tools out there.
15:49
There's all this development in terms of you know these new
15:54
products and these tools and I can I solve this problem and why
15:57
problem, But like, you gotta know that you have a problem to
15:59
solve first, right? And undoubtedly if you're a
16:03
company that's really anywhere in your sort of ML or AI journey
16:07
in terms of adoption, you probably have problems that some
16:09
tool might. But it's just a question of you
16:12
know where, where do you go? Do you bring in that outside
16:14
tool or do you try and like pick away at it internally?
16:19
So yeah, for me it's like is this a one off thing?
16:22
Is this going to be a repeated problem like have I done this
16:24
once before? Is this only the second or third
16:26
time I'm doing this? Then that should probably look
16:28
at how can we maintain this and sustain this in a way that is is
16:34
like repeatable, that we have some standards in place that you
16:39
know things that are alike should largely look alike in, in
16:44
broad strokes. And by that I mean that like the
16:50
mechanisms, the the kind of the structure, how things stick
16:53
together, how they flow, you'd be getting deployment pipelines
16:56
or CICD pipelines or you know how you can sort of compose
17:01
services. The, the structure of that
17:03
should largely look the same because you want it to be easy
17:06
for people to to become familiar with them, right?
17:10
Undoubtedly, you're gonna have people moving into and out of
17:12
working on these systems and you're going to want to change
17:14
them. And they and the easier they are
17:16
to understand and the easier they are for people on board to
17:18
the easier they're going to kind of change, right.
17:20
And some of that requires some of those structures in place.
17:25
You know best practises around, you know, project structure and
17:29
sort of CICD testing and all that kind of stuff.
17:33
But some of it is just like understanding where your team is
17:39
at or where your user. If you're building machine
17:41
learning platform tooling, you're just draw internal where
17:44
they are at and trying to meet them somewhere that's reasonably
17:48
comfortable, right? Because we could.
17:51
We could shoot for perfection. We could shoot for like the
17:53
Absolute by the book this is how things should be.
17:57
But if you're if a lot of your team is coming from a research
18:00
background for example, and they're only like sort of
18:03
familiar with Git best practises for example, it's gonna be
18:05
really challenging for them to go from that starting point to
18:10
to perfection. And there's going to be a lot of
18:12
bumps and there's going to be a lot of like, well, this is in my
18:15
way and I can't really like I want to learn this, but I I
18:18
can't do it fast enough. And so I'm just going to fall
18:20
back to doing it the way I'm comfortable with and I'm going
18:22
to kind of put it to the side and I'm going to hide it, right.
18:26
And so that's, I guess that's part of my approach is trying to
18:29
think about understanding who are the people are, what kind of
18:32
problems are they working on, what tools are they working
18:34
with, what's comfortable today and trying to think about, OK,
18:37
how can I make this easy, right? Do I need to build out some
18:41
template so it's very easy for them to just have all these
18:44
things in a structured way for all their projects?
18:47
Or do I need to develop some internal sticks?
18:50
Maybe those sticks are just simple Python libraries that are
18:53
a bridge between, you know, some new system and kind of what
18:58
they're familiar with today, right?
19:01
And that's kind of where I start.
19:02
Before I say like A there's tool X and we need tool X because
19:08
tool X can do can do ABC123 because it's it's it could be
19:14
it's hard it's hard to bring in a new tool.
19:16
It's hard to onboard users. It's hard to get them to change
19:19
from something they that they are using existing to move to
19:22
something new. And so there has to be
19:24
compelling and it has to be easy, right?
19:27
It has to be like clear tangible outcomes and it has to be easy.
19:32
I mean and you can always come down with the hammer from above
19:35
and say no, this is the new tool you got to use it and and that's
19:39
an approach and there's you know there are times where I where I
19:42
do think that you kind of got to push people a little bit, right.
19:46
You need you need to push them to change because because you
19:52
know we are creatures of a comfort and creatures of habit,
19:54
but but it has to be what you're asking of them has to be
19:59
reasonable. Are there places that you feel
20:04
like you can generalise and say like you were saying, like CI CD
20:09
is always a great idea, right? No matter what your use case is,
20:13
no matter what your end users are doing.
20:16
That is a good thing to have, probably a need to have, not a
20:22
nice to have. Are there other things in the
20:26
mill workflow that you feel like is a must have these days as and
20:31
I understand that. Yeah, 2-3 years ago it may have
20:35
been completely different. You probably were working with
20:37
people or data scientists who were like, hey, we're tracking
20:41
everything in Excel and that was cool.
20:43
That was good enough because the maturity levels may not have
20:46
been there. But now, as we feel like
20:48
maturity levels are much more mature, for lack of a better
20:54
word, what does it look like? What are the need to haves and
20:57
things that you can generalise no matter what, if it's a
20:59
computer vision problem or if it's a just straight tabular
21:03
data problem. Yeah.
21:05
So some sort of deployment pipeline, promotion pipeline and
21:10
deployment pipeline, right. So and part and part of what
21:14
that assumes. I guess, taking a step back,
21:17
part of what that assumes is that you have some standard in
21:20
place, some structure in place for how people develop and and
21:24
iterate on their problems, right?
21:26
And so one of the things that I think about us being standard
21:31
for an MLS stack is a bridge between a bridge from notebooks.
21:34
A bridge away from notebooks, right?
21:35
I mean or or something around horrifying that notebook people
21:41
have ever opinions and we're not going to get into the notebooks
21:44
and production date but people have different opinions about
21:45
this. The way I think about it is that
21:47
like notebooks are a really great tool.
21:49
They're a great tool for exploration, for getting
21:52
something an idea out fast and iterating on it.
21:55
And but I want it to be very easy for people to abstract that
21:59
work, right, so that they can then share it very easily in a
22:02
package, let's say or in you know some you know, container or
22:05
some other shareable asset that other people can come into it
22:11
and have a clear understanding of how to interact with it.
22:14
Meaning that if it's just kind of in, you know in a long stream
22:16
of consciousness sort of notebook, it's not always easy
22:18
for people to come in and put themselves in wherever your
22:20
headspace was and understand your context.
22:22
So pulling that out, abstracting where you can, putting testing
22:26
around it, unit test ideally, but integration test it you know
22:31
where they're appropriate and then having a release process
22:35
for that, right. Every time you having a clear
22:37
sort of policy around this is sort of a branching strategy.
22:41
Again, just trying to make it easy for people to adopt those
22:44
standards. And so sometimes depending on
22:47
the tool you're using, that can be a little bit tricky, right?
22:49
Like some tools are very I've worked with data bricks a lot in
22:53
various contexts and that's a tool that's very interactive.
22:56
Like it's it's very much built around being in the UI and
23:01
pointing and clicking and developing A notebooks and then
23:03
I'm gonna build a workflow that points that notebook and so on
23:06
and so forth. And I think that's great from
23:08
Databricks perspective as a company that wants to sell a lot
23:10
of software and enable users because it's very easy.
23:15
But so then how do I find that middle ground of like, yes,
23:17
that's easy, But also like, it's very easy for them things to get
23:21
lost and for things to not be tracked.
23:22
For us to not be able to reproduce things for us, not to
23:25
be able to deploy things easily and move them from whatever
23:29
we're going to call this dev environment to some higher
23:31
environment like production. And so that's where I talked
23:35
about sort of templates and internal sticks, very simple
23:38
wrappers that kind of foster that process of saying, great,
23:41
you did this cool thing, you sketch it out, you have this
23:43
idea. It should be very easy for you
23:45
to take that idea and encapsulate that in a project
23:48
structure and be very easy for you to sync that with some get
23:54
remote and should be very easy for you to apply that to other
23:57
workspaces or other environments.
23:59
It should be very easy for you to take that idea and
24:03
operationalize it. Like put experiments around it
24:06
and run through experiments and track the outcomes.
24:07
So standards, project structure, get best practises.
24:13
Testing to me is a standard. It's.
24:15
I know it's it's not always easy for people who don't come from
24:19
software engineering or software development background to get
24:21
their head around testing, especially for machine learning.
24:24
But there's so many areas where it's really easy to to put in
24:28
good thoughts and practises and guard against your future self
24:32
and make it easier for other people to understand your CD
24:36
pipeline. And then you know, again, these
24:40
light integrations, right, like make it very easy for people you
24:44
talked about, like experiment tracking Excel.
24:46
People still do that. People still, even when they
24:48
have tools available, still do it just because it's comfortable
24:51
and easy. And so that's where it's like, I
24:53
need to make it easier for you to use a great tool.
24:55
It's so easy to use, but if you've never used it before,
24:59
it's the new thing. It's the cognitive load that you
25:02
have to take on and like inject into your process.
25:05
So how can I make that as easy as possible for you?
25:08
Like ideally you don't have to do anything you know?
25:11
I'm all flows on a great job of like all these auto loggers they
25:14
have now and they work to varying degrees under if I
25:17
understand how you're building things, I understand the
25:19
framework tools you're using. Hopefully it's just it's a no
25:22
for you. It's not even a thing you have
25:24
to think about. It just kind of happened, but
25:26
that's not always possible. Yeah, it reminds me of this
25:29
whole idea of the cultural aspect on things and if somebody
25:33
is comfortable with the Excel then they potentially aren't
25:38
going to realise that. Ohh, wow.
25:40
It's just this easy. And all I have to do is, you
25:43
know, these one or two extra steps, whether that's in Mill
25:47
Flow or Neptune or weights and Biases.
25:49
And now I have this new world that's opened up to me.
25:53
And from there, what's beautiful is what I've seen other
25:57
companies do is they'll have like almost a, I don't want to
26:02
say like peer pressure, but they'll have just like the days
26:04
where different data scientists can hang out with other data
26:08
scientists and they get to exchange notes on what they're
26:11
doing that's working, how they're doing it.
26:13
Or you have something like one of the community members,
26:16
Stefan, He's now an internal developer advocate going around
26:20
teaching the users of the ML platform how to use the ML
26:25
platform better and how can they, how they can leverage it.
26:29
And it's like they don't have any strong rules as to you have
26:33
to use the ML platform. But he sits there and he says,
26:36
well, you know, if you want, you've got all these benefits.
26:39
Here are some of the benefits. And he's teaching these
26:41
different data scientists or consumers of the platform how to
26:46
leverage the platform better. And so I could see that.
26:49
I mean, no. I do want to ask you about
26:51
something that you glossed over, and I know that you love it.
26:54
Your love for tests and explain what the tests look like and how
26:59
you go about writing them and why do you like writing tests so
27:03
much. Yeah, I I will talk about the
27:07
debt debt advocate roles in an important one and I think you
27:10
you hit on, I think it's it's huge because just to kind of put
27:13
the final point on the last that last sequence.
27:16
I think that if you're building, you're like if you're building a
27:20
new platform, right people should want to use it right and
27:23
it shouldn't be painful for them to adopt, right.
27:26
And so that's another kind of how I think about it is how can
27:28
I make this look so cool and so attractive and so easy and so
27:32
beneficial that I don't have to push people.
27:34
Like adoption is something they want to do.
27:37
And so that finding people who are good at that education and
27:40
training piece and can like, hey, here's this full example,
27:43
let me walk you through it. And here's like ohh like how
27:46
easy it was. And here's now I have all my
27:47
experiments and I can compare them and I can.
27:49
Yeah, yeah. Yeah.
27:50
So I think that's a huge role. So I appreciate you saying that.
27:53
Fascinating. What do I love about testing?
27:55
So again, I started my career as an academic.
27:57
I was physics. I was doing kind of doing
28:00
research but running a lot of code, writing a lot of software
28:02
and never wrote a test for a long time and started seeing
28:06
some tests again with with some of the components that that the
28:10
broader team, the software engineers on on my team in grad
28:13
school building. But I didn't really have an
28:15
understanding for kind of where it fit because I wasn't like
28:18
day-to-day in the trenches built like working on these systems.
28:21
I was more consumer. I was kind of helping out some
28:24
of these build trip but when I got to to stats I I primarily
28:30
was a data scientist. I was primarily working on these
28:34
research questions trying specking out new ideas.
28:39
But then when there was the occasions where it's like hey we
28:41
have this whole, this data science pipeline is an
28:44
analytical pipeline basically taking all these raw measurables
28:48
about what happened and and and again in a game you know
28:53
sporting events at the at the game level or even like at the
28:56
you know individual play level and we wanna creating all these
28:59
derived measures for them things like likelihood to score in the
29:03
next seconds or something like that, right.
29:05
That all comes from some enriched on top.
29:09
And there was basically this opportunity like well, we need
29:12
to go and we need to embed with the engineering team that's
29:14
going to build out this ETL process.
29:16
But we need someone there that's going to like sit with them to
29:18
kind of help answer questions, understand.
29:21
And I was just like, yeah, that sounds interesting.
29:23
I'll do that. And that was my first like foray
29:26
or Berry like you know deep dive into software engineering best
29:30
practises and like things like pair programming and testing and
29:36
various types of testing and even like you know various
29:41
stages of deployment pipelines and and sort of like trunk based
29:45
development. I never sort of done that before
29:47
and like just kind of, yeah, everybody's working together.
29:49
But as long as you're sort of doing things, you know, with
29:54
this assumption, with this mindset that we're all kind of,
29:56
you know, working on in this way that it, that it, that it all
29:59
works out. And I think the thing that I
30:02
really like, that I really took away from it was was a few
30:06
things. Hey, I'm Vishnu.
30:08
I'm a data scientist at first hand, and I definitely think
30:12
that you should subscribe to the ML OPS Coffee Sessions podcast.
30:15
It's the best podcast out there to stay on top of what ML OPS
30:18
actually is, to talk to the true thought leaders in the space.
30:22
And oh, by the way, Demetrius. Absolutely hilarious.
30:25
What a weird guy. You should definitely subscribe
30:28
to the podcast. Like 1.
30:35
Umm, I had never seen. I had never written in Java
30:38
before, and all the ETL pipelines were in Java, and so
30:42
I've taken some classes in C Once Upon a time I primarily
30:45
worked in Python, but I was like kind of just like OK, I'm
30:47
pairing with someone like a Java developer Java engineer, having
30:51
never written Java before in my life.
30:52
But I could do it over time, right?
30:54
Because there was this structure to the way the things worked and
30:59
there was this entire testing where I could go look at example
31:03
test. We're trying to make a change to
31:05
this part of the code base. You know, there's this, there's
31:08
this class that does this piece of enrichment with this method
31:10
and we want to go update it to now do something else, right.
31:15
And so they're they're being all these tests.
31:18
They provided that example, they provided that concept context to
31:21
understand, OK, I can kind of look at the syntax now and I can
31:25
the syntax takes time to learn, but I could understand the
31:29
structure and how things were structured and how they, how
31:31
they operated and how they interacted because I had that
31:34
kind of safety net of, of tests, right.
31:37
So that's one thing I really love and why I really advocate
31:40
for testing is and and testing early, like you don't
31:43
necessarily have to do test driven development, but you need
31:46
to, you need to integrate testing early in the process.
31:48
Because the thing you constantly hear especially in the data
31:51
science the community is like, well, if this idea gets a
31:54
production, that's when I'll write tests and that's why I'll
31:57
put tests around one product, you know, whatever.
32:00
And the problem is, is that like that's three months later,
32:03
you've lost all contacts on what you were thinking when you were
32:06
writing that feature engineering pipeline, right?
32:09
Or that transformation logic. And like you don't.
32:12
You could probably come up with some decent examples after the
32:15
fact. But when you're in the context
32:16
of like, oh cool, I had this idea, this is, this is the
32:19
feature we need. I'm going to develop it.
32:21
This is how I'm going to do it. That's when you have the
32:23
context. And so that's like another big
32:26
piece of it for me is that like it saves you time later on.
32:30
It's such a time saver later on and and like I get with ML
32:36
because it's so much of it is speculative that like that is a
32:38
little bit like it's not obvious it's not black and white right.
32:41
There is a little bit of nuance to like do I need to like what
32:45
what level of development of an idea is appropriate and and
32:49
that's where I like try to build bring back to the sort of
32:52
project structure. That's why I try to build all
32:55
these tools around, pulling stuff out of notebooks and
32:57
encapsulating it or abstracting it.
32:58
Because that is the right if you are pulling it into a Python
33:03
module, or if you are pulling it into a project structure that
33:07
you can even go push to a remote like that's the time.
33:10
You need tests. That's great that you have a
33:12
clear idea of alright, it's all fun and games, exploratory mode
33:17
when you're in the notebook and I see 100% this idea of you have
33:23
it so fresh in your mind that it's very clear to you what
33:27
tests need to be written and don't need to be written.
33:29
And so when as soon as you come out of that notebook, you're
33:32
like you're not going anywhere without writing at least some
33:36
kind of test that will help us all get context and understand
33:39
where your head's at. Yeah.
33:41
And there's also like this, just, there's so much good
33:43
cognitive feedback from writing tests.
33:45
Like, I think that people don't understand until you've gone to
33:47
the process of, like, I wrote a test, that test is failing, my
33:52
ID is lighting up red, I adjusted things to make the test
33:55
pass, and now it's all green. I get all these like very
33:57
satisfying little green check boxes on my work, right?
34:00
Like it's seems silly, but it is very satisfying.
34:04
It's like mowing your lawn or something.
34:06
I don't know. It's like breaking leaves or
34:08
not. It's like a very satisfying.
34:11
Yeah, it's a dopamine. Yeah, it's a total dopamine for
34:14
sure. That's basically how developers
34:17
do social media dopamine hits. Basically.
34:22
We don't have to go on Instagram and get likes, we just have to
34:26
make our test pass. So right, more tests, Megan.
34:30
Yeah, it's just right. Five more tests, give more
34:33
dopamine. That's a new shirt.
34:35
I like it. So dude, well, I do want to talk
34:38
a little bit about the like, Brent kind of drifting over into
34:42
the cultural side of things. The thing that you mentioned
34:47
about teams falling down or not performing to their highest
34:53
ability a lot of times comes down to people not understanding
34:58
each other's languages. And so when it comes to machine
35:02
learning projects, I think we all know that there's a lot of
35:05
stakeholders, whether it is on the business side of the house
35:08
or it's on the technical side of the house.
35:10
You just were talking about how data scientists come into things
35:14
with their whole preconceived notions.
35:18
And then you have software engineers who may be coming into
35:21
something with this test driven development.
35:24
Everything has to be tests. Or you have data engineers who
35:27
really like to write and SQL or things like that and just want
35:31
to figure out the best way to move around this data.
35:35
And then you have the plat or the product teams or the product
35:38
owners who have their ways of looking at it.
35:40
And So what roles have you seen people speaking different
35:47
languages in, and how have you remedied this, or how can we
35:50
bridge that gap? Yeah one of the most successful
35:57
from like a from my perspective from a development perspective
36:01
and ML teams that I was part of was that's that's performed.
36:05
We were basically building out a model driven API for sports book
36:11
operators. So you asked about betting
36:12
before. I've never, I've never found
36:14
sports myself but this is a this is a very clear need get you
36:16
know gambling on sports became legal and I think 2018 in the US
36:20
And so we basically had this remit to take some of what we
36:23
were doing and build out a service you know that was trying
36:28
to provide these these predictions of things like
36:33
number of points and individual player was going to score in a
36:35
game right and provide those predictions pregame within some
36:39
window with some likelihood happening, right.
36:42
And so, you know, there was certainly some data science and
36:44
machine learning works to build out those models and how do we
36:47
understand how things happen. And you know sort of generally
36:50
on for the average player versus this specific player and
36:53
understanding some of this, there's a lot of interesting
36:55
data science and machine learning work that went into
36:57
that. But the team that built that out
36:59
of the service, This is meant to be an API consumable by, you
37:03
know, by sports book, by whoever, and had all these
37:09
different sort of triggers for updates too.
37:13
That was kind of one of the complications cause like if
37:16
you're talking about the sports betting world, there's a lot of
37:18
things that can cause it. Huge shifts in what you might
37:20
predict, right? Case in point, like if a LeBron
37:25
James is listed out for a game that's gonna super change your
37:28
predictions for the entire team or and even probably the
37:30
opponent, you know, the opponent team, right.
37:32
So you got to understand how all these different triggers play in
37:36
and be able to update very quickly all these predictions.
37:38
So there's a very like real, just like architectural
37:42
challenge to this. But one of the things was so
37:44
successful about just from a development perspective and
37:48
building out the service was that we were a fully integrated
37:52
team. And that doesn't always work.
37:53
But in this case it worked really, really well.
37:55
So there was data scientists, what you would call ML
38:01
engineers, sort of full stack software engineers, a technical
38:06
lead on the MLS side, a sort of engineering manager type person,
38:11
and a product owner all on one team.
38:14
One of each. Like the invariant degrees.
38:17
So like technical like I was the tech lead on sort of the ML or
38:21
kind of for the whole project more on the ML and and like some
38:24
of the engineering side and there was an engineering manager
38:26
that I partnered very closely with and the PO And so we were
38:30
kind of like a triangle, right of like different spheres of
38:33
influence but how heavy, heavy amount of overlap in various
38:36
ways. So you talk about like like
38:39
defining stories and defining the work and prioritising work
38:41
that was a collaborative process between the three of us.
38:43
We were able to sort of cover gaps and understand like, Oh
38:46
well, we've got to think about if we're going to do that or
38:50
obviously the product owner was the best suited to say, hey, you
38:54
know, I just met with with one of our, you know, sort of
38:57
potential clients or one of our early clients and they and they
38:59
said we're really customers and they said, oh, that like they
39:02
really need this feature on. We kind of got to prioritise
39:04
that. But I don't really know how to,
39:06
you know, how do we set that in and what is that, what is the
39:08
impact of that change, right. Like not everyone's gonna have
39:11
all of that in their head necessarily.
39:12
I think there's some really great technical product owners
39:15
and technical PMS that can do a lot of this.
39:17
But I think kind of that collaborative approach is really
39:19
important. And then we do, we kind of like
39:22
like daily stand up, everybody was there, everybody kind of had
39:25
this shared, right. So all the all of that process
39:28
that development was very shared and there were some cases where
39:32
like you kind of had to look at it and be like well there's a
39:35
front end component to this. We had a front end interface for
39:39
essentially human in the loop predictive process, right.
39:42
And so there was an interface for humans, experts to sort of
39:47
enter things like the amount of minutes a player was going to
39:49
play, like adjust those because that was really hard to predict.
39:53
And so like, maybe we wouldn't have a data scientist or animal
39:57
scientist pair with a full stack engineer to work on like a view
40:01
or react app like that just didn't necessarily make sense,
40:04
right. So there's a little bit of like,
40:06
yeah, we're not all working on the same thing all the time, but
40:08
there was a good amount of overlap and a good amount of
40:10
like pairing and shared context and so.
40:15
I just the there was a lot of things that came out of that
40:18
like one we were able to move really quickly because we were
40:21
kind of all together all had shared contacts to like a lot of
40:25
the sort of ML engineers or ML scientists really learned a lot
40:29
and got upscaled in their software engineering best
40:31
practises. And then similarly, some of the
40:34
sort of software engineers who maybe didn't have any background
40:37
in machine learning and didn't really have never really worked
40:40
with any of the frameworks or tools that like, you know, that
40:43
kind of sits in the middle. They would have exposure and
40:45
they'd have some really great ideas about like ohh well, it
40:49
you know once they kind of understand stood how these
40:51
models are processed or what are the triggers or what are the
40:53
kind of requirements, they'd have some really great ideas
40:56
about how we can break these things apart, how we can start
40:59
architect this system right. And that doesn't always work
41:02
right. You can't always have support a
41:07
team you know that like dedicated A-Team depending on
41:10
kind of your suite of products and how many you have and how
41:14
many how do you do you have the capability to sort of just embed
41:18
a resource like a data science or ML resource with this product
41:21
team. But that was one of the one of
41:26
the most successful sort of from a development and then you know,
41:29
ability to kind of iterate perspective and it really
41:32
reduced that productization or optimization gap that often
41:37
happens when you have data scientists or ML engineer, mill
41:41
scientist or even engineers working on some model working on
41:43
something and they say, OK, great, I'm here like it looks
41:47
good. There's a great idea.
41:48
We want to, we want to put it into production engineering team
41:51
here it is like, what do you need to know, right?
41:54
And then also this whole process of like learning and and getting
41:57
that shared context and like going through iterations of
42:00
like, oh, I did XYZ and it's like, oh, but we looked at the
42:02
data and it's not quite right because you forgot whatever, I
42:05
don't know, whatever the case is, whatever the case may be,
42:08
there's that lack of shared context.
42:10
And so that was the big kind of thing that came out of that was
42:14
there was there was a very minimal gap, there was very
42:17
little translation that needed to happen and people were able
42:20
to then work together rather than saying like going back and
42:23
forth asynchronously. Yeah.
42:25
Over a course of weeks. So some of that is wasted by
42:28
tool, right. Like this was three feature
42:32
stores basically like I think midway through that project is
42:36
when tucked on was like in early trials and they were kind of not
42:40
the first obviously but one of the early ones to really pull
42:43
this concept out very, very widely around feature stores.
42:46
And and I think that that's fostered some of that closed
42:49
some of that communication gap. If you bring in a feature,
42:51
sometimes it's a nice communication medium where data
42:54
scientists can be all know those feature definitions way through.
42:57
But that's that's one part of it.
42:58
That's not everything you know. There's still benefits to
43:02
working. It reminds me of a talk that you
43:07
and Nicholson gave I think probably like no two years ago
43:12
and he was mentioning how he felt like before where you would
43:18
have and he did it very visually.
43:20
He had a slide and he said here's let's imagine you have
43:24
all these different job titles and each one of them has a
43:28
respective like colour like a rainbow on you know like stripes
43:34
as it goes across the slide where all software engineers or
43:40
back end full stack whatever you want to identify as you have the
43:43
red sliver and then the data engineers have a yellow sliver
43:47
of and it's all yellow where they are And then the machine
43:51
learning engineers are purple and the data scientists are
43:55
green. And what he said is how when he
43:59
decided to break up the teams and try to find that shared
44:04
language, and he spoke about it a lot like you did.
44:07
If I'm understanding correctly, it's like, can we create a team
44:10
and embed data scientist with engineers, with machine learning
44:15
engineers and data engineers and software engineers, all of that,
44:19
so that it's its own little unit as opposed to having the
44:23
product, the machine learning product team servicing the data
44:26
and data scientist team. He was mentioning how that when
44:32
that happened, they started to find that shared vocabulary and
44:37
the colours stopped being so defined and it became like a
44:40
gradient as opposed to very clear.
44:43
Like here's the yellow, here's the blue, here's the purple type
44:46
thing. It's like, no, it's a nice mix
44:49
and you're not really clear on the colours because a full stack
44:55
engineer understands what a feature store is and a data
44:58
scientist also understands what a feature store is.
45:00
And so they can save feature store and they both go, oh OK,
45:05
that's like decoupling these feature definitions from the
45:09
code or from the models, whatever it may be.
45:12
And so that just exactly what you said, it reminds me of that
45:17
perfectly like ohh now it's a gradient type of view as opposed
45:20
to very clear like slices of colours.
45:24
Yeah. Or there's even a time where
45:26
like if I were to say surveying or inference model serving or
45:29
model inference like that would like, I don't really know what
45:31
that means. You know, if I'm there would be
45:33
engineers on the team I don't understand.
45:35
And so it took some time, like you said, for us to kind of get
45:37
to a place where where there was that shared language.
45:40
And I think part of the reason I liked it so much is just I, I I
45:43
experienced it first hand, right?
45:45
I experienced everything that I was able to learn and develop as
45:49
a broad range of skills in software development and
45:52
software engineering and ML system architecture from being a
45:56
data scientist. It was like, yeah, I'm going to
45:57
bed with a team with a ostensibly a software
46:01
engineering team and I'm going to just be really confused a lot
46:05
and learn a lot. And you know I think it's to me
46:09
it's it's similar to how you know handful of you know maybe
46:13
time or 15 or I don't know however many years ago DevOps
46:16
and like infrastructure was were dedicated in separate scenes and
46:20
it's a very degrees they still are in a lot of organisations.
46:23
But I think it's much more common today for software
46:25
engineering teams to own their DevOps pipelines, right to say
46:29
whatever you get up action sort of the eye or whatever you're
46:32
using that they understand and own the process of deploying
46:35
their code from local to remote and between environments.
46:39
Right. And similar for like owning
46:41
their infrastructure, right. If you're using cloud and now
46:44
you're using infrastructure as code, like there's just it's
46:47
less common for them to say like, OK, team A, can you please
46:50
go provision me. But you know, you know these
46:53
pods on a cluster or an EC2 instance or whatever it is and
46:56
then I'm going to go deploy my give me access and go no, like
46:59
you, if you're building a thing and it requires infrastructure,
47:02
if it requires deployment, you should own those definitions.
47:04
You should understand how they work.
47:06
You should be able to kind of take your thing from end to end
47:09
and I think that specialisation, that silos those teams off
47:16
breaking, kind of breaking down some of those walls, seeing it
47:18
happening in various ways in data science, machine learning.
47:21
I think it's generally a good thing and it's sort of a similar
47:25
trend a little bit. Yeah.
47:28
Now I want to go through some rapid fire questions with you in
47:32
these last five minutes and I'm going to start with a book that
47:36
has impacted you the most in your career.
47:41
I don't know if it's the most in my career overall period, but
47:44
when I read very recently is Accelerate.
47:47
Is this spelled X sneller rate with an Ohh?
47:51
No, it's actually super alright. Nice.
47:56
Just the word accelerate, yeah exactly.
47:57
So great, great book really you know complement to like the
48:01
Phoenix project or the DevOps handbook any of those like I
48:04
think just as I've kind of come out the other side of maturing
48:08
around sort of building systems and and DevOps and kind of
48:11
understanding that from the first hand.
48:12
It's a it's a great sort of like summary of those principles and
48:15
and it'll look like a very data-driven look at why they're
48:18
what the impact is so. What's the coolest or most
48:21
challenging MLL system you've been a part of building?
48:25
At my last company, legal Plug, strong analytics, data science
48:31
and machine learning consulting firm.
48:33
Very cool, cool bunch of people doing really interesting stuff.
48:35
We were asked to build essentially a a product
48:40
identification service. So you know the ability to like
48:44
snap an image of some store shelf product or something and
48:48
and then be able to identify and the thing was so challenging
48:53
about it was one of the machine learning aspect of just you
48:56
know, the computer. You find these computer vision
48:58
and search techniques and and vector database and being able
49:03
to kind of organise all that. But it also had to be
49:06
compartmentalised by groups of users essentially like some
49:11
contact like a store or a company or whatever so that
49:13
there was very clear guards around data and so just building
49:17
all the systems to kind of like spin up new stack of this.
49:21
So it's really, really challenging and really
49:22
interesting. And then it all got replaced by
49:25
Elms and so. No, no comment.
49:30
Hopefully not. Hopefully not.
49:33
Hopefully, yeah, hopefully not. There's some amount of it.
49:35
I mean, there there's a system aspect, right?
49:37
Like whether you, whether you're running your own models or
49:41
that's an API you're calling, there's still a lot of moving
49:44
pieces that you have to understand how to piece
49:46
together. Yeah, that's a great point.
49:49
So last one for you. I know you're an outdoorsy guy.
49:52
I know you got two dogs and four kids and a wife there.
49:56
What's the most fun or slash scariest hike or backpack
50:01
backpacking trip you've ever been on?
50:03
So I I undergrad was at UC Santa Cruz.
50:06
I was on the West Coast, spent a lot of time in Yosemite.
50:10
So I guess I have a multi party answer.
50:12
One time in the fall a bunch of buddies and I we went and we
50:15
hiked up. We have 70 falls which is
50:17
normally like A to you know in the spring of 2000 foot raging
50:20
torrent of water that falls into the Yosemite Valley.
50:23
But in the late summer and early fall it basically basically dry.
50:27
And so we actually hiked down the riverbed you're probably not
50:29
supposed to do and and sat basically with our feet over the
50:33
edge of Yosemite Falls. It was absolutely terrifying.
50:35
I am not a big fan of heights. So it was like took a lot of
50:38
convincing to get me even close. But it was just this very like
50:42
like the wind coming up. I don't.
50:44
It was a scary, like surreal experience denied another.
50:48
We did another backing trip on the Lost Coast Wilderness, which
50:51
you have to go look up where that is.
50:52
But we ran into a pack of elk along a very narrow trail
50:56
between essentially like the Cliff side and that was on one
51:00
side and then like the actual drop into the ocean was on the
51:03
other. So there was no where we could
51:04
go. And we just kind of had to like
51:06
hang up, hang out with this pack of like 30 or 30 fives, elk or
51:11
massive animals. Like I don't want to spook them
51:14
and have them like run us down. So, you know, but that was
51:18
really, really beautiful 25. Wow.
51:20
Incredible. Well, thanks, dude.
51:22
I think we'll end it there. And for anybody that wants to
51:24
reach out to you, we'll leave all the information in the
51:27
description. This has been awesome.
51:29
I really appreciate you coming on.
51:30
I knew it was gonna be great. I.
51:32
Appreciate it. I appreciate you.
51:33
Like I said, I've been been been around the community for for
51:36
years and and just love what you're doing.
51:37
I love like I go in every day and look at you know Emily's
51:40
question answers and and and sometimes try and help and
51:43
answer but sometimes I'm just reading and learning, learning
51:45
there's so much happening so, so I appreciate that.
51:48
It's that it's a community that's there, the leverage.
51:51
Yeah, And I gotta thank you. I mean, you're just as much a
51:54
part of it as I am, right? Like your answer is you helping
51:57
people also helps me learn. And so I'm very happy that we
52:01
got to do. This.
52:05
Oops, you have to immerse yourself in the ML OPS content.
52:10
The best way to do it is to subscribe to the E-mail OPS
52:14
Community Podcast, so good luck and keep learning.