Latest / Agentic Conversations (formally mlops.community) / Maturing Machine Learning in Enterprise // Kyle Gallatin // MLOps Coffee Sessions #43

Transcript

Maturing Machine Learning in Enterprise // Kyle Gallatin // MLOps Coffee Sessions #43 · Agentic Conversations (formally mlops.community)

0:00
The. Today we've got an excellent
0:19
conversation coming up. Kyle is working on the machine
0:22
learning platform team at Etsy. He's a software engineer there.
0:26
We were just talking before we hit record about how he's been
0:29
burying himself in coding so much that he can't even play his
0:34
guitar. He has not found any time to
0:36
play the guitar in the last couple years, but.
0:39
That is good for us because we don't want to talk about guitar
0:42
today. We want to talk about ML OPS and
0:44
everything under the sun when it comes to ML OPS.
0:48
And I'm excited to get into how you see the current state of
0:53
what is happening in this space. Kyle, I think it would be
0:57
interesting for us to start with your path.
1:02
From being a data analyst to a data scientist to an ML engineer
1:06
and then a platform software engineer, What did that look
1:10
like? And what kind of hurdles did you
1:13
have to overcome as you jumped from one to the next?
1:17
Yeah, definitely. I would say it's a somewhat
1:20
nontraditional path. I'd always like to say I've said
1:23
a bunch of times that I'd tripped and fell into the data
1:25
science space. So I was actually finishing up
1:28
my masters in molecular and cellular biology when I got this
1:33
random job as a data analyst. And my job there was to teach
1:36
myself like R and Python. So analyze the data that we had
1:40
at this biotechnology company. After generating some interest
1:44
in the data space there, I ended up kind of like fostering a love
1:48
for data and data science. I was doing some unsupervised
1:50
machine learning. Got super interested in that,
1:54
went to a boot camp here in New York City, so I did the whole
1:58
boot camp route. Then after some odd jobs ended
2:00
up as a data scientist and then machine learning engineer at
2:03
Pfizer. So kind of from like like more
2:07
and more like analytical stuff and statistical stuff, slowly
2:11
more towards the machine learning driven path.
2:13
And now I'm, you know, purely basically a software engineer on
2:16
the infrastructure side, so very back end.
2:20
There's some. Funny stuff happening right now
2:22
in the MLF community. Slack, which is a group of
2:25
people that decided data scientists need to or they want
2:32
to share horror stories of what they've had with data
2:36
scientists. When it comes to the code that
2:40
data scientists write, I want to get your take on how hard was it
2:44
or what were some things that helped you along the way when
2:48
you were learning to go from that.
2:50
Data Analyst to an actual full-fledged engineer, yeah,
2:55
that's that's a great question because there is a huge
2:58
difference in the in the required skill set and the
3:00
expected quality of those, those skill sets.
3:03
Obviously I, you know, didn't come out of school with a
3:05
software engineering degree and feel like I really lacked some
3:08
of those like fundamental like CS basics just from, you know,
3:12
data structures and algorithms that aren't as important.
3:16
I mean like maybe not even as important in a real software
3:18
engineering job, but like really build the base for like all of
3:22
the code that you learn beyond there.
3:25
And so it's kind of been funny like learning like what you
3:27
would call I guess more advanced data science code, but then
3:31
having to go back to like CS101 on my own and in my own time to
3:35
like also build that basis on the side.
3:39
I've actually had a very similar experience.
3:42
So I I studied bioengineering in my undergrad and and and masters
3:45
and you know, I think sometimes I'm like man, if only I had an
3:48
undergrad CS degree. Not that I would, you know,
3:51
change what I studied. But if I could just, like, tack
3:53
that on really easily, it would be it would make my life so, so,
3:57
so much easier. Yeah, no, tell me about it.
4:00
It's like, I'm I'm very happy with the path I took, but I just
4:03
always wish I'm like, wow, I wish I'd just taken more
4:05
classes, had more of that knowledge, or at least, you
4:07
know, paid attention the one time I took a CS class.
4:10
Yeah, exactly. Like I just want to be the just
4:12
want the freshman sophomore classes.
4:14
That's all I need. Yeah, exactly, exactly.
4:16
That's what I'm missing. Well, in in this production code
4:23
channel that we have. There was a pretty awesome
4:26
question that came up and I want to hear your take on this Kyle.
4:30
It's asking about like if you were to build a SAS app right
4:35
now with some heavy ML services from scratch, what languages and
4:42
frameworks and libraries would you use for the back end if you
4:45
had? You weren't under the gun for
4:48
time. You had your abilities that
4:51
your. Currently using and you could
4:54
really engineer this stuff the way that you wanted.
4:58
How would you go about doing that?
5:01
That's that's a great question. I guess it'll I'm I'm sure I'll
5:04
give different answers to at different points in my career.
5:07
Like, I mean for like a software for just like a language.
5:11
I mean, I might just have to go with Python, just cuz it's so
5:13
ubiquitous. It's easy to use.
5:16
I can write it quickly, I can test it quickly, like easy to
5:18
write for. I'm also like writing a lot of
5:21
Scala lately and it's awesome, you know, high performance, but
5:24
it's much more difficult comparatively to like, do the
5:27
things that I want to do, just because Python is so much
5:30
support for so many different things, especially within the
5:33
machine learning space. But like, yeah, I mean at the
5:37
end, at the end of the day, I feel like I'd end up writing
5:39
more YAML than than anything else, so you know, it all be all
5:45
be templated. Yeah the the YAML thing is
5:50
interesting because you know I think we both, you know I I work
5:54
in ML infrastructure as well and you know it's like sometimes I'm
5:58
like it's just it's so simple it's so templated it feels like
6:03
you know I'm just doing like almost like stenography work
6:05
where I'm filling out a form. But there's there's there's so
6:08
much that is based on that it it kind of blew my mind recently
6:11
like when I was really diving into some like CI concepts and
6:14
trying to figure out like OK, how do I make this whole like CI
6:16
pipeline process scalable for sort of like machine learning
6:19
repost etc. I was like man this YAML stuff
6:21
it's just it's what I could do all day but it's so it's so I
6:25
don't know replicable and and and and crucial to the workflows
6:28
nowadays. Yeah it's it is also everywhere
6:31
I I mean like I think it's like the the Python of markup
6:34
languages right. You know it's like simple, it's
6:35
easy to read like just like colons and and nothing else.
6:39
No brackets or anything like that but it's kind of the yeah
6:44
but like kind of every tool is is is reliant on it and and it's
6:49
cool. I don't know I like it.
6:50
I think it's easy to read and use and it's like a really nice
6:52
introduction to a ton of different concepts where where
6:56
templating is key. So I would like to ask you.
7:00
What are your maybe top two or three tips in terms of working
7:05
with sort of like YAML in general YAML files and like the
7:10
explosion that can happen that you would offer to beginners,
7:13
especially data scientists who are saying OK, I want to get
7:16
into this ML OPS Jitsu. This seems important.
7:19
How should I do it? Yeah, for that, I guess it's a
7:23
good question. I mean like find like a
7:27
technology maybe that you're comfortable with.
7:28
I think like I started with like docker compose, you know like
7:32
just one step above like docker files like pretty simple but you
7:35
can do a lot with them and there's like a really easy
7:38
transition from there to like full scale and Kubernetes.
7:42
I think the only like real second big tip that I have would
7:45
be just like, you know, no need to go light on the the plugins
7:49
for your IDE. I have like a like a rainbow
7:52
indent plugin for VS Code that's perfect for YAML.
7:54
You know, you can just like see like where where things are are
7:57
indented out to and and just let let the ID do the work.
8:01
You know you don't want to like be trying to apply something and
8:04
constantly getting like like errors because it's not in the
8:08
right format or not the right spacing etcetera.
8:10
It's super frustrating. Yeah, that.
8:14
I actually think tips like that are underrated.
8:16
It's it's a great tip because. I used to say this taking it's
8:20
like taking hard classes and in in college is way overrated,
8:23
like nobody ever like no employers are going to be like,
8:25
hey, you know, Vishnu, you took the hardest class for math and
8:28
your sophomore year we're going to give you, you know, a raise.
8:30
Nobody ever does that, right. It's the same way with coding,
8:32
right? Nobody was going to say, wow,
8:35
you did it without any plugins in your VS Code environment.
8:37
Like you were really good at this.
8:39
And I think for beginners in particular, that's a hard bridge
8:42
to cross, right? Because you know people in this
8:45
field, in this data science field, you know all bright,
8:47
ambitious, talented people who want to learn how to do things
8:51
right. And I think Kyle and I are here
8:53
to tell you that doesn't mean you can't do them faster and use
8:56
tools to do it. Ain't no shame in it.
8:58
No, not at all. Like, please, yeah, please take
9:00
the easy way like don't you know, don't try and do
9:04
everything and in them or something like that.
9:06
It you have all these tools available to you, and there's
9:08
definitely no there's no extra points for for doing it that
9:12
way, even if it you know might save you once or twice in some
9:14
some random scenarios. Well Speaking of random tools,
9:18
do you have any others that you really like or plugins or
9:21
anything that you can share with us?
9:24
Let me see, let me like what's what's what's up my Visual
9:27
Studio Code Right now I'm I'm a big Visual Studio Code fan.
9:32
I I definitely use that for most everything.
9:33
Kind of just like install plugins as I go.
9:36
Big fan of the remote SSH plugin.
9:38
Do a lot of remote editing. A lot of the basic ones, you
9:42
know, they'll just pop up and be suggested as you're editing.
9:44
And I'm like, I'm going to try this.
9:46
This is great. I really love those.
9:51
Well, let's talk for a minute about this idea that you were
9:58
mentioning before to me of like where you think the future of
10:03
machine learning, data science and even machine learning
10:07
platforms is going. And how you're looking at the
10:11
bigger picture right now in the ecosystem of machine learning,
10:16
data science, ML OPS and that whole, that whole climate, the
10:21
current climate that we're in, where do you see it moving
10:25
forward? Yeah, it's kind of funny like I
10:29
guess because I've seen it through a couple different
10:31
phases. So we started off in kind of
10:33
data science was the Super immature space and that was, it
10:36
was just like a data science team, but there was no concept
10:39
of like at least formerly like ML OPS and and anything beyond
10:43
that. And like the software
10:44
engineering side of things, data scientists, I don't know felt
10:47
like kind of isolated, siloed and it was more so a lot of
10:51
proof of concept projects and things that were been really
10:55
difficult to get off the ground. And then we kind of came to
10:57
this, you know, ML OPS space was like, OK, lots of these aren't
11:02
making into production. Why aren't they?
11:04
It's because we don't have the operational tools and processes
11:07
to get them there. We can't deploy this into
11:10
production easily or just the data science skill set is
11:13
inconducive to getting them there.
11:16
Software engineers obviously know how to do this, but there's
11:18
a disconnect between the two. That handoff was was really,
11:21
really tough because data scientists and software
11:22
engineers just weren't speaking the same language.
11:25
And now I think we're kind of, you know we're kind of there.
11:27
A lot of team like ML OPS is again like it's ubiquitous, it's
11:32
everywhere, it's a buzzword and it's it's awesome to see it like
11:36
exploding so much in terms of from open source support.
11:39
So just the enterprise platform offerings.
11:43
But even then I think there's like you know kind of a need for
11:45
the the step above that, which I would say is something like
11:47
governance, right, where you actually manage your ML OPS and
11:51
you have, I'll focus on some of the more important non
11:55
functional requirements. Something like observability,
11:58
right, which is like, you know, like a software term, um, it's
12:02
not something that like like inherently I think has been
12:05
phrased a lot in like ML OPS, but like it's something that is
12:08
observability and visibility I think are some of the most
12:11
important things that you can have.
12:13
And that is kind of that next phase of of governance, how you
12:16
manage you know like control all of those operations and
12:20
processes that you have and like how much visibility you have
12:23
over them? Yeah, I I agree first and
12:28
foremost with your overall description of the maturity and
12:30
the process. In the entire space I think and
12:34
I think what you landed on that I think is really interesting is
12:38
I realize now that a lot of machine learning platforms and
12:41
and you know a lot of this ML OPS and and and overall platform
12:45
engineering work has kind of come out of a frustration.
12:49
You know, it's really frustration driven in terms of
12:52
that handoff not necessarily working in terms of, you know,
12:56
groups speaking different languages, having different
12:58
objectives and enough people kind of saying man.
13:01
This sucks, you know, And I think that's the iteration of
13:04
this whole space that we're in right now where a lot of people
13:07
are saying, okay, this doesn't have to suck.
13:09
And going forward, the way we're going to see it is it's going
13:13
to, it's going to move kind of from an efficiency to a
13:15
capability, right. I think that is kind of where I
13:19
think those things like observability, visibility, you
13:23
know being able to like scale your whole your whole stack
13:28
around your model. Easily the same way that you do
13:31
like infrastructure as code. In another regard, you know
13:34
you're going to have your model as code and I think that's
13:36
already happening, right, with with Open AI basically just
13:38
exposing GPT 3 as an API and saying, yeah, you know, there's
13:42
models behind this fence and you know, come use it.
13:45
I think more companies are going to be able to do things like
13:47
that. So you know, I think the
13:51
interesting part here for me and I'm curious to get your thoughts
13:54
is I think that part of. Visibility and really
13:59
understanding the way things work with respect to the model
14:03
and you know data drift and and how the question of robustness,
14:07
that's where I find that maybe the research isn't quite there
14:11
yet. You know if it's like we're kind
14:13
of waiting to see, you know how to do that the right way what
14:17
what what has been your experience with that question.
14:20
Do you think there's maybe some heuristics we can use or or do
14:22
you think we still have a little bit further to go?
14:25
I think I agree that we have a little bit further to go.
14:27
I think it's kind of funny because I mean I'm sure it's
14:30
been said on this podcast even a bunch of times, but the
14:33
technology exists somewhere. It's kind of like how and when
14:38
we apply it. I think a lot of the decisions
14:39
that I've had to make in a day-to-day aren't like I need to
14:43
like build an entirely new technology.
14:45
It's like okay. What is the best way to apply
14:48
this for a machine learning specific operation?
14:52
A lot of things in the machine learning space do map directly
14:54
from software, one to one I think at least.
14:56
And there's a lot of parallels and things that we need to say,
15:00
hey, this is working in software, we should definitely
15:02
be doing this in machine learning, which is also
15:04
software. But there's also some things you
15:06
know, like some considerations. Maybe it's data drift, model
15:09
drift, Maybe it's like the inherent nature of trying to
15:12
serve, you know, a stateless model at scale or something like
15:16
that. But I think that there's some
15:19
considerations there that aren't really fully flushed out and no
15:22
one has had the time or maybe even the inclination to kind of
15:25
go through, like you said and research like, OK, these are the
15:28
most efficient ways to, you know, monitor for data drift or
15:33
model drift or something like that.
15:35
And we don't fully know what the best way to go about that is.
15:38
Once you get to that level, It is kind of territory.
15:41
Even if you could feasibly implement whatever you want, you
15:44
don't know what the best thing might be to implement for your
15:47
use case. Yeah, I see what you're saying
15:51
there and I think this is where I'm hopeful of all of our our
15:55
awesome vendors that are in the ML OPS community, slack and that
15:58
are helping us figure this out together.
16:00
I think. I do think that there are some
16:02
interesting work that they're doing and you know, when you
16:04
have these sort of, you know, it's kind of like.
16:07
You have your, your, your, your deep companies and then you have
16:09
your broad companies, right. And like when you have your deep
16:12
companies like you know, an Etsy or a Pfizer or you know, Tester
16:16
act, which is which is my company or any other company,
16:18
you know, we're trying to figure out in a particular vertical.
16:20
But then when you have these broad companies, you know, like
16:23
Fiddler who came on on our, you know, our podcast recently,
16:26
they're figuring out across a whole bunch of different
16:28
verticals and they're kind of saying, OK, this is what works
16:30
everywhere. This is what aspect of our
16:32
monitoring and observability and that I think hopefully that'll
16:34
trickle out. I guess one of the questions I.
16:38
Go ahead. Do you think it can be that?
16:42
Because it feels to me like there's so many different use
16:44
cases and there is no standardization right now and
16:50
it's really hard to say like this is what works in
16:55
everywhere, right? Like, do you think that will
16:58
ever be a thing or is that something that?
17:02
It's going to have to be like specialized and customized or
17:05
bespoken every time you want to implement it.
17:10
I would, I would kick it to Kyle here and say you've been in,
17:13
you've been in consumer, you've been in Healthcare, you've been
17:15
in bio. I mean what has been your
17:17
experience with that question of what's abstractable and what's
17:20
not? Yeah, it's definitely hard
17:23
question to answer. I mean there's cuz it is
17:25
dramatically different and I don't think that we're at least
17:29
at a place where like 1 size would be able to would be able
17:32
to fit all. I think like the key there is
17:34
kind of I want to think a lot of like the really good vendors out
17:37
there are doing is like making themselves integration for his
17:40
platforms. Like all right, we're going to
17:42
like see what the most common use cases are.
17:44
We're going to say like we're going to solve 80% of your
17:47
requirements, but we're going to allow you to build that, you
17:49
know that remaining 20% because our platform is so flexible and
17:54
integratable with all the systems that you might happen to
17:56
have running. And so I think it's really
17:59
about, yeah, I think there are a lot of commonalities, but the
18:02
the subtle differences, if you try and build for everything
18:05
then then you're going to then you would fail, yeah.
18:10
Yeah. I I see what you're, I see I see
18:12
what you're saying there. It's like if you, yeah, it's
18:14
like taking on too much in a sense and it's like you end up
18:16
becoming it's hard to balance all of that and and and you know
18:20
if you have that sort of flexibility, inherent
18:22
integrations rather than trying to custom engineer everything,
18:25
it becomes a lot easier, you know.
18:28
I kind of have this question just around like this notion of
18:32
machine learning infrastructure and platform.
18:34
Like you know, you have had this transition from like let's say
18:39
user of the platform loosely speaking to like architect of
18:42
the platform, right. And I think there are a lot of
18:45
people in our community who feel similarly, right.
18:47
You you start off because companies are hiring machine
18:50
learning engineers, or they're hiring data scientists and then
18:53
the machine learning engineer you know, usually one or two
18:55
realize wait. We should probably do this in a
18:58
repeatable, scalable, maintainable way, yeah?
19:01
How did that realization come about for you?
19:05
I would really love to dive into this.
19:07
Yeah. I guess it was just like
19:09
probably the first time I went to take some new production or
19:12
maybe it was, maybe it was more so the second, right, Maybe the
19:14
first time, which is kind of like, all right, maybe this is
19:16
just how it is. And then it was the second time
19:19
I was like, all right, why don't we just do that like, but again,
19:22
and I personally, because I didn't at the time, I guess have
19:26
the insights into what was going on like behind the
19:28
infrastructure fail. I, you know, probably didn't
19:31
know the right way to interface with software engineers.
19:33
And at the time, there was no common practice for that, that
19:37
delivery of data science model to software engineer
19:40
infrastructure. So yeah, like I think like a lot
19:43
of people in the slack, I probably like like gained an
19:45
interest in that slide side of things and applied myself there
19:48
a lot. I've now tried to come back and
19:50
say okay. Like when we build, you know,
19:53
platform team, your customers are data scientists.
19:56
Where do you meet them? How do you meet them?
19:59
Like at what? What is your expectation of a
20:01
data scientist? Like what do you want them to
20:02
come to you with? What do you need them to know?
20:04
And then what is the expectation on what, you know you're
20:07
supposed to provide them so that they can?
20:10
Have the best user experience in your platform and deliver models
20:14
as as quickly as possible to production.
20:16
I think that's, you know, I think a common KPI for a lot of
20:18
teams is like timed production for a new data scientist or
20:21
something like that. But it's kind of like thinking
20:24
about in the context of your company, your or team, whatever.
20:29
Where do you meet the data scientists to best foster that
20:32
relationship and best expedite that trip to production along?
20:38
Those Go ahead, Dimitris. Typical zoom meeting.
20:44
So I was just thinking about the place and where the data
20:52
scientist fits in on this and how you've gone from being the
20:58
data scientist or the customer and then actually creating it.
21:02
And we talked a little bit about like.
21:05
The ability to know how to code and know this software
21:09
engineering gets you. So it gives you so many
21:13
advantages when you're a data scientist.
21:16
And so I'm just wondering, and this is something that comes up
21:19
quite a bit is when you're a data scientist, you just stay in
21:24
your lane and you focus on getting the most out of the
21:28
model and really that's it or are you going?
21:32
Into the platform and trying to make it better and maybe working
21:37
with the platform team or if you don't have a platform team or
21:41
are you trying to create that platform like how do you look at
21:44
that? Yeah.
21:48
And if I'm not giving direct answers only because again one
21:51
size isn't fit all, it really depends on the data scientists.
21:54
Cuz I have met data scientists who are very stay in their lane
21:57
and they want to like give you a serialized, you know, model or
22:01
something like that and be like I'm done, that's what you get.
22:04
And like it's everything from there.
22:06
And then there were people who are data scientists like myself
22:09
who are like like this isn't enough.
22:11
I want to understand, you know, the platform.
22:13
I want to actually like learn the things in that platform so
22:15
that I can you know iterate and improve them with platform
22:18
engineers and learn that side of things.
22:20
I actually like software engineering better than modeling
22:22
personally. It's it's more definitive and
22:24
less frustrating in my mind. But but yeah, I think like it's
22:30
it's so highly variable and I guess it's part of like being a
22:33
platform engineer is like just being open to both.
22:35
Like I've I've definitely worked with data scientists who really
22:40
like to like learn Kubernetes or stuff like that.
22:42
I'm like, yeah, I'll absolutely help you with that.
22:44
I think it's cool that you want to learn that and it's helpful
22:47
overall for both kind of collaborating there.
22:50
And there's data scientists who want to be hands off and we have
22:52
to meet them there as well. Yeah, I think one thing that
22:57
I've realized and working at a startup, you know, we have to,
23:01
you know, we have to hire a particular kind of people,
23:03
right? Because working at a startup is
23:04
risky and often very frustrating.
23:07
And I think it's given me a sensitivity to that exact point
23:10
that you have, which is how you build some of these tools really
23:14
depends on the kind of people that you hire and the kind of
23:17
people that come into the door and what they're interested in,
23:20
especially at a smaller company like ours or even some of the
23:22
larger companies that you work on.
23:23
I mean, I'm sure there is no ML team anywhere in the world maybe
23:26
outside of Fang where it's like you know, 100 people, right.
23:30
Most ML teams are very small. And so it's like, who do you
23:32
have and what are you going to do with them?
23:36
Yeah, that ends up being day-to-day.
23:38
Yeah, maybe their PhD, you know, mathematicians and stuff like
23:40
that. And that's, you know, completely
23:42
different from people who maybe came from a boot camp, from a
23:46
variety of different backgrounds or or otherwise.
23:49
And yeah, it's super, super variable.
23:50
But I mean, yeah, that's comes kind of comes down to what we
23:53
were I guess talking about earlier about data science still
23:55
being so poorly defined and so in flux in terms of a definition
24:01
of the domain and as a role and and everything.
24:05
Yeah, for sure, for sure that that that that level of
24:08
variability it's definitely it's a feature, the feature of the
24:11
field that you know hopefully you know we'll we'll get rid of
24:14
over time in this platform engineering realm, what would
24:19
you say maybe the most interesting challenge is that
24:24
you may that you experience and you know some things that come
24:26
to mind or like. You know, CICD and like
24:29
continuous training or data management, distributed
24:32
training, orchestration, like what are the engineering
24:35
challenges that really get you interested in this platform
24:38
question. Yeah, I've always really, really
24:40
loved model serving for some reason.
24:43
I don't know why, like I'd like the maybe just because it's more
24:46
of a an instantaneous thing. You know, it's like an API
24:49
calling back and it works and you and you see the result, um,
24:53
training for me while like has this like a lot of very very
24:56
interesting engineering problems, always been
24:58
frustrating to implement because of the some of the time
25:01
constraints. You know, maybe you kick off the
25:03
job 8 hours later you're like oh man, that failed, like gonna
25:05
have to gonna run again. So I've always found like the
25:10
like the enhancements and performance especially like now
25:13
with so many like large transformer based language
25:15
models and stuff like that. How do we continuously optimize
25:18
models serving to the point where it's feasible and actually
25:22
like generates value to serve these models at scale.
25:25
So I think that's like all of the challenges under that space
25:28
are super, super interesting and there's a lot of really cool
25:32
open source and like enterprise serving frameworks now that are
25:35
fun to play with. And so let's say a community
25:39
member were to come up to you and say, Kyle, you've worked at
25:42
some great companies. You're an expert on model
25:44
serving. How should I?
25:46
You know, my company wants to figure out a strategy for this.
25:50
You know, they told me I need to figure out what our model
25:53
serving platform looks like. What are some of the things I
25:57
should watch out for? And what are some of the tools
25:59
you'd suggest? Anything that you would offer as
26:01
advice? Yeah, that's a that's a great
26:05
question. Um, basically free consulting.
26:08
Yeah, we definitely depend a lot on the company.
26:11
Um, you know, like my first inclination would be like, you
26:15
know, if depending on the resources you have, just
26:18
evaluate a lot of, you know the vendors out there.
26:22
There's a lot of like really, really great tools coming out of
26:25
a lot of different companies. I feel like so many different
26:27
enterprise offerings just to have models serving, but it's
26:30
also built as part of a larger experiential platform or there's
26:34
even things that are more hands off, for instance, like Selden I
26:36
believe, just like a open source framework for deploying models.
26:40
And Kubernetes also offers like Selden deploy, which is I think
26:44
they're enterprise tool which is more UI driven, but a nice way
26:46
to like deploy and manage your models.
26:49
I would say that like the technical aspect of serving is
26:52
going to be easy, right? Like you have something like
26:54
Tensorflow serving, you just train a Tensorflow model, get
26:57
the docker image, then you have an endpoint, but managing that
27:01
and actually like getting to that governance level of how do
27:03
you manage. If you have 50 models, how do
27:06
you keep track of those and how do you manage those M points.
27:10
Make sure you're not wasting resources being cost effective.
27:14
Maybe like role based access controls.
27:16
All of that stuff is like the real stuff that you have to
27:18
watch out for in that space, because otherwise you just get
27:20
this explosion of Technet that is totally unmanageable, yeah.
27:25
Yeah, the nitty gritty for sure. So when it comes to this stuff,
27:32
do you think about like when you're working on the platform
27:37
and you're looking at the platform as a whole?
27:39
Do you have problems when it comes to like plugging in
27:45
different pieces of the puzzle in the platform?
27:49
Because one thing that I've heard many complain about or
27:52
just mention, maybe not complain.
27:55
I'm projecting the complaints on to them, but many have mentioned
27:59
how there's not really standardization yet and you
28:02
don't have. An easy way to do the Lego
28:06
blocks and switch out one piece for another piece.
28:10
Have you found that to be true and how have you dealt with it?
28:14
Yeah, I definitely found it to be true to a degree.
28:19
I guess it depends on the kind of like where you're where
28:23
you're at, Like you're working with a lot of like like
28:26
monolithic in house tools or is your company very like
28:30
microservices Driven? I know, again, like a buzzword,
28:33
but I think there are benefits to working with microservices.
28:36
But yeah, like it it, it's never as easy as, as plug and play,
28:40
right? Like you're like, oh wow.
28:41
Like even if these things are like even if you're told they're
28:44
going to be Lego blocks, they're never Lego blocks.
28:47
He's been like the first two hours trying to like cram 2
28:49
together that you know should go together and just like doesn't
28:52
work for you. So I think, yeah, at the high
28:55
level, I've definitely run into, I've definitely had issues.
28:59
Yeah, putting those pieces of the puzzle together, even
29:02
starting with the first piece of the puzzle, you know, you, you
29:05
always estimate how long a task is going to take in the machine
29:07
learning space. You're like, all right, this
29:08
will take me an hour and then two days later you're like, what
29:13
is happening? Why is nothing working?
29:15
Why do you think that is? Because I've heard that so many
29:18
times. Like, it's really hard to judge
29:21
how long something is going to take in this space and.
29:25
And that's another reason why some people are really against
29:27
doing sprints in the machine learning field because it is so
29:33
hard to judge. Yeah.
29:35
I mean, like, I think like it, it might come down to how you do
29:38
your estimates. Like if you're trying to like
29:41
you know kind of like throw stuff at the wall and see what
29:43
sticks in regards to like if I had to estimate building an
29:46
entire ML platform like I, you know, offhand I would have no
29:50
idea, but I would probably double or triple whatever my
29:52
initial estimate was. But I think it's just kind of
29:56
you know, comes down to agile, right?
29:57
You just want to like take that and like start being like OK
30:00
like what are the small pieces that I actually know how long
30:02
they take or or can at least guess, right.
30:05
It's no big deal if you estimate something is a is a two point
30:08
ticket turns out to be a two week thing.
30:10
Like it happens. It's always going to happen.
30:13
But the the better that you can break down that work into
30:15
smaller pieces the the more accurate you're going to be.
30:18
I think it's especially tricking the ML space because you don't
30:22
know how a model is going to do before you before you train it.
30:25
I remember as a data scientist, people be in like how long until
30:28
you get to this, you know this level of accuracy or like this
30:31
level of acceptance would be like I have no idea, like you
30:33
haven't even given me data. This is all you know, purely
30:37
hypothetical like project planning.
30:39
I I have no idea what's going to happen two weeks from now.
30:42
So I haven't even, you know, been able to train one model
30:45
yet. So.
30:48
I wanna kind of switch gears real fast because we've been
30:53
trying to create this new podcast with one of the
30:56
community members, Fabiana, and we're talking to a lot of people
31:00
about data access and how they go about that.
31:04
And I find that is a really interesting problem to look at,
31:08
especially once you start to get into larger organizations and
31:12
you start to get into. Companies that are dealing with
31:16
private information, or if you're working in the EU for
31:19
example, where there's all these regulations around the data.
31:24
And then I heard horror stories about people waiting around for
31:29
six months to get access to the data and they're just kind of
31:32
twiddling their thumbs until they get that access.
31:35
And I'm wondering how you look at that problem and how you make
31:39
sure that. These horror stories don't
31:42
happen. But you're also not super laxed
31:45
on the data access so that anybody can have access to it.
31:51
Yeah, and I guess that's tough because there's also, I mean,
31:53
like, you know, there's ethical considerations there as well,
31:56
right? I mean, you like, it's from a
31:59
data scientist perspective. You wanna say like, ohh, just
32:01
like give me the data. That's not that big of a deal.
32:02
Like, I'm just a person. Like I can I should be able to
32:05
access this, you know, this regulated data with personal
32:09
identifying information and and work with it.
32:12
But like you know that's that shouldn't be the case and we
32:16
should have like very like strict controls around that and
32:20
you know consumers should understand and know what's
32:22
happening with their data. I like it definitely comes down
32:27
to that kind of governance problem.
32:28
Again, the more like oversight and control and management you
32:33
have of your data, the easier it is to safely provision access
32:37
without you know potentially damaging or lasting effects.
32:44
It's it's not a space that I've had to work in like a lot and
32:46
like well, I guess like, I mean I've definitely seen it a lot in
32:49
in the healthcare space, but fortunately it hasn't been my
32:54
problem to do. Yeah, I think the data access
32:57
question is also interesting technically.
33:01
And you know I think this gets back to your thought kind of
33:04
about platform in a slightly, in a corollary way, you know these
33:07
things, they tend to be built incrementally, right.
33:10
You know it's nobody ever kind of goes out and says hey okay
33:13
designer JC access system. And to end it's going to scale
33:16
with us from one to 1000 engineers and one to 100 data
33:18
scientists, boom, got to do it right.
33:21
And I think that's a challenge that I, I I have a lot is like
33:26
figuring out. What to implement now versus
33:29
like what to implement later, particularly for a system like
33:31
data access where it's like you're thinking a lot about like
33:34
interaction, like what's your interaction model with, let's
33:37
say it's like an API, right. And you know some data centers
33:39
just like pulling it in from from an API, making some kind of
33:42
request. Like how do you go about
33:44
designing that API model or like that interaction model from you
33:48
know, from your standpoint, how do you think about sort of like
33:51
designing interfaces and and and you know, kind of specking out?
33:57
From a technical standpoint, what the different use cases
33:59
look like Yeah, I mean it's that's I guess for me it's just
34:02
all about working with users as much as possible and
34:05
understanding the use case like you.
34:08
I think it's as straightforward as just like get as many use
34:11
cases as you can real use cases as quickly as possible and and
34:15
try and work through them. Because it's more than likely,
34:17
yeah, your first design and assumptions are gonna be wrong
34:20
in in some way, shape or form. Like building a platform for
34:23
data scientists. Like maybe you assume that they
34:26
have some skill they don't, or want some feature that they
34:29
don't. Or maybe it's not important at
34:30
all. But you really need to like, you
34:32
know, get those use cases and understand them to actually
34:36
correctly map out those requirements.
34:37
Design something that actually, you know suits your needs.
34:40
Yeah, I think I think that makes total sense.
34:42
It's, you know, you really have to have that empathy, that
34:44
understanding and that sort of, like intuitiveness around the
34:48
use case in order to be able to, you know, engineer the best way.
34:53
Now I'm gonna ask you a question or a tough question.
34:57
Have you ever had a case with or does anything come to mind in
35:00
terms of maybe where the use case wasn't very clearly defined
35:05
on the part of the user, Maybe there was a little bit of
35:07
incoherence or lack of process? And if so, how did you deal with
35:11
that? Yeah, absolutely.
35:14
I think like more so is a data scientist than anything else.
35:17
I probably have that right because software engineering is
35:21
well defined. If you're building an
35:22
application, there's, I mean one.
35:25
I guess the project managers and software engineers are very
35:27
familiar with that, but so are probably consumers, right?
35:30
Some of they interact with every day.
35:32
Whereas I think when like ML and AI is kind of blowing up as a
35:35
term, people started asking for those things without asking for
35:39
actual, without actually having a use case, right?
35:43
Like, I've definitely had cases of people coming to me being
35:46
like Ohh like I want ML and be like create like for what?
35:50
And like why? And like is this even a use case
35:54
that ML would be remotely good for?
35:56
It's I think like actually, yeah, being able to spot those
36:00
use cases is like, so yeah, it's a skill unto itself, right?
36:04
Like seeing who is just asking for ML and who actually like has
36:08
a problem that's currently tedious, manual, etc.
36:11
That could be automated by some kind of machine learning.
36:14
You're totally right. That's the perfect example.
36:18
It's totally perfect example. I was gonna.
36:21
I was gonna try and steer this ship towards the idea of what
36:26
you feel the next big thing is when it comes to machine
36:30
learning. Maybe beyond ML OPS or within
36:34
the ML OPS ecosystem where? Where do you feel the next big
36:38
thing is? I don't want to default on
36:42
governance again, but I think it's, I think it's going to be
36:44
governance. I think it's going to be machine
36:46
learning in the enterprise actually like like people seeing
36:51
that it's driving value for certain companies and it being
36:55
kind of like getting up there like you know you have data like
36:58
software and now you're gonna have kind of machine learning up
37:00
there. I think at the same time I think
37:02
it's that level of enterprise maturity that we're still not
37:06
currently at, but some companies are starting to reach and really
37:09
set a standard for. I think you know like obviously
37:13
the machine learning like research in itself has been
37:15
exploding for a really, really long time.
37:17
But like the distribution of companies like using or getting
37:21
value for machine learning is huge.
37:23
You've some companies doing like real time training online and
37:27
like like getting all these really great results from like
37:30
real time predictions. And even some companies that are
37:32
still probably questioning the value of of the ML itself And
37:35
like whether or not like machine learning would you know they
37:38
would actually get some kind of return in their investment for
37:42
investing in the machine learning space so heavily.
37:44
A lot of companies like to, you know, like stick a toe in the
37:47
water, hire a data scientist, watch them burn out and then and
37:51
then mix everything. But yeah, I think it's gonna be
37:53
kind of a select few companies setting that, setting that
37:57
standard of what machine learning looks like when it's
38:00
actually driving value and the rest of the market we have to
38:04
meet it. Yeah, that's, I think that's
38:08
spot on. I think that's something I
38:09
really agree with and I think it's something we see a lot in
38:12
the community. You know, the, you know, Etsy is
38:14
one of those companies, DoorDash, Uber, certainly, you
38:18
know, Facebook, Google, etcetera, Apple, they've been,
38:21
you know, I actually think it's kind of interesting.
38:23
I think the Facebook and the Googles and the Apples of the
38:24
world, I think probably justifiably so because they're
38:27
so big, but there's probably a little bit less of true ML op
38:31
stuff I really learned from them.
38:32
You know, I think it's more like.
38:34
The Spotifys, sort of like those midstage tech companies that
38:37
have really given a lot of great lessons in terms of like, how
38:40
does this stuff scale in like a modern way.
38:45
Yeah. And like A and, like a
38:46
manageable way. Sometimes I feel like, yeah,
38:48
sometimes I feel like I read a paper from like a fan company or
38:50
something like that on their architecture or something.
38:53
And I'm like, there's no way we're going to implement this.
38:57
It's just like totally infeasible for wherever I'm at
39:00
right now. But yeah, like.
39:03
Also, something to be said, I think all of these companies are
39:08
tech companies. And what it kind of sounds like
39:12
you're saying, Kyle, is you're expecting more of the companies
39:16
that aren't necessarily tech companies like your Coca Cola's
39:19
or Delta Airlines to also get up there.
39:23
Yeah. There's a huge difference I
39:25
guess between the companies where data scientists are
39:27
embedded in like engineering organizations and like companies
39:32
where like you have these silo data science teams that are kind
39:36
of unsupported from from a lot of different standpoints.
39:39
And maybe those data science teams are more so like the, you
39:41
know, insights, data science and stuff like that.
39:43
Maybe they're working with marketing teams or otherwise.
39:47
But like if they're data scientists that need to like
39:50
create and deploy things to production that actually serve
39:53
as the back ends for, you know, like software, then I think a
39:57
lot of those companies like also have really cool things that
40:00
they can and want to build to automate internal processes, but
40:04
don't yet have the organizational structure or
40:06
support for those data scientists to get to that level
40:08
or or haven't organized in that way.
40:13
Yeah, I've actually, you know, some companies that come to mind
40:17
that, you know, you may not think of as like awesome ML
40:20
companies that I have actually been pretty impressed by like
40:23
Chickfila, John Deere, you know, some like you know some.
40:28
Those two come to mind. Yeah, those are those are the
40:33
two. Corey, Yeah, shout out to Corey,
40:35
right. Yeah.
40:36
No, I mean those great, great, great companies that have really
40:41
started to embrace ML and you know, kind of bake it into their
40:43
entire business model and and their business sort of
40:46
perspective. You know, it's definitely, it's
40:49
definitely out there and that's not to say that like a non tech
40:51
company like. Yeah it's it's not it's not like
40:54
going to succeed at ML. But I think like there's like I
41:00
think some of those companies are definitely doing it right.
41:02
You know you're John Deere's are it sounds like they're they're
41:05
doing it right in terms of how they embed their data scientists
41:08
in the organization support they get them.
41:11
And I mean, like, yeah, I guess there are a lot of companies
41:13
that people don't think of as tech companies that are
41:14
producing cool papers or interesting things within the
41:17
machine learning space. But I guess like from the from
41:21
the inside, I guess that I've seen in some of those companies,
41:23
it can be difficult to get the support you need to, to get
41:27
something into production or even just you know, to have like
41:29
a senior engineer's eyes on something, you know.
41:33
Yeah. I guess just to kind of, you
41:35
know, maybe bring this discussion to a sort of close
41:39
and just kind of, you know, get into our concluding period here,
41:43
you know, this idea of platform engineering, it's new, this idea
41:46
even of like machine learning infrastructure.
41:49
You know, like what really is machine learning infrastructure.
41:51
You know, I think I felt that a little bit lately.
41:55
You know, I think I, I, I, I understand, you know, the, the
41:59
importance of ML Labs. I mean, I'm here hosting a
42:00
podcast about it and hanging out on the community.
42:03
And I see how, you know, this field has to evolve in a way
42:09
that's, you know, probably a lot more like software engineering
42:12
than, you know, maybe some of the more numerical fields that
42:15
you know are out there like, you know, signals and systems
42:17
etcetera. But I think sometimes, you know,
42:20
one of the things I struggle with is really kind of defining
42:23
what something like a machine learning infrastructure really
42:26
looks like on a practical level at a company, right.
42:29
It's, it's kind of like, oh, you know, these are our problems.
42:31
Is it machine learning infrastructure?
42:32
Is it DevOps? Is it, you know, is it a
42:36
software engineering issue? And I guess my question kind of
42:38
to you is, you know, have you dealt with similar challenge in
42:42
terms of defining what the problems are that you need to
42:45
work on or have you felt a little bit that, you know, maybe
42:49
because of the environments you've been in, it's been a
42:51
little bit more clear. How do you think about that?
42:54
Yeah and it has depended based on like what the what the use
42:58
case is. For sure I like.
43:02
I'm trying to think of the best way to answer that because there
43:05
have been times where I'm like, yeah, what is I'm like what are
43:08
we doing? Like, and I'm like, what is this
43:10
year like, you know, like sometimes I felt like I've just
43:13
been following the flow, like I'm deploying a model to a
43:15
Kubernetes cluster because that's what people do this.
43:18
Other times I've felt like I've been doing it because I I need
43:21
to do that. And like, I need, you know, like
43:23
I need this thousand node cluster to scale up that high to
43:26
actually serve the traffic that I'm trying to, that I'm trying
43:28
to meet. And I don't know if I'm
43:30
answering a question at all. But but but yeah machine
43:34
learning infrastructure is like it's it's software
43:38
infrastructure you know it's all of the support that you need to
43:42
actually support your ML software tooling at scale for
43:49
your use case. Yeah I I think I think you did
43:53
answer my question right. It's I think that Kubernetes
43:55
example is a perfect example of something that you know, you
43:59
know, we all feel like, oh man, we should be using cube flow.
44:02
We should be using all these things that are, you know,
44:04
talked about. But it's like what are the
44:06
problems I'm trying to solve here?
44:07
What are we really trying to do here?
44:08
I mean, and if you're asking yourself that question, I think
44:10
maybe you are doing the job of a machine learning infrastructure
44:13
or a platform engineering sort of contributor.
44:18
Yeah, there are definitely times where I've like, you know, I
44:21
started using cube flow when we were leveraging it way back at
44:25
A. Some other place.
44:26
But it was like we started using it and realized that it we
44:30
actually didn't need it. It was like kind of for our use
44:32
case, it it was too much. It was, you know, it was it was
44:35
more difficult to manage than it was than like the value we're
44:39
actually getting out of it. And we're like we could
44:41
literally have like a like for our small data science team have
44:43
a standalone deployment of ML flow and get everything we need
44:47
like out of an MLS pipeline like without any kind of scale or or
44:51
management of like infrastructure resources
44:52
whatsoever. Then there's other cases where
44:55
that's just not even. Close And you have all these,
44:57
you know, workflows running daily and you need that.
44:59
Kind of like a complex workflow orchestrator like that.
45:06
I know there's some kind of adage, but I can't think of it
45:09
right now where it's like you get the tool and then you look
45:14
for the problem or something as opposed to looking for the
45:17
problem and then finding the right tool.
45:19
But someone in the comments will help me out and remind me what.
45:23
Adage that is or what the saying is and how it goes.
45:27
But this has been awesome man. I appreciate you coming on here
45:31
talking with us. I want to mention to everyone
45:33
that has stuck with us, Kyle, your team is hiring, right?
45:38
And you've got a referral code. Are you still?
45:40
There. I mean, it's not directly my
45:41
team, but the company's hiring a lot, yeah.
45:43
And I'm I'm happy to provide, happy to speak to people in the
45:47
in the MLAs community about whatever.
45:52
There we go. So if you're looking to work at
45:55
Etsy, not making handmade baskets or any of that other
46:01
cool stuff, I tried to sell stuff on Etsy once.
46:04
That's a whole another story. But if you're looking to work on
46:08
the software team, then hit up Kyle and the Slack and yeah,
46:14
this has been awesome, man. I love hearing about your path.
46:16
I love hearing about the vision that you have and.
46:19
Really diving into what are some core concerns, What are some
46:24
ways that you can be a better machine learning platform
46:27
engineer? And thank you.
46:31
So much for having me. I really appreciate the time.
46:33
Awesome questions and then and the chat makes me think too.
46:37
It's cool. Yeah.
46:38
Thanks for coming on, Kyle. Really enjoyed it.
46:40
I think. I think our, our community, our
46:42
listeners will get a lot from just some of the technical
46:45
challenge You highlighted all the different aspects.
46:48
I think I feel like we really covered just how broad platform
46:51
is when we talked about Okay, what are your consumers, what
46:54
are the different interfaces, what are the different technical
46:56
challenges like this is a pretty broad field and I'm, I'm very
47:00
glad that you're able to come on.
47:01
So thank you. Yeah, of course.
47:03
Yeah, I know. And it's, it's definitely
47:04
massive. It's not like.
47:06
Yeah, there's so many different spaces you could get into and
47:08
talk about. You talk about Kubernetes
47:10
itself, you know, for so many hours and hours like get into
47:13
networking and like further and further away from the from the
47:16
core. But yeah, no, it's been awesome.
47:18
Thank you so much.