Latest / Agentic Conversations (formally mlops.community) / Systems Engineer Navigating the World of ML // Andrew Dye // MLOps Podcast #136

Transcript

Systems Engineer Navigating the World of ML // Andrew Dye // MLOps Podcast #136 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Andrew died. I'm a software engineer at Union
0:03
aii and typically, drink, Black Coffee in the mornings, but
0:06
would love an Americano like invite.
0:08
David is back here on a podcast with me and for everyone that is
0:14
not watching on YouTube, it just makes me feel like you are the
0:19
Always Sunny in Philadelphia meme right now because you've
0:22
got a ton of math going on, dude, what is that?
0:26
It's some formulas to let's say, Say, find the total compute, the
0:32
total communication, and then measure the intensity.
0:35
The area, average arithmetic intensity.
0:37
It's related to some distributed training research on working on.
0:40
There we go. So in case anybody's been
0:43
wondering where the hell has David been gone to the dark
0:48
side? He's a researcher now.
0:50
He's been. He hung up his ml Ops hat.
0:52
Actually, not really, you're still doing him a lobster on the
0:54
daily, right? Yep.
0:55
At Microsoft heavily. Yep, but now he told me the
0:59
other day, he I've gotten to the point where any problems I
1:02
encounter I've seen them before, so I need to look that stuff up
1:06
anymore. You're just too good, dude.
1:08
You're too good at this. You've graduated ml Ops Academy,
1:12
not at all. I feel like now as I'm diving
1:14
into an area, that's a bit different.
1:16
I feel like I knew all over again.
1:18
It just feels like I'm, I did so much, I don't know.
1:21
There's so many things, I feel like I should know imposter
1:23
syndrome, like you name it, going through it all over again,
1:26
but it's fun. This is, I think this is a comes
1:29
with being In this industry, you either change frequently,
1:32
whether it be roles or jobs or area of interest but or the
1:36
technology and just what's going on in the space Joe.
1:38
So you just everything changes. Yeah.
1:41
And you like to push your limits and see how far you can go and
1:45
see what's right? Learn so it's cool to see.
1:49
I like watching from the sidelines and getting explained
1:52
every once in a while. What all those sigma's on your
1:55
board means summation sums. They yeah.
2:00
It's actually not as hard or complicated of math as looks
2:03
like, but it is intimidating math.
2:07
It's funny that you mention that, but I've always felt
2:10
intimidated by math, growing up, and it was only two.
2:12
I got into the machine learning space, that I started to feel
2:15
more comfortable with it. Dude, I love how you're just
2:18
like, yeah, math scared me until basically.
2:21
I became a rocket scientist and I just got over.
2:24
It's yeah, it's that's for context without getting too deep
2:27
inside. That's how I learned things.
2:28
I like throw myself into Something new and like sink or
2:31
swim and eventually figure out how to swim.
2:33
But he's always like that. What the hell is going on?
2:36
I'm drowning looking at these communities, for help these blog
2:39
posts for help. There you go, we did not just
2:43
talk to each other for this whole our we actually had a
2:45
guest on as much as I would have loved to have talked to you for
2:48
the next hour we brought on Andrew and that was incredible
2:52
man. You got some top takeaways or
2:54
you want me to go first? Why don't you go why don't you
2:56
go start with that? All right, so I love how Andrew
3:00
and when we introduced you to him, you talked about what you
3:04
were working on. You told him about that those
3:06
drawings behind you. And he was like, oh yeah, that's
3:10
funny. Because I work just the level
3:12
below what you're working on right now.
3:14
And so you all were able to bond.
3:16
I feel like on how there was some similarities but it wasn't
3:22
like you overlap that much. And so the problems that he's
3:25
working on at that very low level, I think We don't
3:30
necessarily hear them that much on this podcast and they're
3:35
still very valid. He, as I mentioned, at the end
3:37
of this, it's like he had his experience at Microsoft working
3:42
on hololens. He then went to meta and both of
3:45
those were incredible scale and now he's working at Union, which
3:48
is also production-ready bulletproof and they have to be
3:53
able to put up some serious scale, I know that spotify's
3:56
using them so he was able to Give us his take on why?
4:04
And what you need to keep in mind when you are working at
4:08
these low levels. And why it's so important when
4:11
you are a machine learning engineer and how the two can
4:15
play together nicely? Yeah, and I also like that he is
4:20
a good example of. I think many I'm a lapsed
4:23
practitioners who come into the field from someplace else.
4:26
He has this great experience as a systems engineer doing
4:29
low-level sort of work and then we're going kind of up the stack
4:33
and able to bring those first principles like you said and a
4:37
been able to apply them to a product that is what the users
4:41
interact with. I think that's super interesting
4:44
is super challenging but as we were in the MLS Community.
4:47
This is a space that's always changing.
4:49
We're always learning new things about it, so it's, it's
4:52
encouraging. I think to hear from someone
4:55
that was not pure ml. Ops became the Searing, how
4:59
their learning experience was and how challenging it was.
5:02
Also, how honest, I really, like honesty was about how much we
5:05
depend upon others. I think that's something that we
5:08
often don't talk about, but you don't hear about how many people
5:11
you learn something. From if they hadn't told you
5:13
that you wouldn't be able to finish something.
5:14
There's a ton of that, I think happens in the workplace and It
5:17
works well, when there's a common goal right, which is what
5:20
we're hearing. The tool like Union AI, is it
5:22
forces these various disciplines to work together for a common
5:26
objective and I think it's really cool to hear, ya know
5:29
some training when he started talking about the abstractions
5:32
and how they're thinking through abstraction is because that is
5:35
something that I know so many people think through like what's
5:38
the right abstraction layer and then how you can just create a
5:42
Swiss cheese effect, where you think you have this nice level
5:46
of Action. But then you expose one little
5:49
button that you can play with and then, you know, the
5:52
complications get or it gets a little bit more complex so you
5:56
have to go at it again and try and create that nice abstraction
5:59
layer. So it's cool to see how they're
6:02
thinking through that. But get into this conversation
6:05
with Andrew and huge, shout out to the sponsors of this episode
6:09
Union ml Union aii. So let's get into this
6:14
conversation with Andrew and hope you all.
6:17
Enjoy. And you're welcome, man.
6:24
It's great to have you here. Yeah, thanks for having me,
6:27
dude. So just give us the breakdown.
6:31
I love the background. You have, I think you've got one
6:34
of these. This experience that is super
6:39
vital in the MLM space. I know we were just talking
6:41
about this before we hit record but you are playing at a low
6:45
level, we're going to get all into that and why that is so
6:49
important. Oughtn't for the tools that
6:52
you're working on right now in the tools that you've worked on
6:55
in the past right now, you're currently at Union.
6:57
I know you were at meta before that.
6:59
Can you just break it down? Like how did you even get into
7:02
software engineering? Yeah, absolutely.
7:05
Yeah, I think like out of the gate like I'm a complete ml ml
7:07
Ops Outsider here in 2014. I started working on hololens
7:12
for those not familiar was that a are headset and Microsoft.
7:14
When this is really my first kind of bumping into the ml
7:18
space. I was a little All systems and
7:20
was working on custom silicon sitting between really state of
7:24
the Arts Hardware to do real-time tracking for for AR.
7:28
And then complex algorithms to make that all happen.
7:30
And then do a bunch of cool experience on top.
7:31
And I'm sitting in between, I know nothing about either layer
7:33
above or below me. But working on this chip, that
7:36
is custom silicon with custom instruction set that were using
7:40
to do real-time positional tracking and just getting to
7:43
like really at that layer in the complexities there, but
7:46
understand that the abstractions were building and all this
7:49
complexity. That's happening at the
7:50
hardware. Layer is really empowering.
7:52
These really awesome positional experiences real-time social
7:55
experiences. I remember days sitting in my
7:57
office, like playing this bug shooting game or logs are coming
8:00
out of your walls and I know it's just it was like wild to me
8:03
to see how this put the power of images and data and computation.
8:07
You could disable these really awesome experiences.
8:09
It's like over this time. It really just kind of just sort
8:12
of bumping into what is this ml thing, but ultimately had no
8:15
idea. I'm curious.
8:17
What from your perspective when, as you were learning this from
8:20
scratch like what was the learning curve like?
8:22
Because it seemed like this space, was it still is changing
8:26
a lot, right? Like especially in light of the
8:27
metaverse and some of the other applications that people are
8:30
looking to put together for this, I'm curious.
8:32
What was it like growing in that stage and then going to another
8:35
field? That's all so complicated,
8:37
right? Just too curious to hear your
8:39
experience there. Yeah, I mean, it was really just
8:42
putting myself in collaborating with experts around me.
8:45
Right? Is just learning from Others and
8:47
just soaking up a preparation. Can't that was the case in this
8:51
low level systems firmware role that I had.
8:54
And then in 2017 and I joined meta on switch gears a bit and
8:57
focused on more ml space which we can kind of speak to again.
9:00
Same thing, just learn from bunch of experts in the space,
9:03
highly, iterative knew nothing from the beginning and it just
9:05
been kind of circle. Things up sense.
9:09
I think that's definitely a common experience and maybe just
9:12
this kind of segue into your experience at meta.
9:15
How did you get into the ml space coming from this systems
9:18
kind of low-level firmware space.
9:20
How did you bridge that Gap? Yeah.
9:22
Absolutely for those not familiar the way matter works
9:24
is, you're hired as an engineer for the company and then you go
9:27
through boot camp and team finding process.
9:29
It's really up to you to go engage with teams and figure out
9:32
what you want to do. I knew kind of based on my
9:34
exposure to a mound computer vision, I was interested in
9:36
exploring that space more but really didn't know what I was
9:38
looking for. And ultimately found a team.
9:41
It was responsible for distributed training and what
9:44
would eventually become a? I infrastructure organization.
9:47
And it seemed just like the perfect mash-up between this
9:50
email space that I was interested in learning more
9:51
about what like a bunch of systems problems which I'm
9:54
skilled in and comfortable and passionate about.
9:56
And so I found this nice Middle Ground.
9:58
They really, let me just kind of jump in and start kind of narrow
10:01
and then really grow out from there and kind of explore a
10:04
bunch of different facets, kind of at my pace and did that have
10:09
Anything to do with the pie torch team or were you
10:11
completely separate? Yeah, at this point I think it.
10:14
So I joined early 2017. I think at this point I torch
10:18
had been released but it was part of the research
10:20
organization at that point. We were using in the production
10:23
space reason Cafe to and so a good chunk of my time there was
10:26
on can't be too. In fact I worked on distributed
10:28
data parallel algorithms for and systems objections for Cafe to
10:32
which would end up being the precursor to pie torch DDP for
10:35
those familiar. So I'm kind of involved in some
10:38
of the development. That's base.
10:40
All right, so you mentioned that you tried to surround yourself
10:43
with some smart people that understood where you were going,
10:47
what are some things that you had to learn and you wish you
10:51
would have learned faster or sooner.
10:53
I should say, maybe not faster, because everybody learns that
10:56
their own pace, but things that you recognize the little bit
10:59
later on down the line. And you think to yourself, I
11:02
would have liked to have known that a little bit earlier.
11:06
Yeah, I really saw this from kind of two angles, one kind of
11:10
like off the stack and toward the ml space.
11:12
Like I just had no idea. Like I took Andrew in course and
11:15
just talk, give me some more familiarity around like what is
11:18
deep learning and what our neural Nets?
11:19
And I knew just enough, well that's all like a lot of that's
11:22
pretty academic and simple and when I was working on was
11:25
instead like doing distributed training using model,
11:28
parallelism across scenes of nodes.
11:30
And so things get like way more complex really quickly and it
11:33
kind of Builds on a lot of these Basics that I really didn't
11:35
quite And so I was just kind of flying by the seat of my pants
11:38
leaning on others where I could but also just trusting that
11:41
like, I'll figure it out eventually.
11:42
I didn't get too hung up on. I got to know all this.
11:44
You just like the entire surface is not knowable but I think
11:47
interestingly as well, I had not worked at a service layer before
11:52
a classical a, like I had come from operating systems will
11:54
level systems. So at the same time I'm also
11:56
learning all this, the that classic improve service oriented
12:00
pieces to. And so in met as a leader in the
12:04
space, a ton of amazing experts here.
12:06
That I had an opportunity to learn from.
12:07
So I was really relying a ton. I'm just my intuition and just
12:10
examples that I found and just kind of trying to mimic what I
12:14
could apply my intuition, like what, what I thought was right
12:17
as we move forward and try to scale things.
12:20
And there is one other thing that I was thinking about and it
12:25
was around the type of scale that you were working with.
12:29
Can you tell us the size of? Not only in with the hololens,
12:34
which by the way, I still haven't.
12:36
Ida Holo lens and is ever since 2014.
12:39
Or since I like started seeing it, I was like, this is the
12:42
future. Wow, this is crazy.
12:43
I'm so excited for it, and I have not gotten my hands on
12:46
that, but when you're looking at the scale of what you were doing
12:49
with hololens, and then when you're looking at the scale of
12:52
what you were doing at meta, can you give us a little bit of a
12:54
breakdown on that? Yeah, the problems were really
12:57
different with Holland's you doing a bunch of local
13:00
processing, right? And so latency is are just
13:02
extremely critical for those using have using VR before.
13:05
Like it's really tickle to get your shot Gates down so you can
13:08
update that the renderings. And so talking about optimizing
13:11
latency at all sorts of layers of silicon, the chipper working
13:14
on, had a number of cores and we did like explicit cash
13:18
management to pass messages across chords to just avoid any
13:21
sort of Lindsay's, like, just really interesting problem.
13:23
Still squeeze out and optimize pieces, but it meta things were
13:26
all scale and a more classic sense of just more data, bigger
13:31
models, justjust larger more complexity more compute and
13:34
there's just a whole bunch of interesting.
13:36
Problems that arise in that space to just to try to like
13:39
that. It kind of put some context on
13:41
the scale. When I first joined met in 2017,
13:43
I had an opportunity to work kind of very tangentially on a
13:46
paper at the time, which was training, but imagenet one.
13:48
Can get a set in an hour, which was state of the art of the time
13:51
and not worked in the academic research space.
13:54
Like I was working on some fault tolerance features to move the
13:56
project forward but that was 256 gpus.
13:59
I think people hundreds if I recall it was just a super
14:02
interesting project in a very like practical way for me to
14:05
apply. The distributor learnings I was
14:08
building the fault tolerance mechanisms.
14:09
I was working on 20. Like the, here's how it actually
14:11
maps to the researchers and leveraging, its right.
14:14
The end of the day of investing in liability news, things are
14:16
cool from a computer science standpoint, but the end goal is
14:19
like the train this model and all of the like to system.
14:22
Stuff is a means to an end. I think it's a great point that
14:26
there's yeah. First up, there's a lot of what
14:28
you said that I can relate to just, like figuring things out
14:30
learning from the people around you, but I would love to dive
14:33
into some of the things you said to respect to the work you're
14:35
doing. And I Meta with the distributor
14:37
training. So at the time, right?
14:39
The early days, I think you said that when you started, this is
14:41
when Transformer started little by little kind of creeping into
14:43
the space and model started, getting bigger and bigger.
14:47
And then it became not only the cool thing to do.
14:49
But a necessity, right? To distribute these models
14:51
across multiple devices, can you just from your perspective as
14:55
you started working on the stuff?
14:56
What was that? What was it?
14:58
Like adapting to that Trend? Especially as you're like
15:00
building out this infrastructure and what were some of the
15:02
challenges that you face there? We can probably do have a lot
15:05
deeper there but yeah. Just at a high level.
15:07
Yeah, there's a bunch of different dimensions to speak to
15:09
here, but I think one of when interesting one that I'm sure
15:11
you can relate to is really just around like scheduling and
15:14
resource constraints. Like, how do you even in Meadow?
15:18
We were constrained in terms of how much GPS would get access
15:20
to, how do you schedule fairly so that someone who wants to
15:23
train a really big training job that wants tons of gpus, like,
15:26
how do you optimize their use? You don't want to hold back the
15:28
resources until they run, but hold up for available because
15:31
then you're not wasting the resource in the net efficiency
15:34
of the cluster and so is a bunch of Interest.
15:36
An algorithms around. Okay, let's let the small jobs
15:39
training for a bit but then let the big guy come in when enough
15:42
is ready. And what that requires a bunch
15:44
of like cooperation with training stack to make sure you
15:46
don't like waste progress. They like just a bunch of
15:48
interesting like cascading Fallout from not only just scale
15:53
and but like the heterogeneity of the workloads and it's
15:56
important to have systems that like anticipate this
15:58
heterogeneity like in and isolation.
16:00
Some of these problems are the kind of OneNote.
16:02
There's like one there's one that set of configurations that
16:04
are strung through them, but once you're looking Like a
16:06
basket of these things and there's all these different
16:08
container optimizing for around scheduling.
16:10
And what are the like, what are the neck Network behaviors?
16:13
Where should you locate these things?
16:14
Like, where is the data? There's just a bunch of
16:16
interesting problems in this space.
16:17
It sounds like it's so messy. It's hard to keep track of
16:20
right? The especially from a
16:21
reliability standpoint. What did I mean?
16:23
You were already getting. So what are some things that you
16:25
did to manage? The, I guess the huge search
16:28
base. There's so many parameters, you
16:29
could tune, right? So many things that could mess
16:31
it up. How did you work together as
16:34
specially as a large team to isolate the Things that were
16:36
happening to at least understand.
16:39
Hey, this is where the issue is versus not there.
16:41
Because that, even then, even just, that alone is sometimes
16:43
really difficult, right? Just to identify that it's
16:45
happening at this layer, not at that layer.
16:47
Even though the symptoms may seem like it's happening
16:49
somewhere else. The other sounds really obvious
16:51
in retrospect, but like anticipating failures, and
16:56
behavioral patterns, and instrumenting those and then
16:58
logging them is like Paramount. And it's really hard in the ml
17:01
space because the space moves so quickly and you're just trying
17:05
out new ideas and it throwing them away in the trying
17:07
something else out it's really hard to find the point at which
17:09
it's worth investing the instrumentation or like
17:12
visibility or tests or whatever. Right.
17:13
And so but like then you can kind of get stuck in this like
17:16
Loop of then you're just like iterating and get to go back to
17:19
the beginning because you can't actually diagnose a particular
17:21
problem. And so leveraging tools that are
17:24
able to like put like observability and like failure
17:27
notion kind of fronts on Center and provides like Hooks and ways
17:31
to dig in is really important and then you cook like plan for
17:34
it and especially at the info layer or Charles a little bit
17:36
more stable mentoring that you're investing and
17:38
observability that you can derive patterns from run without
17:42
needing to iterate to learn what went wrong and what you want to
17:45
do next. Obviously important, I think the
17:48
other piece to adhere to is that there's opportunities to look at
17:51
patterns that are across workloads as well.
17:53
I think one of the biases that I observed was a desire to look at
17:57
like single work load logs as opposed to Trends it like time
18:02
series data of a particular job over time or look at like across
18:06
Jobs like how is this one? An outlier, what's different
18:07
about it or they are the network latency is lower.
18:09
Why is that? What can we reason about at the
18:11
data center? I think there's, I think taking
18:14
a step back and zooming out of the logs of particular job and
18:17
trying to understand kind of like, wider execution patterns
18:20
can be really helpful weight. And how were you doing?
18:23
How would you do that? Would you just did you create
18:27
some kind of program, to be able to look at them all?
18:29
Or would you just bring them into Excel and have a look?
18:35
Yeah, yeah. Present it meant of there's some
18:37
great tools and as I'm learning now that I've spent a couple
18:39
months I'm working in the open source space.
18:41
There's some here as well around just instrumentation and is
18:44
really two common types of use. Yeah, tabular logging for kind
18:48
of verbose details of various states are passing through a
18:50
super useful and then more interestingly in the one that
18:53
was maybe less commonly used but I think will be helpful was was
18:56
like time series data point. So just like a mitten counters
18:58
as you got to various events happened.
19:00
You're like walking Lane season. You can look at standard
19:02
deviations and average latency from various things and you can
19:05
use those Has to like glean insights, they can kind of give
19:08
you your next exploration direction.
19:10
Or next thing I tried So there is something that I want to
19:16
transition to an it feels like now is the point.
19:19
Now we've talked a lot about your time at Microsoft.
19:22
We talked about meta and you are now in a new phase and new
19:25
chapter of your life. You're not working at Union and
19:29
shout out to everybody that is working at Union and the people
19:32
behind flight, because we did. So much cool stuff with Caton
19:38
and the union team as when Drew and they are one of Are proud
19:42
sponsors. So we cannot thank them enough
19:45
for being a sponsor. But what is the one thing that
19:49
is interesting to think about? For me is you're not necessarily
19:53
a machine-learning engineer, but you do deal with machine
19:56
learning engineers. And I'm sure that you dealt with
19:58
them at meta and you're continuing to deal with them
20:01
now, at flight and at Union, what does that look like in your
20:06
eyes? How do you see someone that is
20:09
in your position that has your skill set?
20:12
Being able to interact with the machine learning engineers.
20:16
Yeah, so I think coming from meta I have like I had this
20:19
somewhat kind of like narrow perspective on like how things
20:22
were done. What tools do you use?
20:23
There was like one way to do things there was like these like
20:25
layers and I was at times like several steps removed from ml
20:29
Engineers or data scientists in such early this year when I
20:32
started engaging with the wider homologous system and companies
20:34
in the space and ultimately connected with Union, I was just
20:37
amazed at the community and actually how a lot of these the
20:41
like XF and rolls and people that like the different Those
20:46
working in the space, whether there will level systems
20:48
Engineers or their Ops folks. Or ml Engineers or folks really
20:53
are like engage all together in are really working like very
20:57
collaboratively by his. I think that is really well
21:00
embodied in flights Community. It's really awesome to see the
21:03
number of companies that have adopted flights and the
21:05
conversations happening in the selection around.
21:07
Here's the scenarios we're looking to enable is this
21:11
somehow Provide support this? How can we contribute to make
21:13
life better to accommodate these kind of scenarios?
21:15
I think just I think Bridging the Gap.
21:16
You communicate is really powerful because I think it lets
21:19
each side kind of understand the perspectives of the other and
21:23
what are the challenges in this space or what's important to you
21:25
prioritize? And I think the other
21:27
interesting piece about flight in particular is like the tool
21:30
itself facilitates. This kind of like collaboration
21:33
where you can have different teams and personas building
21:37
different parts of orchestration Pipeline and they all get kind
21:40
of stitched together and then can interact.
21:42
So someone can it's some maybe data input for a pipeline.
21:46
Spend like curated by someone deeply familiar with data and
21:49
familiar with like features and such and then someone else who's
21:51
like, optimizing the like performance and behaviors at a
21:54
lower level of the training staff, for various steps in the
21:57
pipeline, the tool kind of becomes this medium to actually
21:59
facilitate that communication. That's really interesting.
22:02
Like I kind of am thinking about like these Mo in four teams that
22:05
or ml Ops teams that kind of form like that, right?
22:08
It's like it's velocity of different skills working
22:12
together for a common goal, and it's the working together.
22:16
Towards that common goal. That makes it ml Ops not the
22:19
things individually right by themselves are kind of separate
22:21
but once they are put together working on a common product or
22:25
for a common goal, it becomes I guess this is the kind of same
22:28
space and it seems like you can get away with not knowing
22:33
everything. When you have a strong team to
22:35
count on it and bounce ideas off of her, maybe this other person
22:38
has this area of expertise and maybe you have this expertise.
22:41
So you need this kind of broad coverage of this sort of scale.
22:46
That you would need to manage these things.
22:47
Do you feel like that's still the case, or is the tooling
22:50
space changing so much that now that may change if the tooling
22:54
is changing? Will that also change the roles
22:56
and the responsibilities of these individuals?
22:58
I'm curious. What your take on that is?
23:00
Yeah, I think we still have a lot to observe here.
23:03
I think things will continue to evolve.
23:04
I think the jury still out. I think think that the pace will
23:07
continue to be really rapid. I think, from a tooling
23:09
perspective, that's important to be opinionated on some pieces
23:13
and maybe like less opinionated. Another is where things are.
23:16
Like less must cauterized it son.
23:17
It's not clear where we're going to end up.
23:19
And I think that looking at something like fight, we
23:21
actually we see this where like flight puts like strong typing
23:25
front and center. Like that is an opinion that the
23:27
tool has and now that becomes this constraint that all of the
23:31
players have to operate and right and actually becomes the
23:34
like some from the language that like they're speaking
23:36
interacting through, but as an orchestration to a plane is also
23:39
an opinionated on like how you get a test done and actually is
23:42
highly extensible and supports a variety of plugins to interop
23:45
with. Various tools, whether they're
23:47
like Legacy tools that you may be using at your company or
23:50
maybe some future tool that isn't invented yet.
23:51
It's built to work well with these.
23:53
And I think as the think as this ecosystem of olives and is your
23:56
noting though a roles folks play, I think it will get in
24:00
particular, will naturally fit in with us.
24:06
Hey, I'm Vishnu. I'm a data scientist at first
24:09
hand, and I definitely think that you should subscribe to the
24:13
ml Ops coffee sessions podcast. It's the best podcast out there
24:16
to stay on top of what ml Ops actually is to talk to the
24:20
truth, thought leaders in the space.
24:22
And oh by the way, Demetrius is absolutely hilarious.
24:25
What a weird guy, you should definitely subscribe to the
24:27
podcast. Something.
24:31
I'm I've been curious about is just kind of going back to the
24:34
distributed training space and we're talking about how ideas
24:37
develop and we kind of come to a consensus and products or tools
24:41
forming. An opinion is important because
24:43
that shapes the direction of things.
24:45
One thing that came to mind was in the distributed training
24:48
space, like, when you started data, parallelism was like the
24:52
standard way to train models, right?
24:53
And then as the models got so big, and there's a limit to
24:56
which data parallelism works. I think it's a 1.3 billion
24:58
parameters, some like that. And then Lining came into play
25:01
and then tensor, parallelism. And it's only now that I'm
25:04
starting to see some consensus on the definition of these
25:07
things. What this means and where works
25:09
and what the trade-offs are. Like when it's good, when it's
25:12
bad, it's just want to chat about that because it seems like
25:15
this happens a lot like these, it takes a while for there to be
25:19
a consensus on what something means, but I find that's pretty
25:23
challenging especially when you're learning about the space,
25:25
right? And you're trying to, I guess,
25:27
develop some sort of expertise, it feels like you're just like a
25:30
moving Target. To speak.
25:32
Yeah I think there's a combination pieces to think.
25:34
They're I think I think one is not like over invest in a
25:38
certain space until it has stabilized a bit.
25:40
Don't don't constrain the system to meet like a certain set of
25:44
assumptions aren't going to quite hold or haven't played
25:46
themselves out yet. I think this also is important
25:48
as you think about abstractions. Like it's, it's critically
25:51
important that when building on top of these very complex
25:54
systems that are orchestrating complex workloads, managing
25:57
data, and all these various scenarios that Emily, Engineers
26:01
and researchers in such a working at a higher level.
26:03
There's of Attraction don't have to like really reason about
26:05
that, but getting those abstractions, right?
26:07
Is really difficult. And I saw this play out a lot at
26:10
meta, I imagine. This is true in other areas of
26:12
the domain as well. We're like you take a stab at
26:14
some like nice clean simple abstraction and then it's like,
26:17
uber. We need this like, one other
26:18
thing to like poke a hole and then you like poke a hole here
26:21
and then all of a sudden like you've reached all of these
26:23
knobs from your level system, like way up into the upper layer
26:26
and then everyone's frustrated. No one understands like what
26:29
that knob does. And it's like, he totally You're
26:30
what happens over here and seen up throwing it away like
26:33
standing up a new thing. And so I think some of that you
26:35
kind of have to accept but I think it's also really important
26:38
to be super intentional about like what layers of abstraction
26:41
you expose, how you expose them and when you jump to a lower
26:46
layer in the stack and I think, as part of this, like there is
26:48
no one-size-fits-all, there is so many use cases and reasons
26:53
for operating at different layers trying to build
26:55
one-size-fits-all is probably like a little goal but being
26:59
intentional when Building tooling and abstractions of who
27:02
you're targeting? What capabilities do you want to
27:05
offer? And then when is it appropriate
27:06
time to work at a different layer of the stack.
27:09
Yeah, that's such a good point man.
27:11
Yeah, and I think it's, I'm curious the kind of leading into
27:14
what you're doing at Union right now.
27:15
I'm curious what sort of abstractions make sense.
27:19
Now given what you're working on which where do you see
27:21
opportunities to improve those abstractions?
27:24
And I see this also related to a big thing.
27:26
I'm learning is framing. The problem is the most
27:28
challenge is almost the most challenging part just like what
27:30
is it that you're trying to actually do?
27:32
There's lots of solutions out there kind of in search of a
27:34
problem but finding the right problem to work on seems to
27:37
really related to I mean or the right abstraction, right that
27:41
you would use to yeah to solve some solution.
27:43
So I'm curious that Union. What sort of work do you guys
27:46
doing there? What sort of abstractions are
27:47
you guys building? Absolutely, yeah, so there's a
27:48
couple, there's a couple things we're investing in this space.
27:51
One is Union ml, which is a layer built on top of flight and
27:56
flank it that offers a higher level abstraction to really
27:59
easily instantiate a models and stained a server to do inference
28:03
on like an awesome way to get started extremely quickly it's
28:07
it's fantastic. I Like the other another
28:09
dimension that Union particular is focused on is removing more
28:14
of the infrastructure management burden from teams, like flight
28:20
flight itself, is an amazing tool but it's a production grade
28:24
system that has to be deployed and managed and configured and
28:27
there's a bunch of knobs and for companies that are like a
28:30
quipped and prepared to take that on, it's super powerful.
28:33
But for a lot of companies like the skillset that folks have I
28:39
don't want to manage infrastructure, they really want
28:41
to just get started and iterate, and ID, and experiment.
28:45
And at Union, we're doing what we can starting with Union Cloud
28:48
to get folks started as fast as they can, and remove, all of
28:52
that burden this, there's a slight cluster running.
28:54
You can hit it next to your workloads.
28:56
You don't worry about managing them and it's available to you.
28:58
There is another piece that I was thinking about and this goes
29:02
back to, when Kate Town's been on here.
29:04
I've Loved learning from him. He mentioned before the last
29:07
time he was on here. Or how when you're doing machine
29:10
learning, you can easily get into some kind of dag.
29:15
Hell. And at the beginning, everything
29:17
looks good and then after a year or so it's just a mess and it
29:22
can that complexity can Spike and it can grow exponentially
29:27
one piece that I was thinking about with what you were talking
29:30
about and Union is your you're able and not just you, but the
29:37
whole Union team. Is really working at such a low
29:40
level for scale. And like you said, for this
29:43
production level, what do you feel like you compare yourself
29:49
to? Because I know there are a few
29:51
other tools out there that are doing this.
29:54
But do you feel like you are, like who when you look around
30:01
and you say, what are the other tools that are doing what we're
30:03
doing without saying? Oh, nobody is because we're the
30:06
best and nobody can touch us. Obviously, who do you compare
30:10
yourself to? Yes, I think it's important to
30:13
think through the Journey of someone ideating and ultimately
30:17
production izing their workload. It's really easy to start with
30:20
some simple tooling and a simple space.
30:22
Maybe it was from just like scripts and I notebook and you
30:25
can get everything kind of running locally and to Kim's
30:27
point, you know, they may be you turn these in a little like mini
30:29
dag and then it's like just a little bit of a eunuch to go.
30:32
Right to go Stitch a couple things together and then like
30:35
quickly it snowballs and all of a sudden it's like whoa, what
30:37
did I build? What am I managing?
30:38
Here I think for me as I think about what flight offers and
30:42
like what union is focused on is around getting the engineer from
30:48
early idea to production seamlessly, I'm in particular
30:51
new to a lot of the like the the tooling or this is actually the
30:54
first time I've used kubernetes and so I'm learning as I'm here
30:56
and I as an infrastructure engineer, I'm like, whoa, this
30:59
is complicated, I can't imagine like a data scientist or
31:02
engineer having to, like, think through a lot of these.
31:04
These like pieces and configuration.
31:06
It's a tool that you can start and do some of the Things you
31:09
can run locally you can but it's but what first principles that
31:12
enable it to go from that to production grade?
31:16
Where all of the like caveat sand Corner cases of all been
31:20
like thought through and handling are like built into the
31:23
product that there's not this like big leap, you have to make,
31:26
you don't have to rewrite your thing in some new tool.
31:28
I think it's what that's something that's really awesome
31:31
about flight and how it was built and in particular and I'm
31:35
sure others. Other listeners know what was
31:36
built and run a production scale with It is now used by 80 plus
31:39
companies across the space, but as you alluded to, there's a
31:43
whole bunch of folks investing in this space, I think there's a
31:46
bunch of Open Source options and then a bunch of companies have
31:49
built their own Solutions. We had when I met it.
31:51
I think there's there are bunch of solutions like airflow, which
31:53
you alluded to, which are commonplace, not necessarily
31:57
because they serve the role of ml orchestration perfectly but
32:01
because there's been an evolution around them.
32:04
And so I think, I think something like, it's like,
32:06
flight is able to bring a targeted purpose-built approach
32:10
to improving the experience right here.
32:12
Flow within looking at the ml space yet is a bunch of players
32:14
in space. I think cuckoo is probably the
32:17
may be the most prominent, but again, I think the jury is out
32:20
here. I think there's there's a I
32:21
think there's a lot to be figured out and at the end of
32:23
the day, the system that supports complexity is highly
32:27
reliable efficient and performance but also provides
32:30
like a nice highly iterable abstraction for folks on is the
32:34
one that will win. Yes, really good answer you
32:36
mentioned that principles often. Form these things like the from
32:41
your experience, in the distributed training space.
32:44
And with the a infrastructure at meta, what are some of those
32:46
principles that you learn from that experience specifically
32:49
that you think can be brought into a tool like Union to
32:53
improve it? Because I feel like the
32:55
distributed training space is like HPC mixed in with deep
32:57
learning rate and a lot of the learnings from systems
33:00
engineering, right is apply to the specific context of deep
33:03
learning or machine learning. But what are some of the general
33:06
principles that are really useful there?
33:08
Sure that you see being applied to a tool like Union, so that it
33:12
is kind of like dutt. The users of this are forced to
33:16
like to use those principles or those best practices so to
33:19
speak. Yeah, I think a couple examples
33:21
that it exists today and I think there's opportunities to extend
33:23
upon these one around like version management repeatability
33:27
and cash ability within flight. This is something that that we
33:31
had and orchestration system at meta.
33:34
I think I kind of took it for granted, or I seemed obvious is
33:37
desirable and wench. With kids on another's that
33:39
Union when I first met them, like this is one of the first
33:42
things they read up which is this is super powerful.
33:44
Ml is highly iterate, like we should be reusing steps along
33:48
the way, we should expect to run, beginning to end.
33:51
We need to go back and rerun this piece.
33:52
Try something new here and so building this in baking this
33:56
into the product. And the way that you actually
33:58
build a pipeline, I think is really cool.
34:00
And Not only was there for thought in doing this the way
34:02
kitten spoke to. This was like oh yeah.
34:04
Like of course we would have this like why would you do this
34:06
any other way? Like why would people not
34:07
leverage this? Interesting is not something
34:10
that I saw leveraged as broadly and my kind of meta.
34:13
And so I think there's I think that the team at Union is really
34:16
a couple steps ahead here and I think a lot of this is based on
34:19
their experience, working at lift and production.
34:21
Izing is there, and I think is a whole bunch of features that
34:25
exist today and the product and get, I think will pull into the
34:28
product over time. That users don't even know how
34:32
powerful it is yet and don't even know what they will need
34:35
when they move from the second idea, phase to production.
34:38
Larger scale doing some these problems?
34:42
Yeah, that's one thing that I love, I know what's coming.
34:45
So we're going to try and prepare Union and flight for
34:49
that future, because it's inevitable.
34:52
When you get to a larger scale, you're going to hit some of
34:55
these hiccups. And these challenges, you can
34:57
speak to an interesting example that we faced while I was at
35:00
met. As soon as your training on tens
35:02
or hundreds of machines, concurrently as part of a
35:04
single, like, step of a pipeline, you just have a whole
35:07
bunch of, like, Problems around like fault tolerance.
35:12
Like, how do you make forward progress?
35:14
When like you have certain types of failure rates or even like
35:17
planned outages? So for instance, when we think
35:18
about how do you roll out, post updates like firmware updates or
35:22
like driver updates to a set of machines.
35:25
If you now imagine that you want to complete this leg roll out
35:27
across the fleet in some time, but you have these jobs that are
35:31
running on tens or hundreds of machines.
35:33
You have to interrupt them to roll this out.
35:36
But if you interrupt them independently, Up doing that
35:39
job, tens or hundreds of times if you don't like aware of what
35:42
like what is this job doing? These nodes are all like
35:44
interconnected. We need to like plan for this.
35:46
How do we cooperatively? Make sure that we're like, you
35:49
know, getting to some saved, check pointed State before we
35:52
didn't interrupt the job so that we don't throw away a bunch of
35:55
GPU compute hours worth of work. There's a bunch of interesting
35:58
complex problems here. You don't know if this is
36:00
something that users of flight have run into yet.
36:03
And I think there's lots of opportunity for these kind of
36:05
things as like compute skills up and job just become Bigger when
36:09
more advanced use cases, one kind of like super specific
36:12
question. But I'm curious because I was
36:13
working on it and my last company is checkpointing.
36:16
So like you mentioned these really long running training
36:19
jobs, you do not want them to fail, you want it.
36:21
And if they do hopefully restart from that checkpoint, that did
36:26
fail is Union working on anything along those lines to
36:29
enable it. Super easy for these
36:31
long-running job skiff. The they do fail to restart from
36:34
there like previous check point. Yes.
36:36
Oh yes. Um, Supply does not feature
36:37
called inter-caste, check, Pointing which does just that
36:39
there's this simple python API which lets you write a file to a
36:43
well-known location and the orchestration system flight will
36:46
feed you back this file restart and so the task can be built to
36:50
directly leverage this. And if you're using something
36:52
like Pine tortures like layers of abstraction built on top of
36:55
this and you can recover, they can kind of plug-in and
36:57
incorporate to this system. That's awesome.
37:00
Any sorry. I'm one last thing.
37:02
What about? And I know there's been lots of
37:03
work in these place. A libraries like deep speed or
37:06
Fair scale. That will take your model and
37:08
automatically partition it for you.
37:10
Is there any interest in combining that sort of
37:13
functionality? It seems like it'd be hard
37:15
though, right? Because that space is changing
37:17
and it seems Union is mostly focusing on the orchestration of
37:20
a lot of these things. But will it also includes some
37:22
of those use cases? Yeah, I think it's a good
37:24
question. I think you need is a young
37:25
company and it is focusing on what it does really well.
37:28
Which is Orchestration layer and executing their in really
37:32
empowering layers above in various Integrations, to no
37:36
leverage, those exploit some of these advancements.
37:38
Like you're noting what the future holds for Union.
37:39
I think I think TVD I think there's a lot of there's a lot
37:42
of room for growth. We have a ton of really awesome
37:44
and smart people at the company that are highly plugged into
37:47
what's going on in the industry. It's pretty cool to see the
37:50
communication within the company and Shane sharing of new ideas.
37:53
New tech that comes out like what's hot right now.
37:55
But I think at the moment, focusing on the skill set that
37:58
the team has Has. And like, what?
37:59
They've? What they're proven themselves
38:01
in a delivered Rajan and expanding on.
38:03
It is with votes. I think it's wise, right?
38:06
You do what you're best at or what you're good at.
38:08
There's too many things going on in this space to take a stab at
38:11
the mall. But it's nice to know that
38:13
there's people that are good at keeping up with all that, right?
38:16
Because I feel like that's even a skill in and of itself just
38:18
like what is going on in the space?
38:20
Like I suck at it. I always find out last minute on
38:23
then when Some people post some paper or something but it just
38:26
so fast moving so I can only imagine it probably wouldn't
38:29
make sense or wouldn't be wise for a company to make that
38:31
they're like business strategy, right?
38:33
We're just going to keep up with everything.
38:36
Yeah, then you end up chasing just the next new thing.
38:41
And all of a sudden you're creating a crypto company.
38:46
Anyways, man, Andrew anything that you would have liked us to
38:50
ask you that we didn't get to think.
38:52
We really covered a ton here has been an awesome conversation.
38:57
Yeah, yeah, yeah. I really enjoyed it.
38:58
I'm looking forward to seeing what comes out of Union and
39:02
hopefully we can have another podcast with you in the future
39:04
where you would. Let us know exactly what you
39:06
guys are up to. Yeah.
39:07
And yet look really looking forward to that.
39:08
Awesome, thank you so much guys. Sweep in take care.
39:13
This is Skylar, I lead machine learning and health rhythms.
39:17
If you want to stay on top of everything happening in ml.
39:21
Ops, subscribe to this podcast now.