Latest / Agentic Conversations (formally mlops.community) / Fixing GPU Starvation in Large-Scale Distributed Training

Transcript

Fixing GPU Starvation in Large-Scale Distributed Training · Agentic Conversations (formally mlops.community)

0:00
These chipsets now have disk, they have SST.
0:03
What if we just cache it? We don't have to make remote
0:07
calls. So the first time in the first
0:11
epoch, when you read the data, just store it locally on your
0:15
GPU, CPU, host, so that next time when you need that data,
0:20
just you have a cache hit, like just bombard it locally.
0:23
You don't have to make any remote calls.
0:31
There's no shortage. Like you're never going to have
0:34
a problem as a software engineer.
0:37
The way that software engineering is going to look in
0:42
a few years is going to be completely different than it
0:44
looks right now or it looked a year ago.
0:48
Yeah, yeah. I'm I'm not buying into the
0:50
whole like, oh, software engineers are cooked narrative.
0:55
Yeah, just my director has flagged me.
0:57
I'm the expensive engineer using Cloud code.
1:00
So by billing numbers are just going on the top.
1:02
It's fine. It's just a few dollars, yeah.
1:06
But yeah, yeah, oh man, when folks start spending more than
1:14
they spend on like engineering resource like.
1:17
Humans, it's too far. I don't think so.
1:20
We are near that. That's going to be then there's
1:23
going to be some real questions that are raised, like is it
1:26
actually should we be doing this with an agent or should we just
1:29
hire a human? Yeah, I think at this point we
1:33
are we we are in a discount phase.
1:35
It's a honeymoon period. Let's see if actual life is
1:38
kicking. I don't know how expensive it
1:40
would be. Yeah, totally.
1:42
Well, dude, you're working at Uber right now on the ML
1:45
infrastructure. You get to go pretty deep on the
1:47
infrastructure. You get to play around with GP
1:49
US. Before that you were at Google.
1:51
You played around with YouTube ads, which are awesome, right?
1:56
And then also the core search ranking, which I think was
2:01
that's like the golden goose of Google.
2:04
And tell me a little bit about what you were doing back in
2:09
those days and your journey so far.
2:11
No, no, that's a good question. Thank you so much for having me
2:14
here. So yeah, like most of the time
2:18
when I OK, so of all those who are listening to me, I'm
2:22
Kashish. I work at the stock engine in
2:24
Uber, leading the marketplace matching team.
2:27
And most of my work is in ML infrastructure domain using
2:30
related to scalability and efficiency.
2:32
Before that, I was in YouTube at SML.
2:35
So I feel like the scale and the problem constraints are very
2:39
different, like working in Google versus working in Google.
2:42
They have of course TPU, GPU, NVIDIA, they also make a
2:46
difference here. But like the problem space, I
2:49
have been dealing with this. Whenever we want to scale the
2:52
model, it's never the model which is the culprit.
2:55
Like many MLE's come here, maybe the architecture is not
2:58
efficient. Maybe we should change the
3:00
layers, the hidden layers or something.
3:02
Or maybe we can have a different model architecture.
3:04
It's never the problem, it's mostly always intersected.
3:08
Or quantizing it, that doesn't matter.
3:11
It doesn't make a huge difference when you look at the
3:14
bigger picture. It's always a constraint on the
3:16
infra and most of the time what I have seen is the data IO.
3:21
Like I have so many stories from my time in Google where the data
3:27
IO but like it's not just a soft engineering problem, it is a
3:30
problem at different stages where it happens.
3:33
Like to give you a very quick examples of two of those
3:36
stories. 1 is like it's a very standard problem.
3:40
Anyone in the industry would like, hey, let's say in a model
3:42
which is a smaller model, not a global model segmented on a
3:46
small subset of data. Now, of course everyone use
3:49
park, everything is columnar. We know that, but it's not
3:53
efficient to read a slice of data from that park a tables
3:57
when you only care about a few rows.
3:59
So it's more like a statement where you read everything 1st
4:02
and filter it out like it's too expensive, you're wasting a lot
4:05
of time in CPU and you're really, really wasting those
4:08
GPU's. Like I feel like these GPU's
4:11
have become so precious, so like a scarce resource that if you
4:16
are not able to feed them and you're starving the GPS, you're
4:19
not doing a good job. And it's not just about
4:21
resources. I feel like the velocity is
4:24
another aspect for at least these industries.
4:28
You cannot, you don't have infinite amount of GPUs, so you
4:31
cannot just throw more GPUs at it.
4:32
And whatever your iteration speed is, you're going to slow
4:35
it down. Like you're going to take a hit
4:37
on your productivity, you're going to take a hit on your
4:39
landings. So it's not just resource, it is
4:42
productivity as well. And.
4:45
What I'm hearing you say there is really like folks being
4:49
sloppy about using the GPUs and really like overcompensating
4:55
with, hey, we can just use this compute and whatever, we're
4:59
going to read the whole file. But it's not that big of a deal
5:04
because we we don't need to worry that much about it.
5:08
But when you're at an organization where everyone is
5:10
fighting for every slice of the GPU, then you have to be very on
5:17
your game about how you're using those or how you're letting
5:22
others use them. Are you?
5:23
Are you like the gatekeeper to the GPU?
5:27
'S you can say that like we want to ensure at least our
5:31
utilization is above 80% otherwise we are wasting the
5:34
resources because like below 80 sounds super bad.
5:39
And like if everyone is reading everything and just throwing
5:41
away the data, then it's more of a infrastructure constraint or
5:44
the problem. It's like you cannot expect ML
5:47
ES to figure that out like they are.
5:49
They have their hands in model architecture, the business
5:54
objectives, how to optimize it, whether the labels are correct.
5:57
Like that's more of a machine learning problem, how the
5:59
attention is working. But do they have to worry about
6:02
how the data is coming to the GPU?
6:05
They should only be able to specify what they need.
6:08
So it's actually a problem then more of a machine learning
6:12
problem in some sense. So like in this case, you can
6:15
actually restructure the data the way it's written on the disk
6:18
so that you can just batch the like, let's say if you know what
6:22
the slices are based on the models, you, all you do is
6:26
rewrite the entire data on the disk such that you can batch
6:29
read the slices. Otherwise you can't do point
6:31
locus. Point locus are super costly.
6:34
So that's what I was saying. Like it's the data problem.
6:36
But I feel like whenever MLE is like, let's say they want to
6:40
have a bigger model, but and they want to read everything in
6:44
the world, they won't be able to scale.
6:46
They will blame their own model. But that's not the case.
6:49
We are not able to feed data fast enough to you.
6:53
OK, so the model's there and it's that's why you're saying
6:57
you're starving the GPU. It's there, but you can't feed
7:00
the data that that model needs. So yeah, like in another case,
7:05
it's like the example I gave is more of a soft engineering
7:08
problem. OK, the way the data is written,
7:10
the data structure, the schema, but let's take an example which
7:14
is very common recommendation models.
7:16
All you care about the query item pairs, right?
7:19
You have a query and you have an item.
7:21
So when you look at the chipset where the software like the
7:24
host, CPU host and the GPU host are connected, the CPU host has
7:29
to move the data to GPU host like as quickly as possible such
7:32
that you can run the forward prog and get the predictions.
7:36
So many a times you have thousands of items per query the
7:40
way you send it like. Again I'm saying this is a
7:42
function of constraint you are working in.
7:45
In this case you have duplicate query features for each item and
7:49
you're throwing it in. This is like a fad BLOB of data
7:52
you're passing and it's expensive.
7:54
Like sure they are connected with 700 terabytes of network or
7:57
whatever, but this is still slow.
8:01
The easy way out is don't duplicate pack and unpack unpack
8:05
on GPU. You can do that, just have one
8:08
query and all the items pass it just before passing it to the
8:12
GPU. For the forward drop, you can
8:14
unflatten it and have like the entire tensor to pass it.
8:18
This way you are saving a lot of time passing the data between
8:22
the CPU and GPU host. So this is more of a
8:25
optimization that you did on the hardware level.
8:28
And it was like for everything, for all ads, because everything
8:31
is recommendation. Well, and are you messing around
8:36
much with tensor RT? So in that case you had Tpus.
8:41
So we were messing a lot with Excelish.
8:44
OK. Yeah, it's a.
8:45
Is that like a different tensor RT basically?
8:49
Yes, yes, yeah. It's like a trend.
8:51
That's the TPU version of tensor RT, yes, but does that even
8:56
matter? I guess is the question because
8:58
you're saying the model isn't the problem.
9:01
Yeah, I feel like these kind of bottlenecks exist everywhere in
9:05
one form or the another and they are solvable.
9:08
Like it's like it's not a function of model, right?
9:12
If you're not passing the data but your utilization is low,
9:15
it's not because your forward prop is expensive.
9:18
GPUs are very good with maths like math metric and they are
9:21
doing what they are supposed to do.
9:23
But if we are not using the passing the data, they can't do
9:26
their job. So that's what I was saying,
9:28
like these kind of problems are independent of the chips you are
9:30
using, but these data, your bottlenecks exist everywhere.
9:34
Yeah, and you're just not optimizing the data to be able
9:38
to get done by the GPUs or the TPUS.
9:42
Like you're not giving it in that way.
9:45
The whole part before it's doing the multiplication, it's doing
9:51
its GPU thing. Is that hard part like you have
9:56
to make sure the data is. Yeah, you just in the perfect
9:59
form. You just need to get the tensor
10:01
to the GPU one way or the other. Rest of the things are super
10:05
fast. I have like really seen like
10:08
sure you can do quantization to optimize a lot of things, but
10:11
the major jump here is getting the data first.
10:15
You should see good utilization if you are able to remove all
10:18
the data. Your bottlenecks here.
10:21
OK, so that was when you were at Google and YouTube and all that
10:25
fun stuff. You have since gone over and
10:29
there's like new stories and new fun.
10:32
Yeah, we have an upcoming blog as well on that.
10:34
I think we want to publish it in a week or two.
10:36
So Google's want to publish it. So yeah, let me give you the
10:39
crux of the problem. So our models were struggling on
10:44
a 100 chips 1520% utilization. That really sounds bad.
10:48
Like 20% of 15% utilization. You're almost wasting 1/5 of the
10:52
GPU and. Yeah, that's sounds like a lot
10:55
of money. Oh yes, a lot of money and a lot
10:58
of time wasted just to train the model.
11:00
So we were like, OK, maybe we are.
11:03
Again, I know I said the model is not the problem, but as a
11:08
software engineer, you always try to remove the variables.
11:11
So we said, OK, let's remove the variable, whether it's the model
11:14
or the infra. So the easiest way was remove
11:18
all the data portal ionics by just loading everything in RAM.
11:22
Let's get a slice of data, put it in the RAM.
11:24
So there's no read or write anything from here.
11:26
We are just reading from the RAM directly less than the training
11:30
on the model, the verdict 85% relation.
11:34
So we understood model is correct, model is not doing
11:37
anything funky which is causing a hit on the GP utilization.
11:42
We are kind of not feeding the data the similar problem we have
11:45
seen before. So then you all like in that
11:50
stage, you are like, maybe there's a easy fix, let's try to
11:53
tweak some levers. Let's increase the number of
11:55
threads of the GOP CP host. Maybe let's increase the
11:58
parallelism, maybe add more queue storage or whatever.
12:01
We did try that. Nothing changed, of course, like
12:04
maybe it's not like it was more like maybe we get lucky life and
12:07
everything works and we just move forward to the problem
12:10
space. But didn't happen.
12:12
So the next one was, OK, how should you even profile this?
12:17
Now we need to understand how do we even understand the data
12:21
bottling where it is. So we added tracing at different
12:24
stages in Petastorm like that's. The workload you're trying to
12:28
profile the workload to see where's thing, where is it
12:31
getting clogged up? Yes.
12:34
So you can consider like a very high level view of Petastorm and
12:37
there's a producer part and the consumer part producer
12:41
responsible for reading the data from some remote file system
12:44
somewhere where data is stored and store it in a queue like in
12:49
a row. The consumer is the one
12:51
responsible for doing the slicing of the batches,
12:54
converting into Tensor and passing to GPU.
12:57
So they are kind of working with each other such that we always
13:01
have the queue full such that we can continue feeding the batches
13:05
like GPU is fast, GPU is the fastest machine here.
13:08
So we need to catch up and make sure we always have the data to
13:11
pass the GPU. So that's how I would say like a
13:14
big level view of PETA Storm, so.
13:17
Petastorm is something you created or it was already there
13:20
that you guys were just maximizing or utilizing.
13:23
So Petastorm is like Uber's open source library for model
13:27
training, distributed model training.
13:29
So it existed before. So I think as the GPU's are
13:33
getting more and more powerful, we started seeing this issue
13:37
more recently. Because basically in this space
13:40
if bright machine which is the GPU is getting more and more
13:43
fast, it's getting faster then you see, OK, we are not catching
13:46
up to it. Yeah, like if you'll think about
13:49
like the previous GPU's, maybe RTX, even before that it might
13:52
not be apparent. There's no mortal record.
13:54
Both the things are operating at a similar speed.
13:57
So you are not wasting anything. But if you go to B200, you're
14:01
going to waste it a lot Like they are very fast.
14:05
They are like 8 chipset like they they will just they're
14:08
hungry. I feel like they're so hungry
14:09
that they will eat up whatever data you're passing.
14:12
So that's what the problem was remote happening now.
14:16
So I was just not a. Question of the problem isn't
14:20
necessarily that you don't have the large data sets.
14:24
No, it's that. It's not that.
14:26
The data sets you have, you're not able to just get them into
14:31
the GPU to do their GPU thing. Yes, that's correct.
14:35
Like you have the data, you have everything, just you are not
14:38
able to feed it. That's the bigger problem, yeah.
14:41
So, OK, so yeah, would you? What happened next?
14:46
Yeah, no, it's, it's interesting.
14:48
So we we added tracing at the producer side.
14:51
We were like, sure we are not able to read the data fast
14:54
enough. Maybe that's the problem.
14:56
The network cost would be expensive which we are making to
14:59
some remote file system where the data is hosted like in
15:01
pocket files. Maybe we're not reading them
15:03
fast enough and up like the queue which should it's supposed
15:07
to be full, it's not, it's empty.
15:09
And that's what we found after tracing and logging everywhere.
15:12
OK, we are not reading the data fast enough from the parquet and
15:18
loading it in queue and that's why consumer can't do shit like
15:21
it. It doesn't have data in the
15:23
queue, how can it pass it to GPU?
15:25
So the utilization was basically a very wobbly in some sense.
15:29
As soon as we test the data, it just passed through it directly
15:31
to GPU, GPU consumes it and the reader threads are still reading
15:36
the data slowly instead. So very standard soft
15:40
engineering problem, like why can't we read?
15:43
There's a fuss, What is the issue?
15:45
So again, like, it's not like the problem kind of escalates.
15:50
OK, data can be different data centers, you can have a copy and
15:53
all that thing. But the idea we've thought about
15:55
was these chipsets now have disk, they have SSD.
16:00
What if we just cache it? We don't have to make remote
16:03
calls. So the first time in the first
16:07
epoch when you read the data, just store it locally on your
16:12
GPU CP host so that next time when you need that data just you
16:17
have a cache hit like book just bombarded locally.
16:20
You don't have to make any remote calls.
16:23
And you don't have a problem with the amount of storage.
16:27
No, that's a good question. So for our case, like at least
16:30
in Uber, like the data is not gigantic, like it's not like
16:35
number of petabytes. So this problem was solvable in
16:38
these constraints. But even if you have more data,
16:41
whatever amount, let's say 50 terabyte of something, you can
16:45
at least catch that. So still that will help you the
16:48
utilization because you don't have to make calls for each and
16:50
every row group. Like it solves the problem for
16:54
us because our data was slow, but for anyone who has more data
16:57
than that, it still solves a lot of problem with getting at least
17:00
some data back to the GPU. Yeah.
17:03
So we implemented it, it was simple, it was nice.
17:06
We have cloud, so it's not like a big deal, but this was like a
17:11
desperate moment for a soft pensioner.
17:13
We were like very excited to launch it and nothing changed.
17:15
Like I was really, really disappointed with what nothing
17:19
changed. Like the queue is full, but
17:23
utilization didn't change. Like that was the moment.
17:26
I will like like you're looking at a soft pension playbook and
17:30
just like, what the hell, what did we do wrong?
17:33
But. This is Yeah, the software
17:35
engineering interview question was total bullshit.
17:40
Yeah, like what did you do wrong?
17:44
Like that's the basic right, that's all.
17:46
Like it's the magic chip you do. Or maybe it was just a magic
17:48
word you say and it used to work.
17:50
Why is it in working? Like is something wrong
17:52
somewhere? But remember a few minutes back
17:55
I said like the first thing we did was removing the variables.
17:58
So we had the headroom of 85%. So that was the only thing which
18:03
was a sparky thing which you could see there is a headroom of
18:06
85%. So we revisited our steps.
18:10
Why did loading the data in the memory worked?
18:13
Like if it works, then our solution should also work
18:18
because we also have a data in the queue which is a RAM.
18:20
It doesn't make sense like why one thing is working and not the
18:23
other thing. And what we found was it's, it's
18:27
a very funny story. So whenever we read the data
18:30
from par K, it's all it's in the PIR format because it's very,
18:34
very optimized for par K just read it very quickly.
18:39
What's Π Arrow. It's Arrow.
18:42
The parch arrow thing, yeah. Yeah, so.
18:45
It's basically reads the columns very quickly and it's like a
18:49
database. You can do slicing and all those
18:52
things. So it's very fast, on par K
18:55
files and that's what someone must have implemented, which is
18:58
the right thing to do. But this is a language that GPU
19:01
doesn't understand. GPU doesn't understand shit.
19:04
What Π Arrow said it case over numpy it case or tensors.
19:08
So this translation. Why?
19:10
This this π arrow to numpy translation is done on the fly
19:15
when we read the data from the queue and when we were doing
19:19
everything in memory, everything was numpy.
19:22
So this small thing, which is the translation from π arrow to
19:26
numpy was eating up entire headroom.
19:29
So it was more of a, how should I say, a double bottleneck.
19:33
First show the data. It was the bottleneck.
19:36
But this is not data rate. This is a transformation from
19:39
the different data types. Because I mean, if GPU
19:42
understand by the sure, that has been like best thing, but it's
19:45
not, it doesn't. It only needs dense tenses to do
19:48
anything. So the idea was again simple.
19:51
We are like, as soon as you understand the problem, you're
19:54
like, gotcha, I know what to do. We can just cast a transformed
19:58
output. We really don't need to do
20:00
transformation every time right? This cache, the cache can always
20:04
have numpy. So your queue is like now a
20:08
full-fledged queue which can directly feed into GPUs.
20:11
So that was the idea and it worked like we were able to come
20:14
to 85% yes and like our training time reduced from a day to like
20:20
an hour or two like that's it you the same set of resources
20:24
and. Dude.
20:26
You like Sherlock Holmes over here trying to figure out what
20:29
the name problem is. Oh, I, I wish like I, we did it
20:32
like last year in Q2, I wish we had more agents like things have
20:37
changed a lot over the year. Like that time we didn't use or
20:41
I would say the agents were not that common like a year back,
20:46
but I, I was. Thinking they would have found
20:47
the problem. That's what I was thinking,
20:49
yeah. Like, I mean, probably I would
20:52
say like if it's able to run some simulations, we don't have
20:56
a simulator, but they can create a simulator of course and join
21:01
it. So like I believe things would
21:04
have been faster and different. Yeah, I mean, you got all that
21:08
idle GPU time, you could just reallocate it to the agent
21:13
simulators. Oh yes, yeah, I know those those
21:16
like we were able to roll it out like publicly now so anyone can
21:19
use it in better strong like who are losing it.
21:22
So yes, so that was good. Dude, that is so incredible.
21:26
I, I got to say I'm I was very invested in the story.
21:30
I thought you were going to give me another turn and twist like,
21:34
no, even that didn't work. And then we found out there was
21:36
another. I would have given up.
21:38
I'm pretty sure at that point I was like, no, this shit is not
21:42
going to work. Everything is broken.
21:44
But no, it worked out. I was happy, yes.
21:47
Oh wow, that so then what's the what are some other war stories?
21:54
Dude, you, you got me. I'm fully hooked on what you've
21:59
been going through here just because I feel like anyone who
22:05
has played around with GP US and had to do anything with them,
22:09
whether it's training or model or just trying to keep them
22:11
running, has war stories and they have scar tissue from it.
22:17
And so there's like I know folks who have PTSD just when they see
22:23
the ohm errors. Oh yes, yes, now.
22:28
Again, how I? Swear.
22:31
No, no, that's, that's very fair.
22:33
Like I, I have some experience or a story which are ongoing to
22:37
serving, like training and serving stories are very
22:40
different. Training is more offline, you
22:42
don't have to worry about a lot of things and you have more
22:45
meeting room in some sense. In serving you are in a war
22:49
between the two main variables. For anyone, 1 is a latency.
22:54
How quickly do you want the results of a model GPU
22:56
inference? And the second phase here is
22:59
should, but how much is the efficiency of utilization?
23:04
Consider a case. If you're very very latency
23:06
sensitive, you want to run the forward Prof on the GPU as
23:11
quickly as possible. But what if you don't have
23:14
enough data to compute the correct like the full batch?
23:17
What would you do? Either you run a padded batch
23:20
like with fake data and justice run it to get the results.
23:23
But that's not helping. Sure you get the results, you
23:26
are good at latency, but you're wasting the GPU's a lot.
23:29
But sure, maybe you can wait long enough so that you have
23:34
enough data to create a batch on the serving side such that you
23:37
can run a forward prop. But then when you say wait
23:40
enough, you're adding latency. So I think it would be a
23:43
function of the use cases like for example, in ads, we are very
23:47
relatively sensitive. Like I think I mean a
23:50
millisecond, some of it cry like it's like we we really need to
23:53
be as fast as possible. In that case we are wasting
23:57
resources. Sure, because our goal was that.
24:01
I mean, there are ways to improve it, but that's always
24:03
like a trade off game. Like how much can you improve
24:06
and what can we do? Blah blah blah this but are
24:09
there. Yeah, on ads, on those use
24:11
cases, are there not use cases where you don't even need GPU's
24:17
or is it you always are going to need those GPU's just because of
24:20
the scale that you're at? Yes.
24:24
So basically on CPU your cost would actually be more than GPU
24:29
if you actually have this model on CPU.
24:31
So consider a case on CPU's you let's say you have 8 threads and
24:37
on each thread you are learning a forward prop on each training
24:40
row sequentially. You're not, you know, you can't
24:43
earn bigger batches on CPU. So the cost, the way it adds up
24:46
is the number of instances you have for that QPS.
24:50
But with GPU, just because it has a capacity, you can have
24:53
bigger batches and less number of GPUs.
24:56
So you can like there's no limitation, like the models were
24:59
not that complex that you can't use CPU for it, but the cost
25:03
adds up. Like they are more costly than
25:05
GPU. Yeah, that's why we were fine
25:08
with the wasted batches still being on GPU.
25:11
Then go on CPU. That's the part.
25:14
Wow, well, that's a that's an interesting trade off right
25:17
there. But sorry, I I distracted you
25:19
get. Back to no, no, no, no, That's
25:21
good actually. And there's also another aspect
25:24
which is killing us now or has killed us before as well, which
25:29
is the IO cost. Again, I told you like data iOS
25:33
everywhere, now you need to get the features.
25:37
If you have a bigger batch, let's say you have a batch of
25:39
10,000, you need 10,000 cost whatever 200 features per
25:43
training low what how many features you have like for the
25:46
model, how will you get that much data?
25:49
If a client is responsible for that, you've been going to pay a
25:51
huge amount of serialization decision cost.
25:54
If client is not responsible or somewhere you need to fetch from
25:57
some Cassandra, some Redis or somewhere you're going to
26:00
bombard that just to create a batch.
26:03
This is a problem like you will have to pay additional cost just
26:07
for data to get the data for that bigger batch.
26:10
But if offline you have caching and everything all ensure you
26:13
can have cache. But it's not like you can scale
26:16
for everything you cannot like online caching will be more
26:19
costly. Wait, so tell me more about
26:23
this. So again let's consider a case
26:26
where you have a model which each training is of a dimension
26:30
200 features. You have 200 features.
26:33
OK, now you are hosting the model on some GPU and you want
26:37
to have a back size of 1000. So you would have a matrix of
26:42
1000 into 200. Who will pass that data?
26:46
Either a client can do it, then it's a very big request over the
26:50
wire which you're going to send to GPU.
26:52
You have to pay that cost. Or what you can do is you send a
26:56
minimal request to the GPU endpoint and let endpoint get
27:00
the other data from some different services and gather it
27:04
there. But again, that is going to be
27:06
costly. You have someone has to pay that
27:08
cost, either a client pays it or GPU endpoint pays it.
27:12
But this will again block you from using the GPU because
27:17
unless you have the full matrix, you cannot run a forward prop.
27:21
It's a similar problem, just in a different domain.
27:24
Like yeah. Different flavor of Yeah, it's.
27:27
A different flavor. Yeah.
27:30
Like, and it's, it's a slightly more difficult problem because
27:34
these are, these features can be real time features, these can
27:37
be, you can maybe we can't even cash them.
27:40
Yeah, like these things can happen.
27:41
Like based on the problem you're solving offline, sure, it's all
27:46
offline. On a fixed set of data, you can
27:48
do more stuff. But online you always want the
27:51
freshest and the recent data. Maybe it's user embedding.
27:54
So you want the latest embedding which is generated.
27:57
So it the cost shall add up. It's when that data is in
28:00
flight. Huh.
28:02
Somebody's got to pay. Yes.
28:05
And so people. Are doing whatever they're good
28:07
at, but we need to give them something.
28:10
Yeah, yeah. Oh man, what is this idea of the
28:14
reproducibility trade off you're talking about?
28:17
And Oh yeah, now that's a good point.
28:21
So basically one other thing which we saw when we working on
28:25
this pedestal thing was we improved the parallelism, they
28:30
improved the utilization by a lot, but we started seeing
28:35
degradation in the models and the degradation in the model was
28:38
not because of assay caching, but it was because we were
28:43
improving the parallelism, increase my parallelism so much.
28:47
There was no determinism in the order of the data which is
28:51
basically considered a case when you have a batch.
28:54
The the skewness in the labels affecting the model quality and
28:59
because you are not guaranteeing any order of data in the
29:02
batches, you are hurting the model quality.
29:04
Like 1 model run on the same data can give you some AUC in
29:09
same run and again on the same data can have a better AUC or
29:12
worse AUC. And you are like OK which one is
29:15
true? You don't know.
29:16
And sometimes the differences are so huge you're like did you
29:18
actually improve the model magically without doing
29:20
anything? It's not possible.
29:22
So we were focusing on more on the optimization efficiency.
29:26
So the Michelangelo team is the one which is like a Uber white
29:29
ML infra team. So they started focusing on
29:31
reproducibility. So what they did was they
29:34
rewired the queues in the PETA storm.
29:38
Before you had a master queue to get all the work done.
29:43
The workers were going crazy. Hey, let's get read the data.
29:46
Let's just put in the queue. There's no order.
29:49
The way they twisted without affecting the efficiency was
29:52
having per worker queues and divide the work in the manner
29:56
that it's always deterministic. You're not starving any worker,
30:00
but you are ensuring this worker gets the task the same way every
30:05
time on the same seat. And workers when their queue is
30:09
getting ready then they fill the master queue.
30:12
And are you creating some kind of like an audit trail so that
30:15
the workers know? Yes, we basically pre allocate
30:19
everything like or pre decide everything before even we start
30:22
training such that we know, OK, this is going to happen for the
30:26
Sun and it's deterministic. So this way at least we even if
30:29
you're going crazy with the parallelism, we are not hurting
30:31
the model quality. And when when you're talking
30:35
parallelism, how much parallelism are you talking?
30:40
And is there a place where that starts to degrade?
30:44
Like, did you find, hey, after a certain point you start to see
30:50
losses? I would say after a certain
30:54
point it was not losses, but no gains.
30:57
I think what internally kernel does is that even if you specify
31:01
let's say 200 threads, it cannot allocate it, it will just ignore
31:05
it. Or maybe it will just do a round
31:06
Robin between the limited threads it has.
31:09
So whatever you specify, if you go outside the constraint or
31:12
whatever the CPU is telling you, the CPU codes are telling you
31:16
it's not going to increase it further.
31:17
It's like you're just telling the CPU core and it's ignoring
31:21
you. It will do whatever it can at
31:23
the Max. You can specify whatever you can
31:26
reduce it. Sure, it will entertain you, but
31:28
if you go over, it will like do whatever you want.
31:30
I'm going to do my best. So that's what we have seen.
31:33
Like there's a good upper bound, but after that doesn't matter.
31:38
Wow, OK, that's fascinating. So creating just a little bit
31:43
more determinism in the way that and orchestrating it almost like
31:47
preemptively and saying, look, here's how it's going to happen.
31:52
And this is the recipe. So that every single time we do
31:55
this, it uses. This recipe gives you that
31:58
reproducibility. Yeah, yeah.
32:00
And this way at least the models are getting the correct, I would
32:04
say quality every time every time and we can have better
32:07
iteration. If something should change, it
32:09
will change other with your children.
32:11
Yeah, yeah. Now you know what's.
32:13
It's so funny because a friend of mine was just putting in a
32:17
WhatsApp group about a, he bought, I think ADGX Spark, and
32:24
he wanted to run a model locally.
32:30
Can I read you this WhatsApp? Yeah.
32:33
Because he's running into something that I think you are
32:36
uniquely suited to figure out the answer to.
32:42
And I was saying to him like, oh, you might want to try Llama
32:45
CPP or you might want to see about VLM.
32:49
But The thing is, after listening to you, I realize it's
32:56
not the model that his problem is so.
32:59
I'm pretty sure, yeah. Yeah, let's see, let's see.
33:02
I'm going to read this in real time and then I'll, I'll give
33:04
you a minute to figure it out, he says.
33:07
I've just purchased the DGX Spark from my office.
33:12
We ran O Llama on it and it pulled Quinn 3.59 B and 27-B
33:20
Deep Seat coder and also GPTOSSI.
33:25
Don't recall how many prams but I think it's the 120B version.
33:31
Now it's exposed to the local network so anyone in the office
33:35
can use O Llama or open AIAPI endpoints with those models.
33:40
I created a docker with clawed code and I tried to run it.
33:44
I use M variables when running the docker to point to the model
33:49
that I want. I tried most of the models, it's
33:54
very very slow. I mean using in it took like 10
33:59
minutes and I had one shell script in my repo.
34:03
I run the O llama run command and tried the model.
34:06
The response was fast. So my only conclusion is either
34:10
the network is too slow, it's one GB or clawed code is really
34:16
slow. Anyone have advice?
34:20
I think that's, that's a really interesting one.
34:23
Like in my opinion, it would be network, I would say, because
34:29
like, even if you are like it, I think all these models also need
34:34
to fetch more tokens or more data right from somewhere.
34:37
I don't know. Like, for example, I if they're
34:39
using some MCP for searching some docs or some of the MCP
34:43
servers, how are they going to reach it to they have to go over
34:46
the network. If your network is not fast,
34:47
they're going to just get stuck at it.
34:50
It would be super slow. Like for example, if I'm running
34:53
a cloud code locally on the machine, but still it
34:55
communicates with other MCP servers.
34:58
So at the end, you need to get the data to the model so they
35:02
can make a something. It makes a decision.
35:05
And if you're slow with that, it could be this.
35:07
Although I'm surprised about the in it part.
35:10
I think I've mentioned like even the in IT is slow, which is a
35:13
little surprising. Like it's not.
35:16
It shouldn't call anyone in practice, in theory.
35:18
Like it's just a start prompt. Yeah, right.
35:24
Yeah. It says and it took like 10
35:26
minutes and I had one shell script in my repo.
35:29
So there is a concept of warm starting these models when you
35:34
run them as inference training as inference.
35:38
I'm not sure if that is something which we missed in
35:41
this case, like the warm start concept is I, I'm not sure if, I
35:46
mean, I'm also not very like expert in that.
35:48
But my understanding is whenever you want to run an inference on
35:51
these GPUs, you need to send some QPS beforehand such that
35:54
the memory is warm started. And if you don't do that, they
35:57
would be very, very, super, super slow.
35:59
So it could be more like, let's say a very good test here that's
36:03
just bombarded with queries for a few minutes and see if the
36:06
more it gets warm start like the GPU which is hosting the models,
36:10
because it will load a lot of things in the memory and the
36:13
next time when you do it really fast.
36:15
So even for the serving GPU's which we have, we maintain a
36:19
QPSA Det or test QPS on those GPU.
36:22
Yeah, such that they always warm like with the fake data and
36:27
whenever you have the actual production request, you're not
36:30
like starting from a Co start. Yeah.
36:32
So that could actually be more because in it is also failing.
36:35
So the maybe GPU is like it's not warm started.
36:39
So you fake it, but the warmth start is basically you faking it
36:44
with a bunch of synthetic data, just throwing it at it so that
36:47
it never fully shuts down. Yeah, yeah, that's correct.
36:52
And you, they are running the model, but you're not using the
36:55
predictions. It's just that you are loading
36:58
everything. It's more like, yeah, you're
36:59
warming the cache or memory in some sense.
37:02
Uh huh. And once again, the key here is
37:06
the answer is not use a smaller model.
37:10
No I don't think like it actually the smaller model will
37:13
waste the GPU of like the capacity.
37:16
I feel like you can use a bigger model like if the more if the
37:20
GPU's have capacity then you should use the bigger model.
37:24
Exactly because I, I was sitting there and I was like, wow, you
37:28
might want to try and use a smaller model maybe I don't know
37:32
and. They were.
37:33
I mean, that could be a very good test in this case, but I
37:36
don't think it will work. Let's I mean that can, we can
37:39
ask them like, let's try it out. And that would be really a good
37:42
definite answer. Again, it's like a detective
37:45
hard kind of thing. You put on a detective hard and
37:48
a Lego. Let's try out a different
37:49
variation series. Maybe everything boils down to
37:51
the same thing. Yeah, You're just testing
37:54
things, testing hypotheses. You're that scientist in the
37:58
lab. You've got your experiment, and
38:01
it's like, is this going to work?
38:03
Maybe. Yeah, let's try it.
38:05
Yes. And at the end when you find it,
38:07
it's like voila, it's done. It's nice.
38:11
It's a very good feeling. I would say yes.
38:13
Yeah, that's that dopamine release right there when you
38:16
find it. And actually when you find it,
38:20
you fix it and then it actually works because you thought you
38:23
found it many times, but it was that false summit trying to get
38:26
to the top of the mountain. Yeah, yeah.
38:28
No, no, I agree. Like the final catch, the gotcha
38:31
is the thing like and it is actually fixed.
38:34
Till then you are like in an illusion.
38:36
No, maybe it doesn't work. Well dude, what else are you
38:40
thinking about these days? Huh.
38:44
For me, like, OK, we are in the world where we are, everyone is
38:50
a building agent. So I'm trying to play around
38:53
with the skills these days a lot like to see like what all skills
38:57
like actually I was following the recent post you had about
39:01
the playground skill. I was thinking to download and
39:03
try it out all these skills. Like I feel at some point in
39:09
future you would have a skill for a junior software engineer
39:12
or maybe senior software engineer, which you can download
39:15
and it has everything. Then yeah, maybe we might reach
39:18
that world. But at like these days I'm
39:20
trying out these although I haven't tried open claw which I
39:24
was thinking about. But The thing is, I have a Quart
39:28
laptop. I can't try it on that I.
39:31
Just am not confident enough in my security chop.
39:35
To try it like, yeah, it's it's a good idea.
39:40
I although I don't even know what I want to ask that open
39:45
claw agent to do. I don't have things like that,
39:48
like hey book me a flight. But no, I want to book it myself
39:52
because I want to have options but I didn't know.
39:55
Yeah, yeah. That is good.
39:58
Going back to skills, it's a really interesting 1.
40:02
Leo just posted in our coding agent conference or coding agent
40:07
channel in Slack. A super cool still actually No,
40:11
sorry I misspoke. David posted about this skill,
40:17
that is, it was created by. A few psychologists and it's a
40:25
way for you to have the option after you have a coding session
40:33
that the agent will come and it will create a learning session
40:39
with you to update you on everything that it's done.
40:42
And then you have like this learning session where it's
40:45
like, hey, here's what I changed, here's how I changed
40:48
it. And it does it in a way that
40:49
helps you learn about the updates from a level of like the
40:56
code base, but also if it introduced new things that you
41:00
may not know about, like I don't know about React.
41:02
OK, cool. Well here's I implemented React
41:05
and here's how I did it. And here's why this matters.
41:07
So it's kind of bringing you along on the journey as you are
41:12
coding. Oh, well, that's so nice.
41:16
Like, so basically it's if I would have used that skill when
41:21
we were debugging this, then I can ask that agent to even debug
41:26
further or maybe have more efficiency or something like
41:28
that, or even like that. Yeah, it can be a debugging
41:31
skill as well, right? Like that could also be
41:33
translated to that because it already has the session.
41:37
Yeah. And I think the the key here is
41:41
that it's helping you learn everything that it did.
41:47
It's like giving you a minute to step back and instead of doing
41:55
work in the code base and having it like whatever change code or
42:01
submit APR, it says I'm going to create a learning journey for
42:06
you so that I can teach you everything that I just did.
42:11
So translating by coding into learning.
42:14
Yeah, in a way or, or at least like, and, and the whole thing
42:19
that I thought was cool about this is that it allows you to
42:23
stay up to date on everything that's changing so that you know
42:28
how very quickly code bases can get away from you.
42:34
And then later on you're trying to debug something and you're
42:38
like, oh, I don't even know what the hell is going on here.
42:41
Actually, I got to really drop in and understand what this code
42:45
is saying. And it's to combat that like so
42:52
that you're not so far out of the loop.
42:54
Now, whether or not you want to do this on every single coding
42:57
session or every single project, that's up to you.
43:03
No, I really loved it. Yeah, it's it's good actually,
43:06
because and especially when you're exploring a new code
43:09
base, I think that will be more beneficial if it is the journey.
43:12
OK, what did we do there? Like, I don't even know this
43:14
code, but I know what I want to do.
43:17
I think that's the part. So how did we reach there?
43:20
That that totally makes sense. Yeah.
43:22
Exactly. So I've seen, I've seen a few
43:25
skills like that where it's almost like reflect back and
43:30
help teach me. Those are cool as as like a
43:33
bucket of skills. The other skills that I have
43:39
been seeing is what I was mentioning to you about.
43:42
And it's not necessarily a skill.
43:44
It's more just like rules, I think is a strong word for it,
43:50
but ways to help set up workflows.
43:55
So there's very common things. And we had Rob on here as I was
43:59
mentioning, and Rob was talking about how with certain things
44:03
that he does a lot, he just set up one click button so that he
44:10
has a little command center and he doesn't have to explain, He
44:15
doesn't even have to tell the agent to invoke a skill.
44:19
He just clicks a button and then or a hot key and boom, the agent
44:24
knows it loads up that skill and knows exactly what to do.
44:30
Oh man, this world is changing so quickly.
44:32
I feel like every other week it's something interesting like,
44:35
yeah, this is actually smart because many a times you
44:38
repeatedly do the same thing every time.
44:40
Like for example, building, testing.
44:42
It's these are simple, but let's say some kind of workflow setup
44:45
or maybe look at XYZ things which you can actually make it a
44:48
part of workflow at a scale. Exactly.
44:51
Yeah. Nice.
44:53
Yeah. And the other one that I've been
44:56
using a lot and, and I guess this is very much on the user
45:00
experience of working with these coding agents is I 99% of the
45:10
time will use a dictation tool instead of actually typing.
45:16
I hardly type anymore. Oh, oh, really?
45:20
Yeah. Yeah, dude.
45:21
Especially when I know what I want to say.
45:25
It's a little harder when you don't have your thoughts clear
45:28
and you're trying to like dictate on the fly what you want
45:31
to say, like for an e-mail or something.
45:34
But when it's like, I know what I want to say and I just boom,
45:38
we'll use it. I have a hot key and it
45:41
automatically starts up my microphone and I'm.
45:44
Talking about it, I never tried it.
45:48
Nice. Man, that is a game changer
45:52
because normally you have in your head what you want, darn,
45:57
before you start to ask cloud code to do it for you.
46:03
So it's a lot easier to go back and forth with it by dictating.
46:08
And someone at the Coding Agents conference during our like Hot
46:12
take session said that one of their biggest hacks is they
46:18
always use voice mode to just brain dump at the beginning of a
46:24
session. And they'll brain dump
46:26
everything in voice mode because it is a way to just get all of
46:31
that context out of you. I see no actually that that
46:35
makes sense, because when you write you rewrite, you kind of
46:39
try to clarify yourself as you are, right.
46:41
And it's slow. Yes, and tiring.
46:45
If it's a big problem, it's. If it's a big problem like you
46:48
need to write the entire design you have in mind into a few
46:51
sentences. But still, it's tiring and you
46:54
may. Now, you don't need to if you're
46:57
just like basically word vomiting at the beginning of the
47:01
session and making sure that it's clear, hey, this is the,
47:05
this is everything for context. And you can say it all.
47:10
You can say it five different ways.
47:13
It doesn't really matter because no amount of talking that you're
47:18
going to do is going to really make a dent in the context
47:22
window unless you're given a sermon or something.
47:25
But I don't know how many people are doing that at the beginning
47:29
of their coding sessions. I don't think so.
47:31
Yeah, that would make some difference, but I don't know how
47:33
much. Maybe it ignores it, but let's
47:35
see. Yeah, that's that's fair.
47:37
Yeah, I'll try that. No, that's thanks.
47:39
Thanks for the tip. No, just this makes sense.
47:41
Yeah, these are these are fun ones and yeah, it doesn't sadly
47:46
doesn't make me any better at coding.
47:52
Like I still am totally encountering the major flaws,
47:59
but one thing we started doing, I, I may have told you this is
48:03
on Fridays, we're having these lunch and learn sessions with
48:06
like the coding agent lunch and learn sessions.
48:09
And those have been super helpful because just getting in
48:12
a virtual room with folks who are also bumping up against
48:19
walls or finding little tricks that work like this, just like,
48:23
oh, hey, at the beginning of the session, you should always brain
48:25
dump with a voice mode. It's like, oh, that's a really
48:30
easy way to do it, and I don't need to change much about my
48:33
workflow to. Make.
48:34
Yeah, nice. But I mean, I'm curious, you
48:38
said that it's not helping you like much in the coding.
48:42
You're still hitting walls. Like what do you mean by that?
48:45
Because I'll still get to a point where I'm like, ah, why is
48:50
this not working? And then I go try to like click
48:54
through the code base and I'm like I.
48:56
Don't actually know. What the hell is going on here?
48:59
And now I really got to debug. And sometimes when it goes
49:04
crazy, it's, it's horrible. Like I sometimes feel like, why
49:08
did you even do that? Yeah, like it's a.
49:11
It's a such a simple thing, like I have to undo everything to
49:14
start from scratch. Well, I think the difference
49:16
between you and me is that you have an opinion about how it
49:20
should be done. Yes.
49:22
And you know that like, why did you even do that?
49:25
And you're like, because you should have done this.
49:27
But when I'm asking why did you even do that, I'm I don't have
49:30
any idea what the other way and better way should have been.
49:35
So it's kind of like, why did you do that?
49:38
What's what were you trying to do there?
49:41
I see. You're fine.
49:42
Yeah, I didn't happen to me. Yeah.
49:43
So like, if I'm touching like I don't, I'm not familiar with Go
49:47
language. So you're changing Go.
49:49
I really don't have any clue. Just make the change, make it
49:52
work. That's that's the end goal I
49:54
have. I'll figure it out later.
49:56
Like first, let's make it work and have to sometimes get a
49:59
scolding to this plot guy. Like what, What are you doing?
50:02
This is not working, but I don't know how much.
50:05
Scolding. Yeah.
50:07
I've tried so many times to tell it to read the docs.
50:10
It doesn't listen. I think they are.
50:13
Sometimes I've seen like they kind of cut cut the context for
50:16
the docs. Like they don't read the entire
50:18
thing, they only read a few things then just go crazy.
50:23
Yeah, they think they know, they know best.
50:25
And and that's another, another trick that Rob was telling me
50:28
yesterday was on all of his files in a code base.
50:36
He makes sure that one of the things that the agents do is
50:39
they always comment like ATLDR, what's happening in this file?
50:44
Because. By doing that, the agents are
50:48
much more likely to be able to search and understand what is
50:55
where and what's working and what each file does.
50:59
Well, that's actually a good idea, actually.
51:01
Everywhere like this is somebody, something which can
51:03
actually like add the end of the day, they're going to summarize
51:06
the doc like they're not going to do.
51:08
But if you're already providing it, it's more like a proper
51:11
context of the doc. And that's a good idea,
51:14
actually. I should ask people to do it
51:15
here. Like we have been using it so
51:17
often now that it should help. Yeah, and he also said every
51:21
folder has a readme. Oh, so that it's the for the
51:26
same reason? Yeah, Oh yeah, makes sense.
51:30
Yeah, you want to give it that context, man?
51:33
Yeah, like sometimes I have to tell.
51:35
Like think five times before you play.
51:37
So I've seen in the prom it's thinking 123.
51:41
Like it's think rethinking everything.
51:43
Like don't give me a reply before you think five times
51:45
already. Like sometimes when I'm
51:47
irritated, like you have to think five times and like I'm
51:50
scolding a child or something. Yeah, that's exactly what I was
51:53
thinking. It's like you have like.
51:55
Count to 5 before you answer so it does that and maybe I've seen
52:00
him give him better answers. I don't know if it's a hack or
52:03
something, but it worked. Yeah, well, it's like that Ultra
52:07
think or the the planning mode. I've heard folks say that they
52:12
stopped using planning mode and. I'll just start from that, yeah.
52:18
Yeah, and and you want, but the basically what you're telling it
52:23
is reason harder. Make sure you're like fully
52:26
reasoning when before you come back to me with anything.
52:30
Yeah, don't waste my time. The resources are cheaper.
52:35
Yeah.