Latest / Agentic Conversations (formally mlops.community) / Introducing DBRX: The Future of Language Models // [Exclusive] Databricks Roundtable

Transcript

Introducing DBRX: The Future of Language Models // [Exclusive] Databricks Roundtable · Agentic Conversations (formally mlops.community)

0:07
up everyone welcome, welcome, what a day to be alive.
0:14
We are here with the Databricks team.
0:17
They've got some huge news that just came out as you probably
0:21
saw last week and after we saw it we said you know what, let's
0:26
have a round table session about this because there is too much
0:31
news to let just slide by. I would like to bring out the
0:37
Databricks team, but I'm going to do it in true MLB's fashion
0:41
and I am going to sing a little song about him to making us
0:47
start off on the right foot. Hopefully you all are ready for
0:51
this. You may not know what you're
0:53
getting yourselves into, but let's get it rocking.
0:56
If anyone has any key that they want me to sing this intro song
1:01
for the Databricks team in Speak now or Forever Hold your peace
1:04
and if you want to prompt me throw it in the chat.
1:06
I am listening. Of course C# is going to be the
1:10
go to 1. Here we go.
1:13
C# minor. Can you all hear that
1:56
this model? And to answer that question
2:01
we've got my good friend Danny Lee.
2:04
Ohh where you at Danny? Hey, Hello.
2:07
Hello. Hello.
2:09
Great C#. Thanks very much.
2:12
Demetrius, always glad to be hanging out with you like usual.
2:15
So. Is you don't know who I am.
2:17
I'm just saying. Hey you, you said the interest.
2:19
I'll do it real quick. Names Dan Italy, but apparently
2:23
a developer advocate at Databricks, Demetrius and I had
2:27
been guilty of doing many fun things.
2:28
Together and we're back at it again.
2:30
That's probably all you really need to know, so.
2:33
What do we call in this model? Dude, that you've got a big LM
2:36
out in the open source land and I think that I pronounced it
2:41
earlier and you said no, that's not what it's called.
2:45
That's right, it's OK Now of course the name is DB RX, but
2:48
I'm trying to make it a thing where we all say Dabba Rex.
2:52
OK, so that's the idea. It's Gonna Stick by Rex
2:56
Navarrete wanted to stick double wrecks.
3:00
Rightly for us, we've got some other people that have worked on
3:04
Dabber X and are able to. We're going to have over the
3:07
next hour or so, we're going to talk with some of the actual
3:13
talent behind it. Look at this.
3:16
Dabba Rex just popped on the screen.
3:18
Oh my God, there we go. First up I'm going to bring on
3:21
to the stage AJ where you at, AJ?
3:24
There he is. What's going on, dude?
3:27
Hello. Tell us.
3:29
What have you What have you been up to?
3:34
So I guess first quick introduction.
3:35
So yeah, I'm AJ, I'm the engineering manager for the
3:38
generative AI training platform team at Databricks.
3:41
And yeah, I mean, honestly for the past few months it's been a
3:43
lot of cranking away making this model train largely running all
3:48
over the back end infrastructure for training it, making sure it
3:51
works while making our making sure the model is actually
3:53
going. You've had your work cut out for
3:57
you, so I'm going to keep bringing the team onto the
4:00
stage. Bendish, I know you're in there
4:03
somewhere. There he is.
4:04
What's up dude? How's it going to be at your
4:06
house? Glad to see you again.
4:07
You too, man, So give us the rundown.
4:10
What have you been up to? Yeah, I mean I think what we
4:13
talked about 2-3 weeks ago, well caught up on a sleep a little
4:16
bit, a little more grey hair after this.
4:18
But yeah, we've yes, my name is Bandish I I'm another
4:22
engineering manager at Databricks and my team basically
4:24
builds all the training tooling like composer, LM Foundry,
4:28
streaming data sets. You know for those folks
4:31
following our open source rules that were used to train DB racks
4:35
or Dopa Rex, I guess we're calling it a little bit.
4:38
We're trying to figure out what to call it.
4:39
We're trying a few different things off size.
4:41
Yeah, so, well, I'm excited to talk about it.
4:43
We got a B. Where you at, dude?
4:46
Where you at? There he is.
4:48
Hey, great to be here. My name is Abby, I'm an
4:51
architect on the NLP team. I've been helping lead a lot of
4:54
the modelling choices and development around X and it's
4:58
been a crazy couple of months to everything from pre training,
5:02
data prep, all the post training as well.
5:04
It's been a huge team effort. Yeah, super excited to be here.
5:08
Excellent. Well, we've also got Davis
5:10
coming on in just the second. We got to kick it off, man.
5:13
Like Denny, can you lead us through the sequence of events
5:18
that has happened over the last month when it comes to Dobber X?
5:22
Well, actually, no. In all seriousness, I I that's a
5:25
great question, but I would rather have bought this.
5:27
Ajay, Ajay or ABI actually take care of the biggest.
5:30
They are integral for its creation.
5:34
I think when we look back at sort of you know, last year, you
5:37
know originally we were a lot of us came from the most high team
5:40
and our goal is to really help people build and train their
5:42
models and build the infrastructure around that.
5:44
And I think we got to this point, we showed that you can
5:46
actually training up to GBP three quality models for a
5:49
relatively low, low price. And as we looked ahead, you
5:52
know, especially after about the car by Databricks, we really
5:54
want to see like can we push the quality to the next level,
5:57
right, can we get to like 3.5 and beyond and what were the
6:00
different things that need to go into that.
6:02
So a lot of it came down to obviously needing to scale up to
6:05
bigger models but to do so in an efficient way.
6:08
So you know, we really like dug into this new mixture of experts
6:11
architecture which I'm sure we can talk a lot about, which
6:14
really helps us train higher quality models without
6:17
increasing the cost too much. And that involved a lot of both
6:20
the research and engineering figure out how to make these
6:23
models average bringing how to actually train them at high
6:26
performance on big clusters. And so we spent a lot of time
6:29
like digging into the research And then I say right about like
6:32
around Christmas time actually that's when we decided, hey, we
6:35
actually want to go for it. We want to try and build a new
6:37
model that's way more powerful than our previous MPT series.
6:41
And that's really kicked off this effort.
6:42
So the past three months or so have been just round the clock,
6:46
you know, research and engineering.
6:47
I'm trying to get double X out. So good.
6:50
So we have a new a new person that joined us.
6:53
Davis, can you give us a quick intro before we dive deeper into
6:57
this? I know it's like we're taking
6:59
one step back and two steps forward, but if we're going to
7:02
get there. OK, what what's the back in two
7:05
steps forward? Sounds exactly like training a
7:07
huge model, so it's it's pretty fitting so.
7:10
Yeah, like we're on point. Yeah.
7:12
So hi, I'm Davis. Good, good to be back here.
7:15
I am a researcher at Databricks as Mosaic AI team, formerly
7:23
Mosaic as well. So I'm always interested in
7:26
hearing and and Danny, feel free to jump in if you have any
7:30
thoughts on this too. But when it comes to training
7:33
one of these models and and thinking about it like you did a
7:37
lot of testing beforehand, right?
7:40
And Abby, you were just talking about how we you at some point
7:45
you were like, OK, we've tested enough.
7:47
Now we think we can go for it. You have to make that gigantic
7:51
investment into training the model.
7:53
You have to be pretty confident that what's going to come out on
7:58
the other side is going to be useful and going to make an
8:02
impact. What are some things that gave
8:04
you all the security to know that you can do this?
8:11
Like if you're going to invest this time and effort and
8:13
resources into it, it's going to be good.
8:16
Yeah. So you know, I think there
8:18
there's like both the the research and the engineering
8:21
risks. I can talk about the research
8:22
and it may banish Raja us talk about the interesting stuff too.
8:26
So you know the basic thing we try to do is really nail down
8:31
the recipe for modelling. So you know the architecture,
8:34
the choices of how much data to use versus the number of
8:38
parameters. And we do a lot of these tests
8:40
at smaller scale and we try to basically benchmark ourselves
8:43
against like other existing models out there and our
8:46
internal benchmarks, you know, our previous MPT models.
8:49
And we tried to see that we're hitting the same quality as
8:51
those other benchmarks would like much less compute our money
8:54
than before. And so we do a lot of those
8:57
experiments, you know, like a smaller scales and eventually we
9:00
start like kind of scaling it up and making sure that we actually
9:02
hitting the quality we expect. And that's what kinds of gives
9:05
us the confidence that like hey, now we're going to really crank
9:08
it up and go 10X farther than we ever have before.
9:10
It's just a lot of those little scaling experiments.
9:13
That's the research. But on the infra, yeah, maybe
9:17
someone else was talking. About, yeah been from the kind
9:21
of the training side, right. I think you know research was
9:24
really kind of pushing the frontier as far as far as kind
9:27
of when we work with our research team.
9:28
And I think that's like very it's a really great product
9:31
collaboration that we do right. We we innovate because customers
9:35
zero is always research for us. And so you know I think sometime
9:39
last year Abby comes up and says hey I got this this MO
9:42
architecture and we had just put out right MPT 30B and kind of
9:46
riding that wave and we were kind of busy with the
9:48
acquisition and it was like that I kind of frame is our Gen one
9:54
stack right. Like that was our dense model
9:55
training stack. We were really good at doing
9:58
this at hundreds of GPUs. We've been working with
10:00
customers and we've kind of proven it out over and over
10:02
again. And that's a major thing that we
10:04
do right. If you don't do this once, we
10:06
have to do it once with our research team.
10:09
That's like creating these new capabilities and like really
10:12
pushing the boundaries of what we can do and then we have to
10:14
repeat that for Databricks customers.
10:16
And so it's kind of that, right. And then scaling in a lot of
10:20
different ways. And so when we build these new
10:23
stacks, right, that's that's really the crux of what's
10:25
happening. So we have this new MOA
10:27
architecture. It, you know, everything breaks,
10:30
right. It's like ohh, we're ready to
10:31
start training this. And it's like not what what do
10:34
you mean? We're nowhere near ready, right.
10:35
We need like, you know, and so a lot of this, you know, when
10:38
we're doing it is almost like, you know, we're we're we're
10:41
already, you know, fly. We're building an aeroplane
10:44
literally while it's flying in the air and on fire, right.
10:46
And but we do that so that you don't have to.
10:49
Right. And and you don't do it later.
10:51
And so a lot of this kind of involved looking.
10:53
Now we have this MO thing like how do we split this workload
10:56
across hundreds of thousands of GP's, right.
10:58
We trained B racks on over 3000 different GPUs.
11:03
What are the things that are going to fail?
11:04
Right. When you go from like 512 like
11:07
block sizes to over 1000? Like things that didn't fail at
11:11
those smaller scales are definitely failing at larger
11:15
scales. You know, and you're kind of
11:17
onion peeling. You're doing a lot of these
11:19
things. And so like a lot of the, a lot
11:20
of the engineering effort starts going into, hey, how are we
11:22
gonna solve these problems, you know, and you start by patching
11:26
and, you know, and then you kind of just build up from there, you
11:29
know, start understanding, hey, like checkpoint loading is
11:31
breaking for some different reason than it was before.
11:35
How do we make this go fast and how do we eventually do it so
11:38
that we're up Time is very long, right.
11:40
And and you know, I think we did pretty well.
11:42
I mean being able to train this model in three months or so is
11:45
is pretty amazing. And you know, I think we're
11:47
going to, as we kind of commercialise the stack and
11:49
start training with customers, it's gonna, it's gonna continue
11:52
to improve, get even better. Alright this is perfect.
11:54
So actually I've got two sets of questions just based on what you
11:58
just said Amish. OK, so but I'm gonna direct it
12:00
to Davis. I wanted you to talk a little
12:02
bit MEX. I'm gonna do that first but just
12:05
as a lead in for the next one. Then AJ, I want to talk to you
12:08
about the hardware failures and the training because that part
12:11
super interesting that bondage was referring to.
12:13
But First things first you know Davis wanted to talk a little
12:17
bit about the memory architecture because you know we
12:19
we we we spunk we we we went around it talking about like
12:23
this idea of mixture of experts. But this is not necessarily the
12:26
most obvious thing that we're talking that typically when
12:29
you're looking at Lamma 2 and even the older 3D models, we're
12:32
talking about dense models and now we'll talk about sparse
12:34
models. Can you talk a little bit about
12:35
that, maybe within the context of the Mega Blocks paper for
12:37
sake argument? Yeah, absolutely.
12:40
So here's one way to think about mixture of experts.
12:45
A simple approach that you could do is you could just train
12:50
different models for different subsets of your data.
12:53
Like you have a model that's good at processing math, model
12:56
is good at processing Python code, whatever.
12:59
You could just have different specialised models and like you
13:04
can run that. That's the thing you can do.
13:06
But you'll hit a fair amount of redundancy.
13:10
Like they're all going to have to have big embedding tables
13:14
with different input tokens. They're all gonna have to figure
13:16
out how to do reference resolution for pronouns.
13:20
There's just a lot of the model that translates across domains,
13:25
and only some of it there is specific to a particular domain
13:29
O in a mixture of experts model. What you do is instead of having
13:32
completely separate models, you're going to have one model,
13:37
but all of the feed forward networks are pairs of linear
13:42
layers in a transformer are going to be duplicated, say 8 or
13:48
in our case 16 times. You have 16 different feed
13:52
forward networks in every transformer block and every
13:56
token is going to go to a subset of these feed forward networks.
14:02
In our case we send each token to four of them.
14:05
So what this means is that we have many different supposedly
14:10
specialised feed forward networks and every transformer
14:13
block and tokens go to the relevant ones.
14:17
There are a bunch of things that are nice about this.
14:21
One is that you have 16 times more parameters.
14:25
That's a lot more model capacity.
14:28
One thing that's not very nice is that you end up with 16 times
14:34
more parameters in each of your layers.
14:36
So that causes a number of technical challenges.
14:39
As far as which experts are given.
14:42
Token goes to you. You learn a so-called routing
14:46
functions that hopefully does something reasonable.
14:48
There are a lot of open questions about how to do this
14:50
really well in the research literature, but that's kind of
14:54
the high level idea. You're adding way more
14:57
parameters, but then at runtime you use them sparsely so that
15:02
each token only sees. So you had just mentioned the
15:07
fact that we're talking about you're using 4 experts out of
15:10
the 16 that are available, right?
15:12
So could you provide what is the benefit, right, Because you
15:15
didn't note the fact that with 16 experts you have that many
15:18
more parameters to work with. So what's the benefit of doing
15:20
for experts out of 16? And then do note the fact that
15:23
we're talking about, for example, like with Minstrel and
15:26
with Grock, it's 2 experts out of eight.
15:29
So I'm just curious from that. Context.
15:31
So why 4 experts out of 16 is really two questions.
15:36
One is why only for experts and the 2nd is why four and not just
15:39
one. So why choose only four?
15:42
Why choose a subset? Or the amount of work you have
15:47
to do computationally? The number of Flops for each
15:51
token you feed into your model is going to be proportional to
15:57
the number of experts that are used.
16:00
O. If I take a token and I feed it
16:02
through 16 different linear layers, that's of course 16
16:06
times as much work as only feeding it through one linear
16:09
layer. So in order to reduce the amount
16:13
of work we do per token, we only routed each token to a subset of
16:18
the experts. Now if that's the case, like why
16:23
not just only do one? Why not maximise the number of
16:27
ratio of parameters to the amount of work you do?
16:30
And yes, there there is that. There's a tradeoff and you kind
16:33
of have to tune it. The trade off is that your model
16:38
still does benefit from doing more work.
16:42
One way to think about this intuitively is that the amount
16:46
of expressiveness you get here is proportional to the like N
16:50
choose K or to better, the entries K of how many experts
16:54
you use. There's only one expert.
16:56
There are only 16 possible ways the token can get processed.
16:59
You have two out of 16 experts, you have 16 * 15 weights hoping
17:04
you can get processed and so on. So if you use 4, you end U with
17:08
this really expressive combinatorial space where each
17:14
token you're still only using a small fraction of the
17:16
parameters, but the number of functions you can express here
17:22
is really, really huge. So you end up with a big model
17:25
capacity win. That still gets you like a lot
17:28
of the benefit of having just using all of the experts or an
17:33
equivalent dense model the same Perimeter account.
17:36
OK. And I think that like the one
17:38
the context that I would add before I switch, I'll switch
17:41
over today is that there's also the benefit of speed, right?
17:46
Basically, by reducing the number of experts going overall,
17:50
you've actually basically get the best of both worlds where
17:53
you basically have the higher volume of quality due to the 16
17:56
experts, but you actually get the better inference speed
17:59
because you're using force. Did I get that correct?
18:03
Just to verify here. Yeah, that's that's a really
18:06
good summary, kind of getting the best of both worlds.
18:10
Cool. I I probably asked Albert that
18:12
question honestly, so that's probably why I don't know.
18:16
Alright. How are the truth?
18:17
OK, yeah. Yeah.
18:19
No, no, no. That's always the truth.
18:20
I'm not pretending I know anything.
18:21
I I literally asked overextend. That tells me.
18:23
And then I just get the site, the site and give me some
18:27
citations that I can act like, I know we're talking about.
18:29
But the one thing I definitely want to talk about, like, OK,
18:33
dude, like paralysing something like this is going to be
18:37
complicated. Like, you got to tell me a
18:39
little bit about the harbour challenges.
18:40
How do you run something like this at scale?
18:42
Because that's all. That's 3072 GB, Houston, you're
18:45
running. This can't be like a like a walk
18:47
in the park, right? Yeah, so this is a fun one.
18:51
So one. Piece is large model.
18:54
Training well, fun and fun and not fun, depending on how you
18:57
see it. But basically one piece of large
19:00
model training that people don't really talk about as much is how
19:03
much of A systems and operations problem it really is when you
19:08
throughout our time, you know, kind of running the Mosaic
19:10
training platform, we we had run across thousands of GPUs before,
19:13
but we had never done a single training on that many GPUs.
19:16
It had been a whole bunch of smaller separate ones.
19:17
And when you hit that scale, as Ben just had mentioned earlier,
19:21
all sorts of things statistically actually people
19:24
say you can you observe roughly one GPU failure every 1000 GPU
19:28
days. And on a 3000 GPU cluster that
19:30
means you're seeing three per day.
19:31
And I can tell you empirically, we actually did see three to
19:34
four per day throughout the entire training run.
19:38
So in order to deal with this, I mean one, you have to have
19:40
really good monitoring. So we were monitoring the GPUs.
19:42
We were monitoring things at the Kubernetes level, which is like
19:44
the foundation of how our platform was built.
19:46
But you'll also have to monitor the entire network because it
19:49
really critical part of training like paralysing this model
19:52
across so many GPUs is that you need to have really high
19:55
throughput communication between all of the GPUs.
19:57
So the network layer is super important, and what you end up
20:00
having to do is basically check every single pair of.
20:03
Machines are like what we call nodes in the cluster to make
20:06
sure that you're actually getting that really high
20:08
throughput. And many times like random
20:10
network links will go down and you'll have to go figure out
20:12
what they are. So there was this whole
20:14
operational thing of building all of these, this monitoring
20:16
layers, all this health checking and then on top of that no
20:20
matter how hard you try to actually catch all these things,
20:23
you will never catch everything. So the next piece of it is, OK.
20:26
Now in the cases where you don't catch the failures, do you have
20:29
the right operations in place to be able to deal with it in a
20:32
very timely manner? So we actually had a dedicated
20:34
on call rotation specifically for this training run where if
20:38
anything ever went wrong, we had a whole bunch of monitors and
20:40
alerts that would be a a 24/7 on call page that would just wake
20:43
people up in the middle of the night and go debug some hardware
20:45
or go back to sleep, wake up again, go do it.
20:48
And it kind of just went on for for weeks.
20:50
That's amazing in fact that you're literally reminding me
20:53
now this is, this is I'm basically ageing myself very
20:58
horribly. I just want to call that out.
21:00
I Did you guys ever deal with like, for example, like HP Super
21:03
Domes like the old school ones? Because that's basically exactly
21:08
the same thing. Like, the idea was that
21:10
incentive bad connections were still really, really new, and
21:13
they're really, really fragile at the.
21:15
Time. So exactly the same concept in
21:17
order just to get up, we were I was part of the SQL Server team
21:20
where we're actually running SQL Server on super dose and that's
21:24
exactly it. Yeah, like basically calls at
21:26
all hours just because then that work like wasn't working and
21:29
then all hell broke loose. And so like I'm just curious
21:33
from your perspective, did this reduce overtime as you were
21:38
training or like you know training Democrats or was it
21:41
pretty consistent anyways just because of the nature of the to
21:44
fit actions of this nature of the GP themselves?
21:47
So we had pretty close relationships with the cloud
21:49
providers and we were able to kind of report all of our
21:51
failures after we detected and mitigated them.
21:54
So we did actually see improvements in health over
21:56
time, but also over time, we got a lot better at monitoring and
21:59
checking for these things automatically because every time
22:01
we had something go wrong, we were able to develop some kind
22:04
of script or something to like diagnose it for future times.
22:06
And now we have this gigantic logbook with all of the things
22:09
that that went wrong. And we're actually actively
22:11
working on automating all of these things so that when we
22:13
have customers training models like this, they will run into
22:16
the same problems. That's nice of you thinking
22:19
about the customers. Yep.
22:21
Workers. Right.
22:23
Yeah, I I'm sure a lot of people really enjoyed that.
22:26
So there's a question coming through in the chat that I
22:29
wanted to ask, which is about have you ever experimented with
22:34
training strategies other than FSP?
22:37
Like using frameworks such as Megatron or Deed.
22:41
Yeah, I know I'll be. I'll be here.
22:43
We wish everyone you want to take in, then I'll maybe add in
22:45
afterwards. Sure, yeah yeah.
22:48
So we primarily use like a Pytorch based training stack and
22:51
we have this library on top called Composer which is open
22:54
source. So anyone please.
22:56
And what we've tried to do with framework is make it as flexible
23:00
as possible for different models.
23:01
And we use composer to train not only our alms like BMX, we also
23:06
use it to train diffusion models.
23:08
We have other future models coming out and one of the
23:11
reasons that's possible in our opinion is is because we use
23:15
Pytorch FDP, which really doesn't enforce like any
23:19
requirements from the model architecture.
23:21
It makes it easy to distribute it across big clusters without
23:26
using 3D parallelism or pipeline parallelism or anything like
23:29
that. And I think that that
23:31
flexibility is what we require for most of all.
23:33
And we found over time that we've been able to optimise FSDP
23:35
to actually achieve very, very close performance to a thing
23:38
like even these 3D parallel strategies.
23:40
And so that's kind of like the tradeoff we've made.
23:43
I think like Deep speed in comparison is basically
23:47
implementing the same algorithm. So whether you use Deep Speed 03
23:50
versus Pytorch FSP, I think you'll see a very similar
23:53
situation. We just tend to use the portal
23:55
all of our natively integrated with things like Pytorch,
23:58
autocast and stuff like that. Yeah, I'll hand over to Banish
24:02
as well, who is also a lot of these different framework.
24:05
We love the simplicity, right. Like FSDP is like very
24:09
straightforward to think about. We're able to keep optimising it
24:12
and keep pushing the bounds. And you know we work very
24:14
closely with Pytorch on this and that's been that's actually been
24:17
a really great partnership. But I think the other side is
24:20
you know I think what actually AJ was talking about like having
24:22
super high speed interconnects really helps and and that's you
24:26
know again not everyone's going to have that.
24:27
So we're not saying like things like oh like pipeline
24:29
parallelism or different parallelism strategies aren't
24:32
necessarily needed. But we, you know when we
24:35
actually build these clusters, when we train on these things
24:38
like having those high performance networks really goes
24:40
a long way. And I think you know as we keep
24:44
pushing the bounds we'll we'll evaluate kind of how do we keep
24:46
pushing the utilisation counts that yeah and you know we're
24:50
really maximising the amount of performance we can get per
24:52
dollar. But so far, yeah, we've we've
24:54
basically gotten away with FTP and then kind of variations on
24:57
that and just really, really, you know, being very intelligent
25:01
about how we optimise things. All right, I'm going to lob
25:04
another one out at you all because the chat has been
25:07
awesome. Keep the questions coming in the
25:09
chat. You all are incredible.
25:12
So what tools do you use to benchmark your LLM inference
25:18
regarding latency, GPU utilisation, et cetera, et
25:22
cetera? So I know the inference people
25:25
have some very specific tooling that we didn't use to pre train
25:29
it necessarily, but it's mostly off the shelf stuff.
25:33
We get a lot of mileage out of the Pi Torch Profiler as well as
25:41
the Pi Torch Memory Profiler. I think those two were about 80%
25:45
of it. Occasionally we would also use
25:48
just like a little bit of our own tooling to say log all the
25:53
distributed calls, but it was mostly off the shelf stuff and I
26:00
think for inference a lot of the challenges there.
26:04
Or maybe less around which tool are you using, because you can
26:07
often indirect to very high performance kernels in TTL or
26:14
other libraries where you can kind of get away with not
26:16
looking at like the hardware perf counters and the L2 cache
26:19
bandwidth and so on a pretty good fraction of the time.
26:24
But it's also hard to get a good representation of your
26:27
workloads, especially when your workloads are very bursty.
26:31
Because, OK, what your inference latency is going to depend on
26:36
your back size. Like, how much work are you
26:37
doing right now? How many other queries are you
26:39
having to answer, and how do you get a realistic setting so you
26:41
can measure that in a situation that's going to be relevant for
26:44
what your customers are seeing? It's not quite an answer, but I
26:48
know we had to build a fair amount of scaffolding around
26:50
that to really get accurate answers to these sorts of
26:53
questions. Yeah, so just to quickly add on,
26:56
add on to what Davis is saying, there's there's like the
26:58
inference latency piece and for inference specifically, there's
27:00
also throughput which matters a lot.
27:02
So one thing we did for serving DBX is actually we built our own
27:06
kind of optimised inference web server that has a whole bunch of
27:09
customizations in it to make the model super performant when it's
27:12
being served. And the way we test this is
27:14
actually the the very simple deploy the model using the
27:17
server and then absolutely bombarded with requests and see
27:19
what throughput we're able to get.
27:21
There's nothing like basically trying to simulate live actual
27:24
workloads and then from there measuring, measuring the
27:26
performance and then optimising from there.
27:28
Yeah, I think the only thing I would add, I mean you know I
27:30
think the inference that is definitely a full kind of end to
27:34
end in that sense like systems problem, right.
27:36
Like just talking about the web server that we have to kind of
27:39
optimise all the way from sort of the REST API call to the
27:43
actual model call itself, right. So I actually I think we were in
27:47
a great blog post about this and it kind of highlights our
27:51
benchmarking or inference benchmarking.
27:52
So you know we look at things like time per output token or
27:55
time to 1st token time for the generate calls to actually
27:59
happen and you know what is compute bound, the other is
28:01
memory bound and also you look at kind of memory bandwidth
28:05
utilisation. So you kind of highlight that
28:07
framework definitely would point of focus to that log.
28:11
And then in terms of actual tooling for the model, right, I
28:13
mean a lot of it is like OK, how do we get those metrics.
28:16
And so a lot of it kind of looks at OK, hey, like what do we
28:20
expect, right, Like if we know that the how much data we're
28:23
moving around and we know that feeds and the speeds of the
28:25
system, what should we actually expect to see in terms of model
28:28
performance? And then we also use a lot of
28:31
the vendor tools, right. Like I know we've actually
28:33
looked at things like insight from NVIDIA to actually model
28:37
kind of like how long certain kernels are taking and and a lot
28:41
of that also happens in partnership with those vendors,
28:43
right. Like we worked very closely with
28:44
the TLM team to really optimise that and yeah, and that's kind
28:49
of how what led to those, you know, those those PR's to TRL
28:52
every bow and the the VLR bills for for both of those solutions.
28:56
Beyond the chinchilla model or beyond the chinchilla scaling
29:02
guidelines, how did you choose the number of tokens to pre
29:05
train on and maybe I'll call out, I'll be on this one.
29:10
Ohh, and just as a quick call out before you do it, most
29:14
people probably here don't even know what the council guidelines
29:16
are, right? They probably haven't read the
29:18
research paper around us. Again, I only know because I've
29:20
asked Alex to tell me so. So what do you want to provide
29:23
the context first. Just want to call that out
29:25
though. Yeah, absolutely.
29:27
So you know when it comes to sharing these models, a lot of
29:31
times you're trying to produce the highest quality forgiven.
29:34
Compute, friendship and compute is kind of basically a product
29:37
between how many parameters are many tokens you use.
29:40
And so if you look back at the history of LM scaling in the
29:44
early days, people are training very, very large models on a
29:47
small number of tokens. Few P3 was kind of like this I
29:51
think 175 billion parameters, only 300 billion tokens and they
29:55
tried to develop scaling laws to protect both the most optimal
29:58
way to assign those two components the the parameters of
30:02
the tokens. Now a few years later there was
30:05
a paper that basically trained a model called Chinchilla and they
30:10
re measured the scaling laws more precisely and they found
30:13
that actually you want to have a lot more tokens than parameters.
30:16
So they came to recommendation that is approximately about a 1
30:19
to 20 ratio. So if you want to train 100
30:22
billion parameter model, you should use actually 2 trillion
30:26
tokens. And so that actually
30:28
dramatically changed how people built models.
30:30
People started building smaller models, trained on much more
30:32
data. Beyond that, people found that
30:36
hey, actually we can extend it even farther, although it's not
30:39
exactly compute optimal, you can train on even even more tokens.
30:43
So you can increase that ratio from 20X to 100X, even 200 X,
30:48
and then you get models like the Llama models.
30:51
So for example, Llama 7B is only a 7 billion power model, but
30:54
trained for 2 trillion tokens. So that's a humongous ratio way
30:58
beyond Chinchilla and so on. So we look at all these
31:01
different strategies and we know for our customers basically
31:04
being on this very efficiently. So the smallest parameter size
31:08
possible is really important. So we basically want to ask
31:10
ourselves what is the highest token to parameter ratio we can
31:14
achieve without getting too much quality away.
31:17
So, so not quite Chinchilla, not quite like extremely large
31:21
amounts of tokens. And we had a lot of experience
31:23
with smaller scale basically at the same compute budgets where
31:26
we tried to shrink the model and increase the token count as much
31:29
as possible to basically see where that trade off was.
31:32
So I think when you look at Java Rexite, it has basically 132
31:38
billion LOL parameters and 12 trillion tokens.
31:41
So it's pretty close to 100 X and I think that's what we kind
31:45
of came to. So if you're interested in
31:48
these, kind of unless there are new papers out there for mixer
31:51
scaling loss, I think there's one even destroy a few weeks
31:53
ago, but that's how we came to it.
31:57
Let's talk a little bit about the data quality like in terms
32:01
of one that recurring themes from the from the blog post was
32:05
just that the higher quality data.
32:07
So we can break it down in terms of for example, the GBT 4
32:10
tokenizer, right from Token, we can talk about from the
32:13
standpoint that we have higher quality data which resulted in
32:15
US doing the Dents MOA versus MTLB, right.
32:20
So I'm just, I'd like to chime in a little bit about that just
32:24
to provide the context that we end up utilising the very data
32:28
tools that Databricks has. I'm not trying to upsell that by
32:31
the way. I'm just calling out that we're
32:32
from Databricks, so cool of using our own tools.
32:35
We end up dog fooding yourself basically to go in and actually
32:38
produce, get high quality data so that we could go ahead and
32:42
actually produce more. And that's more open for anybody
32:46
here right now at this point? I think you should observe it.
32:50
I think when we moved from was it like I I think our our data
32:54
processing literally got 100 X faster.
32:56
Like I think not even exaggerating, It became much
32:59
much much easier to deal with these huge text datasets.
33:02
I think it was pretty remarkable for our data team because
33:07
obviously when you're dealing with many terabytes of data,
33:12
that's not something you want to do on your laptop.
33:14
So getting everyone on board with Spark and having the
33:19
ability to write something you're kind of pandas, Likely
33:23
something, something fairly familiar, but that kind of
33:26
justice works at scale was really helpful.
33:30
We had very interesting time exploring the data, trying to
33:37
run all sorts of different transformations, keeping track
33:41
of everything. Because you try lots of
33:43
different experiments, you try lots of different preprocessing,
33:45
you have this whole data science pipelines to go from raw text to
33:49
this subset of really clean data that you actually want to use.
33:53
So yeah, it ended up working really well getting acquired by
33:59
Databricks. Like honestly, I think that when
34:01
we first found out is a little bit like OHH, data breaks.
34:04
OK, right. Not like the first company you
34:07
think of as deep learning, but in retrospect seeing how much of
34:11
the work is the data pipeline and data management and how much
34:16
alpha there is in doing that really well, it's like ohh, this
34:19
makes total sense. Like I I really see the
34:22
integration here. So it's been a very positive
34:26
experience for everyone who had to crunch large data sets here.
34:29
Bendish, I know you got something.
34:31
For us Ohh yeah sure. So I think when one of the
34:34
things that was that was actually I mean like we're kind
34:38
of talking about why this was like a match rated habit, right.
34:40
Like we, you know and I think we talked about this a few weeks
34:44
ago too like our data processing speed up so much after after the
34:48
data we're acquisition, right, Because it it's just we have you
34:52
know these experts and and this is this is this amazing system
34:55
that in this platform that we built to do this.
34:58
And then once you have that data, you have to pump it into
35:02
3000 plus GPUs and actually train these jobs, right.
35:05
And so there's actually a problem that we had started
35:06
working on at Mosaic. But you know especially when you
35:09
have these cloud built platforms, the underlying
35:12
storage is object storage, right.
35:14
And so when you want to distribute this data right where
35:18
you, I mean at at scale right, we're running one training job
35:21
but we're carving up the work and kind of giving it to these
35:25
different GPUs. And in order to do that, you
35:27
also have to take your data and split it up and send it to these
35:30
different GPUs. So doing that 1000 GPU scale is,
35:33
you know, turns out it's really difficult.
35:35
And also then streaming it over the Internet or streaming it
35:38
over a kind of high bandwidth network also has comes with its
35:42
own set of problems. So for you know, anyone who's
35:45
been on the other side of a DDoS attack, you know, you kind of
35:47
hit these issues especially when you're paying these service
35:49
providers. And so you know, splitting up
35:52
that data and then really streaming it or chunking it up,
35:56
downloading it, making sure that you're downloading stuff that
35:59
you're going to need next in the background of being able to
36:02
shuffle it at scale. And then kind of, you know
36:06
basically you're reorganise it to all these different GPUs.
36:09
You know that's kind of what our streaming data sets library lets
36:12
us do. And and you know it's it's it's
36:14
really just even gotten better and in a more capable right to
36:17
to kind of train the scale of this model with with B Rex.
36:23
I think that the streaming team sounds good.
36:26
Yeah. Ohh yeah.
36:27
I was just gonna say, like out of all the different pieces of
36:29
infrastructure, we had, all the different tools.
36:31
I think streaming is the one piece that didn't break it all
36:34
as we scaled. Yeah, So this is pretty
36:36
unremarkable. It was pretty amazing.
36:38
That's like that's like almost for one person team and a lot of
36:41
sense like that's like you know it's like it is impressive with
36:44
those guys like that that kind to us but like yeah no, I mean
36:48
it's it's a it's an incredible piece of engineering that that
36:51
we kind of built there. Yeah, actually I think you want
36:56
to try it too, so go for. Go another another fun story
36:59
about kind of scaling up our data.
37:00
So our streaming data loader held up and was very scalable.
37:03
Object stores on the other hand not not as much.
37:06
So fun story. We actually have to switch
37:07
object stores at one point and then also implement some hacks
37:10
to be able to download the data and not get throttled or have
37:13
other issues with objects or in the cloud providers.
37:16
Just because we were it. Not necessarily with data, but
37:18
also with checkpoint download. When we're when we're launching
37:20
a whole bunch of training runs. When you start to really scale
37:23
up, you start to make the object or about happy, so.
37:27
Yeah, and. Essentially, you're trying to
37:28
list the, right? So yeah, like the listing alone,
37:31
basically you're doing a DDoS attack directly on the object
37:33
store. Yeah, that's always pretty much,
37:36
yeah. Yeah, you have thousands.
37:37
Of these workers blasting away. So sorry, go ahead.
37:41
You know, I I think it was cool about that is like some of them
37:44
were were things in our libraries where we had done at
37:47
first like what seemed like a natural thing to do.
37:48
Hey, you know every GP should go download the metadata file,
37:51
right? That can't be so bad, right?
37:53
And that's totally fine when you only have under GPU.
37:55
That's why we have 3000 to the object store.
37:58
It looks like you're spamming them and you get like shut down.
38:00
So then we had to like put in all these fixes which which
38:02
surely makes sense, you know, like just one guy downloaded and
38:04
then share the out of date or something like that.
38:07
But now we've actually fixed these, our customers won't see
38:09
these problems at all, which is, which is kind of cool.
38:12
Cool, cool. So before Dimitrios takes over,
38:14
I just wanted to do a quick call up because Davis did bring that
38:17
up. He said you don't want to run
38:18
this on your laptop. Now I did want to clarify that's
38:20
running on your laptop for trainee, which we're completely
38:23
agree with it if you to go get the the quantized version.
38:27
I think Abby correctly is the guy from Apple who actually
38:30
quantized the Uber X down. If you've got what I think
38:34
you're telling me, 96 gigabyte MacBook Pro, you should be able
38:38
to go ahead and run and run imports directly on the laptop
38:40
itself. Yes, yeah, yeah.
38:43
So, so there are at least a couple of high end laptops which
38:45
coaching on it. I think, I think the Llama CVP
38:48
folks are also working on it. So hopefully that will open up.
38:50
Today, yeah, yeah. Yeah.
38:51
Yeah. Yeah, no, I've been playing with
38:53
that myself and my poor will back studio is not.
38:58
Right. You should be able to run I
38:59
think once Llamas CPR that you should be able to run on your
39:04
gaming rig at home right after it's quantized and.
39:06
You gotta like, yeah, I know. But see, I only have 32 units of
39:09
this one. So it's just that excuse for me
39:11
to buy more memory now. Yeah, yeah, yeah, that's all.
39:15
So, yeah, to help me out here. Come on.
39:16
Come on. Exactly.
39:17
Quality. That's, Yeah, that's.
39:18
That's personal. I don't know that's that's just
39:21
32 gigs. I didn't come on.
39:24
I I feel over the horrible right now.
39:27
No back on track the me too. Let's go for but.
39:31
We digress, but there is this data quality issue and there was
39:35
a question coming through in the chat that I think is great,
39:38
which was about thinking about future models is moving towards
39:41
synthetic data modalities on the road map.
39:45
Yeah, I think that's definitely a a researcher.
39:48
We're very interested particularly because as you try
39:52
to to even even larger models, we're starting to run out of
39:55
just data on the Internet. I think one of the core beliefs
39:59
we had with training custom models is that there's actually
40:03
more useful data held privately by enterprises, right.
40:07
And like potentially that's like a like a way for people to tap
40:10
into it. But since I think we found even
40:13
with Drex and we're just trying to explore what this that
40:17
basically being able to take a small data set that you have and
40:21
expand it. And then like fine, that is
40:23
helping a lot. We're trying to figure out ways
40:25
to do that with our customers as well.
40:26
So, so generally, generally like very interested in synthetic
40:30
data and hopefully in the coming months and stuff a lot more
40:33
saved. Ohh, sorry, just had a quick add
40:35
on to that. So I'm on the topic of a lot of
40:38
our research eventually funnelling into the product and
40:40
things we offer to customers. One of the biggest.
40:42
So we actually have a a fine tuning API and coming soon
40:46
ability to fine tune DRX as well.
40:48
And one thing that we really learned through a lot of our
40:50
customer conversations is that customers struggle to have very
40:54
large labelled datasets for supervised fine tuning and
40:57
synthetic data is one really great way to kind of address
41:00
that. And that's something that we're
41:01
actually looking at first first obviously doing all the research
41:03
on and deeply understanding ourselves, but then eventually
41:05
actually building into the offering as well.
41:07
So very much on the road map. Perfect.
41:10
No, so I just realised basically both from what you're talking
41:13
about in terms of the latest research, I don't one thing we
41:16
may or may not have discussed, it was a curricular learning and
41:18
I was wondering if one of you can chime in a little bit about
41:20
that because that actually had to do a lot with the improvement
41:23
for training as well. I just realised because we
41:26
talked about together. We talk about data, but I forgot
41:28
about the curriculum. Anybody wanna chime in on that?
41:33
Yeah I I can try in a little bit.
41:35
So what we found is that that and we did found this through
41:39
small scale experiments that it helps a lot to kind of change
41:42
the distribution of your data towards the end of pre training.
41:46
So we did these experiments when the last 10% or 20% of training
41:49
we would focus the the brain data on less just web scale tax
41:53
and more on like the high quality data or code data that
41:55
we actually wanted the multi really good at And that turned
41:58
out to help performance a lot on the sort of benchmarks that we
42:01
care about. So for DBX broadly we wanted a
42:05
general purpose LLM that was also particularly really good at
42:09
coding and reasoning. So when we when we basically
42:12
tailored the pre training data towards that and again this is
42:15
really easy with our tools because we use these kind of
42:18
streams and you can change the ratio of the streams at runtime.
42:21
That helped a lot in the final performance.
42:23
And I think actually when we talked about with some of our
42:27
reporters as well, we tried out different data mixes.
42:31
So this is actually a pretty exciting moment during the
42:34
Imdbpro X and we found that the Curriculum Learning One worked
42:37
the best. Which is by the way, I don't
42:40
know. I mean that is super exciting to
42:42
me, right? Like like you know, because you
42:44
you hear all these stories about people who like are spent a lot
42:47
of money trying to like take these base models, you know,
42:49
fine tune them or or or change things.
42:52
And I think like if we can get really good at continuing to
42:56
teach these models more and more effectively, you know, and and
42:59
kind of get away from hey, I I added a bunch of my data but the
43:02
model actually got worse and a bunch of other stuff.
43:05
You know, if we can kind of crack that that that's kind of
43:09
amazing and that's like super exciting I think about being
43:11
able to kind of like change the data mix or training like kind
43:16
of look at where it's at. If you have good evals in place
43:18
and that kind of tweak it as you're going along, right,
43:20
you're almost kind of like figuring out what you can
43:22
continue to teach the models. So, so really looking forward to
43:24
kind of like what we kind of keep developing on that front.
43:28
O, there's an awesome question that came through the chat again
43:32
from Max, and I think this one is directed towards you, AJ.
43:35
For a large scale training or generally switch to Slurm
43:40
instead of OK eight, does Kate effect itself affect your
43:44
training, the pipeline? And what factors could you make
43:47
you switch from K to slide? Ohh that that's an interesting
43:50
one. So the reason why we chose
43:53
Kubernetes is actually because we were building this training
43:55
platform to be multi cloud and also to be something customer
43:59
facing and provides A significantly easier to build
44:01
multi cloud. You can run on AWS, Azure,
44:04
Oracle, GCP, wherever much more easily than you can with Storm
44:09
and that was actually the big reason.
44:10
Like at the end of the day when we built our product, our
44:12
research team is is treated just like a customer and to an extent
44:15
and our research team plus all of our customers running multi
44:18
cloud Kubernetes is just significantly easier.
44:20
Now in terms of scalability, yes there's a point there.
44:23
I think the reality is once you really scale up the number of
44:25
machines and the amount of hardware it's managing it, you
44:28
have some challenges in scaling. But those are things that we
44:31
actually built on workarounds for within our own platform to
44:33
deal with. So at the end of the day,
44:35
Kubernetes is simply you can think of as the foundational
44:38
layer that manages the hardware, but we've built a whole stack of
44:41
systems on top of it to really enable this large scale
44:44
training. The whole chat wants to know
44:46
this question which is smaller models on the road map.
44:52
I mean why, you know, we just gotta go bigger.
44:56
No, I mean, yeah, you know, I I think, you know, I think again
44:59
like I think what we're doing is making, we're trying to make
45:02
this available to a lot of different market segments,
45:04
right. And so small bottles, big
45:06
bottles, you know, everything in between I think are things are
45:09
fair game. You could train a smaller model,
45:12
right, with data bricks if that's really what you need.
45:16
But you know, I think longer term we want to support as many
45:19
use cases as we can for our customers and and so that's kind
45:22
of how we'll tackle it. Excellent.
45:24
All right. This is the last one I think
45:26
that we can hit before at the end it's gonna be a rapid fire
45:29
round so that we can all make our next meeting on time.
45:32
And speaking on the topic of fine tuning, if we go with Q,
45:37
Laura or some sort of Laura for do you what's it, what do you
45:41
call it again DBX, DBX? Daba, Rex fan, etcetera.
45:47
You got to work with me about. Going to state one of these
45:50
times it will stick in my head anyway.
45:52
Lightning round Q Laura or some sort of Laura for Dabber X?
45:56
Which framework would you suggest?
45:58
How many GPUs would make sense? I think for Kelora I we are
46:06
working on modifying the Transformers integration to make
46:10
it easier to use. I don't know exactly which
46:13
frameworks are best for right now we are working on Laura Fine
46:17
Tuning Support, just an 11 foundry which is our public
46:20
training repo. So check that out and post an
46:23
issue there depending on what you're looking for.
46:26
But yeah, you can also take a look at just the the
46:28
Transformers repo we were working on or basically right
46:31
now. Excellent.
46:33
So I think that is all for today.
46:38
The last thing Jonathan got back to us in the chat for you AJ
46:41
about the flavour of Kubernetes and was asking was it Open Shift
46:44
Rancher? What?
46:46
What did you use to manage the Kubernetes cluster, if anything?
46:51
So very, very early on we used to use Rancher for a lot of our
46:54
stuff, but we actually replaced large components of Rancher with
46:57
our own with our own stack. Overtime for some of the more
47:01
established cloud providers like 8 WS GCP, we'll just use their
47:05
their managed one and build on top of that an OCI will kind of
47:08
roll one of our own like open source distributions Kubernetes.
47:12
Guys, I feel like we could have continued this for another hour.
47:15
The chat is going wild. We didn't get to half of the
47:18
questions and I appreciate everyone that came and was able
47:23
to talk with me. I also appreciate everyone that
47:26
is in the chat being super active and we're going to be
47:29
doing more of these. So stick around and we will
47:33
probably be emailing you when we have the next one.
47:36
I really appreciate this. If anyone wants to dive in
47:38
deeper, Davis and Bendish and I had a podcast about three weeks
47:45
ago that came out and it talked a lot about the the different
47:49
pains that you all felt when you were going through this.
47:53
So we dove deep into it. I want to thank you guys so
47:56
much. Even though I can only see your
47:59
frozen faces, I'm visualising you, smiling and being very
48:05
happy right now. All right.
48:07
I'll vote for you buddy. We're all happy right now.
48:09
So we're getting and thank you very much to everybody who's a,
48:13
who's watching us. I really, we really appreciate
48:17
you guys diving in and like always to me, Joe, thank you
48:20
very much man. This is always glad to join.
48:23
This was awesome. So much so much fun.
48:25
Thank you for hosting. This was great.
48:27
Thank you all. We will see you all later.
48:29
Have a good one.