Latest / Agentic Conversations (formally mlops.community) / LLM Distillation and Compression // Guanhua Wang // #278

Transcript

LLM Distillation and Compression // Guanhua Wang // #278 · Agentic Conversations (formally mlops.community)

• 0:00
  Yeah, so, so I'm Guanghua from Microsoft DC team.
• 0:03
  I'm a senior researcher here and I drink Apollos mainly I using
• 0:08
  the blue bottle coffee beans. I'd really like it.
• 0:11
  And I drink it maybe like 8:00 or 9:00 in the morning and
• 0:15
  another cup at noon. Optimal throughput is what we're
• 0:20
  talking today with Alex from the Deep Speed team and it was good.
• 0:26
  I was holding on to my seat the whole time.
• 0:28
  As always, I'm Dimitrios, your host for the ML OPS community
• 0:32
  podcast. Listen to this guy's credentials
• 0:35
  and tell me he is not killing it in the LLMAI world.
• 0:41
  The dude created Phi 3. The dude worked on Zero Plus
• 0:49
  Plus, which makes model training time very, very short or shorter
• 0:55
  per SE. But really where he hit the home
• 0:59
  run is where we spent the second-half of this hour talking
• 1:04
  and that is on Domino and what Domino does to bring down the
• 1:11
  training time. It is incredible to hear how he
• 1:16
  did this and what the Deep Speed team is focused on right now.
• 1:21
  I will give a quick plug that if you are looking to get a job,
• 1:25
  the Deep Speed team is hiring aggressively.
• 1:27
  They're looking for senior developers and researchers that
• 1:30
  want to come and do stuff with Domino.
• 1:33
  So let's get into this podcast. And as always, if you liked it,
• 1:39
  share it with just one friend. I'm going to start with this
• 1:48
  question man, this is always on my mind.
• 1:53
  How do you pronounce it? Is it fee?
• 1:54
  Is it Phi? What do you call it?
• 1:57
  Oh, you internally we call it Phi 3, even though it's like I,
• 2:02
  but we call it Phi, not fee. I don't know why it's some kind
• 2:05
  of term. Confuse me so bad.
• 2:08
  It's like what is this is? What's the pronunciation on
• 2:12
  this? But that kind of gives adds a
• 2:13
  little Mystic to it. And can you break down what it
• 2:18
  is? What is fee?
• 2:20
  Oh, fee is like because previously they have something
• 2:24
  called Phi 2 that is a previous version of the small language
• 2:29
  model phase three. And what they mean by fee is
• 2:33
  like they want to mimic the physical environment such that
• 2:39
  the LRM or the small language model can can like reflect some
• 2:44
  physical environment. And take.
• 2:47
  Some action that is their final goal previously, but we are at
• 2:52
  the very beginning and and later on the the team after we finish
• 2:58
  the phase, phase three, the team just do some reorg and we may
• 3:02
  not continue to five four or five five.
• 3:05
  No, No. Oh, so there's not going to be
• 3:08
  any of this? No.
• 3:10
  Because like the the project leader is just a drop to open
• 3:14
  air. He's not in Microsoft anymore.
• 3:17
  Oh, interesting. And the whole thing, I know that
• 3:20
  there was a lot of hype around it when it first came out,
• 3:23
  especially like 5-2, right, because it was so small and it
• 3:29
  had a lot of potential. Yeah.
• 3:32
  And that's the reason right now you can see the Llama 3.2 or
• 3:35
  something, they just so do this kind of small like remodel, not
• 3:39
  the big one, not like a several terms of billion or hundreds of
• 3:42
  billion parameter. And what were you doing to get
• 3:47
  such solid results or performance with such a small
• 3:52
  model? What like when you were training
• 3:54
  it, what were things that you made sure to do differently?
• 3:58
  Yeah, so that's a really interesting question.
• 4:01
  We did a lot of like a fancy trick to to make the small
• 4:05
  language model work. The most important thing I think
• 4:09
  is because I was leading the training team of the five three.
• 4:13
  The most important thing is we need to have very high quality
• 4:16
  data, which means our data has a minimum noise and we need to do
• 4:23
  some kind of data preprocessing, data cleaning all before we do
• 4:27
  the training and that will take a lot of time and a lot of
• 4:31
  money. We also need to purchase some
• 4:33
  high quality data from like the top notch magazine like New York
• 4:38
  Times, Forbes, those kind of. We need to purchase those text
• 4:45
  from them because they are really high quality and usually
• 4:49
  they don't have any wrong information inside of their
• 4:52
  data. That's the most important thing.
• 4:56
  And was it all human created data and all?
• 5:01
  You couldn't use any synthetic data, could you?
• 5:04
  We we have synthetic data, but that portion is not that big.
• 5:09
  Such a data is relative small because those society are just
• 5:14
  trying to mimic some part of the like the original data and
• 5:19
  usually this is just. To me personally I think it's
• 5:22
  just a waste of training time because for the model they just
• 5:26
  see roughly the same pattern compared with the original.
• 5:30
  Like human right data. And so now you're saying that
• 5:37
  like 5-4 and five five are not going to happen because what the
• 5:43
  performance wasn't good enough to make it think that or make
• 5:46
  you think that it was worth continuing with or it was more
• 5:50
  like, let's just hand that off to a different team.
• 5:54
  I think it because like AI inside Microsoft, we have, we
• 5:58
  are very highly competitive. We have several teams doing
• 6:02
  similar thing. It's just some like a company
• 6:06
  wide strategy or policy. They don't want us to continue
• 6:10
  work on that. Right now we only have two lines
• 6:13
  of model we want to further training.
• 6:16
  One is from open airline, the other is from the Microsoft AI.
• 6:21
  Most of us who who lead that team.
• 6:24
  Yeah, yeah, from the Inflexion team, right.
• 6:27
  Yeah. And so this is fascinating to me
• 6:31
  because there is such a large amount of interest that goes
• 6:36
  into the small language models. And if small language models are
• 6:40
  going to be this solution for architectures to do things
• 6:45
  faster and cheaper. And that's what we've been
• 6:49
  promised in a way. Now, whether or not that's a
• 6:53
  reality today is debatable, I think.
• 6:56
  And you really have to do, a lot of you laugh at that, but you
• 7:00
  have to do a lot of work to make the small language model be as
• 7:05
  good as it can be, right? It's not like you can just grab
• 7:08
  it off the shelf and then plug it into your system and it's
• 7:13
  going to be working. Yeah, that's the reason right
• 7:15
  now, at least inside Microsoft for small language model, post
• 7:20
  training is even more important than pre training because we
• 7:23
  need to like learn customised data.
• 7:28
  And now that kind of rolls into the next topic that I wanted to
• 7:32
  talk about, which is deep speed and all your work that you've
• 7:36
  been doing on that, because it feels like deep speed helps out
• 7:39
  with pre and post training, right?
• 7:41
  Yeah, Yeah, yeah, yeah. Right now we also do
• 7:44
  quantization, yeah. So there you go, it's like the
• 7:48
  trifecta. And can you explain to people
• 7:53
  what deep speed is? I think we can call Deep speed
• 7:56
  like a third party library based on Pytorch.
• 8:00
  What we did is that we suppose some new features that Pytorch
• 8:05
  doesn't have previously. The most important one is called
• 8:08
  the 0 optimizer. Basically it's like a data
• 8:12
  parallel training a paradigm, but it is very memory efficient.
• 8:17
  Different from like traditional data parallel training that each
• 8:21
  GPU need to maintain the whole model parameter in 0 strategy,
• 8:26
  each GPU only maintain a portion of the whole model parameter.
• 8:31
  And when one specific layer need to do some computation, every
• 8:36
  GPU do orgasm or do a like a recollect of the weights from
• 8:39
  other GPUs to finish this layer's compute and then just
• 8:44
  erase or dump this layer's weight.
• 8:47
  Once we need to call the computer of this layer again, we
• 8:51
  do all gather again on the ways across GPU.
• 8:55
  So it's memory efficient. That's a huge performance gain
• 8:59
  compared with vanilla Pytorch because in Pytorch previously,
• 9:03
  it doesn't support such memory efficient data parallel training
• 9:06
  paradigm. And the right now they just
• 9:10
  doing something similar called FSDP which is similar as zero
• 9:15
  and but it's a Pytorch native library.
• 9:19
  This is one big piece. And the other thing we did is we
• 9:22
  do some kind of data offloading to further release memory
• 9:27
  pressure on the GPU set. So if a chunk of data doesn't
• 9:31
  need to compute at the moment, we move it from GPU memory back
• 9:36
  to CPU memory such that we have more GPU memory to use for other
• 9:40
  computation. Yeah.
• 9:42
  And after we need to compute some tensors on the CPU set, we
• 9:46
  just move it back to GPU memory and to the computer.
• 9:50
  Once we get the results, we move it back to CPU memory to save
• 9:54
  the memory. So I like that, how it's
• 9:57
  constantly going. Back and forth, back and forth.
• 10:00
  Yeah, yeah, yeah, yeah. Just to make that more memory
• 10:02
  efficient. So that's one aspect of it.
• 10:05
  And then the other aspect was like you said, you almost can it
• 10:10
  it, is it wrong to think of it as you know in databases you
• 10:14
  have sharding, is it similar to that where you're just sharding
• 10:19
  GPU's for their training? And so the 0 is like a database
• 10:23
  sharding, yes? Yeah, yeah.
• 10:26
  So you're, you're taking a GPU and you're saying you don't need
• 10:30
  to have everything, all this information of the model on this
• 10:34
  GPU. You just need to have the parts
• 10:37
  that you're training right now. And then when it gets a little
• 10:41
  bit, when it, when you realise there's certain things that you
• 10:44
  don't need even from that, then you offload those on the CPU,
• 10:48
  right. And yeah.
• 10:50
  Yes, very similar. Yes, if you go there like a
• 10:53
  system research group, we office do this kind of system tricks.
• 10:58
  Yeah. What are some crazy ideas that
• 11:00
  you haven't implemented but you thought about?
• 11:03
  Or you tried to implement and it just didn't make it past?
• 11:06
  Yeah, 1 crazy idea we actually implemented, but not many people
• 11:10
  use it because of the hardware issue.
• 11:12
  So we implement something called 0 Infinity.
• 11:17
  Basically instead of offloading data to CPU memory, we're
• 11:20
  offloading data to disc, because disc has like a much bigger
• 11:26
  volume compared with any memory, CPU or DP memory.
• 11:29
  And by able that feature we can train a large language model on
• 11:34
  a single GPU. That's a crazy thing we have
• 11:38
  done, but that depend on some NVIDIA disc hardware called a
• 11:42
  VME device. Yeah.
• 11:45
  Yeah, a lot of people, they don't have this device, so they
• 11:47
  cannot use this feature. So basically it's offloading you
• 11:52
  onto the Nvme and your able to because the NVME is much larger
• 12:01
  or it's it's much faster. What's?
• 12:02
  Much faster, much faster than traditional PCI based disc.
• 12:06
  Sneaky. Yeah, yeah.
• 12:08
  Yeah, I think so. We had, we had a talk on deep
• 12:13
  speed probably almost a year ago, you know, at, yeah, at the
• 12:18
  conference. And one of the things that was
• 12:21
  said was, yeah, we trained a large language model on a single
• 12:24
  GPU and I was just like, what? No, that's that's impossible.
• 12:28
  Yeah, another like a shortcoming of that is it takes Infinity
• 12:32
  long because we need to move data back and forth and disc
• 12:36
  usually is slow. That's why you called it.
• 12:39
  Infinity like training forever. So it it is possible to train on
• 12:47
  AG single GPU, it's just that it might not be in your lifetime.
• 12:51
  Right, right. Maybe takes tonnes of thousands
• 12:53
  years for big model training. Yeah.
• 12:56
  All right. So then there's the memory
• 12:59
  constraint. And so it's like I'm seeing that
• 13:03
  the work you're doing is is really cool because you've got
• 13:06
  this small language model, you're saying the pre training
• 13:10
  is great, but then we need the post training to be as
• 13:13
  important, if not more important.
• 13:16
  And so you're using deep speed on that post training?
• 13:19
  Yes, we use deep speed for both pre training and the post
• 13:22
  training. And actually we have another
• 13:24
  stage called the mid training, middle mid training.
• 13:28
  That's a weird term, but what it means is we try to support a
• 13:33
  different languages. We call it a mid training.
• 13:37
  It's just the in between of pre training and the post training.
• 13:41
  Post training is for customised data, mid training is just for
• 13:44
  more language support and the pre training is just to pre
• 13:48
  train the model using like the original data we purchased.
• 13:53
  And so you can, you can have that pre training, then you
• 13:56
  stuff in all of the languages to kind of amplify or make it a
• 14:01
  Poly goat and make it speak more languages.
• 14:03
  And then that happens in the mid training.
• 14:06
  Yeah, and then we can when we grab it off the shelf.
• 14:10
  Like if I download 5 from hugging face, is it with mid
• 14:14
  training already? You already know, you already.
• 14:17
  They only support like English or or at the best they the
• 14:22
  support Germany. I remember.
• 14:25
  Yeah, not many languages. Yeah, yeah, yeah, German.
• 14:30
  That's a random one. I mean, I live in German, that's
• 14:32
  great for me. But you would think like Spanish
• 14:36
  or Portuguese does it more. Yeah, we have some like a maid
• 14:40
  training to support that. Also support some like Indian
• 14:43
  language or Chinese. Yeah.
• 14:45
  And then what does the post training look like?
• 14:48
  And can you explain how you guys do it with deep speed and with
• 14:53
  the like, what techniques you're using?
• 14:56
  Yeah, post training mainly lacking evolve like two
• 14:59
  different aspect of the 1st is we have customised like tokens
• 15:03
  or customised text from user both like internal user and some
• 15:09
  external user. They just using it to further
• 15:14
  fun to the model. And in that case usually we
• 15:17
  don't use very long sequence to ensure that our model can
• 15:22
  trained on relatively small cluster, because the larger the
• 15:26
  sequence we need to use more GPU basically.
• 15:29
  Even with the 0 deep speed, Yeah, yeah.
• 15:33
  Yeah, yeah. Because the sequence itself will
• 15:35
  generate some big activation in the forward and generate some
• 15:40
  gradients in the backward. Those activation and the
• 15:43
  gradients size are proportional to the sequence length.
• 15:48
  OK. All right.
• 15:50
  And does that have anything to do with later the context window
• 15:54
  that you're able? To yeah.
• 15:56
  So usually for example we create a 4K sequence, we can do a
• 16:01
  context window of 1K or 2K, that should be fine.
• 16:06
  Another line of post training is we also add quantization on top
• 16:10
  of the base model and try to train on the customised input
• 16:15
  data at the same time we do some quantization like 4 bit or right
• 16:20
  now even 3 bit quantization. Yeah, yeah.
• 16:24
  Not one bit yet though, huh? I don't think so.
• 16:27
  Yeah. I think 3 bit is almost touch
• 16:29
  the ceiling. Yeah, yeah.
• 16:32
  Why is that? Because like the number of bit
• 16:36
  means, the less number of bits you use, the less data precision
• 16:41
  you will have. And the three or four bit of the
• 16:44
  data precision is already not that good.
• 16:48
  Yeah, yeah, even we do a lot of fun tuning on top of that.
• 16:52
  Usually we only expect the model accuracy or the model quality
• 16:57
  slightly below or roughly the same as a base model.
• 17:01
  That is our expectation. But when we try to use something
• 17:04
  like two base or even one base, the model accuracy decreased
• 17:09
  significantly, like only half or even less of the original base
• 17:13
  model accuracy. So again, this goes back to,
• 17:16
  yeah, you can, but do you want to?
• 17:20
  You can train a large language model on a single GPU, but do
• 17:25
  you want to? You can quantize it with one
• 17:28
  bit, but do you want to? Like?
• 17:30
  What's the point of? That I think no one use it.
• 17:35
  Yeah, yeah, exactly. No one's going to use it.
• 17:37
  And so then the but the quantization, I wasn't super
• 17:41
  clear on that it happens after the mid and after that the post
• 17:46
  training or it's part of the post training.
• 17:48
  It's part of the post training maybe.
• 17:50
  Yeah, sorry I may not be very clear.
• 17:52
  So for every waste tensor, we have a contest of waste tensor
• 17:57
  which is the same as the original 1 but with the last
• 18:00
  bits. And during the training we we
• 18:03
  when we pass a gradient to the listensor, we passes through the
• 18:07
  contest with not the original base widths.
• 18:12
  So that's the quantization that goes through.
• 18:15
  Is it just like out-of-the-box Manila quantization or are you
• 18:20
  also? We have someone is like open IS
• 18:24
  version TTQ and the other way also right now we also do some
• 18:28
  research for determining how to round up or round down the the
• 18:33
  quantization value. It is still ongoing.
• 18:36
  I think we may release sometime next year.
• 18:39
  Yeah, that one can achieve higher accuracy compared with
• 18:42
  open STPTQ. Yeah.
• 18:46
  And then afterwards, so you pop out with this very specialised
• 18:50
  model and it is hyper focused or domain specific, right?
• 18:55
  Do you distil it later after that to try and make it smaller?
• 18:59
  You you say it's small enough. This is the whole point of doing
• 19:02
  it. Another issue we found is
• 19:05
  previously for the large library model, this dilation always
• 19:10
  decrease accuracy. That's a main issue.
• 19:13
  So right now we just abandoned the distillation, just our user
• 19:17
  correlation with our model shape, Yeah.
• 19:21
  Even on the the bigger the higher parameter models.
• 19:25
  Yeah, like like several tenths of billion.
• 19:29
  The model also has decreased accuracy.
• 19:33
  Interesting. Because yeah, our black
• 19:35
  production environment, we really want to have high
• 19:38
  accuracy rather than saving some, some, some weights.
• 19:44
  And distillation is not the way to go then because we were
• 19:47
  thinking about it's, it's good that you tell me that now,
• 19:51
  because one thing that we want to do in the ML OPS community is
• 19:55
  create like a hack with me session.
• 19:58
  And we're going to take models and try and make them smaller
• 20:04
  and faster and ideally keep the accuracy.
• 20:11
  And so we had thought about distilling, but now that you
• 20:15
  tell me that, I'm not even going to go down that route.
• 20:17
  And I'm going to say it's all quantization.
• 20:19
  Yeah, I think this situation may work.
• 20:21
  If the downstream tasks is really simple, such as it's just
• 20:25
  a chat bot for a specific person, those kind of very
• 20:30
  simple task, I think this relation should work.
• 20:32
  But for more general task, like a chat bot for a million user,
• 20:38
  yeah, distillation may not work. That's not the way to go.
• 20:41
  OK. And that's it.
• 20:42
  And you've tried, I imagine a lot of different, yes,
• 20:46
  distillation techniques doesn't matter.
• 20:48
  There was a very big team in either who is doing distillation
• 20:52
  for almost 2 years. It goes to a bad end.
• 20:57
  Really. OK, Well, that's a huge signal.
• 21:00
  So basically, anybody who's thinking that distillation is
• 21:05
  going to save you, you got to be OK with letting the accuracy
• 21:10
  slip. Yeah, we saw that.
• 21:12
  For example, if you're using something like a copilot for
• 21:16
  your personal laptop, this relation may work, but if it can
• 21:21
  be more general and provide services for more people, it
• 21:25
  doesn't work really well. And basically after you do this
• 21:28
  relation, the memory inside the model is too small to memorise a
• 21:33
  lot of people's like attributes. OK, so that's why it works with
• 21:39
  a copilot if it's only one person is attributable for.
• 21:41
  Co pilot, we have the distillation quantization
• 21:43
  everything and make the model extremely small, yeah.
• 21:47
  Yeah, so that it can fit on that CPU also, right?
• 21:51
  Yes, we also try to use CPU to do the survey and not the Unity
• 21:54
  issue. Yeah, yeah.
• 21:56
  And what? What's that process?
• 21:59
  Oh, that process is basically we can say doesn't have post
• 22:06
  training. It's just do the post training
• 22:08
  on the fly. When you use a computer, you
• 22:11
  browse a website, you, you, you watch videos, those kind of
• 22:15
  things. We just try to like collect
• 22:17
  those signals and try to customise a personal assistant
• 22:21
  for you. And in that case we also didn't
• 22:24
  upload any data to our server or anything to keep the data
• 22:28
  privacy. Yeah.
• 22:30
  So you're doing Federated learning then?
• 22:32
  Roughly, we we don't even need to communicate with the server
• 22:36
  because like a personal pattern or personal characteristic is
• 22:41
  really easy to learn in your local laptop.
• 22:45
  We don't need to upload anything to our server.
• 22:48
  So the post training is done on the CPU.
• 22:51
  Yeah, on your laptop. Wow, OK.
• 22:54
  And I'm it's like that happens at night when you go to sleep
• 22:58
  and then the laptop kind of gets the updates.
• 23:02
  Happens on the fly when you use it.
• 23:04
  We do some fun. It's in real time.
• 23:06
  Yeah, real time, maybe one or two seconds, Yeah.
• 23:09
  Delay, yeah. How is that possible?
• 23:13
  That's nuts. If you go, we just try to
• 23:15
  collect the very simple signal like you click on what, what
• 23:18
  kind of website at which time frame you watch movies at the
• 23:22
  work time frame, you, you, you may play video games.
• 23:26
  Those are very simple signals for very simple data, which also
• 23:30
  put it and fit in the model to do some fun tuning for human
• 23:35
  activity activity every day. Maybe it's just around 100 or at
• 23:40
  most 1000 activity per day. It's just like a 100 or 1000
• 23:46
  simple data point. We need to find tune.
• 23:50
  OK. And that's why it can kind of
• 23:52
  happen on the fly. But is there a specific like
• 23:57
  package that is constantly retraining it or training it?
• 24:01
  Or is it just that you're throwing that into the context
• 24:04
  window as a? Database, yeah, we throw it into
• 24:06
  the context window once we back to like a 2K contact.
• 24:09
  So we we do a training round, yeah.
• 24:11
  I see. Wow, that's fascinating to think
• 24:14
  about. And the other piece that I
• 24:17
  wanted to know about from you is, are you using Laura's and if
• 24:24
  so, how? What do those look like and what
• 24:27
  is that whole process? Oh for Laura I think it is
• 24:32
  mainly used for post training because yeah Laura it already
• 24:37
  have like a smaller weights module rather than a big chunk
• 24:42
  of weights. It is used for post training for
• 24:44
  like Microsoft internal models as well as some open air models.
• 24:51
  We also we both use Lora and I think it's public and Lora
• 24:55
  itself has a big issue is if the customer data is very different
• 25:03
  from the pre trig data, then the Lora kind of strategy doesn't
• 25:09
  perform really well in terms of model accuracy.
• 25:12
  If the like the downstream, the post training data have similar
• 25:16
  attributes or characteristic compare with pre training, then
• 25:20
  Laura can perform really well. That's what they found.
• 25:24
  Oh so basically you have to have a Laura that fits into the world
• 25:30
  view of that pre trained model. If the Laura is giving new ideas
• 25:37
  to the model, then the model almost rejects it.
• 25:41
  Yeah, maybe. Yeah.
• 25:43
  That's a reason we know like a switch from Laura to some pure
• 25:48
  correlation. OK, the way that I understand it
• 25:52
  is there's like a Laura that's an orchestrator that will tell
• 25:56
  the model which Laura to choose or to like load up real quick
• 26:01
  and add. But you decided that it's
• 26:06
  actually better just to quantize instead of trying to do that.
• 26:09
  Or is it? Laura, because previous Laura is
• 26:12
  designed for a large language model.
• 26:14
  In that case we have a lot of like redundancy.
• 26:18
  I can see in the model with and by adopting Laura smaller width
• 26:22
  rather than the big width, we have a huge performance gap
• 26:26
  without losing accuracy. But right now our model is
• 26:30
  already smaller, much smaller than previous version.
• 26:34
  So Laura, if you're using a smaller tensor to represent the
• 26:38
  original, like a small language models tensor, it will have some
• 26:43
  trouble. Fascinating.
• 26:46
  OK. If you like for Laura, if you do
• 26:48
  it llama llama like A45B or llama 70 B, it works perfect.
• 26:54
  No matter whether the data of post training and pre training
• 26:57
  are different or not, it works perfect.
• 27:00
  But for small language model it's not the same case.
• 27:04
  There's almost like a cut off. Right. 3B.
• 27:09
  You're right, over several billion model it doesn't work
• 27:13
  that well. Yeah, yeah, it's, it's once you
• 27:17
  get over that 5B hump, then you can start saying, hey, yeah, we
• 27:23
  are seeing better effects. But then again, it's the law of
• 27:26
  diminishing returns. Once you go to 30 B, you don't
• 27:29
  really need Laura's, so it's really in that sweet spot of
• 27:35
  like 7 to whatever 17. Yeah, yeah, yeah.
• 27:41
  That's fascinating. I didn't realise that either.
• 27:44
  You. Can see recently there are a lot
• 27:46
  of work on pure quantization, not for Laura, maybe one year or
• 27:50
  two years before it's very popular, but right now for those
• 27:54
  small models, maybe not. Yeah, because so, and this is
• 27:59
  when we're talking about small models specifically,
• 28:02
  quantization is the clear path. Right.
• 28:07
  And we may have different quantization strategy right now
• 28:10
  people are working, right? Like I also heard from Avidia
• 28:13
  AMD those hardware companies, they also do quantization.
• 28:17
  For example, for the Avidia GB 200, they already support IP4
• 28:24
  data type 4 Four dream that. That's awesome.
• 28:28
  Yeah, how and what are some other quantization techniques
• 28:32
  that you have been messing with or have enjoyed?
• 28:37
  One thing we are trying to integrate into deep with the
• 28:42
  Transformer engine from NVIDIA, that is to do FP8 gym
• 28:46
  computation. And previously when we do gym,
• 28:50
  we were always without quantization.
• 28:52
  We always try to do IP16 or BF16 gym to preserve the precision or
• 28:59
  accuracy. But right now with the
• 29:00
  Transformer engine before the gym happen, they downcast the 16
• 29:05
  bits value into 8 bits and doing the gym in really fast amount of
• 29:11
  time and out that they upcast the IP eight data type to back
• 29:15
  to like 16 bit. What is?
• 29:19
  Can you explain the transformer engine for me?
• 29:22
  Yeah, So Transformer engine, for me personally it is just to do a
• 29:28
  data downcast from 16 bits to 8 bits and then do the computation
• 29:34
  like a matrix multiplier or something.
• 29:37
  And after that we get to the FP 8 output, it will upcast it back
• 29:43
  to like 16 bits. So in the middle they only
• 29:48
  compute like 8 bits with which makes it faster than previous 16
• 29:54
  bits computation. And there's no trade off there.
• 30:00
  At least for for right now for the model wave train, we didn't
• 30:04
  see much difference. We may lose some accuracy, but
• 30:09
  that is tolerable. Another is what I have
• 30:12
  implemented called a 0 plus plus.
• 30:15
  That is to do quantization on like gradients and activations
• 30:21
  and ways during training. Because at that moment we still
• 30:25
  train large language model and the communication is a big
• 30:29
  overhead. So the communication is big
• 30:31
  overhead at that moment. So what we can do is we do
• 30:35
  quantization before we do any data transfer.
• 30:39
  Basically like in 0 optimizer, during the first forward, we do
• 30:44
  some weight or gather on the weights.
• 30:47
  So in zero plus plus, instead of doing weights or gather, we
• 30:51
  first do a quantization on the weights to make like the 16 bits
• 30:57
  down to 8 or 4 bits and then we communicate those four bits.
• 31:01
  That can save a lot of time because we just communicate less
• 31:05
  volume of data. And in the backward, we do the
• 31:08
  same thing for the gradient. Before we transfer the gradient,
• 31:11
  we do a quantization and we transfer on the receiver side.
• 31:15
  We do dequantization to upcast for the data and to do the
• 31:19
  training. You remind me of the meme where
• 31:24
  it's like a exhibit, you know? And he's like, yo dog, I heard
• 31:28
  you like quantization, so I put some quantization on your
• 31:31
  quantization forward. Yeah, yeah.
• 31:34
  So we put a quantization before and after communication.
• 31:39
  Exactly. It's like, hey, could we
• 31:42
  quantize this part of the workflow?
• 31:44
  Yeah, let's try it. Can we quantize this data?
• 31:47
  Yeah, let's do it and see every single spot where you can come
• 31:51
  and you can quantize it. It seems like you've you've
• 31:53
  tried to and so far you've had some success with it.
• 31:58
  Then you're not seeing that it's causing.
• 32:01
  Significant issue, some trouble. For example, for weeks you
• 32:05
  already will only quantize to 8 bits, not 4 bits.
• 32:08
  For four bits with quantization we saw the performance
• 32:11
  degradation. And for gradients, because it
• 32:15
  has, it doesn't need a very high data precision, so we can
• 32:20
  contact it to four bits. Wow.
• 32:23
  Yeah, and now this is on pre and post training?
• 32:26
  Or is this mainly on post? It's mainly on pre training at
• 32:30
  that moment because we have some internal workload which have
• 32:34
  like a very low bandwidth interconnect.
• 32:38
  Well, Speaking of bandwidth and let's talk about Domino and what
• 32:44
  you've been doing there because that's like your newest
• 32:47
  brainchild, right? You've your creation and can you
• 32:52
  explain to us what it is first if?
• 32:53
  You want something. Domino is trying to eliminate
• 32:57
  the communication during training, either pre training or
• 33:00
  post training. All communication between GPUs.
• 33:05
  Yeah, or most communication. Right now we can hide around 70%
• 33:09
  up to 100% communication. And it works with deep speed.
• 33:14
  The idea? Right now it works with deep
• 33:15
  speed so. Let me see, Let me see if I
• 33:20
  understand the full picture. It's like I'm training a model,
• 33:23
  but now I don't have to worry about loading up the model on
• 33:29
  every single GPU because I've got deep speed and it is
• 33:33
  sharding, quote unquote, the model weights.
• 33:38
  And so it's training these different layers in their own
• 33:42
  way. And then with Domino, you're
• 33:45
  saying, and you know what, you don't have to be talking back
• 33:49
  and forth to these large clusters like these gigantic
• 33:53
  beefy machines that are training.
• 33:57
  They don't need to constantly be in contact with the other 24,000
• 34:01
  machines because you've got Domino and it just means that
• 34:06
  you have less communication and networking overhead.
• 34:10
  No, no, that's not the case. So what we did is we still have
• 34:13
  communication, but a user cannot see it.
• 34:16
  We had the communication behind the computation such that the
• 34:21
  like a from enter tool and a training perspective.
• 34:24
  You always see that if you're always working on computation,
• 34:27
  the communication is under the hood.
• 34:30
  We still have communication and we have different model sharding
• 34:34
  strategy than deep speed zero. We borrow from the Megatron's
• 34:40
  tensor part of strategy because in Megatron tensor part we see
• 34:44
  that the throughput is higher than 0 or any other strategy
• 34:49
  when you have high bandwidth interconnect between GPU's.
• 34:54
  And why does hiding the communication from me, the user,
• 34:59
  help with throughput? Yeah.
• 35:01
  So previously what we did for transformer training is we train
• 35:05
  a layer, like compute on one layer and then we need to sync
• 35:08
  or communicate for this layer's output.
• 35:11
  That's what we did before. And the the communication
• 35:15
  overhead just stand out as a as a training time.
• 35:19
  For example, you compute for five seconds, and after that you
• 35:22
  need to wait for three seconds for the communication to finish.
• 35:28
  That's what we did before and right now what I did for the
• 35:30
  domino or or what the domino achieve is, which has to hide
• 35:35
  this 3 second communication inside of the 55 second
• 35:41
  competition such that every iteration right now is just a 5
• 35:46
  second previous is like 8 second per iteration.
• 35:50
  Yeah, that's the difference. But so, so it's basically
• 35:54
  borrowing the same time it's communicating.
• 35:57
  It's just within those 5 seconds it's communicating.
• 36:00
  Yeah, the the. Key like attributes of domino is
• 36:03
  previously when we computer layer we need to wait for the
• 36:07
  whole layers output to be generated before we do the
• 36:12
  communication. But right now, after we like a
• 36:15
  computer a few output we started triggering the communication.
• 36:20
  Yeah, yeah. That's.
• 36:21
  OK, like doing some pipelining inside of the computation.
• 36:26
  So the whole message or the whole communication message
• 36:29
  doesn't need to be there. It's you're starting it as it's
• 36:33
  finishing, almost like just in time.
• 36:35
  Yeah, because we need to like do some 10s of sharding to break
• 36:38
  down the data dependency. For data without data dependency
• 36:42
  we can like a send it over without waiting for others.
• 36:47
  All right bro, hear me out. But what if we quantize those
• 36:51
  messages? Yeah.
• 36:54
  So Domino works with quantization.
• 36:57
  Of course, with quantization we can like make the training
• 37:01
  iterative time even smaller for both compute and the
• 37:04
  communication. So that's the reason in Domino,
• 37:08
  our first version, we didn't say we enable quantization, but
• 37:12
  right now we try to incorporate with Transformer engine and also
• 37:16
  my previous like communication quantization strategy.
• 37:20
  So there's inevitably been some wild stuff you've got done
• 37:25
  because of all of the lift it's like you're taking.
• 37:28
  You're not just taking a little bit from here and there, you're
• 37:33
  making some significant increases.
• 37:36
  Yeah, that's the reason like Domino is the next generation
• 37:40
  training platform for deep speed.
• 37:42
  It's a big project and right now we also try to collaborate with
• 37:47
  a lot of industry and active media like AMD, Intel and
• 37:51
  another called the PICA start out for the short video
• 37:54
  generation from Stanford and those are our initial customers
• 37:59
  right now. Oh, that's so cool.
• 38:02
  So the training that you've done, like is there, are there
• 38:07
  any statistics that you can share with us on how much faster
• 38:11
  it is? Oh, compare with like a maybe
• 38:14
  the best throughput like a because right now for if you
• 38:18
  have like a very high bandwidth interconnect between nodes, the
• 38:22
  best solution is to use Megatron.
• 38:24
  The media we can achieve like a one point 3X speed up compared
• 38:29
  with Megatron which is almost like optimal throughput that you
• 38:33
  can get. And this has nothing to do with
• 38:37
  the different. What is it the different wires
• 38:40
  that you're using connecting it like this is the is it Ethernet?
• 38:45
  Which what is it called? The Oh.
• 38:47
  InfiniBand. InfiniBand.
• 38:48
  That's it. Yeah, You don't even have to
• 38:51
  worry about that. You're not dealing at all with
• 38:55
  that type of throughput because of the way that you're
• 38:57
  structuring the software. Yes, yes, we we structure the
• 39:00
  software. We don't touch the hardware.
• 39:02
  Yeah, that's another big attribute for for Domino.
• 39:06
  We also didn't touch the kernel, we didn't modify the kernel.
• 39:09
  So any new kernel we can directly integrate with Domino?
• 39:14
  Oh, and is that how you're able to see this on any type of chip?
• 39:20
  Yeah, because right now we're helping AMD to speed up.
• 39:24
  We also try to run Domino on the MS300 directly.
• 39:28
  We have some speed up without modifying any line of code.
• 39:33
  So it plays in an abstraction above the basically all that
• 39:40
  hardware and all the chip. Basically, we'll just like a
• 39:44
  more smart scheduler for launching compute kernel and a
• 39:49
  communication kernel. Do it.
• 39:51
  You have more wise way more smart way.
• 39:55
  That is fascinating. So now what are some things that
• 39:59
  you've you've been able to see? Like obviously you take any, you
• 40:04
  take 8 seconds down to 5 seconds and that's huge.
• 40:08
  Are there any other numbers that you've seen that you're just
• 40:11
  blown away by? Oh yeah, for like a for because
• 40:15
  previously Megatron from NVIDIA, it doesn't support a multi node
• 40:19
  training. Everything is only happened
• 40:22
  within a single node with only link connection.
• 40:25
  And right now we also extended to multi node training with pure
• 40:29
  like 10s of parallelism. That's another big jump from the
• 40:33
  original 10s of parallel strategy.
• 40:36
  Oh wait I I'm not sure if I understood that.
• 40:38
  So it's the you you created multi node parallelism with this
• 40:43
  and before it was only the Megatron was only supporting
• 40:47
  single node. Yes, because for Megatron it
• 40:51
  doesn't hide the communication but it like a scale up to multi
• 40:55
  node, the communication overhead will be larger.
• 40:59
  For example, in the single node case, we have 5 seconds of
• 41:02
  computer, 3 seconds of communication, but for market
• 41:05
  node the communication could be 22nd or even more.
• 41:08
  Yeah. Yeah, so Megatron cannot hide it
• 41:11
  perfectly. OK.
• 41:15
  So the more nodes that you get, the more that you're, yeah, the
• 41:21
  more you see the benefit. Yes, yes, yes, roughly.
• 41:24
  And we are continue optimising for it.
• 41:27
  For example, if the communication is too big, let me
• 41:30
  do some quantization before we do communication to fix down the
• 41:33
  time. There's them.
• 41:34
  Yeah, yeah. Yeah, oh man, I knew it was
• 41:38
  coming in somewhere. Wait, where are you?
• 41:40
  Quantizing what? Yes.
• 41:44
  So you're quantizing. The with.
• 41:47
  Oh, the weights. OK, Yeah.
• 41:49
  With activations, gradients, whatever takes a long time, we
• 41:53
  quantize it and see how it works.
• 41:57
  And then those the communication comes off the back of it so that
• 42:01
  the communication is able to go just as fast as as the weights
• 42:04
  when they're quantized. Yeah.
• 42:07
  So our perfect ideal case is computation is the same time as
• 42:12
  a communication. So we can do a perfect
• 42:15
  overlapping which one is longer? It will be like the the longest
• 42:20
  for the training. Yeah, OK.
• 42:23
  So it's almost like, yeah, you, you just start the communication
• 42:29
  a millisecond later and it is the same length or it's the same
• 42:33
  amount of time because it couldn't work where it's the
• 42:36
  exact same amount of time, right?
• 42:39
  Because. Slightly longer, yeah.
• 42:41
  Yeah, slightly longer, right? Yeah, that makes sense.
• 42:44
  I really like this idea of sharing time instead of doing it
• 42:50
  so you're doing things synchronously instead of async.
• 42:54
  Yeah. The reason like Megatron doesn't
• 42:56
  do this kind of sharing time strategy is because they didn't,
• 43:00
  at least from my perspective, they may not know how to break
• 43:04
  down the data dependency. That's a key issue.
• 43:08
  We just find a way to do the tensor sharding and after we
• 43:11
  sharded the small pieces, those small pieces do not have any
• 43:15
  data dependency, so we can do the communication on the fly.
• 43:20
  Oh, so you're breaking up the data?
• 43:24
  Even smaller pieces. Yeah, I see.
• 43:29
  And how does that even work? So you're full on, you're
• 43:31
  sharding the tensors so that there's no data dependencies.
• 43:35
  Yes, yes. And because of that, you are
• 43:38
  able to have the communication be right in the.
• 43:43
  Middle of computation, Yeah, OK. And our sharding is very
• 43:49
  general. It can work for any transformer
• 43:52
  models. It's not just for specific
• 43:54
  model. We have a specific sharding I
• 43:56
  didn't know. And you think there's potential
• 44:01
  for gain there if you have different sharding strategies
• 44:06
  you can optimise your sharding? Or is that not the place that
• 44:09
  you're looking to get a little more lift?
• 44:13
  So before we launch this domino product, we already tried tonnes
• 44:17
  of sharding strategy and what we propose is what we think is near
• 44:22
  afternoon. Yeah.
• 44:25
  And so looking ahead, where are you seeing this?
• 44:29
  I I mean, you mentioned the dream is to get the time of the
• 44:36
  two times to be basically the same and the communication is
• 44:40
  just a little bit longer so that you can have it go.
• 44:44
  But what needs to happen to get there?
• 44:46
  Where are you thinking you need to shave off those milliseconds?
• 44:51
  Yes. So the reason we propose a
• 44:53
  Domino project is we see the trend that the internal
• 44:57
  connection, the bandwidth is catching up with the Intranode
• 45:01
  ME link bandwidth. For example, for the H100 DJXH
• 45:05
  100 boxes, the Intranode ME link bandwidth is around 900
• 45:11
  gigabytes per second and the internal, the cross node
• 45:15
  bandwidth of InfiniBand is 400. It's almost at the same level
• 45:20
  even it's just half the bandwidth of Intranode.
• 45:23
  And the next generation of the B100B200 GPU's, we saw that
• 45:28
  their network card for InfiniBand is almost reaching
• 45:32
  800 gigabytes per second. It's almost same as emitting
• 45:37
  900, it's just 100 difference. And because we see this trend,
• 45:41
  we saw that for TP usually it only works for single node but
• 45:46
  for machine node it doesn't work well.
• 45:48
  But in our case, we just try to hide the computation and
• 45:51
  communication together to make it work.
• 45:54
  And this trend it will be continue once we see that there
• 46:00
  is no bandwidth gap between intranode and internode.
• 46:05
  That's the reason why Domino will be more popular later on
• 46:08
  for the later GPUs. OK.
• 46:12
  So once basically the internode catches up, then you're going to
• 46:18
  see Domino Lightly. Shine yeah, lack of a
• 46:21
  competition communication are roughly the same time, yes.
• 46:24
  Or her, yeah. So it's almost not on you to
• 46:30
  create the optimization here. It's more on the hardware
• 46:33
  provider. Yeah, because in this year we
• 46:36
  saw the JB 200, we already see this trend is is become real.
• 46:41
  Yeah, I know. Even if maybe later on they shut
• 46:44
  down this kind of bandwidth matching, they shut down this
• 46:49
  kind of project, we still can do quantization on the
• 46:52
  communication to figure down the communication that both work.
• 46:55
  Yeah, that's right. I instantly thought like, oh,
• 46:58
  the quantization of the communication is where you could
• 47:01
  add a little. List, Yeah, yeah, yeah, yeah.
• 47:03
  And I have the experience. We just try to integrate the
• 47:05
  little plus plus quantitative kernels.
• 47:08
  Yeah, that's not very hard. OK.
• 47:11
  So with 0 plus plus you are already kind of testing that
• 47:15
  out. Yeah, yeah, yeah.
• 47:17
  And we already use 0 plus plus, both internal at Microsoft and
• 47:20
  at LinkedIn. We help them reduce over half of
• 47:23
  the training time, which is big. Wow, yeah, wow, man, you're
• 47:28
  saving so much money. Yeah, yeah, yeah.
• 47:31
  This reason my my work is maybe more important than some other
• 47:35
  people. Yeah, it's incredible to think
• 47:39
  about how much just that shaving 8 seconds down to 5 seconds, if
• 47:44
  you're training a large language model, it's huge.
• 47:47
  Yes, yes, it's huge and ran off for like our big customer like
• 47:53
  Open Eye and Microsoft, Microsoft AI, we always use a
• 47:58
  high performance computing like a little DTX boxes with high
• 48:02
  bandwidth interconnect. So that's what Domino focus on.
• 48:07
  But later on if we want to extend it to a low bandwidth
• 48:09
  interconnect, we also can do quantization.
• 48:12
  So both way works. And it's not, so it's not even
• 48:19
  worth it for you to try and think about how to specialise
• 48:22
  for different hardware. We don't need to.
• 48:25
  Yeah, that's like a beauty of this project.
• 48:29
  We always do it at the tensor level or at the algorithm level.
• 48:34
  We don't touch it down to the hardware because if we target
• 48:38
  down to how your our scope will be limited.
• 48:41
  Yeah, and you're at the whims of other people.
• 48:43
  Yeah, because like previously, like a one or two months ago
• 48:46
  when we tried to adopt the Domino AMD GPU's, I was worried
• 48:51
  I would think maybe it doesn't work or it performed really bad,
• 48:54
  but we just directly run our code, we have performance scans.
• 48:59
  That's the beauty of this project.
• 49:03
  And it's fully open sourced, right?
• 49:04
  Yes. Fully open source.
• 49:06
  Along with deep speed so PRS are open.
• 49:09
  Oh, it's already merged to the master branch.
• 49:12
  Wow, yeah, All right. So if anybody wants to go and
• 49:17
  poke around in the code, it's there.
• 49:19
  It's on GitHub and. You will also have video will
• 49:22
  maintain it. If people have some issues, they
• 49:24
  can raise some issues. We can solve it.
• 49:27
  Perfect dude. It's been so fun talking to you
• 49:31
  because I've just learned a tonne about all this.