Latest / Agentic Conversations (formally mlops.community) / Tricks to Fine Tuning // Prithviraj Ammanabrolu // #318

Transcript

Tricks to Fine Tuning // Prithviraj Ammanabrolu // #318 · Agentic Conversations (formally mlops.community)

• 0:02
  I'm Prisby Raj or I also go by Raj.
• 0:07
  I am an assistant professor at University of California, San
• 0:11
  Diego and also a research scientist at Databricks.
• 0:16
  I was previously with Mosaic before we got acquired.
• 0:20
  So I actually have I have two jobs.
• 0:22
  I do 2 full time jobs effectively and I usually take
• 0:27
  my coffee black but with sugar so usually like no cream but
• 0:34
  we'll have some amount of sugar in it.
• 0:37
  Welcome back to the MLF Community Podcast.
• 0:39
  I'm your host Demetrios, and today talking with Raj all about
• 0:43
  fine tuning, but not just any kind of fine tuning.
• 0:46
  We're talking about Tau fine tuning, the method that he came
• 0:51
  up with and wrote an incredible blog post about.
• 0:54
  Why don't we just start this conversation?
• 1:03
  Do you call it the Dao, do you call it Dao or do you call it
• 1:07
  Tao? Yeah, we're, we're just going to
• 1:09
  call it Tao. It's it's gone through multiple
• 1:12
  naming changes internally. Tao is actually the final
• 1:17
  version, which is what ended up sticking that we came up with
• 1:22
  like maybe a a day before the block post went live.
• 1:27
  So internally, it's actually key to something else in my, in my
• 1:30
  head because for the majority of the time we were, we were
• 1:33
  thinking about it as a different name, but we, we decided that
• 1:37
  this was a better name overall. Because I, whenever I see
• 1:41
  spelling like that, I always think of like the Dow, the
• 1:44
  chain, you know? Yeah, yeah.
• 1:48
  I I think they were just, yeah. We were just thinking like, like
• 1:50
  Tao, yeah. Tao is perfect, So what is it?
• 1:59
  So Tau is basically, at a high level, a way of people fine
• 2:07
  tuning their own models for their own domains, right?
• 2:13
  So there's always been this sort of like stress between, well, do
• 2:17
  you have one model to rule them all or do you have, you know, a
• 2:22
  sort of a separate model for every single individual domain?
• 2:28
  And given the a lot of people's data is private, we kind of seem
• 2:33
  to be gravitating towards the world in which these aren't
• 2:37
  actually necessarily like entirely orthogonal, but they're
• 2:42
  very much in a world where a lot of people do need customized
• 2:48
  models for their data and their use cases.
• 2:52
  And so that's what it is. It's a way of doing that.
• 2:56
  And most importantly, it's a way of doing that without people
• 3:00
  having labels, right? So like the the bane of every
• 3:06
  single, like, you know, machine learning person trying to build
• 3:09
  a custom model is like, Oh no, where am I going to get the
• 3:13
  labels for this data, right? Yeah, these are expensive.
• 3:17
  The annotation costs are going to be super expensive.
• 3:21
  And we've heard this like time and time again from like so many
• 3:25
  customers, right, Like we were. That's actually where some of
• 3:29
  the initial motivation came from, right?
• 3:32
  So we were thinking, OK, well, the initial version was like,
• 3:35
  OK, well, what would it take to have some kind of a system where
• 3:41
  people are deploying their models, collecting feedback in
• 3:46
  some form with these models and then using those to continuously
• 3:51
  like improve them through time, right.
• 3:54
  But not everyone has access to be able to like collect this,
• 4:01
  this sort of like feedback and what not in real time.
• 4:07
  And a lot of people just straight up, they have like an
• 4:10
  idea of the tasks they want to do, right?
• 4:12
  They have some prompts and and whatnot, but they don't have a
• 4:16
  particularly like large data set on how to do it.
• 4:19
  And so the basic there was, OK, you don't have a large data set.
• 4:23
  You also don't have a very labeled data set or you don't
• 4:28
  have any labels because, yeah, takes a lot of time and effort
• 4:32
  to create those labels. So how can we still get
• 4:37
  something of value without any of that?
• 4:40
  Yeah, and it's, it's a little bit like counterintuitive from
• 4:44
  like all the classical machines, like supervised learning
• 4:48
  perspectives, right? That's like, oh, we're talk
• 4:50
  like, you know, always think really hard about your data,
• 4:55
  right? You need like a data.
• 4:56
  And then if you think about how like a lot of the the big
• 4:59
  companies are, are making their models too, right?
• 5:01
  Like they, they do have a ton of like human annotated data,
• 5:07
  right? So that was, that was like the,
• 5:09
  the sort of like initial key challenge that we spent a lot of
• 5:12
  time like brainstorming through is like, OK, well, if the way to
• 5:18
  get people to use something like this, right?
• 5:21
  Like this RL, like XF as a service.
• 5:25
  And so like I'll, I'll get to the, I'll talk about the, the RL
• 5:27
  bits like later, but basically the sort of like continuously
• 5:31
  improving like domain specific specialized models thing then we
• 5:38
  need to like make the barrier to entry as small as possible.
• 5:42
  That's funny. Anybody that has taken the
• 5:44
  Andrew Yang course is like, you can't do that.
• 5:49
  No, it's all about the data. What are you talking about?
• 5:52
  No. Label data.
• 5:53
  Yeah, Well, I it turns out that like it kind of still is about
• 5:58
  the data. It's just like we abstract a lot
• 6:01
  of it away for you. Tell me.
• 6:04
  More about that. And the, the, the data is like
• 6:08
  it's, it's synthetically generated under the hood, so to
• 6:11
  speak, And it's synthetically generated from the, the model
• 6:15
  itself, right? So here's the way I like to
• 6:19
  think about it, right? So, so the way we, we do this is
• 6:22
  very reinforcement learning based.
• 6:25
  And the way I like to to teach reinforcement learning usually
• 6:31
  is you think of this as like personalized supervised learning
• 6:36
  for the model, right? Where if you do normal
• 6:41
  supervised learning, you just have like a ton of labels that a
• 6:44
  human has kind of annotated and you're trying to learn how to
• 6:47
  mimic the human humans labels or whatever.
• 6:52
  Whereas in the reinforcement learning case, the model is
• 6:55
  itself generating data, right? The model is, is making these
• 7:00
  generations. So you have a prompt the model
• 7:03
  like generates like, you know, one or end responses for this
• 7:09
  particular like prompt. And then it gets feedback.
• 7:12
  It's it's like, OK, which one of these things was good?
• 7:15
  Which one of these things was not?
• 7:17
  And then it tries to to learn from that, right?
• 7:20
  And so it's learning from its own mistakes here, right?
• 7:26
  So it's whereas like if you were training it entirely on like a
• 7:31
  human data, it would be learning from a human's mistakes.
• 7:34
  And the, the key aspect of this is the mistakes that like models
• 7:39
  make when they're like reasoning about something is different
• 7:42
  than like how humans do it. So it, it actually turns out
• 7:46
  that in like a lot of cases, the human data is not necessarily
• 7:50
  gold truth. And so that's, that's probably
• 7:53
  like one of the, the key reasons this this actually like works is
• 7:57
  being able to have these models like learn from their own
• 8:02
  mistakes in like a way that's specialized for the the models.
• 8:08
  OK, so breakdown exactly how this is working and what because
• 8:13
  it's all I think we jumped into it saying Tao.
• 8:19
  The Tao stands for something, right?
• 8:21
  And it's not the land of 10,000 things.
• 8:23
  It is test time adaptive optimization, yeah.
• 8:27
  So I, I guess at a, at a higher level, right?
• 8:32
  So it's actually very much the, the process that I was just
• 8:35
  explaining. So people will come and they
• 8:39
  have like a, a bunch of prompts, right?
• 8:41
  So that's, that's the barrier to, to entry.
• 8:43
  Now some customer comes and they say, hey, we have these tasks.
• 8:48
  These are like the, the types of prompts that we want to give
• 8:51
  this model. And then what we do is we we
• 8:55
  take those prompts internally. We have these like, you know,
• 8:59
  our like base models, these sort of like, you know, up to some
• 9:03
  extent pre trade LMS or what not.
• 9:07
  We have them generate a bunch of responses from the the model,
• 9:13
  right? So there's like there's various
• 9:15
  sorts of techniques on, on how to do this.
• 9:18
  But the general gist they all boil down to is you are trying
• 9:23
  to generate different responses. And then we have another model
• 9:28
  which is called a reward model. And this is pretty important.
• 9:34
  So what we have, this reward model will take these like
• 9:38
  responses that, you know, you're generating models, right?
• 9:43
  This like which I'm going to call a policy from from here and
• 9:47
  out to, to try to use more like RL language.
• 9:51
  This reward model is going to take the responses that this
• 9:55
  policy has outputted and then score them, right?
• 10:00
  It's going to be like this is, you know, plus 1 + .5 -, 1 -,
• 10:06
  .5, right? And then you via reinforcement
• 10:13
  learning, we can then train this policy to say, OK, well, some of
• 10:20
  these responses were really good, right?
• 10:24
  And so because these responses were really good, we're going to
• 10:27
  change the weights of the model to up weight those, right, to
• 10:33
  make it more likely to produce outputs that are like the ones
• 10:36
  that have high scores and make the model less likely to produce
• 10:42
  outputs that are like the ones that have received low scores,
• 10:47
  right? And then that process sort of
• 10:49
  like repeats iteratively. Which until now is standard
• 10:55
  reinforcement learning, right? So this is this is mostly like
• 10:59
  standard reinforcement learning, right?
• 11:02
  What I've described, the key things that we've had to do in
• 11:07
  this particular case is, well, one, if you think about it, this
• 11:12
  like reward model that we have is kind of providing labels,
• 11:19
  right? And it's doing a lot of the the
• 11:21
  job of like ranking these like, you know, inputs and out like
• 11:25
  these outputs of the the policy, right?
• 11:28
  It's giving some kinds of like scores.
• 11:30
  And so it's really important to have a reward model that is
• 11:34
  trained really well across like a wide range of, of tasks.
• 11:41
  And so that's like one of the the things that that we talked
• 11:44
  about in the in the blog post, we have this like reward model
• 11:48
  called like ADB reward DBRM. So like a very enterprise
• 11:56
  focused reward model that's able to judge tasks across a wide
• 12:04
  variety of of possibilities. So the reason that this is
• 12:11
  simpler is that it is easier to trade models to judge whether
• 12:18
  something is correct or not as opposed to actually trying to
• 12:23
  come up with the correct answer yourself, right?
• 12:27
  So this this is this like verification problem.
• 12:30
  And so the verification problem is, is easier than the
• 12:33
  generation problem, usually cheaper and cheaper, right?
• 12:40
  And so we spent a lot of effort like gathering the data to make
• 12:45
  as like wide-ranging and generic a reward model as we possibly
• 12:51
  could. And then for the vast majority
• 12:53
  of use cases, at least everything that like we've shown
• 12:57
  in the blog post, we use the same reward model across like,
• 13:02
  you know, text to sequel across like, you know, finance bench,
• 13:05
  which is like, you know, like the the finance domain question
• 13:09
  answering kind of thing. It's actually the same reward
• 13:12
  model. It turns out that same reward
• 13:13
  model is able to judge outputs for like all of those tasks, the
• 13:20
  reward. Model was a fine-tuned model
• 13:27
  that it was like some llama base or something that was fine-tuned
• 13:30
  or it was a completely separate smaller model.
• 13:35
  It was like another model that has that was like a base.
• 13:39
  So think like, you know, in the in the llama style or like
• 13:43
  something like open source style llama DVRX, whatever style, like
• 13:48
  base model that we don't like fine tune on top of.
• 13:52
  So the the conventional wisdom for for training reward models
• 13:55
  is like usually do start with like a pre trained and then like
• 14:00
  somewhat like instruction to model.
• 14:02
  And then you add like an additional like head on top.
• 14:07
  So instead of like mapping to this, you know, giving you
• 14:11
  logits a probability distribution across your entire
• 14:14
  vocabulary to figure out what token to predict next, it's
• 14:18
  giving you a single scale or value now, Yeah.
• 14:22
  OK, now the other piece that I wasn't super clear on here was
• 14:25
  that you're saying the reward model is basically giving you,
• 14:31
  if you squint, some annotated data.
• 14:37
  Kind of it's, it's very sparse annotated data, right?
• 14:41
  So it's it's feedback data effectively.
• 14:44
  So the reason this is like possible is so let's look at
• 14:50
  this in terms of like information density, right?
• 14:53
  So if you have like pure, you know, like imitation learning
• 14:57
  data, right, supervised fine tuning, like sort of data where
• 15:02
  you know, you have to have a human write down exactly how to
• 15:05
  do a task, right? Exactly like all of the, the
• 15:09
  outputs for how to do a particular task given like the
• 15:13
  the input prompt, that's very information dense.
• 15:19
  Whereas the information that this model is giving it is like,
• 15:23
  OK, you're generating this entire like sequence after the
• 15:27
  entire sequence or at like break points in between the the
• 15:31
  sequence. We're just giving you like a
• 15:33
  scalar value right at sort of like, you know, predefined like
• 15:39
  break point. So it's, it's a lot less bits of
• 15:41
  information that it needs to, to be able to predict.
• 15:45
  And so that problem is easier. So now where does the test time
• 15:51
  annotation come in? I feel like there's another part
• 15:55
  to this story. So, so the other part to, to the
• 15:58
  story is that, OK, well, so you have like the, the reward,
• 16:03
  that's like scoring stuff, right?
• 16:06
  But what's it actually scoring, right?
• 16:10
  And what it's actually scoring is what's on the policy side.
• 16:16
  So the policy is generating like responses and the policy is
• 16:21
  generating and you can generate one response and score that one
• 16:26
  response, right? And then learn from that.
• 16:31
  You can generate 2 responses and score that.
• 16:34
  And then the way you do it is also like important, right?
• 16:36
  You can condition the second response that you generate it
• 16:40
  based on the first one, right? It's like, OK, generate
• 16:42
  something different from what I like generated the first time
• 16:46
  around or whatever. But if you think about it, that
• 16:50
  is, that is inference time compute that's being used.
• 16:54
  The more types of responses that you're using, the more types of
• 16:58
  responses you're you're generating, the more you know,
• 17:00
  test time compute, so to speak, is, is actually being used here.
• 17:05
  Now the terminology is it can get like a little bit confusing
• 17:09
  here. And the the reason we used like
• 17:13
  call up this like test time compute is this test time
• 17:17
  compute is just the model during training using additional
• 17:25
  inference time compute, right. But once this like whole
• 17:30
  training process is done and then the models like actually
• 17:34
  like deployed, it doesn't do multiple response generation or
• 17:41
  whatever, right? And so the inference cost for
• 17:45
  the actual user is the same. We are eating up a lot of the
• 17:51
  inference cost during the the training process.
• 17:54
  I see. The additional like adoptive
• 17:57
  test time, like the that bit is eaten up during like kind of
• 18:04
  ahead of time so that like when it's actually deployed, you
• 18:10
  know, you can kind of expect the the same, you know, like
• 18:12
  inference latencies and and whatnot.
• 18:15
  Yeah. So it's not like a deep research
• 18:19
  or whatever R1 reasoning model where you're going to give it a
• 18:24
  prompt and then come back 15 minutes later.
• 18:26
  It is doing that test time compute at the training level.
• 18:35
  Yeah. So, so the, the, the process
• 18:37
  would go right, like customer comes, they have like a bunch of
• 18:43
  prompts that they give us. And then we would do this kind
• 18:48
  of like, you know, like reasoning process at training
• 18:52
  time. And so we would just like drop
• 18:55
  it in and then come back a couple days later, right?
• 18:59
  Or like, you know, whatever amount of, you know, test time
• 19:03
  compute that we want to use for, for this particular task.
• 19:10
  These responses all get like scored in various ways.
• 19:14
  And it turns out like how you generate the responses like also
• 19:17
  matters, right? So there is like very popular
• 19:20
  techniques like best of N, which is you just kind of like
• 19:23
  generate and different responses all ID right, independently
• 19:29
  sampled from from each other. Turns out that's like probably
• 19:32
  not the greatest idea, right? Is like you're actually burning
• 19:36
  a lot of like redundant compute it, it like it, it works.
• 19:40
  But in terms of like trying to be of a somewhat more compute
• 19:43
  efficient it, it gets like a little bit redundant, right.
• 19:47
  So if your, your underlying model doesn't have like very
• 19:50
  much diversity in the range of things that I can say, then if
• 19:56
  you generate like N responses, it'll turn out that like K
• 20:00
  subset of those N responses will be just very similar to each
• 20:03
  other. And you've just burnt a bunch of
• 20:05
  and like the scores for those will all be the same.
• 20:07
  And there's no additional learning signal to be had and
• 20:10
  just trying the same thing over and over again.
• 20:12
  So you have to be a little bit smart about how you actually do
• 20:16
  this kind of sampling. And that's like one of the, the,
• 20:18
  the things that we like internally developed is like
• 20:21
  being smart about this. Because it's it, it's not the
• 20:27
  amount of ones or I guess to put that differently, you quickly
• 20:33
  realize that you, if you're scoring the same thing,
• 20:38
  basically it's the same words, but or it's the same idea and
• 20:43
  different words. And so it's getting the same
• 20:46
  score. And that's not adding any
• 20:48
  richness to that base model. Yeah.
• 20:50
  You're, you're, you're not like extracting any like new
• 20:54
  information, right? From like an information
• 20:57
  perspective, you're kind of like dead of the water.
• 21:01
  So you have to be like careful about how these responses are
• 21:08
  generated in the 1st place, right?
• 21:12
  And how you're doing this like exploration from from the RO
• 21:16
  perspective, such that the feedback that's given by the
• 21:20
  reward model is actually useful. I was figuring you were just
• 21:25
  going to turn the temperature up to 11 and then.
• 21:28
  See what happens, You know, we did, we did try that.
• 21:31
  That was actually probably one of the first things we tried is
• 21:34
  like, OK, let's just turn the temperature off and see.
• 21:38
  But it turns out that doesn't like work particularly like,
• 21:41
  well, it kind of depends on the, the use cases.
• 21:43
  So like if you turn up the, the temperature really high, it's
• 21:46
  like actually, you know, if you're doing like creative
• 21:49
  writing, right? If you want like character AI,
• 21:51
  sort of like chat bots, it could work.
• 21:54
  If you're trying to do like enterprise things, you probably
• 21:57
  don't really like that level of like super high temperature
• 22:01
  stochasticity. So you have to be, you have to
• 22:03
  do things like, you have to like be able to like condition on, on
• 22:06
  previous responses. You have to make sure that like
• 22:10
  you're, you have some definition of like, you know, if you have
• 22:14
  one response and you have like another response, you have to
• 22:16
  have like a definition of like how close or far apart from
• 22:19
  those two, like from each other they are.
• 22:23
  And then make sure that the the next responses that you generate
• 22:28
  are sufficiently different from all the the previous responses
• 22:31
  that you've generated. Have you tried to do this over
• 22:35
  and over? And do you see like loss when
• 22:39
  you're, when you've done it like two or three times?
• 22:42
  Because I can imagine a world where you say, all right, cool,
• 22:44
  let's just set up a retraining pipeline that kicks off every
• 22:49
  week or every couple days or every month, whatever it may be.
• 22:52
  But then you start to see almost like compression loss in a way.
• 22:57
  Yeah, Yeah. That's a great question.
• 22:59
  So like the answer to most things in ML and it depends,
• 23:06
  right. And so I'll tell you cases it
• 23:09
  works and sort of some cases where where it's not so great,
• 23:14
  right. So it turns out that actually
• 23:16
  like one of the things that we did in Tau is we just kind of
• 23:22
  like ran this process sort of like iteratively more and more.
• 23:25
  And we're like, oh damn, the, the numbers just going straight
• 23:29
  up right. Like it just keeps, yeah, this
• 23:32
  is the best graph ever. It's just going up and to the
• 23:34
  right, like just continuously this, this just keeps working.
• 23:39
  And this was like like a a year, year and a half ago at this
• 23:44
  point. We're like, this is great.
• 23:46
  You know, our reward model works.
• 23:48
  Like all of this works, but it turns out there's two things
• 23:53
  that that could go wrong here, right?
• 23:55
  One, is this like obvious issue or in hindsight, obvious issue
• 24:02
  of reward hacking where it turns out that like, you know, like
• 24:05
  your reward model is not entirely like 100% accurate,
• 24:09
  right? There's some amount of like
• 24:11
  noise in it. And so if you spend too much
• 24:15
  like compute and you're just doing this like like looping
• 24:20
  around, at some point you've extracted all of the useful
• 24:24
  signal from the reward model and now you're kind of just learning
• 24:28
  the remaining like error noise. Right.
• 24:33
  Interesting. And this is like this is, you
• 24:36
  know, like I, I don't know what overfitting really means
• 24:38
  anymore, But it like for the, for a classically trained ML
• 24:44
  person, this would be the, the way to think of it is like this
• 24:46
  would be like an RL ish version, possibly of, of overfitting
• 24:52
  where you've just, you're really good at optimizing for the
• 24:57
  reward for a particular like model.
• 25:00
  But because that itself is just a proxy for what like a customer
• 25:04
  or someone might want at the end it what does it really do so
• 25:07
  well downstream? It's almost like I think about
• 25:12
  when you're squeezing a lime and in the beginning you get all
• 25:15
  that juice and then later you're like trying to squeeze, squeeze,
• 25:18
  squeeze and it's so much effort just for like 1 drop and really
• 25:23
  it would be even better. Yeah, and it starts getting
• 25:26
  bitter to play if if you start squeezing at the the end, right,
• 25:30
  like and and like it, it's somewhat well known, but it's
• 25:35
  it's really interesting to kind of see it and practice.
• 25:38
  And it's like a very much like a like a somewhat of an art to
• 25:41
  figure out like, oh, when what is that tolerance threshold,
• 25:46
  right? What is, what is that cut off?
• 25:49
  And how do you get that cut off in like a generic way such that
• 25:53
  you know multiple different customers can use a singular
• 25:56
  board model? Because, well, what would you do
• 26:00
  then? Do you just swap out the base
• 26:03
  model and like start over from zero again?
• 26:06
  Or how do you? Well, I mean, you have, you have
• 26:09
  checkpoints from, from like sort of like all through the, the
• 26:13
  training process, right? So you could roll back to a, to
• 26:17
  a previous checkpoint like ideally like the, the way this
• 26:21
  would work, right? The, the ideal loop of what this
• 26:24
  is would be, you know, you have some kind of like generating
• 26:30
  model that burns like a bunch of infants compute.
• 26:33
  And simultaneously you have this like reward model that's trade.
• 26:36
  And then at some point you cut it off, you just deploy the
• 26:39
  model and then you collect new feedback so that you can
• 26:42
  actually like, you know, update your reward model.
• 26:44
  And therefore now there's more signal in the reward model.
• 26:47
  And now that there's more signal in the reward model, right?
• 26:50
  You can now train again on it, right?
• 26:52
  That would be the, the, the ideal loop is you're doing this
• 26:56
  like cycle of like redeployment. And this is what you were
• 26:59
  talking about, right? This like redeployment thing and
• 27:02
  like retraining across like deployments only works really if
• 27:07
  you are able to get new signal in between each.
• 27:11
  Right. And that new signal questions,
• 27:13
  Yeah, I get it. Yeah.
• 27:15
  If it's the same questions, right and you've kind of
• 27:17
  optimized up to a certain extent already, you're not going to get
• 27:20
  anything like more after the these sort of like initial
• 27:24
  training run because we're already doing like a lot of that
• 27:26
  like optimization for you and like we're figuring out
• 27:30
  dynamically like what those like thresholds and stuff are.
• 27:35
  So, so like ideally, right, if somebody comes to us and they're
• 27:39
  like, OK, so you know, we have these prompts.
• 27:41
  This is this is cool. We will then use this like, you
• 27:45
  know, like tile method overall, you know, figure out like, you
• 27:48
  know, all these sorts of like thresholds, optimize against
• 27:51
  this reward model that we have dynamically and then we'll give
• 27:55
  you and like, you know, the the model goes back to them, right.
• 28:00
  The reason you would want to then like retrain it like
• 28:02
  further would be actually, you know, we have these new tasks
• 28:09
  now that we want the model to be able to do.
• 28:11
  Here's a bunch of new prompts. And we are reasonably confident
• 28:17
  that these new prompts are actually doing something
• 28:19
  different from the old prompts. And to, to use your own words,
• 28:22
  they're not like the same thing, but in different words because
• 28:27
  we, we kind of count for that too.
• 28:30
  Sort of under the hood of like taking your prompt data set and
• 28:34
  then just making sure we're covering all our bases of like,
• 28:37
  oh, here's like all the ways that like people could possibly
• 28:39
  say or, or ask for these like same set of prompts.
• 28:45
  Yeah. And then so if you have like a
• 28:47
  bunch of new tasks that you're reasonably certain is like
• 28:50
  different from the capabilities the model has already like been
• 28:53
  trained for, then you come back to us and say, OK, well, we
• 28:57
  would like to optimize again, please.
• 29:00
  And then you do it right. And that would be the the reason
• 29:03
  to to retrain the the model. So what are some of the results
• 29:09
  that you saw when you did this, like compared against different
• 29:13
  ways of fine tuning, I guess would be the main thing.
• 29:17
  And most of the time, right? And I'm sure you've seen this a
• 29:22
  lot, it's just try and optimize the prompts, try and prompt tune
• 29:26
  it to the best of your ability because you get such a fast
• 29:28
  feedback loop. And then if you can't do that,
• 29:31
  then figure out if you need to go in and you need to fine tune
• 29:35
  and then what kind of fine tuning with how many GPU's and
• 29:39
  all that, what, what data, what labels, all of that fun stuff.
• 29:43
  There's so much of a headache that comes along with fine
• 29:46
  tuning and not doing it right. And then later after you fine
• 29:51
  tune you, you realize the model actually performs worse.
• 29:54
  What's going on? And so here I see the value prop
• 29:58
  really clearly. You say, look, the label data
• 30:01
  you don't need to worry about and will actually just give us
• 30:05
  your prompts and we'll make a model that works better on these
• 30:08
  prompts. Yeah.
• 30:10
  Yeah, so again, you know, the the standard machine learning
• 30:13
  like a dodge of there's no free lunch.
• 30:17
  Is is is very much true, Ray. So, you know, these prompts can
• 30:22
  be from like multiple different tasks and we can make your model
• 30:27
  better at these like multiple different tasks, but you're
• 30:33
  trading off performance somewhere else, right?
• 30:35
  So like you're, you're like, you know, like most of the, the
• 30:38
  customers will come to us. They'll have like a bunch of
• 30:40
  like enterprise based, like prompts, but you know, you're,
• 30:43
  you're probably, you know, like reducing the, the models like
• 30:47
  underlying creative writing, right?
• 30:50
  Which they may or may not care about.
• 30:52
  Yeah, which they, like, probably don't really care about, right?
• 30:55
  And if they do care about it, they can come back to us with
• 30:57
  like creative writing prompts as part of it, and then we can
• 31:00
  optimize for that too. Give me my financial statements
• 31:05
  in the style of a Bob Dylan zone.
• 31:08
  Yeah, exactly. That'd be cool actually.
• 31:12
  Yeah, maybe I should figure out ways of making stands that way,
• 31:18
  but yeah. The other thing that just popped
• 31:21
  up there in my head, as you were saying that is do you need a
• 31:25
  certain amount of diverse prompts or does it perform
• 31:30
  better if you give it a much bigger set of prompts?
• 31:33
  Because you were saying at the end of the day, all of the
• 31:36
  output they get scored, you reach that threshold.
• 31:41
  So in my mind I'm wondering, Oh well, if the input is more vast,
• 31:45
  then does that mean that you can get more out on the other side?
• 31:49
  And I'm sure you looked at that. Yes, we, we absolutely did did
• 31:55
  look at that. I guess the, the, the high level
• 31:57
  here is that like it is absolutely true that the bigger
• 32:02
  your prompt data set and the more diverse your prompt data
• 32:05
  set, the better the performance is going to be.
• 32:08
  And for the longest time actually, we only found that we
• 32:12
  were able to get like good performance, at least good
• 32:17
  performance means like we could train like a relatively small
• 32:20
  parameter count model to match the performance of like 4 O or
• 32:25
  something on, on a particular task if we had like a really big
• 32:28
  prompt set, like a really big diverse sort of a prompt set.
• 32:32
  But really big is like 1000. Or yeah, big would be in like in
• 32:36
  the in the thousands. OK.
• 32:39
  Right, but that's a pretty big barrier to entry again, right.
• 32:46
  If we're like asking for like no labels, but we're like, OK, only
• 32:49
  prompt. But actually the asterisk is
• 32:53
  that like you have to give us like, you know, 10,000 prompts
• 32:56
  or, or whatever. That's that's a pretty big
• 33:00
  barrier to entry. And so we we don't require that.
• 33:02
  We found a way around that basically such that for the, the
• 33:07
  sort of like end customer, they can give us just like a, a sub
• 33:12
  sample of the kind of range of things that that they think the
• 33:16
  model can be used for. And then we'll, we'll take it
• 33:22
  from there and we'll figure out how to like, you know, get that
• 33:26
  kind of like, you know, diverse, like prompt set or or whatever
• 33:31
  and learn from it. Yeah, so it's like I come to you
• 33:34
  with my 3 prompts and you're like, hopefully.
• 33:38
  You have more than three, but like.
• 33:42
  You got to work. Yeah, but you know, like if you
• 33:46
  come to me with like only three prompts, like I, we, we would do
• 33:49
  a great job of that. That'd make our life easy as
• 33:53
  like, I just needed to do well on like these three types of
• 33:56
  prompts. I was like, OK, sure.
• 33:59
  Right. But you know, like, you know,
• 34:02
  we'd be OK with like that range, right?
• 34:03
  It's like I'll like, like, you know, 10 to 50 prompts even,
• 34:08
  right? Like that, that range of stuff.
• 34:10
  And then, you know, we can go from there and internally like
• 34:15
  get to the stage where we see like prompts that are a lot more
• 34:21
  along this line and, you know, much bigger sort of like data
• 34:28
  set, much more diverse data set, so on and so forth.
• 34:33
  And so the thing that is also fascinating here is that you
• 34:37
  were able to get such high performance from small models.
• 34:43
  And I wonder if you had no real constraints on how you were to
• 34:52
  do it. But I just told you, hey,
• 34:54
  smallest model, best performance, highest accuracy on
• 34:59
  X amount of tasks, how would you go about it?
• 35:04
  And it can be anything from, all right, I'm going to distill this
• 35:07
  model and then I'm going to find Jordan and then I'm going to
• 35:09
  whatever you want. Like how would you look at that
• 35:12
  and try? Or is it not even worth the
• 35:15
  headache of doing all these extra hoops that you jump
• 35:21
  through and you just say I'm going to do Tau and it's going
• 35:25
  to be good enough? I actually think like, I'm, I'm
• 35:28
  going to do Tau and it's, it's probably going to be good enough
• 35:31
  because like we, I think we've done a pretty good job of like
• 35:34
  figuring out what the, the, the sort of like optimal scenario is
• 35:40
  even given kind of like all of these constraints, right?
• 35:43
  So these constraints are like fairly realistic, but under the
• 35:49
  hood, we were kind of like relatively unconstrained, right?
• 35:53
  We have no compute constraints and things like that under under
• 35:57
  the hood. And so like I, I think we've,
• 36:01
  we've done a, a good job of like being unconstrained as much as
• 36:05
  we can on our side where as still accepting constraints and
• 36:11
  making sure that the customers are like don't have like a
• 36:15
  particularly high barrier to entry.
• 36:20
  Yeah. So I actually wouldn't change
• 36:22
  too much, honestly, I think, I think like if I'm thinking about
• 36:26
  it, if you're like, oh, you're like fully constrained, how
• 36:29
  would I do it? That's actually kind of the
• 36:30
  question I asked myself. And because that's what would be
• 36:34
  ideal, right? And it's like, oh, because I, I
• 36:36
  guess the, the, the worst feeling would be like, oh, like
• 36:41
  somebody will, will come to you and they'll be like, oh, OK,
• 36:43
  well, you know, we have this particular task like, you know,
• 36:48
  we want to, we want to train models on, on this task or
• 36:51
  whatever. And then we're like, OK, here's
• 36:56
  this model that like kind of works Asterisk, right?
• 37:01
  But if we were more unconstrained, it would do
• 37:04
  better, right? And it's like that's that's a
• 37:05
  scenario I wanted to avoid. Yeah, yeah, yeah.
• 37:09
  I hear you. So the other interesting piece
• 37:14
  here is all right, does the how small of a model can you get to
• 37:19
  do how great of things or how good can you get these small
• 37:23
  models? Yeah, yeah, again, it's, it's a,
• 37:30
  it's a no free free lunch question.
• 37:31
  But it turns out that like the smaller models, like the, the
• 37:34
  sort of like 8B range of stuff, you can get them to do pretty
• 37:40
  darned impressive things, right? And again, it like, it really
• 37:47
  depends on the, the difficulties of the, the actual like tasks
• 37:52
  that people are trying to do. But what we found is that like
• 37:59
  in a narrow range, if you like take away it's like broad
• 38:02
  intelligence and specify it for like, you know, sort of more
• 38:06
  narrow data intelligence, so to speak.
• 38:10
  The the small models are are actually perfectly fine that the
• 38:15
  relatively smaller models, right.
• 38:17
  So there's like a threshold at which they're like, you know,
• 38:19
  good enough. Now, obviously, like, you know,
• 38:24
  you might be like, oh, you know, like Raj, does that mean you
• 38:27
  don't believe in scaling? That's like that's not true.
• 38:29
  I do believe in scaling, right. I just like scaling is like an
• 38:34
  axis that, you know, the more complicated A task you have, the
• 38:39
  the bigger at the model that you will need in order to to be able
• 38:43
  to like achieve it. But not everyone needs.
• 38:47
  So like you, you see the new Llama force, right?
• 38:50
  The the the behemoth is 2 trillion parameters.
• 38:56
  You know, only a fraction of that are like are active, but
• 39:00
  that's still going to be like a massive pain to to deploy for
• 39:03
  like the vast majority of people.
• 39:05
  Yeah. People are not stoked.
• 39:07
  People are not stoked about the the size of the Behiba even even
• 39:11
  like Scout is like relatively like large right and.
• 39:15
  Compared. To this, yeah.
• 39:20
  So like, you know, like people have these kinds of like, you
• 39:24
  know, like cost quality, like, you know, trade-offs and every
• 39:30
  single customer, I guess like lies somewhere different on that
• 39:35
  Pareto frontier of of cost quality.
• 39:39
  And so when you're thinking about setting up any sort of a
• 39:41
  product where you're trying to like get people to make their
• 39:46
  own customized models, like things like, you know, adaptive
• 39:51
  compute or what not are great because they basically like
• 39:54
  allow them to control where on the the quaff cost quality, like
• 39:59
  frontier they are. It's like I'll spend.
• 40:04
  More cost burn, more compute during training, get better
• 40:08
  quality at the end. Yeah, Yeah, that's a great point
• 40:13
  is how OK you may need to optimize for a certain place in
• 40:22
  this whole life cycle on you're OK with spending a little bit
• 40:26
  more money on that training to fine tune because later it means
• 40:31
  now you have that small model. And I imagine, have you played
• 40:35
  around with trying to like compress these models or distill
• 40:39
  these models even more to make them even smaller?
• 40:43
  We haven't like, yes, internally, like we've played
• 40:47
  around a little bit of like distilling it down from like the
• 40:50
  AB size even further. And you can like still do pretty
• 40:56
  reasonably well for certain tasks on those.
• 41:00
  But you you start seeing like pretty sharp drop offs.
• 41:03
  If you're like doing like a 3B, like a 1B or like a 3B sort of a
• 41:08
  model. Even for for some of the
• 41:11
  benchmarks that we were like thinking of or that we showed in
• 41:14
  the blog post, like the the text, the sequel, like bird
• 41:17
  sequel and and what not, the drop offs were were pretty high.
• 41:21
  But for some of the other benchmarks that it worked OK,
• 41:24
  right. So I guess it's like again, it,
• 41:26
  it kind of depends a little bit on how much or like how
• 41:31
  difficult the, the task is. But my my impression is that the
• 41:37
  difference between 3B and 8B in terms of performance is
• 41:46
  disproportionately high compared to the cost difference.
• 41:50
  Oh wow. OK.
• 41:53
  Yeah, which is kind of why we presented a lot of results at
• 41:56
  the, the the 8B scale, right. So it's like, oh, it was, it was
• 42:01
  not, it was not a linear correlation like the, the, the
• 42:04
  performance dropped off a lot more than like, you know, say
• 42:07
  like you know, 3/8 of the compute or whatever.
• 42:11
  Yeah, and the cost wasn't outrageously different.
• 42:14
  Yeah, the, the cost isn't like outrageously different between
• 42:17
  like a 3B and like an AB. Like right now of course there's
• 42:23
  like all sorts of like inference time inference things that you
• 42:27
  can do, you can do like multi tenant serving, like all that
• 42:30
  kind of jazz. But like we, we felt that this
• 42:33
  was like a reasonable trade off. Like an 8B was like a a
• 42:36
  reasonable trade off that is like still like fast and
• 42:41
  lightweight enough to deploy in like a lot of scenarios, but
• 42:45
  it's still able to perform pretty well, right.
• 42:51
  And the beautiful thing about Tau is that like it just like it
• 42:55
  just weren't like it's it's size independent, right?
• 42:57
  Like like we have results on like 70 B, we have results like
• 43:01
  internally on like 4 O 5B and it just keeps working even better
• 43:07
  and better like did. You notice any drastic
• 43:12
  differences or notable differences?
• 43:14
  Maybe not drastic between the models that you were using to
• 43:19
  fine tune on and some took better to this method than
• 43:22
  others. Yes, kind of.
• 43:27
  So it it really boils down to the models being able to
• 43:33
  generate diverse responses. So there are certain models that
• 43:40
  are out there that are kind of been RLHF to death, right?
• 43:48
  And so, so, so OK, so like from like my, my, my primary field of
• 43:53
  research is like like RL. And so like what, what is RLHF
• 43:56
  really doing here kind of underneath, right.
• 43:59
  If you have some kind of like distribution.
• 44:02
  So the pre training lets you learn all these like token
• 44:06
  distributions over the entire like Internet language
• 44:10
  distribution, like supervised fine tuning, like makes that
• 44:15
  distribution like smaller, right?
• 44:16
  It's like, OK, well, I'm going to bias you towards this like
• 44:20
  range of instructions that I think the the model is most
• 44:23
  likely to hear people like say to it and interact with it.
• 44:28
  Now, like RLHF, there's been like a lot of sort of
• 44:31
  contemporary work that's been out in the open, which it shows
• 44:35
  that it makes these probability distributions super spiky,
• 44:38
  right? It's like instead of now being
• 44:41
  able to generate like a wide range of of responses, you are
• 44:46
  now super constrained, right? You like can only say like one
• 44:50
  thing. And This is why you get all
• 44:51
  these like responses of like the the safety types of of
• 44:54
  responses. Like I can't do that for you,
• 44:57
  right? Like he it's, it's kind of been
• 44:59
  beaten out the, the model that's not great from the, the
• 45:05
  perspective of doing further like optimization with like RL,
• 45:11
  right? So if you have like a, a, a
• 45:14
  reward, like I was saying, right, like a lot depends on
• 45:17
  being able to extract good training signal from this like
• 45:21
  really great reward model that we have.
• 45:24
  And which means that the less, the, the more diverse, the
• 45:30
  underlying sort of like base model that we're, we're using is
• 45:34
  able to generate responses, the better it is, right.
• 45:39
  So that's, that's, that's part of the, the, the trick is like,
• 45:42
  you know, don't use things that are already like to RLHF.
• 45:47
  They won't take particularly well to to this method because
• 45:51
  everyone's that they've already been sort of like, you know, RL
• 45:54
  trained for a different set of tasks and now they're no longer
• 45:59
  particularly great at this. Do you remember way back in the
• 46:03
  day two years ago when the Mistral, first, the first
• 46:08
  Mistral model came out and it had pretty much no RLHF on it?
• 46:14
  And I think that's why people loved it so much was because it
• 46:17
  was like, whoa, we get to do it ourselves.
• 46:21
  That's amazing. Yeah.
• 46:24
  And, you know, there's, there's, there's pros and cons of, of
• 46:27
  releasing like a model with like, you know, no, no RLHF.
• 46:31
  But I it certainly became really popular in, in certain subsets
• 46:34
  of the, the open source community that wanted to use it
• 46:37
  for, you know, all sorts of like use cases that like a model like
• 46:41
  Llama was kind of too RLHF to to do well on, right.
• 46:49
  But yeah, so that's, that's kind of the, the, the thing to, to
• 46:52
  think about here, right. But well, I guess to this is
• 46:56
  from like a researchy kinds of perspective, from a customer
• 47:01
  perspective, they don't really have to think about it because
• 47:03
  we've already thought about it for them.
• 47:06
  Yeah, you've already tested a bunch of different models.
• 47:09
  Yeah, we, we already, we already have a pretty good idea of like
• 47:12
  the, the distributions and like, like which models are spiky,
• 47:18
  which are not if they are spiky, how to get rid of some of the
• 47:22
  spikiness in the distribution such that it's more useful to,
• 47:25
  to generate, you know, like a a diverse range of things.
• 47:30
  We we've kind of already taken care of this with, with internal
• 47:33
  research. OK, we've been dancing around
• 47:36
  this and really talking about how the sausage is made, but
• 47:43
  what we didn't talk about is the actual results.
• 47:47
  So maybe you can give us a quick TLDR of like like how is this
• 47:52
  better? In what ways?
• 47:53
  I guess is the is the big question that's going through my
• 47:55
  mind right now. Yeah, yeah.
• 47:58
  So, so the one thing that it's like super obvious, right?
• 48:02
  The the very obvious, like baseline is just like supervised
• 48:05
  fine tuning, right? It's like if you had supervised
• 48:10
  fine tuning, like labels. It turns out that it does way
• 48:15
  better than supervised fine tuning with labels, even though
• 48:18
  this doesn't have labels, right? That's pretty clear.
• 48:22
  We tested it on like, you know, like text to sequel and then
• 48:25
  like some other benchmarks that we, we developed that other
• 48:29
  people in my team have developed previously, which is like the
• 48:32
  enterprise arena. And we have our own version of
• 48:35
  like finance bench and and what not, which test all these sorts
• 48:40
  of like, you know, question answering, open and close book
• 48:42
  question answering scenarios in very like enterprise ways.
• 48:48
  Like I, I would never go back to like supervised fine tuning,
• 48:53
  right? Like it, it's, I, I think it
• 48:55
  right. I, I think it's, it's, this is
• 48:57
  like just way better unless you have like a ridiculously like
• 49:02
  massive data set of, of labels that's like rich and what not,
• 49:08
  which like effectively nobody does.
• 49:13
  And even then, like Tau is not orthogonal to that.
• 49:18
  Like you could still run Tau on top of that, right?
• 49:22
  Because if you have like a label data set with like prompts and
• 49:26
  responses, guess what? You also have you have the
• 49:28
  prompts, which is all Tau needs, right?
• 49:33
  So, so you can, you can still run like Tau together with the
• 49:38
  the supervised fine tuning step up front.
• 49:40
  The, the other interesting thing under interesting thing is that,
• 49:47
  you know, the models are actually like I'm, I'm going
• 49:50
  back and looking at the, the sort of like exact numbers.
• 49:54
  But it turns out that like, you know, even the 8B models end up
• 49:59
  being comparable to like 4 O for that like narrow like task,
• 50:06
  right, for that like range of, of tasks on some tasks, right,
• 50:10
  right. So, so the, the AP models like,
• 50:14
  you know, if you have a task like finance venture or what
• 50:17
  not, we'll do as well as like 4 O or like, you know, O3 like
• 50:23
  mini or or whatever, which are like an order at least order to
• 50:29
  two orders of magnitude bigger and parameter count.
• 50:33
  Now for the tasks that are like a little bit more complicated,
• 50:37
  you see, like, you know, for like bird sequel or, or
• 50:40
  whatever, which I yeah, is is more complicated of a task.
• 50:46
  You see that the performance of like the AP models are still
• 50:49
  slightly less than like 4 O or O3 mini, but that gap narrows
• 50:55
  and basically vanishes when you get to 70 B.
• 50:59
  Oh nice. Right.
• 51:02
  And so, so this is what I was I was talking about is like it's,
• 51:07
  it's where you are on the the sort of like cost quality, trade
• 51:12
  off rate, what would be good enough for you.
• 51:16
  And so if you are a person that is doing like financial QA in
• 51:21
  the line of like finance bench AP is probably good enough for
• 51:26
  you, right? If you're doing something
• 51:29
  significantly like more complicated, right?
• 51:33
  You'd probably need something that's like a little bit like
• 51:35
  larger sized in order to be able.
• 51:39
  And like, if, if you're doing something that's like a little
• 51:41
  bit more complicated and you want to absolutely at like O3
• 51:45
  mini level, right, then you're, you're going to like need to
• 51:48
  spend a little bit more on, on training at.
• 51:53
  The interesting thing I will say also is the, the comparison that
• 51:58
  we did do, we didn't like show results of like the actual like
• 52:01
  inference time comparison cost. We kind of just presented them
• 52:04
  as all equal. But O3 Vinny is really doing
• 52:08
  like test time inference, right? They're they're doing like a lot
• 52:12
  of like additional compute where is like we've baked a lot of
• 52:15
  that like additional compute in during training time itself.
• 52:19
  And so like the the real correct sort of to speak comparison
• 52:23
  would be like probably like 4 O in terms of like test time
• 52:26
  latencies after the model is is deployed.
• 52:32
  Yeah. So it's like what's, what's your
• 52:33
  threshold? How complicated is the task that
• 52:37
  you're doing? But the the results are pretty
• 52:41
  promising in the sense that like, you know, with like A7DB
• 52:46
  model, even the seven DB is like still like smaller than four O
• 52:53
  and like O3 MIDI and and all of these things you can erase and
• 52:59
  basically like get to near parody.
• 53:03
  And then, well, how you control your fate, right, to sort of
• 53:09
  speak right, Like you have this exact model.
• 53:13
  You know, nobody does anything under the hood that changes like
• 53:18
  the the model behavior. They let the models don't swap
• 53:21
  out under the API hood. If like if you give a particular
• 53:26
  prompt like one day and the model does a certain way, you
• 53:30
  can build the guarantees around that, right?
• 53:33
  The models going to continue behaving the same way because
• 53:36
  you know. Depreciating a model after six
• 53:39
  months? Then no.
• 53:40
  Depreciating a model? Yeah, like 4.5 is like
• 53:44
  depreciating already, which is crazy.
• 53:46
  Dude, well, this is this is awesome, man.