Latest / learning with yacine show / Pretrained LLMs Are Surrounded by Task Experts feat Yulu Gan from MIT
Transcript
- 0:00SPEAKER_01: Today, we're going to dive into the strange geometry of pre-trained neural network,
- 0:04SPEAKER_01: lost plane, and what they mean
- 0:06SPEAKER_00: for post-training.
- 0:07SPEAKER_01: Yes, we're going to dive straight into the curvature of the distribution of
- 0:11SPEAKER_01: the pre-trained weights themselves,
- 0:13SPEAKER_01: and it's going to make a whole lot more sense in a minute.
- 0:15SPEAKER_01: The old mental model I had for pre-trained neural network was that
- 0:19SPEAKER_01: the pre-training Like recipe,
- 0:22SPEAKER_01: make them land into a single set of pre-trained weights that was optimal
- 0:27SPEAKER_01: for many tasks.
- 0:28SPEAKER_01: That was the mental model I had.
- 0:29SPEAKER_01: But after going through the neural ticket paper,
- 0:31SPEAKER_01: this one, by Yulugan, my view has been kind of radically changed.
- 0:36SPEAKER_00: The new metal model I have in my mind now is that
- 0:39SPEAKER_01: for whatever reason,
- 0:40SPEAKER_01: the model during pre-training is getting pushed toward a region that is pretty flat,
- 0:45SPEAKER_01: like a bassin,
- 0:46SPEAKER_01: I don't know how to say it in English,
- 0:47SPEAKER_01: where we're just surrounded by like specialized experts,
- 0:51SPEAKER_01: A bit like this valley in Chilliwack, British Columbia, that is surrounded by
- 1:06SPEAKER_01: So Yulu,
- 1:06SPEAKER_01: the first author of this paper,
- 1:08SPEAKER_01: demonstrated that with various very elegant experiments.
- 1:29SPEAKER_01: Which we're going to go through in depth today.
- 1:31SPEAKER_01: But the most interesting one in my opinion is that they made an algorithm
- 1:35SPEAKER_01: which will randomly perturb the weights and then sample them together
- 1:39SPEAKER_01: with great performances.
- 1:40SPEAKER_01: So like this revolve in post training, GRPO,
- 1:43SPEAKER_01: ES and a whole bunch of other algorithms
- 1:46SPEAKER_01: while being It's like almost too dumb simple.
- 1:49SPEAKER_01: Like literally they will jiggle the weights and like put them together and
- 1:52SPEAKER_01: that will be the end result.
- 1:54SPEAKER_01: So in this conversation,
- 1:55SPEAKER_01: we're going to start the gentle walkthrough of the paper for people who are newer
- 1:58SPEAKER_01: to this type of literature.
- 2:00SPEAKER_01: So you don't have to read it before watching this video.
- 2:02SPEAKER_01: Then we'll get to a whole bunch of existential questions I had that I asked Yulu about.
- 2:07SPEAKER_01: Like first, why do these tickets exist at all after pre-training?
- 2:12SPEAKER_01: And what it is about pre-training That fill the space around a checkpoint
- 2:17SPEAKER_01: with expert in the first place.
- 2:18SPEAKER_01: And then we'll try to take a step back to what it actually imply.
- 2:22SPEAKER_01: This kind of structure of the distribution of weight for post-training,
- 2:26SPEAKER_01: for distillation, for continual learning.
- 2:27SPEAKER_01: One quick thing before we start.
- 2:29SPEAKER_01: The team built a really beautiful, very visual project blog at tickets.mit.edu.
- 2:35SPEAKER_01: Link below.
- 2:36SPEAKER_01: You should check it out.
- 2:37SPEAKER_01: Like a very visual and will help you kind of understand at the deeper level what
- 2:41SPEAKER_01: the results are showing.
- 2:42SPEAKER_01: It's kind of a good companion website to the paper.
- 2:44SPEAKER_01: Before we get into the interview,
- 2:46SPEAKER_01: I want to talk a bit about RC,
- 2:47SPEAKER_01: the OpenAI lab that is making this interview possible.
- 2:50SPEAKER_01: If you are familiar with the OpenAI model landscape,
- 2:53SPEAKER_01: you know it's pretty much dominated by lab outside of North America.
- 2:56SPEAKER_01: Well,
- 2:57SPEAKER_01: RC is a very capable AI lab that's not I had the chance to interview the CTO, Lucas,
- 3:01SPEAKER_01: and the pre-training lead,
- 3:02SPEAKER_01: Varen, about their Trinity Large Thinking model, which is a 400B sparse MOE.
- 3:14SPEAKER_01: Working with the team is how open they are,
- 3:16SPEAKER_01: like yes with the model weights,
- 3:18SPEAKER_01: but also with like the whole training methodology and how humble they were about
- 3:22SPEAKER_01: where it worked and where it didn't work and where there's room for improvement.
- 3:25SPEAKER_01: So this level of transparency is very unheard of in the space.
- 3:28SPEAKER_01: They are currently working on another run of the family of model which I'm super pumped about.
- 3:32SPEAKER_01: You can get this model on open router, right?
- 3:34SPEAKER_01: Right here, very easy to get set up.
- 3:36SPEAKER_01: If you're into a personal agent, you can get it set up within your Hermes setup pretty easily.
- 3:41SPEAKER_01: You can also try it out and use it straight within the Trinity platform right over here.
- 3:45SPEAKER_01: One way I'm using these models in my workflows for like knowledge type
- 3:49SPEAKER_01: of work is I'm using them extensively to kind of take a whole bunch of my notes
- 3:54SPEAKER_01: as I'm navigating to the material, let's say for this interview.
- 3:57SPEAKER_01: And then I give it to it to kind of go and do a bigger search to hook it into
- 4:02SPEAKER_01: the literature and like the broader information.
- 4:05SPEAKER_01: Let's copy paste that.
- 4:06SPEAKER_01: So these are kind of most of my notes about like this interview I did
- 4:10SPEAKER_01: when I was reading the paper and like structuring all the links I had.
- 4:14SPEAKER_01: And then you will go and do some deep research on the subject.
- 4:18SPEAKER_01: So once it's done,
- 4:19SPEAKER_01: I have like a bunch of more grounded information about like the author,
- 4:24SPEAKER_01: about like the different paper.
- 4:27SPEAKER_01: And what I usually do after that is ask it to kind of figure out all of the cluster
- 4:33SPEAKER_01: of questions I can ask that are interesting,
- 4:35SPEAKER_01: that is grounded within like this core piece of knowledge that will be interesting
- 4:39SPEAKER_01: to dig in with respect to the literature.
- 4:41SPEAKER_01: So I have my little prompt for that and then it will do it by trying
- 4:45SPEAKER_01: to create questions that are interesting and that are not answerable I try to ask it
- 4:50SPEAKER_01: to generate as much question as possible to really exhaust all
- 4:54SPEAKER_01: the possible like angle and then what I do personally is I look at all
- 4:58SPEAKER_01: the different themes and I eliminate those that are like kind of trivial or not
- 5:03SPEAKER_01: that interesting And I focus on some that are much more related to the intuition
- 5:09SPEAKER_01: behind that is not explicitly said in any of the literature I've read.
- 5:15SPEAKER_01: And then I start to kind of fuse them together and then like,
- 5:18SPEAKER_01: or kind of generate my own question based with like some of these question stubs,
- 5:23SPEAKER_01: right?
- 5:23SPEAKER_01: And that's it.
- 5:24SPEAKER_01: Big thanks to RC for making this interview possible.
- 5:27SPEAKER_01: Very cool lab to keep an eye on and to follow on Twitter.
- 5:29SPEAKER_01: Let's now dive into the interview.
- 5:31SPEAKER_00: Hi, everyone.
- 5:32SPEAKER_00: My name is Yulu Gan,
- 5:33SPEAKER_00: and now I'm a second year PhD student at MIT CSAIL,
- 5:40SPEAKER_00: advised by Philip Isola and Tommaso Poggio,
- 5:43SPEAKER_00: and my research interest is around At a very high level,
- 5:51SPEAKER_00: the intuition is that pre-training does not just give us a good model,
- 5:56SPEAKER_00: it actually gives us a bunch of task experts.
- 6:00SPEAKER_00: So instead of saying, hey,
- 6:02SPEAKER_00: here is a pre-trained model,
- 6:04SPEAKER_00: and now we need to teach it a new skill,
- 6:07SPEAKER_00: we can ask,
- 6:08SPEAKER_00: are there already many nearby versions of this model that are better
- 6:12SPEAKER_00: at different tasks?
- 6:14SPEAKER_00: And what are we finding in the paper is that the answer is yes.
- 6:19SPEAKER_00: If we randomly perturb the weight of the large pre-trained model,
- 6:23SPEAKER_00: many of those nearby models are not just worse copies.
- 6:27SPEAKER_00: Some of them just become They may improve on one task while getting worse
- 6:33SPEAKER_00: on another, but they can become a coding expert, a math expert, a writing expert.
- 6:39SPEAKER_00: And all that suggests that the pretreat model is sitting among many local experts.
- 6:45SPEAKER_00: So that's a high level intuition about the paper.
- 6:49SPEAKER_01: A kind of logical step from why
- 6:54SPEAKER_01: evolutionary strategy even works with these big models, right?
- 6:58SPEAKER_01: It's because they're so densely packed.
- 7:00SPEAKER_01: Can you just tell us what's the logical kind of steps that you took from the paper
- 7:06SPEAKER_01: about evolutionary strategy to now with just looking literally at the shape of
- 7:12SPEAKER_01: the Lawson's case?
- 7:13SPEAKER_00: Yeah,
- 7:14SPEAKER_00: I think the connection between the evolution strategy paper and the secret paper is
- 7:19SPEAKER_00: that they are both treating the post training as a selection problem,
- 7:25SPEAKER_00: but in evolution strategy paper,
- 7:27SPEAKER_00: like you sample a positive So it's population-based and it's multi-step.
- 7:32SPEAKER_00: And RandOps is like a more,
- 7:43SPEAKER_00: Extreme version of that.
- 7:45SPEAKER_00: So instead of doing many iterations with a small population,
- 7:49SPEAKER_00: we do one large round with many samples, and we do not update the base model.
- 7:55SPEAKER_00: So no iterations at all.
- 7:57SPEAKER_00: We just select the best ones and ensemble them.
- 8:00SPEAKER_00: So this suggests that the experts or the solution density around
- 8:05SPEAKER_00: the base model is extremely high and dense.
- 8:09SPEAKER_01: Yeah.
- 8:09SPEAKER_01: I mean, I have so many questions.
- 8:11SPEAKER_01: A good flow right now will be like that you present a bit like the research
- 8:15SPEAKER_01: with your slide deck.
- 8:16SPEAKER_01: And I'll interrupt you, Atraat, to ask my question.
- 8:20SPEAKER_00: Since I have already talked about the intuition and the flow of the paper,
- 8:25SPEAKER_00: maybe I will skip some of the introduction, but I will just start.
- 8:30SPEAKER_00: So another analogy maybe used for have intuition about the paper It's like
- 8:37SPEAKER_00: if we think about finding a diamond in the grids and we have the grid size of 13
- 8:44SPEAKER_00: and it's very hard if we just have one diamond in these grids and it's very hard
- 8:51SPEAKER_00: to Find these grids.
- 8:53SPEAKER_00: But if we have many,
- 8:55SPEAKER_00: many diamonds in the grids,
- 8:57SPEAKER_00: and then by a random click on the grids,
- 9:01SPEAKER_00: we have a much larger chance to get the diamond.
- 9:07SPEAKER_00: So this is an analogy that we can perturb the model by clicking the grid here,
- 9:16SPEAKER_00: and the star here is the base model.
- 9:21SPEAKER_00: Actually, this is a long question that people ask for a long time.
- 9:28SPEAKER_00: I mean, it's back into 1999.
- 9:32SPEAKER_00: Jurgen, Seb, and Banjo, they tested this.
- 9:36SPEAKER_00: Random guessing algorithm on neural networks.
- 9:39SPEAKER_00: And they do exactly this thing that they repeat randomly initialize the weights
- 9:46SPEAKER_00: until the resulting network happens to classify our training sequences.
- 9:51SPEAKER_00: In their paper,
- 9:54SPEAKER_00: they said random guessing cannot be viewed as a reasonable learning algorithm,
- 10:00SPEAKER_00: and they would never use random guessing in realistic applications.
- 10:06SPEAKER_00: It's impossible to train a neural network by guessing.
- 10:10SPEAKER_00: And this makes a lot of sense because if we consider the neural networks,
- 10:16SPEAKER_00: they will have the curse of dimensionality that is easy to get one point right.
- 10:24SPEAKER_00: But if it goes to two dimensional problem, and it's very hard to get the two points right.
- 10:32SPEAKER_00: And for those large language models, they have billions or trillions So, almost impossible.
- 10:38SPEAKER_00: Also, people were talking about how powerful the gradient descent is,
- 10:45SPEAKER_00: and someone, they tested, they used CLN on
- 10:54SPEAKER_00: And then they find that the random step is literally a 185 sigma event compared to
- 11:04SPEAKER_00: the gradient step.
- 11:06SPEAKER_00: So it's almost impossible if you want to So back to the correlation between
- 11:12SPEAKER_00: the CK paper and the evolution structure.
- 11:26SPEAKER_00: Actually during our work in our explorations in
- 11:30SPEAKER_00: the evolution strategy we find we can use very small population size
- 11:35SPEAKER_00: that is around 30 and the model on premise is around And this can work.
- 11:42SPEAKER_00: So this suggests that the reward landscape of
- 11:45SPEAKER_00: the large language model should be much more denser than we thought.
- 11:50SPEAKER_00: But during that, I mean,
- 11:52SPEAKER_00: in the evolution strategy paper,
- 11:55SPEAKER_00: We didn't go very deep into,
- 11:57SPEAKER_00: we didn't look into the reward landscape or loss landscape of
- 12:01SPEAKER_00: the large length model.
- 12:03SPEAKER_00: So in the CK paper,
- 12:05SPEAKER_00: actually,
- 12:05SPEAKER_00: the first thing we do is that we look into the waste space and the reward landscape
- 12:12SPEAKER_00: of the large length model.
- 12:13SPEAKER_00: If we look at the loss landscape of the pre-training objective,
- 12:19SPEAKER_00: it's expected You can see a very wide region of low loss area.
- 12:26SPEAKER_00: But if we look at the accuracy landscape,
- 12:29SPEAKER_00: we can see a very surprising phenomenon where you can see
- 12:33SPEAKER_00: the landscape is very warm.
- 12:36SPEAKER_00: Here, actually,
- 12:37SPEAKER_00: each point here refers to a model,
- 12:40SPEAKER_00: model's weights, and we do random projection from the model weights to this 2D map.
- 12:47SPEAKER_00: So each point refers to the model weights,
- 12:51SPEAKER_00: and the value of each point here is the accuracy on some tasks.
- 12:56SPEAKER_00: For example, this figure is for GSM-AK mass reasoning dataset.
- 13:01SPEAKER_00: The value here is the accuracy of the math reasoning dataset,
- 13:06SPEAKER_00: and each point refers to a different perturbed model.
- 13:09SPEAKER_00: So this figure means if we have a pre-trimmed model and then we add some noise onto
- 13:14SPEAKER_00: that,
- 13:15SPEAKER_00: it's more likely you have a higher accuracy than the base model
- 13:20SPEAKER_00: on some certain tasks.
- 13:22SPEAKER_00: This is a scaling effect because if we did this analysis on
- 13:28SPEAKER_00: a very small model like 0.5 billion model,
- 13:31SPEAKER_00: you can see that if we just perturb the model, you will just get a worse solution.
- 13:38SPEAKER_00: But as a model scales up to a larger size and then you can find you have
- 13:46SPEAKER_00: a much more warmer landscape here.
- 13:49SPEAKER_00: And also if we map this 2D map into the 1D slides we can find, we can see it's more clear.
- 13:59SPEAKER_00: Like from the zero point billion model to 32 billion model,
- 14:05SPEAKER_00: the accuracy of the base model,
- 14:08SPEAKER_00: which is a middle point in all of these sub figures,
- 14:13SPEAKER_00: and it shifts from the local maximum to the local minimum.
- 14:18SPEAKER_00: That is,
- 14:19SPEAKER_00: if you have a very small model,
- 14:21SPEAKER_00: you perturb the model and all the solutions will be But if you have a larger model
- 14:28SPEAKER_00: and then you add some noise into the model and almost all very large amounts of
- 14:34SPEAKER_00: the perturbed model can improve the performance.
- 14:39SPEAKER_00: And this is also consistent with different datasets and also with different models.
- 14:46SPEAKER_00: This is tested on And this is tested on the newest JAMA 4 model from Google.
- 14:53SPEAKER_01: So I had a question about this actually.
- 14:57SPEAKER_01: I don't know if you have an answer for interpreting why as we go to
- 15:04SPEAKER_01: a bigger model like that, we get into this local minima.
- 15:08SPEAKER_01: Is it because where the pre-training weight stops is that they're a generalist
- 15:14SPEAKER_01: at roughly everything, but they're not like a specialist?
- 15:17SPEAKER_01: Or can they actually be an expert in one of these different areas that you've tested?
- 15:23SPEAKER_00: Based on this observation,
- 15:26SPEAKER_00: we find that the base model is good,
- 15:30SPEAKER_00: but it ends up like it's not a specialist on the different tasks,
- 15:35SPEAKER_00: and it's somehow good at every...
- 15:38SPEAKER_00: Every task, but if you just perturb it, it can get better on some certain tasks.
- 15:44SPEAKER_00: So we have a hypothesis that maybe pre-training is just doing something that
- 15:52SPEAKER_00: because it's trained
- 15:58SPEAKER_00: And it's just trying to find some good points that's good for all the tasks.
- 16:06SPEAKER_00: So like find a main point.
- 16:08SPEAKER_00: That is good for all the tasks.
- 16:10SPEAKER_00: And then Post Train is to select the best model for some certain tasks.
- 16:18SPEAKER_00: And this can be done very easily, like by just do a searching, a Gaussian searching window.
- 16:25SPEAKER_01: I don't know if it's just like a matter of the shape being like this or the fact,
- 16:32SPEAKER_01: because I think you have an experiment where you do it from first principle, right?
- 16:38SPEAKER_01: Is it also connected with the fact that now post-training and pre-training,
- 16:43SPEAKER_01: mid-training are kind of starting to get blended together?
- 16:46SPEAKER_01: Is the pre-trained model at the end of the day in this spot where it is easier to post-train?
- 16:52SPEAKER_01: A fact of the whole training regiment?
- 16:54SPEAKER_01: Or even if you're not even thinking about post-training,
- 16:57SPEAKER_01: it will end up there because of just the scale of it.
- 17:01SPEAKER_01: What's your thought here?
- 17:03SPEAKER_00: I think it's hard to have a pre-trained model that can solve all the tasks very well
- 17:10SPEAKER_00: because based on the hypothesis I mentioned before,
- 17:13SPEAKER_00: if you have huge amount of tasks during pre-training, you can
- 17:20SPEAKER_00: It can be very hard to reach the best performance on every single task,
- 17:26SPEAKER_00: but you can just do an average performance on them.
- 17:30SPEAKER_00: For example, you can do an average
- 17:39SPEAKER_00: So that's the reason why I think we should have post-training,
- 17:43SPEAKER_00: but in a very easy way,
- 17:45SPEAKER_00: like to perturb or apply run-opt algorithm on it,
- 17:50SPEAKER_00: and then you can just find Domain specific expert for you.
- 17:55SPEAKER_00: And we also define a solution density to describe this phenomenon.
- 18:02SPEAKER_00: So the solution density is the probability of a random guess.
- 18:07SPEAKER_00: Increasing performance by some margin.
- 18:10SPEAKER_00: So here we can see a very clear trend that if we scale the model size,
- 18:18SPEAKER_00: and then we can get higher solution density.
- 18:20SPEAKER_00: And it's also validated on several different tasks like mass programming,
- 18:26SPEAKER_00: writing, and chemistry.
- 18:28SPEAKER_00: And so to summarize the finding we have found is When you have a small model,
- 18:37SPEAKER_00: if you add some perturbation into the model,
- 18:40SPEAKER_00: you can likely have some Experts on some tasks.
- 18:44SPEAKER_00: But if you have a very large model,
- 18:47SPEAKER_00: a very good pretrained model that have seen many,
- 18:50SPEAKER_00: many data and many,
- 18:52SPEAKER_00: many tasks,
- 18:53SPEAKER_00: and then you add some little noise into the model,
- 18:57SPEAKER_00: you can just get a domain expert in,
- 19:01SPEAKER_00: for example, in chemistry, in writing, in programming or in math or in other tasks.
- 19:07SPEAKER_00: One following up question is that we wonder Do all the perturbations help or hurt in
- 19:12SPEAKER_00: the same way?
- 19:14SPEAKER_00: So there's two hypotheses around that.
- 19:17SPEAKER_00: One hypothesis is that it's a generalist.
- 19:21SPEAKER_00: So one certain perturbation can be So this can be beneficial for all the different tasks.
- 19:28SPEAKER_00: Like here we tested on four math tasks and one programming task, one writing and chemistry task.
- 19:39SPEAKER_00: The answer is that it's more like the specialist.
- 19:55SPEAKER_00: I think it's a figure three or four.
- 20:01SPEAKER_00: In our paper, we show that the x-axis here is different tasks or different datasets.
- 20:10SPEAKER_00: And why axis is here is relative performance.
- 20:14SPEAKER_00: And we show that here each line is a certain perturbation.
- 20:18SPEAKER_00: And here we can see that the model is more like a specialist because for each line,
- 20:24SPEAKER_00: it looks like a zigzag curve.
- 20:26SPEAKER_00: And it can be,
- 20:28SPEAKER_00: for example,
- 20:28SPEAKER_00: the red line can be very good at Olympia bench mass data set,
- 20:34SPEAKER_00: but can be very poor at chemistry data set.
- 20:38SPEAKER_00: You may wonder why this happens.
- 20:41SPEAKER_00: So we did a 1D toy experiment.
- 20:45SPEAKER_00: So here I will introduce a toy example, the toy experiment setting.
- 20:53SPEAKER_00: So here we are trying to predict the 1D signal and we have a
- 21:01SPEAKER_00: So we have four-layer MLP neural networks, which is a toy model.
- 21:06SPEAKER_00: And the input is Y context here.
- 21:10SPEAKER_00: So we give some context just as a prompt we give to the large-length model.
- 21:15SPEAKER_00: Here we have some context,
- 21:17SPEAKER_00: and then we are trying to predict the next token,
- 21:21SPEAKER_00: or here is the next value actually, to predict this 1D signal.
- 21:30SPEAKER_00: Behavior is very similar to the large length model.
- 21:34SPEAKER_00: So we have auto-regressive setting,
- 21:37SPEAKER_00: and we have the context, and we have to predict the next token.
- 21:42SPEAKER_00: And in this setting, actually we find the same thing.
- 21:46SPEAKER_00: So, we find three regions here.
- 21:53SPEAKER_00: The first one is,
- 21:55SPEAKER_00: we called it the needle in the haystack region,
- 21:59SPEAKER_00: where you You didn't pre-train the toy model.
- 22:04SPEAKER_00: That is, you just do random initialization to the toy model, and then you test it.
- 22:12SPEAKER_00: For example,
- 22:12SPEAKER_00: here is you let the model to predict And then it's because
- 22:21SPEAKER_00: the base model can do it very bad.
- 22:25SPEAKER_00: And so no matter you add noise or you add some other message,
- 22:31SPEAKER_00: So we call this region the needle in the haystack.
- 22:37SPEAKER_00: But things changed if you have a pre-train model.
- 22:41SPEAKER_00: So here we pre-train the toy model on a mixture For example,
- 22:48SPEAKER_00: linear functions, sine waves, square waves, and other different signals.
- 22:55SPEAKER_00: And if we have this pre-trained model,
- 22:57SPEAKER_00: and then we add random perturbs,
- 23:01SPEAKER_00: So here, the random perturbs are shown as the gray lines.
- 23:06SPEAKER_00: And the model can have very different behaviors.
- 23:10SPEAKER_00: Like you can see here are many very different behaviors.
- 23:17SPEAKER_00: And then we can select the top k perturbs among the random perturbs,
- 23:24SPEAKER_00: which is the right line here.
- 23:26SPEAKER_00: And so we can see this top k perturbs What it perturbs actually is looks near to
- 23:34SPEAKER_00: the ground truth.
- 23:36SPEAKER_00: So this is what we call the thick region.
- 23:38SPEAKER_00: That is,
- 23:39SPEAKER_00: once we have a good pre-train model,
- 23:42SPEAKER_00: and then we add noise into this pre-train model,
- 23:45SPEAKER_00: The model can have diverse behavior or diverse predictions of functions.
- 23:53SPEAKER_00: And the third region,
- 23:56SPEAKER_00: we call it the plateau region,
- 23:58SPEAKER_00: which is we predict Retrain the model on the linear functions and then do a test on
- 24:06SPEAKER_00: the linear functions.
- 24:07SPEAKER_00: So you can see the base model is actually very good or perfect at this task.
- 24:15SPEAKER_00: If you still add random perturbations onto it,
- 24:19SPEAKER_00: it will just go into some wrong directions and you can always select
- 24:26SPEAKER_00: the top K perturbs and you can always get a perfect But that just also didn't matter
- 24:33SPEAKER_00: because the pre-trained model is good enough and it's perfect for this task.
- 24:38SPEAKER_00: So this suggests that pre-training on a mixture signals may be essential.
- 24:45SPEAKER_00: And we regard pre-training more like learning a prior over this mix.
- 24:52SPEAKER_00: And post-training is just to pick out the specific tasks.
- 24:58SPEAKER_00: Another interesting thing is that-
- 25:02SPEAKER_01: I just want to go back to this.
- 25:04SPEAKER_01: This one was super interesting for me because you have the mess that is happening at
- 25:09SPEAKER_01: the 32B level, which is very hard to figure out.
- 25:13SPEAKER_01: Then you have these very small models that still exhibit this based
- 25:16SPEAKER_01: on your specific training regimen.
- 25:22SPEAKER_01: But I still don't get why they do happen when it's mixed.
- 25:29SPEAKER_01: Is it a similar kind of behavior where since you have all
- 25:32SPEAKER_01: of these different data points,
- 25:34SPEAKER_01: the base model will be at like a region in the weight that will be kind of okay
- 25:41SPEAKER_01: at everything but not an expert, right?
- 25:44SPEAKER_01: Just to maximize like the performance.
- 25:46SPEAKER_01: What's the rational here on the 1D specifically?
- 25:51SPEAKER_00: Yeah, so I think it's more about like the,
- 25:57SPEAKER_00: if we want to enter the CK region,
- 25:59SPEAKER_00: you need a good preaching model,
- 26:03SPEAKER_00: but good means that it have seen a huge amount of data,
- 26:09SPEAKER_00: like here we use the many different types of the functions,
- 26:17SPEAKER_00: and once you have that,
- 26:20SPEAKER_00: If you have good data and you have trained that good pre-trained model,
- 26:24SPEAKER_00: then you can enter the CKD region.
- 26:27SPEAKER_00: And it cannot happen if you do not have that or if you have a model is already
- 26:35SPEAKER_00: a special list on this task.
- 26:37SPEAKER_01: Right.
- 26:38SPEAKER_01: Okay.
- 26:39SPEAKER_01: So like this, I get,
- 26:41SPEAKER_01: but then, um,
- 26:42SPEAKER_01: I don't know, like, um,
- 26:43SPEAKER_01: if this is the case and it's like about like the data and like how you're, you're,
- 26:47SPEAKER_01: uh,
- 26:47SPEAKER_01: you're having good pre-training,
- 26:49SPEAKER_01: uh, like why are the AB model not entering that ticket reason, right?
- 26:54SPEAKER_01: Like, uh, they're very sparse.
- 26:55SPEAKER_01: Is it because like,
- 26:56SPEAKER_01: they're not seeing, um,
- 26:59SPEAKER_01: enough or like, yeah, here, here, like,
- 27:02SPEAKER_01: What's the gap between this small model that is very purpose-built,
- 27:06SPEAKER_01: that's seen enough,
- 27:07SPEAKER_01: and then this bigger model 32B that has,
- 27:11SPEAKER_01: they're entering that regime, and then the 8B in the middle that is still not there.
- 27:16SPEAKER_00: Basically, your question is more about the large language model size, right?
- 27:20SPEAKER_00: Like for the 0.5b model,
- 27:25SPEAKER_00: you think it already see a lot of data,
- 27:31SPEAKER_00: but you are wondering why the model size matters a lot?
- 27:35SPEAKER_01: Yeah, exactly.
- 27:35SPEAKER_01: Well, I mean,
- 27:36SPEAKER_01: it is connected to this smaller one because these are,
- 27:41SPEAKER_01: if I understand it correctly, these are not LN.
- 27:43SPEAKER_01: It's much smaller.
- 27:45SPEAKER_01: Toy model.
- 27:47SPEAKER_01: Yeah, these are some like MLP layers.
- 27:51SPEAKER_01: So it's something fundamental,
- 27:53SPEAKER_01: the ticket region, to what the model is seeing in the pre-training.
- 27:58SPEAKER_01: But then you have the white 8B,
- 28:01SPEAKER_01: or the smaller than that, that are not entering that region either.
- 28:05SPEAKER_01: So I'm just trying to figure out the connection here.
- 28:09SPEAKER_01: Why is the smaller LLM
- 28:18SPEAKER_01: Not entering that.
- 28:20SPEAKER_01: Is it a matter of like the way they were trained?
- 28:24SPEAKER_01: Just like the configuration of their parameters?
- 28:29SPEAKER_01: What do you think here?
- 28:31SPEAKER_00: Yeah, that's a very interesting question.
- 28:35SPEAKER_00: Actually, so here, in this toy experiment, we used the same size.
- 28:42SPEAKER_00: So the model size are the same.
- 28:45SPEAKER_00: So here we just figure out how the pre-trained data affects this CKD phenomenon.
- 28:55SPEAKER_00: Yes, as you said,
- 28:56SPEAKER_00: like for,
- 28:58SPEAKER_00: so I think my guess is that,
- 29:00SPEAKER_00: for example, those 0.5 billion model, they are so small.
- 29:06SPEAKER_00: And so if we train it on a huge amount of data and they will try,
- 29:11SPEAKER_00: They will try very hard to adapt to this data,
- 29:17SPEAKER_00: fitting this data, and try to minimize the loss.
- 29:21SPEAKER_00: And so somehow it will be,
- 29:25SPEAKER_00: Very sensitive and it's just like some students,
- 29:31SPEAKER_00: they are trained very hard,
- 29:33SPEAKER_00: but they are not capable for observing or to fit in all the data or all
- 29:43SPEAKER_00: the exams they will take.
- 29:47SPEAKER_00: But larger model may have a bigger capacity to maintain some subnets or
- 29:59SPEAKER_00: subsets of
- 29:59SPEAKER_00: the weights to solve one specific task.
- 30:03SPEAKER_00: So that's probably the reason why I think the smaller model cannot do it.
- 30:11SPEAKER_01: Right.
- 30:12SPEAKER_01: And based on this,
- 30:14SPEAKER_01: do you think that there's a smaller model that are trained from scratch and stuff,
- 30:21SPEAKER_01: do you think a distilled model would exhibit a different kind of landscape here?
- 30:28SPEAKER_00: I think, yeah.
- 30:29SPEAKER_00: So for example,
- 30:30SPEAKER_00: for clan families,
- 30:31SPEAKER_00: we all know that smaller models are distilled by other larger models.
- 30:41SPEAKER_00: I think,
- 30:42SPEAKER_00: for example,
- 30:43SPEAKER_00: the smaller model,
- 30:45SPEAKER_00: the 0.5 billion model and the 32 billion model,
- 30:49SPEAKER_00: they are all distilled from bigger models.
- 30:53SPEAKER_00: So they are under the same setting.
- 30:55SPEAKER_00: And I think it will be interesting to see what if we have a model that is trained
- 31:02SPEAKER_00: from scratch without any distillation from larger models.
- 31:07SPEAKER_00: See their behaviors.
- 31:09SPEAKER_00: But the QAM model families and the JAMA model families,
- 31:16SPEAKER_00: they are highly likely or distilled from huge models.
- 31:22SPEAKER_00: Yeah,
- 31:22SPEAKER_00: and here another interesting thing is that it also holds on other types of models,
- 31:30SPEAKER_00: for example,
- 31:30SPEAKER_00: diffusion model, if we do text to image diffusion models and then add some
- 31:39SPEAKER_00: And we can find it can change the colors or the styles of the images while
- 31:48SPEAKER_00: the image looks still okay or looks good, I think.
- 31:54SPEAKER_00: And one interesting phenomenon here is that some certain perturbation like
- 31:59SPEAKER_00: the random one, random two, They are consistent in the style or in the color.
- 32:07SPEAKER_00: For example,
- 32:08SPEAKER_00: the perturbation one seems this perturbation loves the color blue,
- 32:14SPEAKER_00: and the two likes the color white.
- 32:21SPEAKER_00: Seems they have the consistent styles that applies to different prompts here.
- 32:28SPEAKER_00: Okay, and then I would introduce the run-opt algorithm.
- 32:32SPEAKER_00: So we already see that The solution density is very high and we will have
- 32:41SPEAKER_00: a very good diversity if we add different perturbations into the model.
- 32:50SPEAKER_00: We have a very simple algorithm we named RAN OPT.
- 32:54SPEAKER_00: It's extremely simple that you just need to guess random Gaussian perturbations of
- 33:00SPEAKER_00: the weights and then pick the top K on a different value.
- 33:05SPEAKER_00: So you can just write the pseudocode in just a few lines.
- 33:24SPEAKER_00: First, select top case seats based on the performance on a validation dataset,
- 33:30SPEAKER_00: and then you assemble them.
- 33:32SPEAKER_00: It can just get very good performance.
- 33:37SPEAKER_00: Then I will show you some results here.
- 33:41SPEAKER_00: This is a figure 1c in our paper.
- 33:46SPEAKER_00: We can first see this part in the red box.
- 33:53SPEAKER_00: First, we tested it in the countdown task and we used the OMO3 7B Instruct model.
- 34:02SPEAKER_00: Let's first see this.
- 34:05SPEAKER_00: So we have three data points.
- 34:07SPEAKER_00: It's a one pass inference where k equals one.
- 34:11SPEAKER_00: So no ensemble here.
- 34:13SPEAKER_00: So it's just a random guessing for ranopt.
- 34:18SPEAKER_00: And these two lines,
- 34:20SPEAKER_00: The vertical line is the evolution strategy, and the blue line is the GRPO.
- 34:27SPEAKER_00: And you can see that after around 500 training steps,
- 34:33SPEAKER_00: the evolution strategy We can reach similar performance to the Reinopt.
- 34:40SPEAKER_00: And let's look at this part.
- 34:44SPEAKER_00: So this part is we tested on a set of 50 past inference.
- 34:51SPEAKER_00: So it means we do an ensemble with the top 50.
- 34:56SPEAKER_00: So then we can get around 85 percentage accuracy.
- 35:03SPEAKER_00: The purple dashed line and the blue dashed line is ES and GRPO, but also use the ensemble.
- 35:16SPEAKER_00: For example, you can use the checkpoints along the GRPO's trajectory and do an ensemble.
- 35:26SPEAKER_00: And it also helps a lot on the performance if you do ensemble to evolution strategy
- 35:32SPEAKER_00: or GRPO, but it still cannot reach the 85 performance of the Randopt.
- 35:42SPEAKER_00: And also Randopt have lower training flops because we just have one iteration.
- 35:50SPEAKER_01: Yeah, I think I will skip this.
- 35:53SPEAKER_00: And we also did a speed run to let the model to achieve 70% accuracy on countdown on eight.
- 36:06SPEAKER_00: And you can see
- 36:12SPEAKER_00: the run-opt can have both lower work-lock time
- 36:16SPEAKER_00: and lower total compute regarding the training flops.
- 36:20SPEAKER_00: So we also run it on 200 H100 GPUs.
- 36:25SPEAKER_00: So under this setting, we can just use 3.2 minutes to get
- 36:32SPEAKER_00: To get a 70% accuracy on countdown.
- 36:34SPEAKER_00: And if you use a PPO, you need more than 10 hours to get that.
- 36:41SPEAKER_00: And there can be some issues or concerns around that.
- 36:47SPEAKER_00: For example, you may be wondering, is it just a random noise fixing the format?
- 36:53SPEAKER_00: We tested it.
- 36:55SPEAKER_00: And we find that, yes,
- 36:58SPEAKER_00: adding random noise can sometimes just fix the format of those measuring data sets.
- 37:05SPEAKER_00: And we also find if you use GRPO,
- 37:08SPEAKER_00: you will also have We have a large amount of portion that is improved by the format.
- 37:15SPEAKER_00: But there is still the case that is around 16.7 percent.
- 37:24SPEAKER_00: This is a percentage accuracy that is improved by purely you correct the answer,
- 37:30SPEAKER_00: but not just by correcting the format for an opt.
- 37:34SPEAKER_00: Also, you may wonder, is it sandbagging phenomenon?
- 37:40SPEAKER_00: This phenomenon is introduced by a paper last year, I think.
- 37:44SPEAKER_00: They show that nowadays the companies are releasing models that are So good
- 37:51SPEAKER_00: and they want the model to be not that good and they have some techniques like
- 37:59SPEAKER_00: to inject some special recipes.
- 38:08SPEAKER_00: And
- 38:13SPEAKER_00: this can be recovered by adding noise.
- 38:16SPEAKER_00: But actually,
- 38:18SPEAKER_00: we find this is not a case for the CK phenomenon because we also tested on
- 38:23SPEAKER_00: the OMO model, which is a fully transparent model.
- 38:27SPEAKER_00: And you can check the training code in their code base.
- 38:30SPEAKER_00: It didn't have these techniques.
- 38:33SPEAKER_01: Could you just go back to where we see like if it's a formatting and like
- 38:40SPEAKER_01: the formatting aspect has a big impact in the DRPO.
- 38:48SPEAKER_01: If I understand this correctly,
- 38:50SPEAKER_01: the dash red is the regression in the post-train model that are introduced,
- 38:57SPEAKER_01: but they still have some gain back.
- 38:59SPEAKER_01: So you lose a lot and then you gain back based on the baseline.
- 39:05SPEAKER_01: But it seems like a run opt with 50 has almost no regression and then has
- 39:12SPEAKER_01: the formatting boost from From GRPO,
- 39:16SPEAKER_01: was the GRPO also at 50, or was it lower than that, in different groups?
- 39:20SPEAKER_00: The question is basically about why the ensemble of the run-ups have a
- 39:27SPEAKER_01: lower... Regression, yes.
- 39:30SPEAKER_01: Because it's almost like you're just taking the best of all of the different elements.
- 39:37SPEAKER_01: And there's basically no regression here,
- 39:40SPEAKER_01: which if you compare to GRP,
- 39:41SPEAKER_01: which is not the case, like with GRP, there is some regression that you get here.
- 39:46SPEAKER_01: And then you get the bonus from mainly the formatting aspect.
- 39:49SPEAKER_01: But here you get the bonus from the reasoning that is a bit better.
- 39:53SPEAKER_01: And the same thing as the GRP with the formatting.
- 39:55SPEAKER_00: Ensample helps because it reduces the risk of relying on one perturbation.
- 40:05SPEAKER_00: I think a single perturbation may improve one task but hurt another
- 40:09SPEAKER_00: and also this This is the cause of the regression here.
- 40:15SPEAKER_00: But if you ensemble many selected perturbations,
- 40:18SPEAKER_00: the useful signal can add up while some of the regressions cancel out.
- 40:25SPEAKER_00: Simply sampling the base model 50 times can help too.
- 40:31SPEAKER_00: And we also have those results in our paper.
- 40:36SPEAKER_01: Yeah.
- 40:37SPEAKER_01: It looked a bit like some inference time kind of scaling methodology
- 40:42SPEAKER_01: where you can sample multiple times.
- 40:45SPEAKER_01: Have you benchmarked in a similar way how different it is from benchmarking it
- 40:53SPEAKER_01: from sampling,
- 40:55SPEAKER_01: let's say,
- 40:56SPEAKER_01: 50 times in the output space versus the wait space and comparing what's
- 41:01SPEAKER_01: the difference between these two and maybe assembling the output here?
- 41:05SPEAKER_01: I think you call this like the TTMV baseline in the paper, right?
- 41:11SPEAKER_00: Oh yes, we have the baseline in the appendix table 4, I think.
- 41:16SPEAKER_00: It's a TTMV baseline.
- 41:18SPEAKER_00: So TTMV is the test time majority vote.
- 41:21SPEAKER_00: So it's basically just doing ensemble to the base model.
- 41:26SPEAKER_00: So it's a baseline we want to compare with because we are doing random guessing and
- 41:31SPEAKER_00: then do ensemble.
- 41:33SPEAKER_00: So that baseline is we didn't do random guessing,
- 41:37SPEAKER_00: we just do Ensemble on the base model and we find that the performance is worse than
- 41:45SPEAKER_00: the Reynolds because this also should suggest that we need the diversity for
- 41:52SPEAKER_00: the ensemble.
- 41:53SPEAKER_00: So you may also wonder if this is just because the batch size
- 41:58SPEAKER_00: or group size is not large enough.
- 42:00SPEAKER_00: Because usually if you have a larger batch or group size, you can have better performance.
- 42:06SPEAKER_00: And so here in the paper,
- 42:09SPEAKER_00: in the appendix figure,
- 42:10SPEAKER_00: we did a sweep across the batch or group size and tested on the GSMA-K dataset
- 42:19SPEAKER_00: for Our finding is that no larger batch or group size does not always lead
- 42:28SPEAKER_00: to better performance.
- 42:30SPEAKER_00: You can see here is a trade-off here.
- 42:32SPEAKER_00: If you look at the green circle here,
- 42:35SPEAKER_00: So if you increase the group size for GRPO,
- 42:39SPEAKER_00: you can increase the performance, but you cannot always do that.
- 42:45SPEAKER_00: But for the random opt, we didn't have the batch size or group size.
- 42:52SPEAKER_00: We didn't have this hyperparameter.
- 42:53SPEAKER_00: We just have the population size.
- 42:55SPEAKER_00: But you can see, we also try different learning rates here.
- 43:01SPEAKER_00: You can see,
- 43:02SPEAKER_00: no matter how we tune the hyperparameter,
- 43:05SPEAKER_00: The parameters through an opt is always better than those algorithms.
- 43:11SPEAKER_00: And you may also wonder, isn't the magic just the ensemble?
- 43:15SPEAKER_00: Also, this is a,
- 43:16SPEAKER_00: I think it's a figure,
- 43:18SPEAKER_00: like a figure, maybe figure eight or figure nine in the appendix.
- 43:24SPEAKER_00: And we show that if you just do random guessing,
- 43:27SPEAKER_00: you do not do The ensemble, it can already improve the base model performance a lot.
- 43:33SPEAKER_00: And ensemble also helps a lot.
- 43:36SPEAKER_00: Actually, the ensemble part introduced a major limitation.
- 43:40SPEAKER_00: So you need basically,
- 43:42SPEAKER_00: if we use And then you need 50 more times for passes during the inference.
- 43:54SPEAKER_00: And then we introduce the distillation solution In the paper,
- 43:59SPEAKER_00: I think it's section 8 or section 9.
- 44:03SPEAKER_00: And basically,
- 44:04SPEAKER_00: we just first filter and keep only hard examples and use the top 50 models
- 44:12SPEAKER_00: to generate response So we can get responses on those hard examples and
- 44:15SPEAKER_00: then we distill this into a single model so it can remain the performance very well
- 44:22SPEAKER_00: and then we can just get a single model, we do not need to do an ensemble.
- 44:28SPEAKER_00: Also, there is a follow-up paper I think came out last week.
- 44:35SPEAKER_00: They observed that run-op's top-k perturbations share a low-rank structure across tasks.
- 44:42SPEAKER_00: So they designed a compatibility-aware and
- 44:48SPEAKER_00: reward-weighted strategy in
- 44:50SPEAKER_00: the shared low-rank sub-base instead of assembling.
- 44:55SPEAKER_00: You can
- 44:58SPEAKER_00: also use their way to get rid of assembling and just use a single model.
- 45:03SPEAKER_01: It seems to be, I don't know, a bit similar in some shape or form to self-distillation, right?
- 45:10SPEAKER_01: But at this point,
- 45:11SPEAKER_01: you're using the distribution of parameter space around the pre-training way,
- 45:18SPEAKER_01: the teacher in some way, right?
- 45:20SPEAKER_01: And then the SFC on the base model.
- 45:23SPEAKER_01: Consolidating rewarded perturbations for post training.
- 45:38SPEAKER_01: When you're doing the distillation on this distribution,
- 45:43SPEAKER_01: what happened to this weight space of the distant model afterward?
- 45:49SPEAKER_01: Did it just level up the whole thing and now it's not as spiky with the expert around?
- 45:55SPEAKER_01: Have you tried to visualize that part that you did at the start?
- 45:58SPEAKER_00: I didn't look into that, but I think that's very interesting.
- 46:03SPEAKER_00: And I would say distillation compresses the behavior of the selected experts back
- 46:09SPEAKER_00: into a single model.
- 46:11SPEAKER_00: And the ensemble contains many different specialists, but it's
- 46:18SPEAKER_00: So distillation gives us a way to transfer that ensemble behavior into one model.
- 46:25SPEAKER_00: But I would not say distillation fully levels the basin around the base model.
- 46:31SPEAKER_00: We have not directly shown that.
- 46:33SPEAKER_00: But I think it does combine the behavior of the multiple nearby expert into one.
- 46:39SPEAKER_00: And I think it will be very interesting to explore that,
- 46:42SPEAKER_00: like how to leverage the basins around the base model.
- 46:46SPEAKER_01: I think it will be very interesting to visualize this because like if you look at
- 46:51SPEAKER_01: the different possibility here, it could level the thing, right?
- 46:54SPEAKER_01: It could make like every different perturbation kind of the same around this
- 47:00SPEAKER_01: or it can make it like it can kind of I destroyed the ticket.
- 47:05SPEAKER_01: Maybe the ticket doesn't exist anymore.
- 47:07SPEAKER_01: And then it looked more like the AD model now, which are sparser.
- 47:13SPEAKER_01: The performance will be better,
- 47:14SPEAKER_01: but this will be sparser because that point is the point where
- 47:19SPEAKER_01: the information is all distilled around.
- 47:24SPEAKER_01: It will be pretty interesting to see that result.
- 47:27SPEAKER_00: In the paper,
- 47:29SPEAKER_00: we show that Randopt can achieve similar performance as standard approach like PPO,
- 47:35SPEAKER_00: GRPO, or ES.
- 47:37SPEAKER_00: And if we want to just achieve very high accuracy,
- 47:41SPEAKER_00: we think there is There's no clear winner
- 47:44SPEAKER_00: when giving all methods k test time samples.
- 47:48SPEAKER_00: Because if you enter the CK region, maybe it doesn't really matter which algorithm you use.
- 47:58SPEAKER_00: Maybe they can just work.
- 47:59SPEAKER_00: And if you want fast wall clock training, runoff is a great choice, I think.
- 48:05SPEAKER_00: And if you want fast inference,
- 48:07SPEAKER_00: you better distill it or use the strategy I mentioned in It's not just a crane,
- 48:14SPEAKER_00: similar effects also happen on Lama, Omo, and also Lama.
- 48:27SPEAKER_00: So the Gemma 4.
- 48:29SPEAKER_00: And it's not due to instruct tuning.
- 48:32SPEAKER_00: Similar effects also happens on the OMO bass model.
- 48:36SPEAKER_00: So we also tested a OMO bass model.
- 48:39SPEAKER_00: And all the other models are in Instruction tuned model.
- 48:44SPEAKER_01: So that's pretty cool.
- 48:47SPEAKER_01: But given this whole setup,
- 48:52SPEAKER_01: a good post-training methodology here would be to kind of do roundup,
- 48:56SPEAKER_01: and then you distill it into a model,
- 48:58SPEAKER_01: and then you use that model at inference, right?
- 49:00SPEAKER_00: Yep.
- 49:01SPEAKER_01: Because if I understand correctly,
- 49:03SPEAKER_01: the perturbations are kind of spread throughout the whole model,
- 49:05SPEAKER_01: or are they specific to different layers?
- 49:08SPEAKER_00: We apply the perturbations to the whole model, so all the parameters.
- 49:12SPEAKER_01: Do you think there is some layer that matter more to perturb than others or like
- 49:19SPEAKER_01: the effect throughout?
- 49:20SPEAKER_01: Yeah, that's my point.
- 49:23SPEAKER_01: Is it like the earlier layer?
- 49:24SPEAKER_01: Is it like specific segment over there that matter more?
- 49:28SPEAKER_01: Have you kind of looked into that part?
- 49:31SPEAKER_00: Actually,
- 49:32SPEAKER_00: I didn't look into it,
- 49:35SPEAKER_00: but I guess maybe some expert like they lie in different layers or maybe it's like
- 49:46SPEAKER_00: a More global control over the four parameters and that can all happen.
- 49:55SPEAKER_00: And yeah, I think this is a very good directions for the future.
- 50:02SPEAKER_00: And also one observation I have is that if I just perturb some certain layers like
- 50:11SPEAKER_00: the MLP layers and also only perturb the attention layers,
- 50:18SPEAKER_00: it can still work but just hurt a little bit of the performance.
- 50:22SPEAKER_01: Right, right, right, right.
- 50:24SPEAKER_01: I think, like,
- 50:24SPEAKER_01: with the paper that you shared,
- 50:26SPEAKER_01: I think it kind of makes sense that, like,
- 50:28SPEAKER_01: it is low rank in nature and you don't need to perturb all,
- 50:31SPEAKER_01: but you can perturb, like, some part of it.
- 50:33SPEAKER_01: I should read it a bit in deeper detail because they say
- 50:37SPEAKER_01: that they're using one-tenth of their random perturbation budget
- 50:40SPEAKER_01: and exceed single inference by whatever point.
- 50:43SPEAKER_01: It seems to go into that direction of, like,
- 50:46SPEAKER_01: not having to perturb everything, but just, like, a specific sub-portion of it.
- 50:50SPEAKER_01: Right now, we have these models that get outputted, right?
- 50:52SPEAKER_01: And it's their position in the weight landscape that is the model.
- 51:00SPEAKER_01: But I think your philosophy is a bit different here.
- 51:02SPEAKER_01: I think you started with This philosophy of having like a distribution of weight,
- 51:08SPEAKER_01: that's the artifact, the model that makes more sense, right?
- 51:14SPEAKER_01: And then you can sample a class.
- 51:18SPEAKER_01: Do you think we should be,
- 51:20SPEAKER_01: when we like get these models and like finalize the training and stuff,
- 51:24SPEAKER_01: Map around them what's going on and have these different flavors.
- 51:31SPEAKER_01: What's your thought here?
- 51:32SPEAKER_01: Because if we have to figure this out,
- 51:38SPEAKER_01: where the experts are,
- 51:39SPEAKER_01: you have to run the whole battery of tests with the different benchmark.
- 51:45SPEAKER_01: So what's your thought process here for how we should be
- 51:50SPEAKER_01: Like releasing models now if we take more of a distribution perspective.
- 51:55SPEAKER_00: I think in principle, yes,
- 51:57SPEAKER_00: if we can map the,
- 51:59SPEAKER_00: for example,
- 52:00SPEAKER_00: if we can map the basin around the model,
- 52:03SPEAKER_00: it can help us understand better which directions maybe can improve which task.
- 52:11SPEAKER_00: But I just think in practice,
- 52:13SPEAKER_00: fully mapping
- 52:17SPEAKER_00: the way space or the accuracy landscape is probably very hard.
- 52:23SPEAKER_00: That's because the parameter space is just too And also the landscape depends on the task.
- 52:30SPEAKER_00: So I think instead of trying to map everything,
- 52:35SPEAKER_00: we may need some probes,
- 52:36SPEAKER_00: for example,
- 52:37SPEAKER_00: random sampling,
- 52:38SPEAKER_00: layer-wise perturbation,
- 52:40SPEAKER_00: And also,
- 52:47SPEAKER_00: I think it's very important to analyze the solution overlap among different tasks.
- 52:54SPEAKER_00: So this can help us.
- 52:56SPEAKER_00: So we have to know for a certain task which direction we want to go
- 53:02SPEAKER_00: and this will also hopefully can
- 53:07SPEAKER_00: find some certain layers or certain parts in
- 53:11SPEAKER_00: the weights that is
- 53:16SPEAKER_00: responsible for some tasks and that if we can
- 53:22SPEAKER_00: And then we maybe can even find a generalist by adding noise to different parts of the model.
- 53:30SPEAKER_01: Right, right, right.
- 53:32SPEAKER_01: I mean, if you're distilling at the end of the day,
- 53:35SPEAKER_01: to inference it, it doesn't matter too much, right?
- 53:38SPEAKER_01: You're going to be able to have a much better generalist model
- 53:42SPEAKER_01: in these different benchmarks that make sense.
- 53:45SPEAKER_01: I was thinking along this line about like,
- 53:51SPEAKER_01: would routing,
- 53:52SPEAKER_01: just routing the different queries and stuff like that,
- 53:55SPEAKER_01: be better you think than just like straight up assembling,
- 53:58SPEAKER_01: let's say,
- 53:59SPEAKER_01: or like by Because let's say you know this model is good at math or writing
- 54:05SPEAKER_01: because you benchmark it ahead of time.
- 54:08SPEAKER_01: You receive a query,
- 54:09SPEAKER_01: you're able to figure out is the math query,
- 54:12SPEAKER_01: is the writing query, and you route it to the right expert by category.
- 54:18SPEAKER_01: Strictly speaking,
- 54:19SPEAKER_01: are we going to be in the same regime as Ensembling or
- 54:24SPEAKER_01: the distilling version will be covering that part, you think?
- 54:29SPEAKER_00: Yeah, I think the things you mentioned sounds very like a mixture of expert stuff,
- 54:36SPEAKER_00: like you have some gates that can control which subset or which model you are using.
- 54:45SPEAKER_00: And this also reminds me, like,
- 54:50SPEAKER_00: I think it's a very,
- 54:51SPEAKER_00: it can be a very interesting direction to explore the ranopt with
- 54:56SPEAKER_00: a continued learning setting because other...
- 54:59SPEAKER_00: Yeah, under the setting you mentioned,
- 55:02SPEAKER_00: we can find a better way to enable the model to adapt very fast to different tasks
- 55:10SPEAKER_00: and hopefully can reduce the forgetting problems.
- 55:14SPEAKER_01: Yeah, it's kind of like a mixture of perturbation,
- 55:16SPEAKER_01: where you have a router in there that can figure out
- 55:22SPEAKER_01: Roughly speaking, what type of perturbation, because it's the same model.
- 55:25SPEAKER_01: You don't have to keep a copy of everything.
- 55:26SPEAKER_01: You just have to know what perturbation you need to do.
- 55:29SPEAKER_01: And then depending on whatever this task is,
- 55:31SPEAKER_01: you do that perturbation and you're going to just shift your weight toward
- 55:35SPEAKER_01: that expert for now.
- 55:37SPEAKER_01: And then you can move around a bit like this.
- 55:42SPEAKER_01: That would be so cool to do.
- 55:43SPEAKER_01: And then it will get closer to adding that artifact being not just
- 55:49SPEAKER_01: a fixed weight model, but more of this distribution in the landscape.
- 55:57SPEAKER_01: That's literally what it is.
- 56:00SPEAKER_01: It is a distribution of stuff.
- 56:04SPEAKER_01: Okay, this is really cool.
- 56:05SPEAKER_01: I have a question about scale.
- 56:08SPEAKER_01: What's the biggest that you've tested here?
- 56:10SPEAKER_01: I think it was 32B, right?
- 56:11SPEAKER_01: Yes, it's 32B.
- 56:14SPEAKER_00: Do you think?
- 56:15SPEAKER_01: I don't know.
- 56:16SPEAKER_01: What do you think will happen at 400B and stuff?
- 56:20SPEAKER_01: Do you think it's even more densely packed, a bunch of experts?
- 56:24SPEAKER_01: Is it even possible, feasible to run this experiment on the 400B?
- 56:27SPEAKER_01: What's your thought here?
- 56:29SPEAKER_00: I strongly believe that if we have 400B or even bigger models,
- 56:34SPEAKER_00: the CK phenomenon will be clear,
- 56:39SPEAKER_00: and the solution density will be higher than the 32B model.
- 56:44SPEAKER_00: And it will be much, much more easier to do post training.
- 56:50SPEAKER_00: And you can find many good solutions by just random guessing.
- 56:55SPEAKER_00: And of course, I think we can run this on the...
- 57:02SPEAKER_00: So relatively, it's much cheaper than four-chain networks.
- 57:17SPEAKER_00: Like people post-training the models using RL, RenOpt will be much more cheaper.
- 57:24SPEAKER_00: Also,
- 57:25SPEAKER_00: you do not need to store solvents of four models if perturbations are generated
- 57:30SPEAKER_00: from RenOpt.
- 57:31SPEAKER_00: We just need to restore the random seats.
- 57:34SPEAKER_00: That's very true.
- 57:35SPEAKER_01: Okay, nice.
- 57:36SPEAKER_01: Are you planning on testing that out or no?
- 57:38SPEAKER_01: I'm very curious about that part.
- 57:39SPEAKER_01: I guess we know that these bigger models are much more gigantic even than the 400B.
- 57:44SPEAKER_01: But seeing the progression from the smaller 8 to the 32,
- 57:50SPEAKER_01: I'm very curious to see what happens when you have such a massive model,
- 57:55SPEAKER_01: like an order money to the bigger.
- 57:57SPEAKER_00: Yeah.
- 57:58SPEAKER_00: I think maybe I will find some time to try and maybe
- 58:02SPEAKER_00: the accuracy landscape will just looks like a whole red region.
- 58:08SPEAKER_01: Yeah.
- 58:09SPEAKER_01: I had a question about the connection from this to continual learning.
- 58:12SPEAKER_01: I think we alluded to it a bit,
- 58:15SPEAKER_01: but do you have an opinion about how this type of algorithm or distillation can link
- 58:23SPEAKER_01: to continual learning and enable it in some way?
- 58:27SPEAKER_00: Yeah, I think it's very interesting to think about continued learning.
- 58:32SPEAKER_00: I believe runoff can help in continued learning.
- 58:36SPEAKER_00: And some interesting aspects, maybe like in continued learning, the circuit is no longer fixed.
- 58:42SPEAKER_00: As a model changes, I mean,
- 58:44SPEAKER_00: like model will change to some task expert to solve the tasks
- 58:49SPEAKER_00: in your learning setting.
- 58:51SPEAKER_00: And as a model change, the local neighborhood will also change.
- 58:57SPEAKER_00: I think one interesting idea is that and also very straightforward is
- 59:02SPEAKER_00: to repeatedly sample the expert around the base model and then select the useful one
- 59:07SPEAKER_00: for the new tasks and then you just need to repeat the progress and to do it in
- 59:14SPEAKER_00: the continued learning setting.
- 59:19SPEAKER_00: I think you can just imagine the loop like sample nearby experts,
- 59:26SPEAKER_00: ensemble them,
- 59:27SPEAKER_00: use the ensemble as a teacher,
- 59:29SPEAKER_00: distill it into a student if we want to relate this to
- 59:34SPEAKER_00: the self-distillation pipeline.
- 59:40SPEAKER_00: And yeah, and then we just repeat from the new students and sample them.
- 59:46SPEAKER_01: Yeah.
- 59:46SPEAKER_01: That's kind of what I was thinking and like,
- 59:49SPEAKER_01: how's it like with kind of this whole self distillation loop,
- 59:52SPEAKER_01: maybe you don't have to do all this work prior to figure out where the And
- 59:55SPEAKER_01: then based on that, you can, you, you, you, you, you, you, you, you, you, you,
- 1:00:10SPEAKER_01: You know a bit more, okay, this one is like this expert for that.
- 1:00:15SPEAKER_01: I don't know.
- 1:00:16SPEAKER_01: It seems like there is a discovery of the skill set that is around that model, right?
- 1:00:25SPEAKER_01: And it's kind of discovering like these different tools that is using it.
- 1:00:30SPEAKER_01: And at some point,
- 1:00:30SPEAKER_01: there's maybe some consolidation that should be happening with all of
- 1:00:35SPEAKER_01: the different skill sets to kind of elevate.
- 1:00:37SPEAKER_01: And this is why I'm saying that it would be very interesting to see what What
- 1:00:40SPEAKER_01: the landscape look like around the self-distilled renda because
- 1:00:46SPEAKER_01: if it becomes drastically sparser,
- 1:00:50SPEAKER_01: I mean, it has some implication toward this whole loop.
- 1:00:53SPEAKER_01: If it's still the wiggle, you still have the same overall shape.
- 1:00:59SPEAKER_01: I mean, it can keep continuing like this.
- 1:01:01SPEAKER_01: And maybe at some point you saturate and like whatever you are in there is good enough, right?
- 1:01:06SPEAKER_01: But maybe it's like a signal for like,
- 1:01:09SPEAKER_01: yeah, it's time to explore like a different type of tasks which are harder, right?
- 1:01:16SPEAKER_01: And then you have kind of this auto-curriculum Going on,
- 1:01:20SPEAKER_01: because you're sampling the neighborhood, right?
- 1:01:23SPEAKER_01: And the neighborhood is telling you if there's any gain in continuing what you're seeing.
- 1:01:28SPEAKER_01: I'm pretty sure if you give a very hard task,
- 1:01:31SPEAKER_01: a benchmark to these models, it will just look like low accuracy everywhere, right?
- 1:01:36SPEAKER_01: But as they gain more capacity,
- 1:01:39SPEAKER_01: they will be able to do some of these and then the landscape will look
- 1:01:42SPEAKER_01: a bit different.
- 1:01:43SPEAKER_01: But then if you continue the subdivision loop,
- 1:01:47SPEAKER_01: I get the intuition that It seems like there's a direct connection
- 1:01:51SPEAKER_01: toward
- 1:01:54SPEAKER_01: continual learning by properly wrapping this landscape exploration.
- 1:02:01SPEAKER_01: Anyway, I'm super interested about this stuff.
- 1:02:05SPEAKER_00: I agree.
- 1:02:05SPEAKER_00: I think we already have many very good directions to explore,
- 1:02:11SPEAKER_00: and you already have specific plans and strategies to achieve that.
- 1:02:20SPEAKER_01: Okay, this is good.
- 1:02:21SPEAKER_01: You mentioned the Baldwin effect framing here.
- 1:02:24SPEAKER_01: I think in the conclusion and stuff, I really like that.
- 1:02:28SPEAKER_01: Could you say more about your thinking about this effect?
- 1:02:32SPEAKER_01: I think that is biological toward these models.
- 1:02:37SPEAKER_00: That's also I think one of my favorite parts in the paper to do this analogy
- 1:02:44SPEAKER_00: because you know I'm very interested in evolution and that's also
- 1:02:49SPEAKER_00: the reason why I get into this research line by doing evolution strategies last year
- 1:02:56SPEAKER_00: and I think it began like many years ago like Just being so excited
- 1:03:02SPEAKER_00: about learning evolution stuff.
- 1:03:05SPEAKER_00: And yeah,
- 1:03:07SPEAKER_00: the borrowing effort is a very nice analogy,
- 1:03:11SPEAKER_00: I think,
- 1:03:12SPEAKER_00: because it says evolution may not directly It can simply encode every behavior.
- 1:03:17SPEAKER_00: Instead, it can select for organisms that are good at learning during their lifetime.
- 1:03:25SPEAKER_00: And I think for large language models,
- 1:03:28SPEAKER_00: pre-training may play a very similar role for neural network.
- 1:03:35SPEAKER_00: Pre-training may not directly solve every downstream task,
- 1:03:39SPEAKER_00: but it can put the model in a position where many downstream solutions are nearby
- 1:03:46SPEAKER_00: and very easy to reach.
- 1:03:49SPEAKER_00: Actually,
- 1:03:50SPEAKER_00: the pre-trained model is more like they are trying to find a good initialization for
- 1:03:57SPEAKER_00: the different tasks, and then we do post-training to select them.
- 1:04:02SPEAKER_00: So it's very similar to the Baldwin effect.
- 1:04:09SPEAKER_00: And it's also very similar and related to another research line, the meta-learning stuff.
- 1:04:17SPEAKER_00: Many years ago,
- 1:04:18SPEAKER_00: people studied and proposed methods around the meta-learning and to make
- 1:04:26SPEAKER_00: the pre-trained model
- 1:04:30SPEAKER_00: It's more like a good initialization rather than a specific solution for some tasks.
- 1:04:37SPEAKER_01: Yeah,
- 1:04:37SPEAKER_01: I think it's beautiful because I'm getting very excited about self-disciplination,
- 1:04:43SPEAKER_01: but just like cultural learning in general,
- 1:04:45SPEAKER_01: because it seems that it's very, very likely to be
- 1:04:54SPEAKER_01: This whole paper has almost looked like a setup for this to happen.
- 1:05:00SPEAKER_01: You have the model that is in the goods.
- 1:05:02SPEAKER_01: If it's big enough,
- 1:05:04SPEAKER_01: you have this other capacity and it's closer to being able to be able
- 1:05:11SPEAKER_01: to use these different skill sets.
- 1:05:14SPEAKER_01: The only thing that it takes is just Random perturbation and being able to know
- 1:05:19SPEAKER_01: which perturbation is good or not, right?
- 1:05:20SPEAKER_01: Because when you're doing random, right?
- 1:05:23SPEAKER_01: Distillation, you already map the 500 expert, right?
- 1:05:26SPEAKER_01: If I see it properly.
- 1:05:29SPEAKER_01: So you already map them, right?
- 1:05:31SPEAKER_01: It seems that there's a way to kind of...
- 1:05:35SPEAKER_01: Kind of self-evolve,
- 1:05:36SPEAKER_01: not just the model on a specific benchmark or anything like that,
- 1:05:41SPEAKER_01: but self-evolve the model capability to understand its landscape.
- 1:05:46SPEAKER_01: Because if it is able to understand the different Because now we're a bit more meta
- 1:05:51SPEAKER_01: at this point.
- 1:05:52SPEAKER_01: It's not about the model weight here.
- 1:05:55SPEAKER_01: The model is the landscape around the weights.
- 1:05:59SPEAKER_01: If it's able to learn and understand the weights and evolve to be able
- 1:06:03SPEAKER_01: to use them properly,
- 1:06:05SPEAKER_01: Depending on what type of benchmark that matters at this point in time,
- 1:06:10SPEAKER_01: it seems that logically the continual learning aspect is going to be a given.
- 1:06:16SPEAKER_01: Because then you're going to be able to use whatever and at some point kind
- 1:06:19SPEAKER_01: of compress and adjust all of the information into the weight.
- 1:06:25SPEAKER_01: The last question I wanted to...
- 1:06:28SPEAKER_01: To get your point,
- 1:06:30SPEAKER_01: I think you've done some experiment over there about
- 1:06:37SPEAKER_01: task that was too novel,
- 1:06:39SPEAKER_01: right?
- 1:06:41SPEAKER_01: What do you think about that?
- 1:06:44SPEAKER_01: Being able to discover novelty just by the random exploration of the weight.
- 1:06:51SPEAKER_01: Do you think that this is possible in this regime or is still kind of bounded
- 1:06:54SPEAKER_01: to what it's been seeing in the pre-training?
- 1:06:58SPEAKER_00: Yeah, so the question is about like does a random perturbation unlock very new skills, right?
- 1:07:06SPEAKER_00: So I think I would be careful here.
- 1:07:09SPEAKER_00: I do not think the paper,
- 1:07:12SPEAKER_00: in our paper,
- 1:07:12SPEAKER_00: What seems to happen is that I think different perturbations change the model's
- 1:07:29SPEAKER_00: And some perturbations make the model more likely to follow the right format,
- 1:07:34SPEAKER_00: and some makes it better at a particular reasoning pattern,
- 1:07:39SPEAKER_00: and some may recover answers that the base model So I think since
- 1:07:46SPEAKER_00: the pre-trained model have seen so many data,
- 1:07:49SPEAKER_00: so it's very hard to train, sorry,
- 1:07:52SPEAKER_00: so it's very hard to say if both the RAN opts and those RAN opts RL algorithms
- 1:07:59SPEAKER_00: or evolution strategy algorithms unlock brand new skills.
- 1:08:05SPEAKER_00: Yeah, maybe one hypothesis is that, okay, the pre-trained model just see all of the data.
- 1:08:12SPEAKER_00: So that's the reason why.
- 1:08:12SPEAKER_00: That's why we can do post-train very easily.
- 1:08:16SPEAKER_00: But I think I would be very interested to explore more about if using Renopt
- 1:08:25SPEAKER_00: or ES can unlock new The skills that RL or other algorithms can't.
- 1:08:31SPEAKER_00: But now what we report is more like the accuracy on the task.
- 1:08:37SPEAKER_00: And people say it's an additional dataset that people didn't, sorry,
- 1:08:43SPEAKER_00: The model didn't see it in the pre-training,
- 1:08:45SPEAKER_00: but I think the model have seen some relevant or similar data set,
- 1:08:51SPEAKER_00: so it's not guaranteed.
- 1:08:55SPEAKER_00: It's a very new skill that learned in the post-training, I think.
- 1:09:00SPEAKER_01: So yeah, I just want to know to finish, what's next on your side and this direction overall?
- 1:09:08SPEAKER_01: I asked you that with evolutionary strategy and it was very interesting
- 1:09:13SPEAKER_01: because you were mentioning this research.
- 1:09:15SPEAKER_01: So what's next on your plate here?
- 1:09:18SPEAKER_00: I think based on evolution strategy and run-off, there are so many interesting directions.
- 1:09:25SPEAKER_00: And I think for me,
- 1:09:28SPEAKER_00: the next step,
- 1:09:30SPEAKER_00: I'm excited about some For example,
- 1:09:35SPEAKER_00: I think first,
- 1:09:36SPEAKER_00: the underlying mechanisms,
- 1:09:39SPEAKER_00: like where do SIGITs actually live,
- 1:09:43SPEAKER_00: and are they in certain layers, heads, MLPs, or some lower rank directions?
- 1:09:49SPEAKER_00: And another thing I think is about the algorithm,
- 1:09:54SPEAKER_00: like can we make the run-out much cheaper using better sampling strategies
- 1:10:00SPEAKER_00: and also leverage the low-rank searching routine or distillation or other methods.
- 1:10:08SPEAKER_00: And so I think I'm also...
- 1:10:11SPEAKER_00: I feel excited about the pre-training part.
- 1:10:14SPEAKER_00: Like, can we train the model from scratch using the yacine or the ranopt?
- 1:10:24SPEAKER_00: But that sounds crazy.
- 1:10:25SPEAKER_00: And I think the more practical thing is that maybe,
- 1:10:31SPEAKER_00: like, can we train the model to have thicker and more useful local neighborhoods?
- 1:10:37SPEAKER_00: And maybe we can introduce this objective into the pre-training and that can help.
- 1:10:44SPEAKER_00: And I also think continued learning is very exciting.
- 1:10:47SPEAKER_00: Like instead of maintaining one model,
- 1:10:49SPEAKER_00: maybe we should maintain an evolving population,
- 1:10:52SPEAKER_00: just as you said,
- 1:10:53SPEAKER_00: an evolving population of the nearby experts and we can distill them
- 1:10:58SPEAKER_00: or we can just keep them.
- 1:11:01SPEAKER_00: Yeah, so I think in this paper,
- 1:11:05SPEAKER_00: Renopt is just a probe and there are so many interesting directions to be explored.
- 1:11:10SPEAKER_01: It's great.
- 1:11:11SPEAKER_01: For me,
- 1:11:12SPEAKER_01: the one thing that I was excited about the paper is that it's very exploratory in
- 1:11:18SPEAKER_01: the sense of there's very few biases in this paper.
- 1:11:22SPEAKER_01: Evolution strategy is now just one of the
- 1:11:27SPEAKER_01: It's not that thing.
- 1:11:29SPEAKER_01: So very low biases.
- 1:11:30SPEAKER_01: And then you even went into the fundamental of it with the 1D.
- 1:11:36SPEAKER_01: So that was very beautiful to see.
- 1:11:38SPEAKER_01: But in my view,
- 1:11:40SPEAKER_01: it can connect very easily to the pre-training,
- 1:11:44SPEAKER_01: the continual learning,
- 1:11:45SPEAKER_01: the self-destillation type of literature that seems to work super well.
- 1:11:51SPEAKER_01: But I feel like there's a connection too.
- 1:11:54SPEAKER_01: To all this stuff.
- 1:11:55SPEAKER_01: So I'm always pretty excited about this.
- 1:11:58SPEAKER_01: Yolo, thank you very much for coming a second time here.
- 1:12:01SPEAKER_01: It was awesome.
- 1:12:03SPEAKER_01: It was a great paper.
- 1:12:05SPEAKER_01: I'll put all of the information that you provided also with this other paper in the description.
- 1:12:12SPEAKER_01: And we'll let you know as soon as it's up.
- 1:12:16SPEAKER_00: Yeah.
- 1:12:16SPEAKER_01: Thanks a lot for having me.