Latest / learning with yacine show / Why Self-Distillation Is Taking Over LLM Post-Training (w/ the Researchers Behind It)
Transcript
- 0:00SPEAKER_01: Currently,
- 0:00most post-training of large language model is done via
- 0:04reinforcement learning
- 0:05SPEAKER_01: method like GRPO.
- 0:06SPEAKER_01: The way it works is you take a problem,
- 0:08SPEAKER_01: you have the model generate a bunch of rollouts,
- 0:10SPEAKER_01: you score each of the rollout with a verifier usually,
- 0:13SPEAKER_01: did you get the right answer,
- 0:15And then you update the weights based on which attempt were
- 0:18good and which were bad.
- 0:19SPEAKER_01: The issue is that this reward signal is extremely sparse.
- 0:23SPEAKER_01: You get one score per entire rollout.
- 0:25So the model has to figure out on its own which of the token in
- 0:29a thousand token
- 0:30SPEAKER_01: chain of thought actually matter.
- 0:32SPEAKER_01: This is a pretty brutal credit assignment problem.
- 0:35SPEAKER_01: In this video,
- 0:35we're going to explore a family of methods that kind of
- 0:37sidestep this bottleneck
- 0:39SPEAKER_01: entirely using something called self distillation.
- 0:42SPEAKER_01: The core idea is that in many of these environments,
- 0:45SPEAKER_01: you actually already have rich textual feedback.
- 0:48Things like compiler error,
- 0:49runtime exception,
- 0:51judge evaluation,
- 0:52that sort of stuff.
- 0:53SPEAKER_01: These explain why an attempt failed and current
- 0:57SPEAKER_01: The current RL method just don't use that information.
- 1:00SPEAKER_01: And in the Xtreme case, they collapse all of this into a binary reward.
- 1:04SPEAKER_01: What self-destination does is it takes the model own rollout,
- 1:08SPEAKER_01: give it back to the same model,
- 1:10SPEAKER_01: but now condition on whatever feedback you got from the environment,
- 1:14SPEAKER_01: and it let it
- 1:16SPEAKER_01: re-evaluate its own tokens.
- 1:18SPEAKER_01: In this distillation setup,
- 1:19SPEAKER_01: the teacher and the student are literally the same model,
- 1:23SPEAKER_01: but the teacher just sees more context.
- 1:25SPEAKER_01: This creates a dense,
- 1:26SPEAKER_01: token-level learning signal,
- 1:28SPEAKER_01: while at the same time being cheap to produce,
- 1:30since it's literally just a single forward pass over the
- 1:33existing rollout.
- 1:34SPEAKER_01: This approach was developed by two groups working together.
- 1:37We have the people at ETH Zurich with Jonas Hubater who
- 1:42developed the SDPO for the
- 1:43SPEAKER_01: reinforcement learning setting and then we have Eden
- 1:46Shenfield at MIT who developed the SDFT for continual learning
- 1:51from demonstration.
- 1:52Both papers dropped around the same time in January 2026 and
- 1:56they actually met
- 1:57SPEAKER_01: earlier on in 2025 to discuss this idea.
- 2:01SPEAKER_01: What you like about this paradigm is that it's simple.
- 2:03SPEAKER_01: It bootstraps learning using the model's own in-context learning ability.
- 2:08SPEAKER_01: On the SDPO side,
- 2:10it reaches GRPO accuracy 6 times faster in wall clock time and
- 2:14produces reasoning
- 2:15SPEAKER_01: traits that are up to 11 times shorter.
- 2:17SPEAKER_01: On the Continual Learning side, SDFT
- 2:20let a single model learn multiple skills sequentially without
- 2:24forgetting the
- 2:24SPEAKER_01: previous one,
- 2:25SPEAKER_01: which standard SFT completely fails at.
- 2:27SPEAKER_01: This family of methods are already getting picked up in production,
- 2:30SPEAKER_01: with systems like OpenClaw RL and Frontier Open Source Model like GLM-5,
- 2:34SPEAKER_01: who are using similar approach in their post-training pipeline.
- 2:37Jonas and Idan will walk us through both methods and how
- 2:41self-distillation works at
- 2:42SPEAKER_01: a high level,
- 2:43SPEAKER_01: and I'll be asking them a whole bunch of questions.
- 2:44SPEAKER_01: Thank you to everybody that sent questions my way.
- 2:47SPEAKER_01: during live stream.
- 2:48SPEAKER_01: It was super helpful.
- 2:49SPEAKER_02: Enjoy.
- 2:50SPEAKER_02: So, hi.
- 2:50SPEAKER_02: Happy to be here.
- 2:51SPEAKER_02: I'm Idan Shenfeld.
- 2:54SPEAKER_02: I'm currently a PhD student at MIT working with Professor Pulkit Agawal.
- 3:00SPEAKER_02: My research is focused on reinforcement learning algorithms,
- 3:04SPEAKER_02: mainly for LLMs,
- 3:05SPEAKER_02: but also for robotic applications.
- 3:08SPEAKER_02: Before my PhD,
- 3:09SPEAKER_02: I did research on reinforcement learning in my undergrad,
- 3:13SPEAKER_02: then worked on autonomous driving for a bit,
- 3:15SPEAKER_02: was part of General Motors' big autonomous driving project,
- 3:19did a lot of cool applications of reinforcement learning for
- 3:22the real world there,
- 3:23trying to solve problems that actually involved a lot of moving
- 3:27parts like other
- 3:28SPEAKER_02: drivers,
- 3:29SPEAKER_02: for example,
- 3:29SPEAKER_02: pedestrian,
- 3:30SPEAKER_02: etc.
- 3:33SPEAKER_02: I was also at DeepMind,
- 3:34SPEAKER_02: was part of the post training team as an intern at Summer 24 Man,
- 3:41SPEAKER_01: you were everywhere
- 3:44SPEAKER_01: How old are you?
- 3:46SPEAKER_01: You're already done.
- 3:47SPEAKER_02: A bit older than the other student.
- 3:49SPEAKER_02: A bit older than Jonas.
- 3:51SPEAKER_02: That's for sure.
- 3:52But I wanted to ask you this because I saw that you were doing
- 3:56work that was more
- 3:57SPEAKER_01: practical and then doing the research stuff.
- 4:00What would you say is the biggest difference in mentality that
- 4:03you have when you do
- 4:04that type of more practical minded work that you know is going
- 4:07to go into a car or
- 4:09SPEAKER_01: something like that and the more research
- 4:13Long-term type of stuff like how's the switch going in your in
- 4:16your in your work?
- 4:18SPEAKER_02: Yeah,
- 4:18so I think when you work on an actual product actual
- 4:21application what you really
- 4:22care about is performance Understanding is just a tool to get
- 4:28the right performance
- 4:28and sometimes you know like as much as understanding is fun and
- 4:32research is fun
- 4:32Just going through a data set and cleaning it give you way more
- 4:35performance.
- 4:36SPEAKER_02: Yeah
- 4:38So it's kind of you need to kind of stop yourself from focusing
- 4:41too much on what is
- 4:43you know like cool or fun or new and do the like you know grind
- 4:47work first and only
- 4:49then like put effort well I feel and this is one of the reasons
- 4:52I like like went
- 4:53from industry to PhD that as a researcher they don't care about
- 4:57a specific
- 4:58application the thing that we bring into the world is new
- 5:02understanding is new
- 5:03SPEAKER_02: knowledge
- 5:04SPEAKER_02: and this require a completely different perspective.
- 5:07SPEAKER_02: I don't care about being state of the art on any benchmark.
- 5:11i also think and this is in general good that the community
- 5:14kind of went away from
- 5:15that if you saw papers from you know 2000 i don't know 18 19 20
- 5:20it was all about
- 5:20being number one on the image net leaderboard stuff like that
- 5:24and i think that
- 5:24these days you know people still use benchmark like they
- 5:27produce numbers on aim and
- 5:29stuff like that but it's not really about being number one
- 5:31because there is always
- 5:32SPEAKER_02: a bigger model that will be number one
- 5:35it's all about what kind of new understanding new knowledge new
- 5:39perspective you
- 5:39give to the community and I think that this is much more fun
- 5:43than just you know
- 5:44SPEAKER_02: playing the numbers game
- 5:46SPEAKER_01: Yeah, yeah, yeah.
- 5:47SPEAKER_01: I agree here.
- 5:48SPEAKER_01: I totally agree.
- 5:50SPEAKER_01: I also agree that like,
- 5:51SPEAKER_01: to be honest,
- 5:51like because I have like a consulting practice where I
- 5:54literally just go with like
- 5:55businesses and we try to implement the stuff and they always
- 5:58want to go with for
- 5:59SPEAKER_01: the flashy thing.
- 6:00SPEAKER_01: But most of the gains are just like,
- 6:03SPEAKER_01: Hey,
- 6:04SPEAKER_01: let's,
- 6:04SPEAKER_01: let's look at each of the data points one by one.
- 6:07SPEAKER_01: Right.
- 6:07SPEAKER_01: Yeah.
- 6:07SPEAKER_01: And see like, and see how all messed up they are.
- 6:11SPEAKER_01: And then like you clean it up and then you automatically get
- 6:14SPEAKER_01: a massive gain, and then you're kind of done.
- 6:17SPEAKER_01: Everything else is extra.
- 6:19SPEAKER_01: Cool.
- 6:19SPEAKER_01: This is really nice.
- 6:21SPEAKER_01: I had the other question, but I'm going to leave them for later.
- 6:24SPEAKER_01: Jonas, you can go ahead.
- 6:25SPEAKER_00: Yeah, awesome.
- 6:26SPEAKER_00: Also great to be here.
- 6:28SPEAKER_00: I'm Jonas Hibota.
- 6:29SPEAKER_00: I am a PhD student at ETH.
- 6:32SPEAKER_00: in Zurich, working with Andreas Krause.
- 6:35SPEAKER_00: Yeah,
- 6:35my research has primarily been focused on something that's
- 6:39called test time
- 6:39SPEAKER_00: training,
- 6:40which is effectively this idea that you can continue training a
- 6:44model at inference
- 6:46SPEAKER_00: once it's been given
- 6:47SPEAKER_00: the task.
- 6:48And my primary motivation is the question of how can we have
- 6:51models that are at
- 6:53inference time deployed in a new environment and then continue
- 6:58learning and
- 6:58SPEAKER_00: exploring within that new environment to eventually solve very hard
- 7:02tasks,
- 7:02which initially were completely out of distribution of their
- 7:05initial abilities,
- 7:07right?
- 7:09And that entails a lot of different questions,
- 7:11and obviously it touches on continual learning.
- 7:14SPEAKER_00: It touches on this question, how do you
- 7:17SPEAKER_00: Do you do effective exploration?
- 7:19SPEAKER_00: And a lot of other questions as well.
- 7:21SPEAKER_00: A lot of my prior work,
- 7:23SPEAKER_00: let's say,
- 7:24SPEAKER_00: to this was primarily focused on this question of how do you do effective
- 7:27SPEAKER_00: exploration within that new environment?
- 7:30SPEAKER_00: Yeah.
- 7:30SPEAKER_00: Before that, I studied computer science in Zurich and in Munich.
- 7:38SPEAKER_01: Good.
- 7:38SPEAKER_01: And I had a question about like your efficiency learning at test time.
- 7:43SPEAKER_01: active fine tuning of LM influence, sorry, active fine tuning of LM.
- 7:48It feels really like an extension,
- 7:52SDPO,
- 7:53of that stuff that you were doing.
- 7:55SPEAKER_01: Were you really influenced by that work and just like
- 7:58SPEAKER_00: the next logical step?
- 8:01SPEAKER_00: To some extent, yes, right?
- 8:02SPEAKER_00: Like this other work that you mentioned,
- 8:05SPEAKER_00: in that context,
- 8:05SPEAKER_00: we were asking,
- 8:07SPEAKER_00: okay,
- 8:07how can you continue training and improving LLM at test time
- 8:11once it's given a task
- 8:13SPEAKER_00: through gradient descent?
- 8:14There's obviously,
- 8:15there's several aspects that are important when you want to
- 8:18improve your LLM.
- 8:19SPEAKER_00: There's important, what data do you learn from?
- 8:23SPEAKER_00: And that's the question that we asked in that work.
- 8:25But then,
- 8:26as we discovered in subsequent works,
- 8:28that's not the only question.
- 8:29SPEAKER_00: Another very important question is what loss are you optimizing?
- 8:34SPEAKER_00: And so,
- 8:34SPEAKER_00: especially when you learn a test time,
- 8:36SPEAKER_00: it's very important that you,
- 8:39SPEAKER_00: one,
- 8:39SPEAKER_00: are very efficient in turning your data into gradient updates,
- 8:44SPEAKER_00: right?
- 8:45um purely also from a practical perspective if you want to run
- 8:48these algorithms and
- 8:49do interesting things with these and demonstrate these
- 8:51interesting environments and
- 8:52we are compute bound we have to make the most out of the signal
- 8:56that we get that's
- 8:57number one and number two is I think in settings we also
- 9:03usually just need better
- 9:06uh signals and you know just extract as much signal from our
- 9:10environment as
- 9:11SPEAKER_00: possible
- 9:11SPEAKER_01: Yeah, yeah, I agree.
- 9:13SPEAKER_01: I really like how like,
- 9:14SPEAKER_01: because in my view,
- 9:15SPEAKER_01: like the RL stuff,
- 9:17SPEAKER_01: there's just so much,
- 9:19SPEAKER_01: especially when it's verifiable reward,
- 9:20like there's just so much extra things that like an actual
- 9:24human being that would
- 9:26SPEAKER_01: go through the same process.
- 9:28SPEAKER_01: will look at this stuff, right?
- 9:29SPEAKER_01: They will look at the error trace and they will look like at the,
- 9:32SPEAKER_01: I don't know,
- 9:32SPEAKER_01: the documentation,
- 9:33SPEAKER_01: like the feedback from like a demo or something like that.
- 9:36SPEAKER_01: And then we'll just integrate it in like its learning procedure.
- 9:39SPEAKER_01: 100%.
- 9:40SPEAKER_01: But just giving this the end signal like, hey, you messed it up.
- 9:43SPEAKER_01: You're like, oh, okay.
- 9:44SPEAKER_01: Sorry.
- 9:44SPEAKER_01: All the rollouts.
- 9:46SPEAKER_01: Bad.
- 9:47um it always felt a bit wasteful and there were that we're kind
- 9:50of missing
- 9:51something um and like i i i like i like the the all the studies
- 9:57where there was a
- 9:58teacher that was trying to generate rollouts but then like
- 10:01you're all in token
- 10:02space and it's just weird like what you're gonna do to match
- 10:06the uh the two kind of
- 10:07SPEAKER_01: stream
- 10:08SPEAKER_01: So now I was really excited when I saw like how you guys were doing it.
- 10:12SPEAKER_01: That was pretty good.
- 10:13The original motivation was actually for me coming from pretty
- 10:16much that angle
- 10:18because we were working on coding at the time and it just
- 10:21seemed obvious that you
- 10:23SPEAKER_00: know
- 10:24SPEAKER_00: At that time, the coding agents were not that good as they are now.
- 10:27SPEAKER_00: And so when I was trying to make ChatGPT code,
- 10:30SPEAKER_00: it was like this back and forth,
- 10:31SPEAKER_00: ChatGPT generating some code,
- 10:32SPEAKER_00: I'm running it,
- 10:33SPEAKER_00: I'm pasting back the error.
- 10:35SPEAKER_00: And you know, it was at the time where it started to get decent at that.
- 10:39SPEAKER_00: The intuition was,
- 10:39SPEAKER_00: okay,
- 10:40SPEAKER_00: it's already very good at,
- 10:42SPEAKER_00: or somewhat,
- 10:42it's starting to become very good at understanding what are the
- 10:46sources of the
- 10:47SPEAKER_00: errors it makes.
- 10:48SPEAKER_01: Yeah,
- 10:49SPEAKER_00: and I then actually before we jump into the presentation,
- 10:51there was also your paper or else razor why online
- 10:54reinforcement learning forget
- 10:55SPEAKER_01: less.
- 10:56That's also something that is that got me even more excited is
- 11:00that like you with
- 11:03SPEAKER_01: this type of self distillation method,
- 11:05it seems I think it's easier to even like different learning
- 11:09that you you you you
- 11:11SPEAKER_01: put there is some forgetting as as the thing goes,
- 11:15SPEAKER_01: but
- 11:16It seems to be the type that is normal I would say for like a I
- 11:19don't know a human
- 11:21like if you were to learn like four things in a row right okay
- 11:25you will like forget
- 11:26a bit the first thing and stuff but you will not
- 11:29catastrophically forget all the
- 11:31things that was this work also like helping you like go into
- 11:36that same direction
- 11:37SPEAKER_02: Yes, that was a big influence.
- 11:39SPEAKER_02: My RL Razor paper was a big influence on SDFT.
- 11:43SPEAKER_02: Basically,
- 11:43I finished the RL Razor paper and our conclusion there was that
- 11:48on-policy methods
- 11:50SPEAKER_02: tend to forget less compared to offline methods like SFT fine tuning.
- 11:55SPEAKER_02: And I was like, okay, that's very cool.
- 11:56SPEAKER_02: We mainly focused on RL as the main on-policy method that people use.
- 12:01But I'm like,
- 12:02okay,
- 12:02my conclusion and my research can't be,
- 12:04oh,
- 12:04let's just do RL and
- 12:06throw any other kind of learning signal to the garbage you know
- 12:09that's not
- 12:09something that is like there as you said there's so much
- 12:12learning signals like yeah
- 12:16feedback and so on and I'm like okay we need to find an
- 12:19algorithm that is like on
- 12:21policy like RL but is able to kind of like learn from very
- 12:26diverse kind of signals
- 12:29SPEAKER_02: and that's where I like
- 12:31SPEAKER_02: started to work on this idea and using in context learning you know with
- 12:35distillation to create the self distillation and actually like
- 12:38allow me to put a
- 12:39SPEAKER_02: small story here about how me and Yona started to collaborate on that.
- 12:43SPEAKER_01: Go ahead.
- 12:44So we know each other from conference like few years already
- 12:47and we met at NeurIPS
- 12:49SPEAKER_02: this December and we sat down and you know as usually like
- 12:52we kind of like update each other on what like exciting project
- 12:56you were working on
- 12:57and Jonas open his laptop and show me some slide on like it was
- 13:00the beginning of
- 13:01SPEAKER_02: this SDPO work and I'm like no way that's the exact formulation
- 13:08I came up with and I'm opening my laptop and showing him like
- 13:11you know the
- 13:12beginning of like an overleaf like few paragraphs of ideas and
- 13:15few experiments like
- 13:17we both stumbled on the same like you know idea here um
- 13:21different perspective I'll
- 13:22SPEAKER_02: say
- 13:22like yonas was focused more on the textual feedback that led to
- 13:25the sdpo paper i
- 13:27was focusing more on learning from expert demonstration that
- 13:30led to the sdft paper
- 13:31but that's how we are like okay we are working on the same core
- 13:35algorithmic
- 13:36innovation that's what i found like so exciting at that time
- 13:41because i think we
- 13:42were both thinking and part of the motivation was that this
- 13:45could be a learning
- 13:46SPEAKER_00: paradigm that could enable learning really from
- 13:49SPEAKER_00: very rich diverse data.
- 13:51SPEAKER_00: But this really showed it to us, right?
- 13:53SPEAKER_00: Because we basically arrived at the same underlying learning algorithm,
- 13:58but coming from completely different directions and working on
- 14:02very different
- 14:02SPEAKER_00: problems.
- 14:03Yeah and I mean like because I also really like to like learn
- 14:10from the old older
- 14:11paper not really for the knowledge because like when you when
- 14:15you push too far down
- 14:17there's a lot of noise because they're gonna say like oh we
- 14:20think this and then you
- 14:21read it like 30 years later and like dude no like that was not
- 14:24it right but like
- 14:25you you see some of these convergence of idea brewing that you
- 14:30know that in like a
- 14:31SPEAKER_01: year from now
- 14:32they're gonna get it right they're gonna get I don't like that
- 14:35a very good example
- 14:36of this is like just like the all the resnet highway network
- 14:41kind of moment in in a
- 14:43machine learning because there was all this convergence of
- 14:45ideas and then like the
- 14:48every time like it's like different lab that not necessarily
- 14:50talk to each other's
- 14:51and then they don't necessarily like also reference each other
- 14:54but then they come
- 14:55SPEAKER_01: into the same direction
- 14:57and then the kind of paradigm that usually win is like the one
- 15:00that can distill the
- 15:01idea to the simplest atomic unit right and then this then get
- 15:05like tested on and
- 15:06like in many different form which is kind of what I felt here
- 15:11because like you're
- 15:12SPEAKER_01: in MIT he's in ETS Zurich
- 15:15um i was trying to find the connection and like hey wait so
- 15:18like these guys are not
- 15:19100 related i know this this was also exciting uh validation uh
- 15:24from like a an
- 15:24historical perspective okay cool uh this is pretty good guys i
- 15:28want to say
- 15:28something about your previous point regarding how ideas kind of
- 15:32converge yeah i
- 15:33think like resnet is a great like example another great example
- 15:37is attention right
- 15:38yeah essentially all you need is not the first attention paper
- 15:41but it's you know
- 15:42SPEAKER_02: There are many.
- 15:43SPEAKER_02: I think when it comes to our work on self-distillation,
- 15:47there is a core idea that I start to see coming up recently and
- 15:52Omar Khataib with
- 15:53SPEAKER_02: his RLM work also kind of pushed that.
- 15:57Which is that we go to the point where models can start to be
- 16:02the force that push
- 16:04their own learning algorithms They have enough capabilities
- 16:07that then you talk to
- 16:08JGPD Which means that we should not train them the same way we
- 16:12train three-layer
- 16:13SPEAKER_02: MLP We should use their own capabilities,
- 16:16SPEAKER_02: their own reasoning to drive new learning paradigms
- 16:19Exactly and I think like like like two three years ago like in
- 16:23context learning
- 16:24like it was I felt it was something very powerful and tells you
- 16:28a lot about the
- 16:29model right because it knows a lot and he has like he learned
- 16:33all these functions
- 16:33that they can recombine so theoretically if you give it like
- 16:37some some like a
- 16:39direction and new information it's not like it's not a dumb
- 16:43rock like you will be
- 16:44SPEAKER_01: able to
- 16:45it will be able to kind of go and go in a direction and you
- 16:49like these model that
- 16:49SPEAKER_01: we are using cloud and stuff like as soon as they go and then they search
- 16:53information on the web and stuff like it's pretty good like it
- 16:57will it will be able
- 16:58to like recover and then and iterate and and do their stuff um
- 17:02so if there's a way
- 17:03to just like make like it's more of like allowing them to do
- 17:07the learning right a
- 17:08bit like uh ilia said like these models just want to learn like
- 17:11it just allowing
- 17:12SPEAKER_01: them
- 17:13SPEAKER_01: to have the chance to incorporate the right signal and the learning.
- 17:18SPEAKER_01: I think that's kind of one of the bottleneck.
- 17:20And if you allow them to do that,
- 17:22then they're able to learn a bit like reasoning.
- 17:25SPEAKER_01: Reasoning is like,
- 17:26SPEAKER_01: in my view,
- 17:27SPEAKER_01: as you just start to loop back and back and let them just do their stuff,
- 17:32SPEAKER_01: they kind of were able to lift a constraint from themselves,
- 17:36SPEAKER_01: which is like,
- 17:36SPEAKER_01: I'm going to get an answer
- 17:38and then that's it and then we're done right it's now it's more
- 17:40like I'm gonna get
- 17:42an answer and then like put out some scaffolding and then like
- 17:45we use that
- 17:45scaffolding to be able to generate the right stuff and then I
- 17:49output the stuff
- 17:50SPEAKER_01: there's a lift directly in performance all across because kind
- 17:53SPEAKER_01: This constraint that we've put on the main structure is lifted.
- 17:57SPEAKER_01: I feel it's literally the same thing.
- 17:59It's like this constraint of like,
- 18:01okay,
- 18:01now what if the context window reset?
- 18:04SPEAKER_01: Oh, you have to do this again?
- 18:05SPEAKER_01: Eh.
- 18:05SPEAKER_01: But like,
- 18:06SPEAKER_01: now it's like,
- 18:07you were able to learn from these and then you were able to
- 18:09move your weight into
- 18:11kind of like a peak in the valley that like make more sense for
- 18:14like these specific
- 18:15SPEAKER_01: tags,
- 18:15SPEAKER_01: a specific user.
- 18:17This makes a lot more sense because like you don't have to mess
- 18:20around with like
- 18:21crazy memory scheme and stuff like it will have a better sense
- 18:25about like what to
- 18:25do next and if like you like if if you you weave it in so that
- 18:30like there's there's
- 18:31not too much catastrophic forgetting to happen it's a very
- 18:35viable in my opinion
- 18:36SPEAKER_01: kind of pattern to have these model have
- 18:40SPEAKER_00: Yeah, so we put together some slides.
- 18:43What we'll talk about will be a little bit of a unified
- 18:46perspective of these three
- 18:47SPEAKER_00: papers which we put out,
- 18:49SPEAKER_00: which as we discussed,
- 18:50SPEAKER_00: they're all on the same algorithm.
- 18:52SPEAKER_00: The algorithm is exactly the same,
- 18:54SPEAKER_00: in fact,
- 18:55but they touch on three different perspectives of how that
- 18:57algorithm can be useful.
- 18:59and really special thanks go to Thomas who has also been
- 19:04leading one of these three
- 19:06papers and a super exciting one and we'll talk also we'll cover
- 19:10that as well what I
- 19:11would like to start with is this observation that in many ways
- 19:15current learning
- 19:16paradigms are imperfect and that surfaces in different ways so
- 19:20as we talked about
- 19:21SPEAKER_00: one big issue
- 19:23SPEAKER_00: in many learning paradigms is that they lead to catastrophic forgetting.
- 19:26SPEAKER_00: So meaning as you try to learn a new task,
- 19:29you have this catastrophic behavior that you become
- 19:34significantly worse on previous
- 19:36SPEAKER_00: tasks that previously were
- 19:38SPEAKER_00: and that's obviously not something that we want.
- 19:40Then the second problem I think surfaces in a little bit more
- 19:45in a little bit more
- 19:46subtle way which is that most current learning paradigms they
- 19:49require very careful
- 19:50designing of the data that you put into them so this could be
- 19:54SFT data sets or this
- 19:56could be RL environments and I think this shows predominantly
- 20:01in the fact that
- 20:01SPEAKER_00: there have been
- 20:03a surge in startups or even like scale.ai now a bit older and
- 20:07already acquired but
- 20:08really a surge in companies that whose primary aim is to
- 20:13organize the data and I
- 20:16think a dream of many is to have eventually a learning paradigm
- 20:20where the models
- 20:21are able to make sense of the raw data itself as they interact
- 20:24with the environment
- 20:25SPEAKER_00: and go out in the world and find the data
- 20:27And then the third one of course is this observation that often
- 20:31systems have some
- 20:32kind of brittle generalization that if you prompt them in
- 20:35exactly the right way
- 20:36they do what you want them to do but if you just change your
- 20:39prompt a little bit
- 20:40SPEAKER_00: they seem not to be able
- 20:42SPEAKER_00: yacine.
- 21:12SPEAKER_00: So the thing that we will talk about is this thing called on-policy
- 21:16self-distillation and we really view this as a new learning
- 21:19paradigm that enables
- 21:20continual learning from arbitrary data and makes some
- 21:24improvements on the three
- 21:25SPEAKER_00: desiderata that we outlined
- 21:28what's your take on forgetting because like continuous learning
- 21:32is like is one thing.
- 21:33But if we're talking about like a system that kind of is is
- 21:37learning from your your
- 21:39your pace and like your way of working here from like a human
- 21:43or like a process.
- 21:44SPEAKER_01: But like I said, the process in the human is flawed, right?
- 21:47And then like,
- 21:48I'll give the finally like that was the wrong direction we need
- 21:50to go there.
- 21:50What's your your take on on that stuff like the the forgetting
- 21:54the adaptation in general?
- 21:56Yeah I can answer on that so for me at least like the ability
- 22:01to learn without
- 22:02forgetting is almost a must in the world where you know we have
- 22:06AI models that are
- 22:07SPEAKER_02: actually being deployed like think about let's say
- 22:11SPEAKER_02: Claude or Chagipti.
- 22:13According to recent numbers,
- 22:14they process around 3 billion messages a day.
- 22:17SPEAKER_02: Not learning from these messages, it's a waste of data.
- 22:21SPEAKER_02: User interaction,
- 22:22environment interaction is going to be the largest source of
- 22:25data that we have to
- 22:27SPEAKER_02: train models.
- 22:28SPEAKER_02: But unlike other sources of data,
- 22:30SPEAKER_02: which we have in some fixed container and we can choose when and how we
- 22:36We can use this data is the streaming data every day you get it
- 22:40a bit by bit and
- 22:41therefore you have to do like you know this kind of continue
- 22:44learning which is
- 22:45essentially learning without forgetting you have to be able to
- 22:48update yourself
- 22:49every day a bit becoming a bit better aggregate another new
- 22:52skill another new
- 22:53SPEAKER_02: capability
- 22:54um without like you know start to degrade on what you know
- 22:59already uh this is true
- 23:00in the general sense of you know let's improve the model from
- 23:03three billion
- 23:03messages and this is also true in the small sense of if i have
- 23:07my own you know open
- 23:08claw and i want my open claw to be like tailored to my
- 23:11preferences um also it will
- 23:14not get like one big data set of my preferences it will get
- 23:17every day i'll tell it
- 23:18SPEAKER_02: a new thing that it needs to remember um
- 23:21and this is just very different than the way we think about you
- 23:24know classical
- 23:25machine learning when we start assume a data set IAD from some
- 23:29source you know and
- 23:31therefore forgetting is something that really need to be dealt
- 23:35with our agenda for
- 23:37this talk we'll try to keep it somewhat short is first of all
- 23:40to cover what is
- 23:40SPEAKER_02: self-distillation what is the core ideas behind it
- 23:44and to kind of go through the different parts of self of post
- 23:48training that we
- 23:50believe self distillation can take part of which to be honest
- 23:54is most of self
- 23:55distillation so we'll cover stuff that like how self
- 23:58distillation can be used to
- 23:59learn as um from demonstration as a replacement for sft from
- 24:04verifiable rewards
- 24:05SPEAKER_02: replacement to rl and also to learn from new kind of um
- 24:10SPEAKER_02: learning signals such as rich environment,
- 24:12feedback and real user conversations that are not able with the
- 24:16current methods.
- 24:17SPEAKER_02: So let's start with like in context learning.
- 24:19SPEAKER_02: So in context learning,
- 24:21SPEAKER_02: I think these days we almost take it obvious like,
- 24:23SPEAKER_02: okay,
- 24:24SPEAKER_02: everyone know that it's happened.
- 24:26but like I still remember three years ago when like people
- 24:28start to realize that
- 24:29this is happening it just seems magical like you just put data
- 24:32into context and the
- 24:33model just change its behavior and is able to learn from
- 24:37whatever data you put into
- 24:39SPEAKER_02: context and it doesn't have to be
- 24:40really nice form like example it can be hints it can be
- 24:44feedback it can be random
- 24:45instructions and it's also generalized very well but and this
- 24:49is a big but it's
- 24:51very transient like who if today I put some context into the
- 24:54model tomorrow I
- 24:55and start a new session and the context is not there,
- 24:58the model goes back to how it was.
- 25:00SPEAKER_02: There's nothing that just stays there.
- 25:03And the second is that,
- 25:04which is another big limitation,
- 25:06is that context windows are bounded.
- 25:08SPEAKER_02: And in the end,
- 25:09SPEAKER_02: if,
- 25:09SPEAKER_02: again,
- 25:10SPEAKER_02: I want to become better at every possible task on Earth,
- 25:13SPEAKER_02: I cannot put examples for every possible task in context.
- 25:16SPEAKER_02: And therefore,
- 25:16SPEAKER_02: like,
- 25:17SPEAKER_02: when we looked in context learning,
- 25:19SPEAKER_02: we're saying,
- 25:20SPEAKER_02: okay,
- 25:20SPEAKER_02: we have something that is really good at our hand,
- 25:24but we need to somehow compress it we need to somehow take it
- 25:28and put it into the
- 25:29model weight to make it stopping the transient and become
- 25:32something more like that
- 25:33stays there even when you remove the context and this led us to
- 25:37the core idea of
- 25:38self-distillation i'll now cover like briefly the algorithm
- 25:42itself we take an llm
- 25:44and we operate it in two modes the first mode is the student
- 25:48mode we just get the
- 25:49SPEAKER_02: input prompt some question from a user x
- 25:52and it's output a response Y from a second mod is a teacher mod
- 25:57where in addition
- 25:58to the input prompt the same LLM also get an extra contact see
- 26:04this extra contact
- 26:05again can be expert demonstration instruction feedback whatever
- 26:08but the important
- 26:09SPEAKER_02: thing is that now this model is conditioned on another input
- 26:14SPEAKER_02: the context and this automatically changed its output distribution.
- 26:17SPEAKER_02: Now the output distribution,
- 26:19the responses that the teacher would have produced if we sample
- 26:22from it are kind of
- 26:23different from the student and therefore the core idea in
- 26:26self-distillation is just
- 26:27SPEAKER_02: to use this teacher
- 26:29and do teacher-student learning all paradigm in machine
- 26:32learning where you just
- 26:33minimize the sum distributional measure and this in our case we
- 26:36chose the reverse
- 26:37KL between the teacher and the student and take the gradient of
- 26:41course only through
- 26:42the student and not through the teacher because the teacher is
- 26:45the one that guided
- 26:46SPEAKER_02: the learning this is a distillation algorithm
- 26:49uh very similar to what people have been using quite a lot big
- 26:52to small model
- 26:53distillation but with one important change that the teacher is
- 26:57the one that is
- 26:58changing um the the model itself is the one that guiding the
- 27:02learning um and why we
- 27:04like it is that this is much more similar to how human learns
- 27:08like humans don't
- 27:09learn by looking at some example and just trying to mimic it
- 27:13you know one-to-one um
- 27:15What we usually do is like kind of we observe we condition our
- 27:19short-term memory
- 27:20condition our actions on whatever we see in front of us
- 27:24whatever feedback we got
- 27:25from the environment and we use that to improve our behavior so
- 27:29kind of the force
- 27:31that push the improvement in humans behavior is humans
- 27:34themselves and we want to
- 27:36SPEAKER_02: give models the ability to do the same
- 27:38At the same idea let's say like following this idea how do we
- 27:41do it in practice
- 27:43given the teacher and the student we can compare their output
- 27:47distribution for
- 27:48every single so we go over some assistant answer in this case
- 27:53the user said answer
- 27:54with yes or no is what are we yet and the assistant produce an
- 27:57answer that is much
- 27:58more than the user asked yes the water is wet and the user said
- 28:02then I said yes or
- 28:03SPEAKER_02: no only
- 28:05In hindsight if we take the full conversation put it on context
- 28:08and ask the model
- 28:09or like let's go token by token here over your assistant and
- 28:13see how now that you
- 28:15know that the user said afterwards I said yes or no only the
- 28:18probabilities will
- 28:19change stuff like yes will become more probable in hindsight
- 28:23and everything else
- 28:24become less likely in hindsight and this is the core algorithm
- 28:29object that we're
- 28:30SPEAKER_02: working on with the log probability ratio
- 28:34SPEAKER_02: between the original model and the improved policy.
- 28:37This is actually very interesting because when you take this
- 28:41log probability ratio,
- 28:42there are two perspectives that you can use about how we can
- 28:45use it in learning.
- 28:46One is that we say that this is just a token level advantage or
- 28:50a token level award.
- 28:52This is similar to how in normal GRPO where we train LLMs,
- 28:56the advantage is the,
- 28:58you know,
- 28:59the reward of this answer minus the average of the rewards
- 29:03across all answers here
- 29:04we just say okay instead of using that signal let's use another
- 29:07signal but plug it
- 29:09back into the rl algorithm at the same time we can use look at
- 29:13it as distillation
- 29:14algorithm where we want to if we sum this log probability
- 29:18across the entire like
- 29:19token space we get just the kl measurements um between as i
- 29:24said before the student
- 29:25SPEAKER_02: and the teacher
- 29:26Now one nice thing that we like about the self distillation
- 29:30perspective is that
- 29:32unlike RL where you take you get feedback only on the tokens
- 29:36you sampled if you
- 29:37take the full KL you take you get feedback you get a learning
- 29:41signal over the full
- 29:43SPEAKER_02: token distribution which is much much more informative and
- 29:48uh you know if we had the whole conversation about like how
- 29:51normal rl rlvr is just
- 29:53like one bit of information per trajectory if we do this like
- 29:57rich advantages we
- 29:58get more bits because now for every token we get some feedback
- 30:02but if we do the
- 30:03full kl that's a very rich signal to learn from and we'll see
- 30:07later how it also
- 30:08SPEAKER_02: leads to um
- 30:09faster convergences so another nice thing is that we said okay
- 30:13we let the model
- 30:14decide how to learn and when to change its behavior like how
- 30:18much can we trust it
- 30:19like how maybe this process is very noisy so in one of our
- 30:23works led by Thomas
- 30:24SPEAKER_02: Mahdid yacine.
- 30:47SPEAKER_02: and this is similar to before,
- 30:49SPEAKER_02: we have like a multi-turn conversation with the user give feedback.
- 30:55SPEAKER_02: Now,
- 30:56SPEAKER_02: sometimes the user give relevant feedback,
- 30:58SPEAKER_02: relevant user follow up and sometimes the user asks
- 31:01related question after that and one I think that we noticed
- 31:05regarding in context
- 31:06learning in this experiment is that the model will decide to
- 31:09change its original
- 31:11answer only if the user gave relevant feedback so you see above
- 31:15basically the color
- 31:16is corresponds to how much a log ratio change the probability
- 31:20is changed which
- 31:21means that the model decided to that there is another answer
- 31:26that will be better
- 31:27SPEAKER_02: given the feedback
- 31:29And you see that there is much more color on the top plot than
- 31:32on the bottom plot
- 31:33SPEAKER_02: Basically,
- 31:34SPEAKER_02: if the model didn't got any relevant feedback,
- 31:38it will not change its answer Which means that this in-context
- 31:42learning only push
- 31:43SPEAKER_02: the learning,
- 31:44SPEAKER_02: only change the model,
- 31:46SPEAKER_02: only when we have relevant feedback
- 31:48The whole system seems to hinge on in context learning ability
- 31:52of the model,
- 31:53right?
- 31:53SPEAKER_01: Like it's leveraging this in order to kind of like guide the trajectory.
- 31:59SPEAKER_01: And one of the questions was like,
- 32:01SPEAKER_01: what if the model is not able to kind of understand the correct answer,
- 32:06SPEAKER_01: right?
- 32:06SPEAKER_01: Like how to like, I don't know, like
- 32:09it's not usually like not have enough capacity because they're
- 32:12too small but it
- 32:13just doesn't understand what what is the correct answer it will
- 32:17still kind of judge
- 32:18the token output like is this going to lead to kind of just the
- 32:22the direction being
- 32:23SPEAKER_01: like random or like what's happened in this specific case
- 32:26Yeah in that case if the model like understood that there is
- 32:29some feedback that it
- 32:30should change its behavior but it didn't really understand how
- 32:34and like you know is
- 32:35able to push it into the good direction uh then yes like the
- 32:38update can be quite
- 32:39random the nice thing i'll say is that one um when you take you
- 32:45know usually you do
- 32:46like training over a whole data set of examples even if some of
- 32:50them the model was
- 32:51SPEAKER_02: not able to understand
- 32:53you will be able to understand on others and the you know
- 32:55average gradient will
- 32:56still push you towards positive directions and second that like
- 33:01look at the models
- 33:02that we get every year we have new models that even with the
- 33:06same number of
- 33:06parameters just have much better capabilities so if i compare
- 33:10you know these days
- 33:12SPEAKER_02: quen 3 8b to like the first llama 8b
- 33:15SPEAKER_02: The in-context learning capabilities of the new coin are much,
- 33:18much better And I expect this to continue growing The new model
- 33:24of next year and
- 33:25SPEAKER_02: the year after that will just have better and better in-context learning
- 33:28SPEAKER_02: capabilities and will be able to understand users better
- 33:32Maybe one thing I would add to that is I think the other aspect
- 33:37that this points to
- 33:38which is quite important is the feedback itself and I think in
- 33:42the same way that as
- 33:43humans right we also rely on the feedback that our environment
- 33:46gives I think to me
- 33:48this points to a super interesting field which I think will
- 33:52open up in the future
- 33:53is the question of how does the model best like seek out the
- 33:56right information from
- 33:57SPEAKER_00: this environment
- 33:58as well as right in the same way that when we want to learn
- 34:02about a particular
- 34:02problem we have to learn about where to look where to get
- 34:05guidance from if we
- 34:07search for people that help us yeah people that help us become
- 34:10better at XYZ we try
- 34:12to find the right people that actually give us the feedback
- 34:15that we that our policy
- 34:17needs you know yeah and I think that will be super interesting
- 34:20in the future as a
- 34:21SPEAKER_01: human like uh whenever I'm trying to learn and I had to learn something
- 34:26SPEAKER_01: The hardest learning I had to do was not because the material was hard.
- 34:31SPEAKER_01: It's just because I had to fight to get a signal.
- 34:36SPEAKER_01: I just had the end signal.
- 34:39SPEAKER_01: I knew it was something that I had to do a lot of work to get to.
- 34:43SPEAKER_01: Sometimes there is nobody to ask for.
- 34:45You just have to try a bunch of stuff and then you get some
- 34:48more idea about what's going on.
- 34:51But sometimes it's just like the environment was not well set
- 34:56to give me the signal.
- 34:56SPEAKER_01: But as soon as I get the signal, I'm like, oh, okay, that's it.
- 35:00SPEAKER_01: I just get another textbook, look at it.
- 35:02SPEAKER_01: I'm like, okay, that's just the stuff.
- 35:04SPEAKER_01: It's not that hard.
- 35:06SPEAKER_01: So yeah, it's true.
- 35:09The richness of the feedback is really important for learning
- 35:13like an organic being.
- 35:15I do agree and maybe you know the same way we see like a lot of
- 35:19environment design
- 35:21that go into RLVR these days we'll see a lot of like
- 35:24environment plus feedback
- 35:25SPEAKER_02: design that go into like training data loops in the future
- 35:29um we have one um rubini carmona uh he's asking like does this
- 35:34paradigm skew the
- 35:35model representation toward meta condition without the
- 35:38substance of actual
- 35:39commission and i ask him like what he means by actual
- 35:42connection and he says like
- 35:43if the representation of a task or ability a get progressively
- 35:47higher while losing
- 35:48the initial specificity he's talking about like uh if the model
- 35:52is not like losing
- 35:53SPEAKER_01: a bit of specificity to kind of go a bit more higher level
- 35:56I think there are two kind of loops here there's the
- 35:59metacognition loop which is
- 36:01like kind of like like equivalent to in context learning like
- 36:04how much the model
- 36:05can learn from feedback and kind of like guide itself and
- 36:09there's the inner loop of
- 36:10SPEAKER_02: the specificity
- 36:11which is like how much it can actually solve a problem like
- 36:15solve a question and I
- 36:17feel like here we mainly take advantage of the metacognition
- 36:21loop to improve the
- 36:22specificity I believe that there are other works and this is a
- 36:26very exciting like
- 36:27line of research to see how we can improve also the outer loop
- 36:30how to make
- 36:32SPEAKER_02: the model more aware to the fact that it's being used in this kind of
- 36:38self-distillation training so that it will be able to provide
- 36:42better signal for
- 36:43SPEAKER_02: itself.
- 36:44SPEAKER_02: Actually,
- 36:45Joe Parry from my lab here at MIT released a paper around a
- 36:49year ago about
- 36:49SPEAKER_02: self-adapting LLMs that did exactly that.
- 36:52basically an outer metacognition loop and inner RL loop where
- 36:57it's basically
- 36:58trained the model to give better feedback to itself for
- 37:02training so I believe this
- 37:04is like combining that with a self distillation kind of
- 37:08algorithm so back in the
- 37:10self adapting language model it did mainly supervised learning
- 37:14in the inner loop
- 37:15but combining the two ideas of improving the outer loop and
- 37:18improving the inner
- 37:19SPEAKER_02: loop can be quite powerful
- 37:21what are you guys thought because we're talking about
- 37:24continuing learning maybe
- 37:24you're gonna you're gonna talk about it a bit but like very
- 37:28very long horizon hard
- 37:31task where the feedback is rich right like you are able to get
- 37:35some feedback about
- 37:36SPEAKER_01: like if you're going a bit in the right direction, right?
- 37:40SPEAKER_01: What do you think here?
- 37:43Would it be a useful kind of methodology to kind of refine the
- 37:46model as it is like
- 37:49SPEAKER_01: going through the context
- 37:51SPEAKER_01: in this,
- 37:52SPEAKER_01: let's say in Rnest,
- 37:53SPEAKER_01: like RLM or whatever it is,
- 37:55SPEAKER_01: and then is able to kind of,
- 37:56SPEAKER_01: without blowing its context window,
- 37:58SPEAKER_01: manage it properly,
- 37:59SPEAKER_01: but like kind of run these experiments,
- 38:01get some feedback and know if that gets going into the right
- 38:04direction and like
- 38:05SPEAKER_01: inject this into its weight.
- 38:07SPEAKER_01: Do you know that this,
- 38:08SPEAKER_01: do you think that this has like some utility in like this more automated
- 38:13SPEAKER_01: research-ish,
- 38:14SPEAKER_01: like a long context hard task?
- 38:17SPEAKER_00: Yeah, I think we'll have some early examples of that later.
- 38:21SPEAKER_00: that we'll cover.
- 38:24SPEAKER_00: I fundamentally agree.
- 38:25SPEAKER_00: I think 100%.
- 38:28SPEAKER_00: To me,
- 38:28something that also seems interesting is it seems to me kind of
- 38:32intuitively that as
- 38:34SPEAKER_00: humans,
- 38:34SPEAKER_00: we do this reflection on
- 38:36SPEAKER_00: various,
- 38:37SPEAKER_00: let's call it horizons,
- 38:38SPEAKER_00: over various frequencies,
- 38:41SPEAKER_00: both in terms of very narrow and short-term feedback that we get from our
- 38:47SPEAKER_00: environment as we interact with it,
- 38:49SPEAKER_00: like immediate responses that the environment gives,
- 38:52SPEAKER_00: as well as over longer periods of time where,
- 38:56SPEAKER_00: I don't know,
- 38:56SPEAKER_00: we recapitulate,
- 38:57SPEAKER_00: okay,
- 38:58now that I talk to my supervisor about what I've done the last
- 39:02two weeks and
- 39:08That's what was in my mind because like every time I was doing
- 39:13research like
- 39:14looking back when I was done I was like that was like max four
- 39:18weeks worth of work
- 39:19right but throughout I was getting all sorts of signals I had
- 39:24to entangle and try
- 39:25to couple up the feedback in order to figure out like the
- 39:29direction like and like I
- 39:31I knew when I was off off right and the direction even though I
- 39:35was dumb right I
- 39:36SPEAKER_01: knew like
- 39:37this was not the right direction for sure right so I had to
- 39:40kind of like go over
- 39:41there and kind of mess up a bunch of stuff here get enough
- 39:44feedback to like hey
- 39:45it's it's here and I just continue and then I got the stuff and
- 39:48then when I linked
- 39:49them up it was like three four steps um but I was able to do it
- 39:53but I needed the
- 39:54feedback from the environment so this is kind of what I was
- 39:56thinking like in a in a
- 39:58SPEAKER_01: setting where
- 39:59SPEAKER_01: you can get even if it's complex feedback,
- 40:01SPEAKER_01: like very,
- 40:02SPEAKER_01: very detailed one,
- 40:05you might be able to kind of steer the model to just have in
- 40:08this hand side
- 40:10SPEAKER_01: direction,
- 40:11SPEAKER_01: right?
- 40:12SPEAKER_01: Because like,
- 40:13let's say there's a discovery that we want to make now the
- 40:15model has it in its data
- 40:17SPEAKER_01: set and it was trained with it.
- 40:18SPEAKER_01: It knows right now it knows, right?
- 40:20SPEAKER_01: But before it knew, right?
- 40:22SPEAKER_01: It's not terribly different.
- 40:24SPEAKER_01: The model from like now then, then like when it was trained,
- 40:28SPEAKER_01: So being able to kind of get the signal there,
- 40:30maybe we'll be able to arrive at these kind of result if the
- 40:35feedback is rich
- 40:36SPEAKER_01: enough.
- 40:36SPEAKER_00: Yeah, 100%.
- 40:37SPEAKER_00: I think it's a very, very important point that you make there.
- 40:42SPEAKER_00: To me,
- 40:43it's certainly the case that especially as we do as humans
- 40:48reflection over longer
- 40:50SPEAKER_00: time horizons,
- 40:51we do this form of meta update of our strategy where we say
- 40:56once we made the
- 40:57discovery and we reflect on how in the future we can make these
- 41:03discoveries better
- 41:04we don't say okay in hindsight I should have just you know
- 41:06straight one-shotted my
- 41:08discovery what we do is we try to understand okay what kind of
- 41:11systems can I build
- 41:13SPEAKER_00: or how can I improve my way of doing research
- 41:18SPEAKER_00: so that I'll make discoveries faster.
- 41:20SPEAKER_00: And we are aware,
- 41:21SPEAKER_00: right,
- 41:22through some form of trial and error and reasoning that this
- 41:25does not come through
- 41:26SPEAKER_00: better one-shotting,
- 41:27but through building better kind of systems and better ways of
- 41:31probing and getting
- 41:32SPEAKER_00: to answers quickly.
- 41:33SPEAKER_00: And so, yeah, I think, like, that's super interesting.
- 41:36SPEAKER_00: I'm very sure that over this,
- 41:38SPEAKER_00: like,
- 41:38SPEAKER_00: over this year and also longer term,
- 41:41SPEAKER_00: we'll see much more results in that.
- 41:46So before going into like specific examples of learning we want
- 41:50to touch back into
- 41:51the R-DZT Rata basically what we expect from a true continue
- 41:57learning algorithm and
- 41:59see how the algorithm that I just described kind of like check
- 42:03any one of these
- 42:04requirements so as we said before what we want is basically
- 42:08performance we want an
- 42:09SPEAKER_02: algorithm like the model to learn we want no catastrophic forgetting
- 42:14SPEAKER_02: and we want to be able to learn from arbitrary non-curated data.
- 42:19SPEAKER_02: Let's start with performance.
- 42:21SPEAKER_02: So,
- 42:22SPEAKER_02: as I mentioned before,
- 42:23SPEAKER_02: our algorithm is an on-policy algorithm,
- 42:25SPEAKER_02: which means that it's a feedback-style algorithm.
- 42:28SPEAKER_02: The model,
- 42:29SPEAKER_02: the student,
- 42:30SPEAKER_02: in a student mode,
- 42:31SPEAKER_02: rolls out its own trajectories,
- 42:33SPEAKER_02: it tries to solve whatever task,
- 42:36SPEAKER_02: and going over this trajectory,
- 42:37SPEAKER_02: it gets feedback.
- 42:39SPEAKER_02: Now,
- 42:40SPEAKER_02: there is very classic results from 2010 from the Dagger paper,
- 42:45SPEAKER_02: a very famous paper in robotics that
- 42:50show that basically if you do on policy learning you're
- 42:54supposed to get much better
- 42:56just performance in your task and why is that let's look at the
- 42:59other option to do
- 43:00like teacher forcing basically to just look at teacher
- 43:04demonstrations and to
- 43:05understand the pain point here let's look at this like two
- 43:08example of like
- 43:09autonomous driving so we have this car driving this loop and it
- 43:13follows like it
- 43:14have a bunch of expert demonstrations in blue here and it's
- 43:18trying to learn from
- 43:19SPEAKER_02: them
- 43:20the problem is that all of these expert demonstrations are only
- 43:24of a car driving
- 43:25quite well in the middle of the road what will happen that
- 43:28during inference time
- 43:29you know you deploy you learn you deploy the model now the
- 43:31model is not perfect and
- 43:32sometimes it will get very close to the wall but this area of
- 43:36being very close to
- 43:37the wall has no coverage in your training data basically it
- 43:41creates some train test
- 43:43SPEAKER_02: distribution shift
- 43:44SPEAKER_02: between the training and the inference.
- 43:49And you get to an area where you didn't learn over and
- 43:53basically you're screwed.
- 43:54SPEAKER_02: The model does not know what to do.
- 43:56SPEAKER_02: Now, this is kind of extreme examples.
- 44:00but we see it's also true in any kind of learning as long as
- 44:04you have this like
- 44:05sequential decision making this you know when the model output
- 44:10a sequence of
- 44:11actions or a sequence of tokens as in LLMs if you just follow
- 44:15the teacher you get a
- 44:16SPEAKER_02: lack of robustness that lead to just poor performance and this was
- 44:22known also for like normal on policy distillation versus
- 44:25offline distillation from
- 44:27a big model to a small model that on policy distillation just
- 44:30give you better
- 44:31performance and therefore we that is one of the reasons we
- 44:34chose to focus on on
- 44:36policy self-distillation another one and I think we touched it
- 44:40a bit before about
- 44:40my our razor paper where we showed that on policy learning
- 44:45tends to forget less so
- 44:47SPEAKER_02: we did a very like simple experiment where we took the same task
- 44:51the same set of prompts and train a model with either RL or SFT
- 44:56and measure not
- 44:56just how much we improve on the new task as we see in the
- 45:00x-axis on the plot on the
- 45:02left but also how much we forget we took a set of eight known
- 45:06benchmarks like if
- 45:08eval and truthful qa mmlu and so on and we basically evaluated
- 45:14like how much
- 45:14degradation of capabilities we have when we learn a new a new
- 45:18task in this case
- 45:19SPEAKER_02: tool use and
- 45:20something that we found is that RL tends to learn without
- 45:25forgetting while for SFT
- 45:26in order to improve your model you need to sacrifice prior
- 45:30capabilities and this is
- 45:31by the way not only true for LLMs we try it with robotic
- 45:34foundation models and even
- 45:35SPEAKER_02: with three layer MLP on MNIST and just to try to give like a
- 45:40short intuition about why it happened is because that in a lot
- 45:44of these problems
- 45:45there's not only one policy that can get to let's say 90%
- 45:50success on the new task
- 45:52it's a whole set of them and which one you converge to will
- 45:56affect how much
- 45:56SPEAKER_02: forgetting you'll have and RL just have a tendency um
- 46:01like own policy methods such as RL have a tendency to converge
- 46:05to one that are as
- 46:07SPEAKER_02: close as possible to the original policy even without any explicit KL
- 46:12SPEAKER_02: regularization or something like that so this kind of like
- 46:17implicit bias towards like minimum change keep the models from
- 46:23forgetting more and
- 46:24more and we took advantage of that and said okay if our
- 46:28self-distillation is an
- 46:29on-policy self-distillation it will also forget less than if
- 46:33we'll do like let's
- 46:34say offline self-distillation we're just generating data from
- 46:39the teacher and
- 46:40SPEAKER_02: learning from them
- 46:41and so like we know that like we can get good performance and
- 46:44we can learn without
- 46:45forgetting it another the question is like can we learn from
- 46:48arbitrary kinds of
- 46:50data and I think this is what we really work on quite a lot in
- 46:54our series of papers
- 46:56to to show that this is possible that this is not just like a
- 47:00one algorithm that
- 47:01learned from one type of learning signal such as verifiable
- 47:05rewards but this is
- 47:07SPEAKER_02: like a really
- 47:08core and versatile kind of learning paradigm and we go through
- 47:13some of these
- 47:14examples now yeah so the first and the most I think intuitive
- 47:18one that we like to
- 47:19SPEAKER_02: start from is learning from demonstrations basically you have the expert
- 47:22SPEAKER_02: demonstration you can put it in context and ask the model to you know
- 47:29the teacher model to produce its own response which means that
- 47:33given that it
- 47:33condition on the right response it will be a true one and this
- 47:37is like a different
- 47:38use of that this is the same data that we use in sft but a new
- 47:43algorithm that
- 47:44SPEAKER_02: learned from it and as we see in the plot here on the right
- 47:47we see two things again we have in the x-axis the new task
- 47:50accuracy and on the
- 47:51y-axis the prior task performance and we see that SDFT our
- 47:56self-distillation
- 47:57algorithm is able to both get better new task accuracy again
- 48:02because of that train
- 48:03test distribution mismatch that does not happen in non-policy
- 48:06learning and
- 48:07SPEAKER_02: and learn with almost no forgetting,
- 48:10SPEAKER_02: which is exactly what we want to get from a continual learner algorithm.
- 48:15And this also enables us to do what we call like a real
- 48:19continual learning experiment.
- 48:20SPEAKER_02: Basically,
- 48:21SPEAKER_02: not just improving,
- 48:23SPEAKER_02: taking one task and saying,
- 48:24SPEAKER_02: oh,
- 48:24SPEAKER_02: look,
- 48:24SPEAKER_02: we learned without forgetting,
- 48:26but actually taking a model and trying to like improve it on a
- 48:29series of tasks.
- 48:30SPEAKER_02: So we have tool use,
- 48:31SPEAKER_02: scientific QA,
- 48:33SPEAKER_02: medical questions,
- 48:35SPEAKER_02: and we're asking,
- 48:35SPEAKER_02: okay,
- 48:36SPEAKER_02: can we get a model that just
- 48:37goes data set by data set and improve on all of them and here
- 48:42we see the on policy
- 48:44self distillation is actually able to do it like use every time
- 48:48you change from one
- 48:51try that data set to another you have a slight degradation of
- 48:55performance but
- 48:56SPEAKER_02: overall
- 48:56SPEAKER_02: Mahdid yacine
- 49:23I really like that figure because it showed really clearly the
- 49:28difference between the two.
- 49:30SPEAKER_01: But I was really curious.
- 49:32SPEAKER_01: It's just a gut feeling here, right?
- 49:34SPEAKER_01: How much data set,
- 49:36SPEAKER_01: diverse data set,
- 49:37do you need to pile up here in order to make the own policy
- 49:41self-determination
- 49:43SPEAKER_01: crumble?
- 49:44SPEAKER_01: Do you think it will show massive degradation at some point?
- 49:50SPEAKER_01: or you still think it's going to be robust?
- 49:53SPEAKER_02: That's a really good question, right?
- 49:55SPEAKER_02: We still need to like push it in scale to see where things break.
- 50:00That is kind of like the downside of doing kind of academic
- 50:03research where at some
- 50:04SPEAKER_02: point you're saying,
- 50:04SPEAKER_02: okay,
- 50:05SPEAKER_02: I ran out of compute.
- 50:08But I do think that over like my closest point of comparison is
- 50:13I think Nvidia had
- 50:15the Cascade NemoTron paper where they did RL on a series of
- 50:19tasks and if I remember
- 50:21correctly they had six different tasks and each one of them had
- 50:24was much bigger
- 50:25data set than the one that we use you know if they did tool use
- 50:29they did like a
- 50:30SPEAKER_02: proper you know like
- 50:32tens of thousands of Tullius problems and there was on policy
- 50:36rl they were able to
- 50:38show that like you can keep learning and basically have no
- 50:41degradation at all even
- 50:42SPEAKER_02: after like if i remember correctly like thousands of gpu hours
- 50:48SPEAKER_02: And since the core mechanism,
- 50:49SPEAKER_02: the on-policy learning mechanism is the same mechanism,
- 50:54SPEAKER_02: I expect SDFT to do the same,
- 50:57SPEAKER_02: being able to just keep aggregating performance.
- 51:00SPEAKER_02: Of course,
- 51:00SPEAKER_02: you can see,
- 51:01SPEAKER_02: at some point,
- 51:02SPEAKER_02: when you move from tool use to science,
- 51:04SPEAKER_02: for example,
- 51:04SPEAKER_02: you see the tool use plot drop a bit by around 10-15%.
- 51:10SPEAKER_02: So I feel like this will happen anyway,
- 51:12SPEAKER_02: you know,
- 51:12it's kind of you move from your local minima to a Neighborhood
- 51:16that is still good,
- 51:17SPEAKER_02: but not as good.
- 51:18Yeah You kind of sacrifice a bit of like specialization for
- 51:21like generalization,
- 51:23SPEAKER_01: which is which is like fair game,
- 51:24SPEAKER_01: right?
- 51:25I think it's good because even with human like one guy that
- 51:29just like look at one
- 51:30thing right versus like somebody that has like a 17 different
- 51:34skill set and
- 51:36uh okay like the generalist will be like more useful generally
- 51:38but like when you
- 51:39get to something that is that need require like just doing one
- 51:42you you're much
- 51:43better getting a specialized like a person but like looking at
- 51:47this plot and then
- 51:48SPEAKER_01: looking at the run on the right i'm thinking like
- 51:52SPEAKER_01: During the frontier pre-training,
- 51:56SPEAKER_01: post-training pipeline,
- 51:58does it make sense to weave in RL a bit earlier or
- 52:02self-distillation like this
- 52:03SPEAKER_01: earlier in the process versus just jamming
- 52:06SPEAKER_01: supervised fine tuning in there.
- 52:08SPEAKER_02: I think it's completely makes sense.
- 52:10SPEAKER_02: And this is actually correspond to what we see frontier labs start to do.
- 52:15SPEAKER_02: Basically,
- 52:15SPEAKER_02: if you look at like recent paper,
- 52:18SPEAKER_02: you see that they kind of took all of their SFT data and pushed it into
- 52:22SPEAKER_02: pre-training.
- 52:24So now pre-training is kind of a mixture of like random
- 52:27internet data and a very
- 52:28SPEAKER_02: well structured high quality SFT data sets.
- 52:32and so this all happen at once so you don't have this
- 52:35sequential training because
- 52:37they know that the sequential training is bad and then after
- 52:40that you just move to
- 52:41a big RL training and in the RL training since it's on policy
- 52:47you can allow
- 52:47yourself to have different phases right math and then code and
- 52:51etc but at least
- 52:52SPEAKER_02: they stop doing like a separate SFT at least some of them
- 52:56One caveat which I would add here is that in some cases you
- 53:00might expose yourself
- 53:03sequentially to environments which partially contradict each
- 53:06other We have one
- 53:08example in Thomas' paper actually where you interact with users
- 53:11and it can be the
- 53:12same user and the preference changes but in the beginning that
- 53:15user wants very
- 53:17SPEAKER_00: detailed answers and eventually that changes to very concise answers
- 53:21SPEAKER_00: And of course there's no way to be good with respect to both.
- 53:23SPEAKER_00: They're contradictory, right?
- 53:26And so I think to some extent that's driving most of the
- 53:31degradation you would see
- 53:32SPEAKER_00: in on policy methods as you stack up your training.
- 53:36And so a question if you are a model designer is either how do
- 53:44I sort or organize
- 53:46my tasks in such a way such that I minimize the adverse effects
- 53:54so that I kind of
- 53:54prioritize my tasks correctly so that I train last on the tasks
- 53:59which I want to
- 53:59SPEAKER_00: prioritize most
- 54:00SPEAKER_00: Or alternatively,
- 54:01and this is I think what I've seen from a lot of the recent
- 54:06frontier labs is that
- 54:07SPEAKER_00: they have some relatively small multi-stage training step at the end
- 54:15where they mix so they do the sequential on policy learning for
- 54:19the bulk of the
- 54:19training and then they do have a very small stage where they
- 54:21take some data from
- 54:22all of these different environments and then train with respect
- 54:26to a checkpoint
- 54:28they do on policy distillation with respect to a checkpoint
- 54:30that was already good
- 54:31at that task to just it's just a way of re-weighting your
- 54:35priorities at that point
- 54:36it's not a way of of trying to learn a new a new skill anymore
- 54:39but it's just a way
- 54:40SPEAKER_00: okay
- 54:41Maybe I first trained on tool use but then in the end I trained
- 54:44on science and
- 54:45medical but it's actually not the case that I want the model to
- 54:48focus on medical
- 54:49and kind of sacrifice its tool use priority but I want to have
- 54:54a more balanced I
- 54:55think like it made me think so because I've talked to the
- 55:00founder the head of AI at
- 55:02SPEAKER_01: the math academy
- 55:04and it was like it's very heavy on this is human stuff right
- 55:07but it's very heavy on
- 55:08like having this hierarchical kind of learning paradigm which
- 55:12is kind of like the
- 55:13same thing here like you would like put the the different
- 55:16training blocks in a way
- 55:18SPEAKER_01: that
- 55:19So if tool use is going to be needed in one dataset environment
- 55:23and dataset later
- 55:24SPEAKER_01: on,
- 55:24you much better learn it here because otherwise you're going to
- 55:26learn tool use
- 55:27SPEAKER_01: while you're supposed to do like browsing and stuff like that.
- 55:30SPEAKER_01: So I think that makes sense.
- 55:31SPEAKER_01: And like, I really like the idea of
- 55:33So the second kind of learning that we'll touch upon is
- 55:35learning knowledge and
- 55:55This is really cool because knowledge for example is something
- 55:58that you cannot
- 55:59learn with normal RL It's kind of like doesn't matter how many
- 56:04you know like what
- 56:05kind of verifiable reward I'll have if for example I'll take a
- 56:09model with at the
- 56:11end of 2024 knowledge cutoff and we'll try to train it on a
- 56:15data set of like what
- 56:17SPEAKER_02: happened in 2025 as we did
- 56:20like no matter how many you know how much exploration it will
- 56:23do how many guesses
- 56:24it will have what will be your group size it will not be able
- 56:27to kind of like guess
- 56:28correctly so this is basically a kind of learning that we're
- 56:31not able to do um with
- 56:34SPEAKER_02: on policy algorithm before
- 56:36but now with self distillation it is completely possible you
- 56:39just put the text of
- 56:40whatever new information new knowledge you want to give to the
- 56:44model you give it to
- 56:45the teacher you put it in context and it does just train itself
- 56:50we compared it in
- 56:51our paper to um few baselines first of all a normal you know
- 56:55like in context
- 56:57SPEAKER_02: learning or rag um we also compared to cpt continued
- 57:03continual pre-training which is a very standard of just next
- 57:06token prediction on
- 57:08the document and also what we call like supervised training but
- 57:12is like kind of
- 57:13self-data a self-study algorithm that was popularized recently
- 57:18by microsoft which
- 57:20is taking the document generating a huge data set of
- 57:24synthetically generated
- 57:26questions on it and then train the model to answer on them
- 57:30using normal sft
- 57:32and of course we compare it to our own policy self-distillation
- 57:37so the base if we
- 57:39look at the result the base model of course is not aware at all
- 57:43of the new like
- 57:44SPEAKER_02: information from 2025 so it's get zero but
- 57:47in like what we call accuracy both strict and lenient accuracy
- 57:52just we ask the
- 57:53model questions about this event that happened during 2025 and
- 57:57expected to answer
- 57:58SPEAKER_02: and also on OD accuracy what is OD accuracy here we checked a bit for
- 58:02SPEAKER_02: generalization so this is not like a direct question about you know
- 58:07what happened in this storm during 2025 but it's more like a
- 58:11questions that need to
- 58:12use the knowledge from in a more general aspect for example
- 58:17what were the 10
- 58:18biggest natural disasters in the last 10 years where the new
- 58:23knowledge should
- 58:25change the way you answer them because some of these 10 biggest
- 58:28natural disaster
- 58:29SPEAKER_02: happened in the last year
- 58:31but it's not directly you don't like you know it's kind of you
- 58:34don't ask kind of
- 58:35the model specifically on events which require this is a way to
- 58:39see how much the
- 58:40new knowledge was actually incorporated into then what the
- 58:44model actually knows
- 58:45about the world its world model and then we compare it to like
- 58:49CPT which do
- 58:51SPEAKER_02: terribly
- 58:52like barely able to learn anything SFT which we know like from
- 58:57previous work
- 58:58they're doing kind of well are able to get like around 80%
- 59:01strict accuracy but
- 59:03doing not that well on the OOD accuracy so basically if you do
- 59:06SFT you kind of
- 59:07memorize answers this is a very known problem with like offline
- 59:13algorithm where you
- 59:14SPEAKER_02: do teacher forcing
- 59:15Well when do you do SDFT algorithm you are able to get better
- 59:19accuracy but also
- 59:21mainly better OD accuracy which means that you actually
- 59:24incorporated the new
- 59:25knowledge into the model So to preface that I want to kind of
- 59:29start by discussing
- 59:31in which ways on policy methods or current on policy methods
- 59:34have been bottlenecked
- 59:35and that touches also on what Edan mentioned on why for example
- 59:38they would not be
- 59:39SPEAKER_00: able to learn knowledge
- 59:40So the first way in which they were bottlenecked is that they
- 59:43methods such as GRPO
- 59:45or general methods in RLVR so reinforcement learning with
- 59:48verifiable rewards
- 59:50receive only a scalar signal per rollout from their environment
- 59:54and that of course
- 59:54SPEAKER_00: bottlenecks
- 59:55SPEAKER_00: how much they can learn from their environment.
- 59:57SPEAKER_00: For example,
- 59:58SPEAKER_00: if you want to learn knowledge through a scalar signal,
- 1:00:00SPEAKER_00: that will be very difficult.
- 1:00:02And then the second bottleneck is that then this already weak
- 1:00:06signal is used for
- 1:00:08SPEAKER_00: rollout level credit assignment,
- 1:00:10meaning the policy is not trained in particular on a specific
- 1:00:14token or shown that a
- 1:00:16SPEAKER_00: specific token was good or bad,
- 1:00:17SPEAKER_00: but it's just being shown that the entire rollout was good or bad.
- 1:00:20So there's here's one example that illustrates this and we also
- 1:00:24have this in one of
- 1:00:25our papers and the example is the question is write a python
- 1:00:30function that returns
- 1:00:31all numbers from one to n answer briefly and then in a normal
- 1:00:36process and also in
- 1:00:37SPEAKER_00: our training
- 1:00:38And on policy self-distillation,
- 1:00:39we would sample a response from the model.
- 1:00:42SPEAKER_00: Let's say it would be this Python program.
- 1:00:44SPEAKER_00: And as you can see,
- 1:00:45SPEAKER_00: it returns a list that ranges from one all the way through n,
- 1:00:49SPEAKER_00: so including n.
- 1:00:50SPEAKER_00: And then a feedback could be not to include n.
- 1:00:54and so what would GRPO do or RLVR methods typically it would so
- 1:01:00in GRPO's case it
- 1:01:01would just say all tokens were bad so it would receive a signal
- 1:01:06a negative reward
- 1:01:08SPEAKER_00: or
- 1:01:08or at least on average with respect to the rollout group might
- 1:01:11be negative.
- 1:01:12And so in that case,
- 1:01:13then it would just down weight the probability of all tokens.
- 1:01:17SPEAKER_00: It would make all tokens less likely.
- 1:01:19But instead,
- 1:01:20what on policy self distillation would do is would look over
- 1:01:23the
- 1:01:23SPEAKER_00: The generated tokens and for each ask,
- 1:01:26SPEAKER_00: in hindsight,
- 1:01:27SPEAKER_00: given that you know the feedback was not to include n,
- 1:01:31SPEAKER_00: would you have still generated that token?
- 1:01:33And as you can see here,
- 1:01:35SDPO,
- 1:01:35and this is a real example,
- 1:01:36this is with QUEN38B
- 1:01:38cpo here doesn't change any of the next token predictions for
- 1:01:43any of the tokens
- 1:01:44except for this one token plus that follows the n right because
- 1:01:49that plus token was
- 1:01:50SPEAKER_00: the cause why we included n and so this this is
- 1:01:53SPEAKER_00: So that's basically the intuition.
- 1:01:54SPEAKER_00: So both STPO uses this richer signal,
- 1:01:58SPEAKER_00: this textual signal that describes,
- 1:02:00SPEAKER_00: okay,
- 1:02:01SPEAKER_00: we don't want to include n in our list that we return.
- 1:02:03SPEAKER_00: And it produces denser credit assignment because it actually says,
- 1:02:08SPEAKER_00: okay,
- 1:02:08SPEAKER_00: here,
- 1:02:08SPEAKER_00: this token you should change.
- 1:02:10But as Idan also mentioned before,
- 1:02:12it doesn't only do that over the generated tokens.
- 1:02:17SPEAKER_00: It does this over the entire vocabulary.
- 1:02:20SPEAKER_00: So at every next token prediction,
- 1:02:23the hindsight policy or the teacher asks okay any of the
- 1:02:28possible next tokens here
- 1:02:29at that position should they now become more likely or less
- 1:02:33likely so then in this
- 1:02:35example you would get but let's just look at the position of
- 1:02:37the plus because
- 1:02:39SPEAKER_00: that's maybe most intuitive
- 1:02:41you would not only say that the plus here should become much
- 1:02:44less likely you would
- 1:02:45also say the alternative how you should have continued at that
- 1:02:48position should be
- 1:02:49to just have closed the list right and complete it and that
- 1:02:53becomes much more
- 1:02:54SPEAKER_00: likely as is indicated by this blue color
- 1:02:57so how does this manifest so if we look here now as an at an
- 1:03:01example where there's
- 1:03:03this type of behavior but aggregated now across multiple
- 1:03:06rollouts you see this
- 1:03:07interesting pattern so so grpo basically asks was a response
- 1:03:12better or worse than
- 1:03:14average across a group rollout and then would make all tokens
- 1:03:17either more likely
- 1:03:19SPEAKER_00: which would be blue here or less likely
- 1:03:22SPEAKER_00: And in contrast,
- 1:03:23SPEAKER_00: SDPO,
- 1:03:24SPEAKER_00: how we called the on-policy self-distillation in this RL context,
- 1:03:29would ask was this particular token good or bad in hindsight
- 1:03:33given the additional
- 1:03:34SPEAKER_00: context and then the policy would comment on each
- 1:03:37individual token or even on each individual possible next token
- 1:03:43prediction.
- 1:03:43SPEAKER_00: And so one natural setting that we looked at was the RLVR setting.
- 1:03:48SPEAKER_00: So the typical setting where RLVR methods are applied,
- 1:03:52SPEAKER_00: such as GRPO,
- 1:03:55SPEAKER_00: which is the setting where you learn from success and failure,
- 1:03:57SPEAKER_00: where you have some environment that just tells you whether you're
- 1:04:01SPEAKER_00: response was correct or your response was wrong.
- 1:04:05SPEAKER_00: Our first question was, okay, even in this kind of environment where
- 1:04:09SPEAKER_00: where we don't get any additional signal from the environment,
- 1:04:12SPEAKER_00: can we still benefit from better credit assignment?
- 1:04:15SPEAKER_00: And so how you would do self-distillation here is very simple.
- 1:04:18SPEAKER_00: So you would do similar as in GRP,
- 1:04:20SPEAKER_00: you would do multiple rollouts per question,
- 1:04:22and then you would have some correct attempts and you would
- 1:04:25have some wrong
- 1:04:26SPEAKER_00: attempts.
- 1:04:26And then you can just put the correct attempts in the context
- 1:04:29of the teacher for
- 1:04:30SPEAKER_00: the wrong attempts.
- 1:04:31In the same way as Idan described how you would use
- 1:04:34self-distillation when you
- 1:04:35SPEAKER_00: actually already have correct demonstrations.
- 1:04:38SPEAKER_00: Here you would use these correct demonstrations,
- 1:04:40but these would be generated by the model itself as it explores
- 1:04:44in the environment.
- 1:04:45SPEAKER_00: And so here's one example.
- 1:04:48SPEAKER_00: This is on a chemistry reasoning dataset.
- 1:04:50SPEAKER_00: And what we saw here is that we saw two things.
- 1:04:53SPEAKER_00: So we saw that SDPO both converges here to a much higher accuracy,
- 1:04:58SPEAKER_00: but it also converges much faster in training wall clock time,
- 1:05:03SPEAKER_00: which this fast convergence we generally saw because as we discussed
- 1:05:08it just provides much denser credit assignment so a much richer
- 1:05:11update signal and
- 1:05:13another thing which I don't know we don't go in here right now
- 1:05:15is that we also
- 1:05:16typically saw that it learns more efficient reasoning so it
- 1:05:19uses more or it uses
- 1:05:20rather less of these typical reasoning tokens such as hmm and
- 1:05:25weight etc which GRPO
- 1:05:27tends to produce and another thing that I think I've seen in
- 1:05:31the paper is that like
- 1:05:33SPEAKER_01: they're also way shorter than like the GRPO one
- 1:05:37uh do you have any thought about like why exactly like I know
- 1:05:41that like the the
- 1:05:42just standard grpo formulation like when it's wrong it's going
- 1:05:45to be wrong for
- 1:05:46longer and stuff there's bias for that uh but why is it like
- 1:05:50that much uh more
- 1:05:52SPEAKER_01: efficient shorter
- 1:05:53SPEAKER_00: Yeah, it's a great question.
- 1:05:54SPEAKER_00: I think it touches on what we discussed earlier,
- 1:05:58SPEAKER_00: namely that in hindsight,
- 1:06:00SPEAKER_00: when you kind of critique how you would have solved the problem,
- 1:06:05SPEAKER_00: you often think,
- 1:06:06SPEAKER_00: okay,
- 1:06:07SPEAKER_00: I could have shortened
- 1:06:08SPEAKER_00: Mahdid yacine
- 1:06:23SPEAKER_00: Okay,
- 1:06:23here this was like a circular reasoning loop that wasn't
- 1:06:27unnecessary and it's
- 1:06:29SPEAKER_00: penalizing the policy for that.
- 1:06:31At the same time though,
- 1:06:33I think it's also the case that this points to a particular
- 1:06:37problem in
- 1:06:38SPEAKER_00: GRPO,
- 1:06:39SPEAKER_00: which has also been widely studied at this point,
- 1:06:42SPEAKER_00: that even though,
- 1:06:43SPEAKER_00: you know,
- 1:06:44I think parts of the core intuition when GRPO was first
- 1:06:48introduced and then also
- 1:06:49SPEAKER_00: when the DeepSig R1 paper came out was this idea,
- 1:06:51SPEAKER_00: okay,
- 1:06:51SPEAKER_00: wow,
- 1:06:52we can see that our policy learns to produce longer and longer
- 1:06:55responses and has
- 1:06:56SPEAKER_00: these kind of tokens and starts to actually quote unquote think.
- 1:07:01SPEAKER_00: a lot of work since then has shown that,
- 1:07:02SPEAKER_00: you know,
- 1:07:03SPEAKER_00: often these like not always,
- 1:07:05SPEAKER_00: but sometimes these think additional think tokens,
- 1:07:08SPEAKER_00: they're not necessary for good generalization.
- 1:07:12SPEAKER_00: And they are kind of, I think they're surfacing this form
- 1:07:15of weak credit assignment that GRPO performs in the sense that
- 1:07:20GRPO will just
- 1:07:21SPEAKER_00: upweigh things that tend to work better on average,
- 1:07:25SPEAKER_00: on a very coarse average,
- 1:07:27which are then sometimes these approaches which just do think
- 1:07:32things five times
- 1:07:33SPEAKER_00: over in the same way.
- 1:07:35and we did saw this in the paper that we got the circular
- 1:07:39reasoning in GRPO even
- 1:07:41explicit circular reasoning where GRPO or the model trained
- 1:07:44with GRPO would say I
- 1:07:45SPEAKER_00: am running in circles or things like that maybe one
- 1:07:51which we don't have to go in that much detail because we
- 1:07:53already covered it at this
- 1:07:55point which I wanted to mention is that what we generally saw
- 1:07:58in the RL case but
- 1:08:00also in the in the cases that Eden described before where we
- 1:08:04compared against SFX
- 1:08:06SPEAKER_00: learning from demonstrations is we saw that as you scale models,
- 1:08:09you get better in context learners and those translate to
- 1:08:12better self teachers.
- 1:08:13So better teacher signals that then lead to better student
- 1:08:16models as we train them
- 1:08:17SPEAKER_00: with own policy self distillation.
- 1:08:19SPEAKER_00: I mean, that's a very intuitive thing.
- 1:08:21So maybe the the one major thing which we want to cover which
- 1:08:24we're very excited
- 1:08:25SPEAKER_00: about is how these methods can really uncover new data or unlock new data
- 1:08:31modalities that we can use for training and one very natural
- 1:08:34one is this idea of
- 1:08:35learning from which rich feedback so here's just one very
- 1:08:39practical example to
- 1:08:40illustrate this where the question is how much impulse did
- 1:08:42thrusters generate for
- 1:08:43the Mars climate orbiter and let's say the model answer would
- 1:08:48be 100 pounds for
- 1:08:49SPEAKER_00: seconds
- 1:08:50And as I mentioned,
- 1:08:52in normal ROVR,
- 1:08:53the model would just receive a binary reward.
- 1:08:55And so in this case,
- 1:08:56because the answer is wrong,
- 1:08:57it would be just a negative reward.
- 1:09:00SPEAKER_00: But in many cases,
- 1:09:01SPEAKER_00: in many real environments,
- 1:09:03SPEAKER_00: you would have some richer signal,
- 1:09:05SPEAKER_00: some denser feedback loop,
- 1:09:08where the feedback would be much more informative as to what
- 1:09:11the model actually did
- 1:09:12SPEAKER_00: wrong.
- 1:09:12SPEAKER_00: So here in this case, that it should answer in Newton seconds.
- 1:09:15and there's many examples of this kind there's and two examples
- 1:09:20we looked at
- 1:09:21primarily there's code environments for example which produce
- 1:09:24runtime errors failed
- 1:09:25unit tests etc and there's real user conversations and I want
- 1:09:29to focus maybe on the
- 1:09:31real user conversations so one really nice experiment this was
- 1:09:35led by Thomas
- 1:09:40can we learn from raw user conversations in the wild and can
- 1:09:45they actually improve models?
- 1:09:48And what we did here is we took 14,000 real-world user
- 1:09:51conversations from a dataset called
- 1:09:53wildchat which was produced some years back by Allen AI and
- 1:09:58then what we did is we
- 1:09:59did kind of the natural thing which we discussed we split this
- 1:10:03into triplets where
- 1:10:04each triplet consists of the prompt or the history of the
- 1:10:07conversation up to that
- 1:10:08point then the model response and then the follow-up user
- 1:10:11response which would
- 1:10:13indicate maybe what was an issue a possible issue with the
- 1:10:17previous assistance
- 1:10:18SPEAKER_00: response and we would train with on that with on policy self-distillation
- 1:10:22and so what we then did is then we evaluated the trained model
- 1:10:27on a diverse suite
- 1:10:28of benchmarks so for alignment for instruction following for
- 1:10:32reasoning for creative
- 1:10:33writing and also for knowledge here with MMLU Pro and what we
- 1:10:37saw here is that on
- 1:10:38several of these benchmarks and here particularly for alignment
- 1:10:42reasoning and
- 1:10:43creative writing the model substantially improved here this is
- 1:10:46with QUEN38b but we
- 1:10:47SPEAKER_00: also tried this with other models and this was very surprising to us why
- 1:10:51SPEAKER_00: Well,
- 1:10:51SPEAKER_00: because this data that we trained on is in some sense free data,
- 1:10:55SPEAKER_00: as Idan also mentioned before.
- 1:10:56This is data which we just get through deploying our model and
- 1:11:03letting it interact with users.
- 1:11:04SPEAKER_00: It's not the user conversation.
- 1:11:06SPEAKER_00: And of course, the big companies that perform a lot of inference
- 1:11:21SPEAKER_00: yacine
- 1:11:36explicit rewards just by raw interaction with the environment
- 1:11:39and receiving textual feedback.
- 1:11:41So this is like one example of how you can learn from this raw
- 1:11:45feedback from
- 1:11:45SPEAKER_00: humans,
- 1:11:46SPEAKER_00: but it's like a cross population of humans,
- 1:11:48SPEAKER_00: right?
- 1:11:48SPEAKER_00: So it's still a typical post-training objective.
- 1:11:51The one last thing which we wanted to discuss here in these
- 1:11:56slides is how can we
- 1:11:58SPEAKER_00: use on-policy self-distillation
- 1:12:00for a really quote unquote continual learning system that is
- 1:12:04deployed in the wild.
- 1:12:07And I'm going to give a few examples of what does that even
- 1:12:11mean and how could such
- 1:12:13SPEAKER_00: a system look like.
- 1:12:14So one early example that we started looking at was how can we
- 1:12:18discover solutions
- 1:12:20SPEAKER_00: to very hard problems.
- 1:12:21SPEAKER_00: So we didn't want to go with
- 1:12:22SPEAKER_00: super hard problems.
- 1:12:24So what we did is we took problems from coding tasks from
- 1:12:26LiveCodeBench that the
- 1:12:28SPEAKER_00: model was not able to solve across a lot of attempts.
- 1:12:32And then what we were interested in is how quickly would the
- 1:12:35model discover the
- 1:12:36SPEAKER_00: solution to that task?
- 1:12:37SPEAKER_00: And how we quantified that was through this discovery at K.
- 1:12:40SPEAKER_00: metric.
- 1:12:41So this is just the probability of solving a task within k
- 1:12:45environment interactions,
- 1:12:47right?
- 1:12:47SPEAKER_00: So this is what I'm going to plot here.
- 1:12:49SPEAKER_00: And the tasks we considered here were really hard tasks.
- 1:12:53SPEAKER_00: So these were tasks where the pass at 64 was less than 3%,
- 1:12:57SPEAKER_00: meaning that if you were to 100 times each time sample 64 solutions,
- 1:13:02SPEAKER_00: only in three of these 100 cases,
- 1:13:04the model would actually have sampled any correct solution in
- 1:13:08these 64 attempts.
- 1:13:09and then the simplest baseline is what's typically called best
- 1:13:13of k which just
- 1:13:14repeatedly samples solutions from the base model and we wait
- 1:13:19until it has found a
- 1:13:20solution and so actually for best of k this discovery at k
- 1:13:23metric is the same as
- 1:13:24SPEAKER_00: pass at k
- 1:13:25because here there's no sequentiality it's just repeated
- 1:13:28sampling and in some sense
- 1:13:30the best of k baseline here is what would correspond to if you
- 1:13:34were to try to run
- 1:13:35something like GRPO here right because GRPO does not have a
- 1:13:38learning signal any
- 1:13:39SPEAKER_00: learning signal until you get the first solution and
- 1:13:43of course the feedback from the environment here would just be
- 1:13:46runtime errors and
- 1:13:47code unit tests and so what we saw is that in on these hard
- 1:13:52tasks running self
- 1:13:53SPEAKER_00: distillation got a significant speed up over best of k and also in
- 1:13:57SPEAKER_00: and another baseline,
- 1:13:58SPEAKER_00: which is just a multi-turn baseline,
- 1:14:00which keeps all of the conversation history with the
- 1:14:03environment in context until
- 1:14:05SPEAKER_00: it runs out of context and then has a first in first out queue.
- 1:14:08SPEAKER_00: And so what this told us, this experiment, is that
- 1:14:12self distillation can really learn to solve hard tasks even
- 1:14:15before it ever solved
- 1:14:17the task right just by the teacher providing directionally
- 1:14:20accurate feedback that
- 1:14:21points towards how to solve the task and this was really one of
- 1:14:24the first ways in
- 1:14:25which we applied self distillation online in a continual way
- 1:14:30when given one
- 1:14:31particular task and so based on this people took this way
- 1:14:35further and this is super
- 1:14:36SPEAKER_00: exciting so I think very shortly after we put out this paper
- 1:14:40SPEAKER_00: Someone on Twitter had this idea of continual code.
- 1:14:45SPEAKER_00: So like cloud code, but running a local model.
- 1:14:49SPEAKER_00: And that model is actually learning as you go.
- 1:14:51SPEAKER_00: It's not just saving things in context or scaffolding.
- 1:14:55SPEAKER_00: It's actually updating the model parameters.
- 1:14:57so here you have a gif running and basically as he interacts
- 1:15:01with the model
- 1:15:02whenever the model does something that they don't want the
- 1:15:05model does some update
- 1:15:07so here it's it's thinking and then he will reject the edit and
- 1:15:12he will say that he
- 1:15:14SPEAKER_00: wants the helper to be minimal
- 1:15:16SPEAKER_00: and some other instruction.
- 1:15:17SPEAKER_00: And then the model would actually do a training step.
- 1:15:19So it was a first sketch of an idea of like a continual
- 1:15:22learning system that
- 1:15:23SPEAKER_00: actually updates weights as you interact with it naturally.
- 1:15:26SPEAKER_00: And then I think over the last few weeks, this has been picked up by
- 1:15:31a library called OpenClawRL which is running on policy
- 1:15:35self-distillation under the
- 1:15:36hood but is extending this way beyond just coding agents but
- 1:15:40putting this into
- 1:15:41OpenClaw so having your agent interact with whatever tools you
- 1:15:45give it access to
- 1:15:47and then having the agent actually learn over time in an
- 1:15:49asynchronous fashion.
- 1:15:50And so that's all these things we're super excited by because
- 1:15:54these really point to
- 1:15:55a future where you have a model that learns online as you are
- 1:15:59interacting with it.
- 1:16:00SPEAKER_00: And so this was kind of an overview of that.
- 1:16:04Just one more mention is that while there have been the three
- 1:16:08papers that Edan and
- 1:16:11SPEAKER_00: Thomas and I have been leading,
- 1:16:13there's also been a lot of other really cool research that has
- 1:16:16come out and here's
- 1:16:17just a few papers all research that has come out in the last
- 1:16:21month or so so yeah
- 1:16:23that's all and thanks a lot I'll also add one thing that both
- 1:16:28SDFT and SDPO the two
- 1:16:30SPEAKER_02: versions of the self-distillation algorithms
- 1:16:33SPEAKER_02: are available with TRL.
- 1:16:36In the last week,
- 1:16:37Hug and Face people merge implementation in their code base.
- 1:16:41So,
- 1:16:42you know,
- 1:16:43it's make it much easier for everyone to play with these ideas.
- 1:16:46SPEAKER_01: Thanks a lot for the presentation you asked.
- 1:16:48SPEAKER_01: There's Shannon Sands from New Research.
- 1:16:51I want to know like,
- 1:16:52okay,
- 1:16:53like this is what you guys have been working on.
- 1:16:55SPEAKER_01: What are you guys working on now in this direction?
- 1:16:58SPEAKER_01: Can you share a bit with this?
- 1:17:00For me one of the kind of like there are two things that remain
- 1:17:05to do at least in
- 1:17:06my eyes in this field one is just scaling up and that kind of
- 1:17:11hard to do in the
- 1:17:11academia and we hope that you know various frontier labs will
- 1:17:14just take this idea
- 1:17:15SPEAKER_02: and scale them
- 1:17:17The other one is which is more of like not in the scale up but
- 1:17:22more in the last
- 1:17:23example that Jonas gave where like you have one user interact
- 1:17:27with one agent and
- 1:17:29SPEAKER_02: you want to improve from this conversation
- 1:17:31is on the opposite side is about sample efficiency where you
- 1:17:35don't have scale you
- 1:17:36just have one user providing a few points of feedback and the
- 1:17:39question is how can
- 1:17:40you learn from that and like in context learning can learn from
- 1:17:43that like we know
- 1:17:44that put even one sentence you know into the context and it
- 1:17:47will change the model
- 1:17:48SPEAKER_02: behavior quite well
- 1:17:50SPEAKER_02: But self-distillation,
- 1:17:51SPEAKER_02: although it's able to propagate the same kind of behavior change,
- 1:17:56SPEAKER_02: it's still limited by the fact that it's doing gradient descent,
- 1:18:00SPEAKER_02: basically.
- 1:18:00And we know that gradient descent only takes a small push the
- 1:18:03model a bit at a time
- 1:18:05SPEAKER_02: that's inert in the algorithm.
- 1:18:07SPEAKER_02: So currently we're looking into ways to make the supervision even denser.
- 1:18:12to make the update such that even with a single point of
- 1:18:17environment feedback you
- 1:18:18can change the model quite a lot without forgetting so you know
- 1:18:22an extreme case
- 1:18:23would be that if the user say never use the word the letter F
- 1:18:27that's it the model
- 1:18:28SPEAKER_02: will stop using the letter F
- 1:18:30is one update to the weights that's kind of for me that's the
- 1:18:33dream of course this
- 1:18:34is like a silly example but like you know kind of the same way
- 1:18:37that I don't need to
- 1:18:38tell you things thousand times in order for you to learn the
- 1:18:41concept I want models
- 1:18:43not to be needed to tell thousand times in order to learn
- 1:18:45content so this is kind
- 1:18:47of like what excites me these days and we have some ideas that
- 1:18:50hopefully in the
- 1:18:51SPEAKER_02: next months or so we'll share about how to do it
- 1:18:53SPEAKER_00: Yeah, same here.
- 1:18:55SPEAKER_00: So I do think this is maybe the central,
- 1:18:59SPEAKER_00: at least one of the very central immediate kind of questions.
- 1:19:05SPEAKER_00: And I think this is one,
- 1:19:06like this question of can we have a learning paradigm that can
- 1:19:09actually do
- 1:19:10SPEAKER_00: parameter updates,
- 1:19:11SPEAKER_00: but be as learning efficient or as sample efficient as humans are,
- 1:19:16SPEAKER_00: right?
- 1:19:16SPEAKER_00: This is like a big...
- 1:19:18SPEAKER_00: and has been a big goal of the field.
- 1:19:21SPEAKER_00: And I do believe that, right?
- 1:19:23So I think part of the magic of in-context learning has been
- 1:19:27that in-context
- 1:19:28SPEAKER_00: learning is somehow that,
- 1:19:29SPEAKER_00: right?
- 1:19:30SPEAKER_00: It is somehow extremely sample efficient.
- 1:19:33And I think that's what has been so exciting about it and also
- 1:19:37so useful about it.
- 1:19:39SPEAKER_00: But also, as we mentioned, it's inherently transient.
- 1:19:41SPEAKER_00: So the question is, can we do lasting learning?
- 1:19:45but in the same sample efficiency as say in context learning or
- 1:19:49humans is a very
- 1:19:51SPEAKER_00: appealing question.
- 1:19:53SPEAKER_00: And I do think like there's a lot of promise there.
- 1:19:56And then of course there's other things like also the thing
- 1:19:59which I sketched out
- 1:20:00earlier a bit was this question of how can a model elicit the
- 1:20:05or like find the
- 1:20:06SPEAKER_00: right feedback or seek out the right feedback from its environment.
- 1:20:11SPEAKER_01: There was a question from Jasper Lou from Figma.
- 1:20:16SPEAKER_01: He's asked,
- 1:20:16SPEAKER_01: like,
- 1:20:17has there been any work exploring this in a more subjective
- 1:20:20feedback space where,
- 1:20:22SPEAKER_01: like,
- 1:20:22SPEAKER_01: the feedback is,
- 1:20:23SPEAKER_01: like,
- 1:20:24SPEAKER_01: not necessarily,
- 1:20:24SPEAKER_01: like,
- 1:20:25SPEAKER_01: hard feedback.
- 1:20:27you showed that with the wild chat but like is there any hope
- 1:20:35when the feedback is
- 1:20:36not clear at all like when the model needs to kind of interpret
- 1:20:39it or like generate
- 1:20:40SPEAKER_01: its own feedback
- 1:20:42SPEAKER_00: I think yes.
- 1:20:43SPEAKER_00: There has been several things which we looked at.
- 1:20:46SPEAKER_00: So one in terms of subjective feedback,
- 1:20:49something which I didn't mention or which we didn't mention in
- 1:20:52the presentation was
- 1:20:53part of what Thomas also looked at was personalization to user
- 1:20:57preferences.
- 1:20:58SPEAKER_00: So
- 1:20:59as opposed to the wild chat example where we just did general
- 1:21:02alignment to a large population.
- 1:21:06SPEAKER_00: The question of,
- 1:21:06SPEAKER_00: okay,
- 1:21:06SPEAKER_00: if you interact with the user and the user exhibits certain preferences,
- 1:21:09SPEAKER_00: like I don't want emojis or I don't like sycophancy,
- 1:21:13SPEAKER_00: right?
- 1:21:14SPEAKER_00: or I want short responses.
- 1:21:17The user,
- 1:21:18like the model is able to pick this up through on policy
- 1:21:20self-distillation.
- 1:21:22SPEAKER_00: And so that is like a subjective type of feedback.
- 1:21:25SPEAKER_00: And then in terms of an imperfect type of feedback,
- 1:21:28I think that's more what we try to look at in the context of
- 1:21:31this solving hard
- 1:21:32tasks in life code bands thing where the feedback is inherently
- 1:21:36incomplete.
- 1:21:37SPEAKER_00: Like it may be directionally right,
- 1:21:40SPEAKER_00: but if you just see a runtime error,
- 1:21:42SPEAKER_00: it doesn't tell you yet how to solve the problem.
- 1:21:44and it might constrain your solution space in some way but it
- 1:21:48doesn't leak the
- 1:21:49answer and i think both are important and there's lots of cool
- 1:21:52other things we can
- 1:21:53do i'll add that in cases where the feedback is not like very
- 1:21:57clear where like one
- 1:21:58need to reason and extract you know kind of what is the actual
- 1:22:01change that needed
- 1:22:02to be done a very cool idea that one can do and we didn't like
- 1:22:06had time for it but
- 1:22:07i'm sure it will work and i hope someone will take up on that
- 1:22:10is just to let the
- 1:22:11SPEAKER_02: model reason a bit you know
- 1:22:13give it the context and just before you get to the part where
- 1:22:16there is a new answer
- 1:22:17that you do self distillation over give it the opportunity to
- 1:22:20do some chain of
- 1:22:21thought understand a bit what is the thing you need to take out
- 1:22:26of this additional
- 1:22:27information yeah and I'm sure it will work I'm sure it will
- 1:22:30improve things and
- 1:22:31again I'm sure we'll also see a paper doing that in the next
- 1:22:34few months someone
- 1:22:36SPEAKER_02: would take up on that
- 1:22:38this is good and also like I don't like the the quality aspect
- 1:22:46of the demonstration
- 1:22:47and like what's your thought on that like it and let's say you
- 1:22:53had like a hundred
- 1:22:55SPEAKER_01: okay like not high signal feedback versus like
- 1:23:01SPEAKER_01: 10 very excellent one.
- 1:23:03I know like the model like like to have multiple exemplar and
- 1:23:07then like iterate on that.
- 1:23:11Which in the current paradigm of self distillation,
- 1:23:13which will it prefer here?
- 1:23:14SPEAKER_01: Like the high quality one or like the repeated feedback ones?
- 1:23:21At least I think from the experiment I did that like the more
- 1:23:25the hundred,
- 1:23:27SPEAKER_02: you know,
- 1:23:27SPEAKER_02: more examples,
- 1:23:28SPEAKER_02: but like medium quality.
- 1:23:31will give you better performance eventually as long as there is
- 1:23:34some signal that
- 1:23:35you can extract even just telling the model you know here's
- 1:23:38some feedback think
- 1:23:40again about what you did give me a better answer will push you
- 1:23:43forward um and but
- 1:23:46i'll say that this is not really about the objective it's just
- 1:23:48because as we
- 1:23:49mentioned gradient descent learning likes a lot of example
- 1:23:53likes coverage it's help
- 1:23:54SPEAKER_02: for generalization help for wider minimas stuff like that
- 1:23:59so you have to do a lot with the optimizer that we use yeah i
- 1:24:02would say i would i
- 1:24:04would agree with that i think it's it becomes a little bit of a
- 1:24:08of a subtle
- 1:24:09question when you consider that the feedback that you get or
- 1:24:13that your student gets
- 1:24:15it it need not necessarily come from new data that you attain
- 1:24:19from your environment
- 1:24:20SPEAKER_00: so for example in edun setting when you have expert demonstrations
- 1:24:24you could like solutions like ground truth solutions which may
- 1:24:27be very detailed you
- 1:24:29could have your model sequentially generate attempts train with
- 1:24:33the feedback then
- 1:24:35generate attempts again right which may be a bit better but not
- 1:24:38fully solved it yet
- 1:24:39and then get feedback again and so these are these are examples
- 1:24:42i think where right
- 1:24:43now you would still prefer to get feedback like a lot of times
- 1:24:48right but the
- 1:24:50SPEAKER_00: feedback does not require in that context another data point
- 1:24:54SPEAKER_01: True, true.
- 1:24:55It reminds me of how I was learning,
- 1:25:00I don't know,
- 1:25:03verifiable elements like math or physics.
- 1:25:07SPEAKER_01: I had this kind of red,
- 1:25:08SPEAKER_01: green,
- 1:25:09yellow method where I would just try the exercise and if I
- 1:25:13completely messed it up,
- 1:25:15SPEAKER_01: I would put the red there,
- 1:25:16SPEAKER_01: but I would still look at the answer and try to figure my way out,
- 1:25:20SPEAKER_01: right?
- 1:25:21and I'll just move on and if like I kind of didn't get it right
- 1:25:25but like I knew
- 1:25:26exactly why I put the yellow there but then all the green ones
- 1:25:30that I got first try
- 1:25:31I never do them again like I got them so what's the point right
- 1:25:34but then I just do
- 1:25:35iterative passes on like yet under the yellows and then
- 1:25:39iterative passes on the red
- 1:25:41SPEAKER_01: and I still got like good learning signal out of it.
- 1:25:44And at some point there's like the three last one that are so
- 1:25:47kind of all messed up.
- 1:25:49And in this specific case,
- 1:25:51I know that like I need to get better feedback.
- 1:25:53SPEAKER_01: So then that one get a better feedback from the teacher or whatever.
- 1:25:57but I feel like it's the kind of same situation where like you
- 1:26:00have multiple I got
- 1:26:01multiple exemplar of the same stuff right but with different
- 1:26:05understanding right
- 1:26:06and one while you go through like the the rest of the training
- 1:26:10data set and then
- 1:26:10you come back to those that you messed up you're not gonna like
- 1:26:14it with the
- 1:26:15feedback will be different because you're gonna do it a bit
- 1:26:17differently you can
- 1:26:17learn something in this specific exercise that will help you in
- 1:26:23those that you're
- 1:26:23SPEAKER_01: struggling
- 1:26:26I feel it's like a similar kind of vibe on the on the on this
- 1:26:30and it also reminded
- 1:26:31SPEAKER_00: me of how I how I studied for exams
- 1:26:35SPEAKER_01: And last question.
- 1:26:37SPEAKER_01: Oh, I want you to think hard here.
- 1:26:39SPEAKER_01: Like,
- 1:26:40do you think there is a setting where like self distillation
- 1:26:45will be kind of
- 1:26:47SPEAKER_01: strictly poor than like the standard oil VRO GRPO?
- 1:26:51SPEAKER_01: Like,
- 1:26:51do you think that there's a setting here that like self
- 1:26:55distillation will struggle
- 1:26:56SPEAKER_01: compared to GRPO?
- 1:26:58SPEAKER_02: One setting that is kind of obvious is when you have weak models, right?
- 1:27:02If you're trying to improve a 350 million parameter model with
- 1:27:06a very weak in
- 1:27:08SPEAKER_02: context learning capabilities,
- 1:27:09SPEAKER_02: it will not work the best.
- 1:27:10SPEAKER_02: I'll say that like,
- 1:27:12SPEAKER_02: um,
- 1:27:13SPEAKER_02: we tried,
- 1:27:14SPEAKER_02: um,
- 1:27:15SPEAKER_02: here in MIT to apply self distillation to robotics,
- 1:27:18SPEAKER_02: to vision language action models,
- 1:27:20SPEAKER_02: VLAs,
- 1:27:21SPEAKER_02: and it just doesn't work.
- 1:27:23Like this model does not have in context learning capabilities
- 1:27:26strong enough.
- 1:27:27um to actually learn while you know jobio even simple reinforce
- 1:27:33can improve this
- 1:27:33models quite a lot so that i'll say that the obvious uh answer
- 1:27:38um more than that i
- 1:27:40can say that there are cases where i don't know if it will do
- 1:27:44poor but it just not
- 1:27:45SPEAKER_02: really needed if you just want to kind of sharpen the distribution
- 1:27:49SPEAKER_02: you know,
- 1:27:50like whatever you want to learn is already kind of like inside
- 1:27:54your distribution,
- 1:27:56SPEAKER_02: you get,
- 1:27:56SPEAKER_02: let's say,
- 1:27:57SPEAKER_02: always like around 50,
- 1:27:58SPEAKER_02: 60% success rate.
- 1:28:00SPEAKER_02: And you just want to push the 50 to the 100 on this,
- 1:28:03SPEAKER_02: you know,
- 1:28:03SPEAKER_02: inside the group,
- 1:28:04SPEAKER_02: like you sample a group for a prompt,
- 1:28:07SPEAKER_02: you get out of like,
- 1:28:08SPEAKER_02: you know,
- 1:28:09SPEAKER_02: you're like,
- 1:28:10SPEAKER_02: however many like responses you get most of them correct.
- 1:28:13And you just want to make sure that you don't let's make sure
- 1:28:15all of them will be correct.
- 1:28:17then self-distillation is not really like maybe it will work
- 1:28:21but i'll not say like
- 1:28:22avoid the complication you know right pure grpo um because then
- 1:28:27there is nothing to
- 1:28:28gain from like you know some external knowledge additional
- 1:28:32knowledge like really
- 1:28:33good credit assignments stuff like that that were
- 1:28:36self-distillation really shines
- 1:28:38SPEAKER_02: to not help you that's not what you need
- 1:28:39SPEAKER_00: Yeah,
- 1:28:40SPEAKER_00: maybe to pick up on that point,
- 1:28:43SPEAKER_00: I think like normally in RL,
- 1:28:46SPEAKER_00: like there's this trade-off between methods that are unbiased,
- 1:28:51SPEAKER_00: but exhibit a lot of variance and methods that are maybe biased,
- 1:28:54SPEAKER_00: but have much smaller variance.
- 1:28:57SPEAKER_00: And in many ways you can think,
- 1:29:00self distillation in that lineage of kind of a method that may
- 1:29:05be a bit biased
- 1:29:06depending on how good your models and context learning ability
- 1:29:10is etc but which has
- 1:29:11much less variance in its update signal that of course is a
- 1:29:16trade-off right and I
- 1:29:19SPEAKER_00: think
- 1:29:20SPEAKER_00: It depends often on which one is better.
- 1:29:24SPEAKER_00: It depends on whether you're compute bound, whether you're data bound.
- 1:29:27SPEAKER_00: It depends on how much compute you have available in total.
- 1:29:31SPEAKER_00: And so right now,
- 1:29:32SPEAKER_00: as Eden said,
- 1:29:33if you have a large pipeline and you have unlimited compute and
- 1:29:40you have your data
- 1:29:40SPEAKER_00: environment set up that give you like 50% success rate in every group,
- 1:29:46you should run an unbiased method probably and that will be
- 1:29:52tends to be more stable
- 1:29:53SPEAKER_00: right long term I don't it's very hard to predict what will happen
- 1:30:03SPEAKER_01: That makes a lot of sense.
- 1:30:03SPEAKER_00: But yeah,
- 1:30:04I think that the things which like we're or I personally at
- 1:30:08least I can say that
- 1:30:10I'm most excited about are the settings where there isn't even
- 1:30:15a method that can
- 1:30:17SPEAKER_00: learn from that type of data effectively.
- 1:30:19SPEAKER_01: So could you give an example?
- 1:30:22SPEAKER_00: Yeah, for example, learning from user
- 1:30:24conversations or like learning in this kind of online fashion
- 1:30:28in like Open Claw RL
- 1:30:30where you just interact with your environment and your
- 1:30:32environment returns you
- 1:30:33SPEAKER_00: whatever and yeah.
- 1:30:35Because then you increase massively the amount of data that you
- 1:30:39can shove into the
- 1:30:41SPEAKER_01: model and still make it learn in a stable fashion.
- 1:30:45And if there were one thing that we learned over the past two
- 1:30:49decades in deep
- 1:30:50SPEAKER_02: learning,
- 1:30:50SPEAKER_02: that's all about scale.
- 1:30:52SPEAKER_02: Find ways to shove more and more data into your model.
- 1:30:56cool fantastic guys like this was absolutely awesome really
- 1:31:00thankful for you to
- 1:31:01have taken the time to come and answer all of these questions
- 1:31:05and that's it for
- 1:31:05today folks if you're interested in diving more into this whole
- 1:31:09method of self
- 1:31:09distillation check out the links in the description I try to
- 1:31:12put as much paper that
- 1:31:13are relevant as possible and if you have any questions don't
- 1:31:16hesitate to shoot them
- 1:31:17SPEAKER_01: in the comment have a great week