Latest / learning with yacine show / What Continual Learning Actually Means feat. Ilija Lichkovski from Prime Intellect
Transcript
- 0:00SPEAKER_01: Continual learning is one of those problems in ML
- 0:02SPEAKER_01: that everybody agrees is important, but almost nobody agrees on the definition of.
- 0:07SPEAKER_01: Right now we have a wave of things branded
- 0:09SPEAKER_01: as continual learning like memory scaffold,
- 0:11SPEAKER_01: skill files,
- 0:13SPEAKER_01: self-destillation,
- 0:13SPEAKER_01: real-time RL,
- 0:15SPEAKER_01: Replay method,
- 0:16SPEAKER_01: regularization,
- 0:17SPEAKER_01: gradient projection,
- 0:19SPEAKER_01: they're all pointing roughly at the same intuition,
- 0:21SPEAKER_01: which is that models should be able to keep learning after they've been deployed.
- 0:25SPEAKER_01: Even Sutton had to do the confusion by saying that they should be all rebranded
- 0:30SPEAKER_01: as Just learning.
- 0:31SPEAKER_01: Which I kinda agree.
- 0:32SPEAKER_01: So in this video,
- 0:33SPEAKER_01: I'm sitting down with Elijah,
- 0:34SPEAKER_01: who is working at Prime Intellect on efficient weight update for a financial model,
- 0:38SPEAKER_01: and who wrote what I think is the cleanest attempt so far at actually specifying
- 0:42SPEAKER_01: the continual learning problem from first principle.
- 0:44SPEAKER_01: He will show us hyper network results on class incremental learning,
- 0:48SPEAKER_01: including a very interesting negative one,
- 0:49SPEAKER_01: the dependency graph benchmark he's been building,
- 0:52SPEAKER_01: and some preliminary runs We're also going to get sidetracked,
- 0:58SPEAKER_01: talking about cortical cell in a petri dish, learning Pong
- 1:02SPEAKER_00: for some reason.
- 1:03SPEAKER_01: Elijah's core idea is that a functional learning system have five requirements,
- 1:07SPEAKER_01: and most methods miss at least one of them.
- 1:09SPEAKER_01: It should handle non-stationary distribution, so distribution at shift in time.
- 1:13SPEAKER_01: It should preserve general capabilities.
- 1:15SPEAKER_01: It learns sequentially, not in like a multitask setting.
- 1:18SPEAKER_01: It has to be efficient,
- 1:19SPEAKER_01: because when you have infinite compute, it makes the problem kind of trivial.
- 1:22SPEAKER_01: It should support compositional transfer,
- 1:25SPEAKER_01: which is combining primitive,
- 1:26SPEAKER_01: you encounter a completely different training stage in later stage.
- 1:29SPEAKER_01: Every paper mentioned in this conversation is linked in the description, did my best.
- 1:33SPEAKER_01: There are a lot of them, and they're...
- 1:34SPEAKER_01: All worth reading.
- 1:35SPEAKER_01: By the way,
- 1:36SPEAKER_01: if you enjoyed these expert interviews,
- 1:38SPEAKER_01: do consider subscribing to the channel and big shout out to the 40 people
- 1:42SPEAKER_01: that are helping out by being a member of this channel.
- 1:44SPEAKER_01: Love you all.
- 1:44SPEAKER_00: Hello, hello.
- 1:45SPEAKER_00: So yeah, my background is in physics.
- 1:47SPEAKER_00: I think for your audience,
- 1:48SPEAKER_00: it's probably obvious why Physicists gravitate towards machine learning, right?
- 1:53SPEAKER_00: Physics is this very old field.
- 1:54SPEAKER_00: You want to make a contribution,
- 1:56SPEAKER_00: you have to pay your dues for hundreds of thousands of years of research
- 2:00SPEAKER_00: and really catch up to get to the frontier,
- 2:02SPEAKER_00: and then maybe you can make some kind of contribution.
- 2:05SPEAKER_00: But it's a mature field, right?
- 2:07SPEAKER_00: It's not a field that the foundations are being laid off now.
- 2:11SPEAKER_00: Which is exactly what machine learning is.
- 2:12SPEAKER_00: So that attracted me a lot.
- 2:14SPEAKER_00: I did all kinds of stuff during my studies,
- 2:16SPEAKER_00: but afterward I think I settled pretty well in machine learning.
- 2:20SPEAKER_00: I actually started off doing AI safety research.
- 2:23SPEAKER_00: I like to think that if you're serious about your beliefs about AGI and
- 2:28SPEAKER_00: the implications and how close we are,
- 2:30SPEAKER_00: some part of you will need to think about safety.
- 2:32SPEAKER_00: Because if all these claims are true about transformative super intelligence,
- 2:36SPEAKER_00: then of course it's going to have implications.
- 2:39SPEAKER_00: So yeah,
- 2:40SPEAKER_00: I started off in safety,
- 2:41SPEAKER_00: did some research there on some steering methods,
- 2:44SPEAKER_00: then some reinforcement learning, interpretability stuff as well, some legal safety.
- 2:50SPEAKER_00: Well, not very related to my background at all,
- 2:52SPEAKER_00: but yeah,
- 2:53SPEAKER_00: for the last probably year and some change,
- 2:56SPEAKER_00: I've been really interested in continual learning.
- 2:58SPEAKER_00: Basically in AI,
- 3:00SPEAKER_00: we always have this like final frontier,
- 3:03SPEAKER_00: you know, the goalpost shifting It has happened for decades.
- 3:06SPEAKER_00: And we always think there's this next thing.
- 3:08SPEAKER_00: And of course, there's that feeling of continuing learning.
- 3:10SPEAKER_00: My work recently has been on benchmark continuing learning.
- 3:13SPEAKER_00: But as we will discuss,
- 3:14SPEAKER_00: benchmarking it is kind of like getting your foot in the door
- 3:18SPEAKER_00: in having some good ideas about how to tackle it.
- 3:20SPEAKER_00: Because a lot of the field has been talking about continuing learning,
- 3:23SPEAKER_00: but there was very little consensus about what it means and
- 3:27SPEAKER_00: about what it shouldn't mean.
- 3:28SPEAKER_00: So I find myself having pretty strong opinions about that.
- 3:31SPEAKER_01: That's fantastic.
- 3:32SPEAKER_01: So before we even start this up,
- 3:33SPEAKER_01: I was curious to hear your thoughts because that's not
- 3:35SPEAKER_01: the first time I've seen this pattern of people that have studied physics and then
- 3:40SPEAKER_00: that just go on to do like master learning related stuff.
- 3:44SPEAKER_01: Like one that I know is the founder And he did like a bunch of computational
- 3:49SPEAKER_01: or science stuff,
- 3:50SPEAKER_01: but then like he just went on into machine learning and then just built that thing
- 3:55SPEAKER_01: that is now used everywhere.
- 3:56SPEAKER_01: I don't know.
- 3:57SPEAKER_01: He felt very fluent, like in machine learning.
- 3:59SPEAKER_01: It felt like his, his own old background made him like sink into ML pretty easily.
- 4:05SPEAKER_01: So what's your take on that?
- 4:06SPEAKER_00: Yeah, that's a good question.
- 4:07SPEAKER_00: I'm curious to hear your opinion as well.
- 4:09SPEAKER_00: I have landed on the fact that a lot of what physics is concerned with,
- 4:13SPEAKER_00: which is building models of reality that compress the most information in
- 4:17SPEAKER_00: the leanest form, is kind of the same objective as machinery.
- 4:20SPEAKER_00: As a physicist, I say this often, you pride yourself in being lazy, right?
- 4:25SPEAKER_00: You have the biologists,
- 4:26SPEAKER_00: the status game there is how much do you know,
- 4:28SPEAKER_00: and there's no way around learning a lot and having a good intuition about how
- 4:32SPEAKER_00: the different systems in the natural world work.
- 4:34SPEAKER_00: In physics,
- 4:36SPEAKER_00: It's actually a flex to show up to an exam not having memorized the formulas and
- 4:40SPEAKER_00: to be able to derive them on the spot.
- 4:42SPEAKER_00: So in a way,
- 4:42SPEAKER_00: it teaches and rewards this idea that you want to compress observations about
- 4:47SPEAKER_00: the world in the most lean representation.
- 4:50SPEAKER_00: And of course, general form.
- 4:51SPEAKER_00: And I think in a way,
- 4:53SPEAKER_00: physicists really crave this kind of work where they get to take epistemic leaps.
- 4:58SPEAKER_00: But of course, physics now is like the king of sciences, right?
- 5:01SPEAKER_00: It's a mature field.
- 5:02SPEAKER_00: It's one of the crown jewels of scientific endeavor, I think.
- 5:06SPEAKER_00: But lately you hear a lot of academic physicists sounding the alarm that there's
- 5:10SPEAKER_00: a bit of a stagnation in physics.
- 5:11SPEAKER_00: And maybe a part of the attraction of physicists
- 5:16SPEAKER_00: to machine learning is it's part reactionary, right?
- 5:18SPEAKER_00: It's part frustration with a slow pace and the credentialism and really wanting
- 5:23SPEAKER_00: to join a nascent field where you get to lay down a brick or two and have
- 5:27SPEAKER_00: a contribution.
- 5:28SPEAKER_00: What do you think?
- 5:28SPEAKER_01: Yeah, that's true.
- 5:29SPEAKER_01: I think like ML is great because you could go fully empirical
- 5:34SPEAKER_01: and still make awesome, great discovery fast.
- 5:37SPEAKER_01: And I think just from experience,
- 5:38SPEAKER_01: I'm not a physicist,
- 5:39SPEAKER_01: but from my experience being close to these people is
- 5:42SPEAKER_01: that their theoretical background is very, very solid.
- 5:45SPEAKER_01: It's almost like there's rarely anything else that you will need.
- 5:49SPEAKER_01: Maybe a bit of like, I don't know, like neuroscience knowledge or stuff like that.
- 5:53SPEAKER_01: But even then it's not like needed.
- 5:55SPEAKER_01: So you're both well prepared and then it's just more fun because you can just go and do stuff.
- 5:59SPEAKER_01: But I also seen the same pattern with the computationalist actually,
- 6:03SPEAKER_01: that like gravitational And then just dealing with
- 6:05SPEAKER_01: the raw intelligence versus dealing with, I don't know,
- 6:18SPEAKER_01: Yeah.
- 6:39SPEAKER_00: That makes sense.
- 6:39SPEAKER_00: It also, of course, helps that there is proximity in the subject matter itself, right?
- 6:43SPEAKER_00: So in machine learning,
- 6:44SPEAKER_00: I think I've developed a bit of a love-hate relationship with bio-inspired methods.
- 6:49SPEAKER_00: Because on one hand, human brains are an existence proof, hopefully, general intelligence.
- 6:54SPEAKER_00: I think, of course, you have the bitter lesson on the ML side, right?
- 6:56SPEAKER_00: You don't want to overhand engineer things, and you want to let the model learn itself.
- 7:03SPEAKER_00: I think there is still room though for people with an innate understanding
- 7:08SPEAKER_00: of what is happening in the brain and the synaptic mechanisms of memory and whatever
- 7:12SPEAKER_00: to really maybe make a contribution.
- 7:14SPEAKER_00: And there's plenty of examples.
- 7:16SPEAKER_01: Yeah, there's plenty of that.
- 7:17SPEAKER_01: And also there's plenty of reason not to do this, right?
- 7:21SPEAKER_01: Because you don't have the same constraint.
- 7:23SPEAKER_01: Because the brain is one thing, but there's a living thing with actual physical constraint.
- 7:27SPEAKER_01: If you engineer it differently without these constraints, you can remove a lot of the stuff.
- 7:31SPEAKER_01: But I usually like to gravitate towards the two end of the spectrum.
- 7:35SPEAKER_01: I really like when it's liquid AI where the guy just did a lot
- 7:38SPEAKER_01: of just neuron spirit stuff for like the Sakana folks with
- 7:41SPEAKER_01: the continuous start machine.
- 7:42SPEAKER_01: Even the HRM people where they try to kind of cram as much as possible.
- 7:46SPEAKER_01: I think it creates like a good balance between just going for
- 7:52SPEAKER_01: the brain doesn't matter to like,
- 7:53SPEAKER_01: actually, yes,
- 7:54SPEAKER_01: the brain is the only thing that is general that we,
- 7:56SPEAKER_01: that we know in the moment allegedly.
- 7:58SPEAKER_01: I think that we can go into the presentation and I'll be surfacing a bunch of different things.
- 8:02SPEAKER_00: So, you call this a lecture.
- 8:07SPEAKER_00: I'm going to try to make it more fun than that.
- 8:09SPEAKER_00: Let's start from the ground up.
- 8:10SPEAKER_00: Basically, we're going to be talking about lifelong learning, continual learning.
- 8:13SPEAKER_00: In large neural networks,
- 8:15SPEAKER_00: it will be focused on foundation models,
- 8:17SPEAKER_00: because there, the forgetting problem is really interesting, right?
- 8:21SPEAKER_00: My goal here is to have a teapot
- 8:36SPEAKER_00: This is a deep-shaped approach.
- 8:37SPEAKER_00: I don't want to bore you with every single detail.
- 8:39SPEAKER_00: This is not like a survey.
- 8:41SPEAKER_00: I want to motivate some examples that are, I think, illustrative in how to think about it.
- 8:46SPEAKER_00: Let's think about what we are doing with machine learning and specifically deep learning.
- 8:51SPEAKER_00: So the idea with a learner is that it's the probability distribution over some data.
- 8:56SPEAKER_00: And this is how we've been thinking about learning Problem for decades.
- 9:00SPEAKER_00: You have some distribution of data,
- 9:01SPEAKER_00: and then your goal is to iteratively come to a good approximation of that.
- 9:07SPEAKER_00: Again, you can find many examples where people haven't done this.
- 9:11SPEAKER_00: Of course, machine learning is a field with a lot of diversity.
- 9:13SPEAKER_00: But in general,
- 9:14SPEAKER_00: I think the methods that have dominated the zeitgeist have been about distribution
- 9:19SPEAKER_00: that you're randomly sampling,
- 9:20SPEAKER_00: and crucially, the samples that are independent and identically distributed, right?
- 9:24SPEAKER_00: It's a fixed distribution.
- 9:25SPEAKER_00: And this assumption Like you kind of don't think about it if you're training a model
- 9:30SPEAKER_00: to categorize cats and dogs or look at medical images
- 9:35SPEAKER_00: to classify cancerous tissue versus non-cancerous tissue.
- 9:38SPEAKER_00: So here the very problem definition means that you have some distribution
- 9:41SPEAKER_00: where some things mean cancer and some things don't mean cancer.
- 9:44SPEAKER_00: And then you're training the model to learn this distribution.
- 9:47SPEAKER_00: And this has served us very well.
- 9:49SPEAKER_00: And I think this is the part of the tech tree that we've been going down.
- 9:53SPEAKER_00: And we've had great success.
- 9:55SPEAKER_00: The difficulty comes when you introduce a temporal component,
- 9:59SPEAKER_00: when you deal with non-stationarity.
- 10:02SPEAKER_00: In your data.
- 10:03SPEAKER_00: And here,
- 10:04SPEAKER_00: the methods that we're using,
- 10:06SPEAKER_00: again, they take samples that are independent and identically distributed.
- 10:09SPEAKER_00: So that means that they take a fixed distribution.
- 10:11SPEAKER_00: And nothing,
- 10:12SPEAKER_00: when you're calculating your gradients or when you're training your network,
- 10:15SPEAKER_00: cares about any other time point.
- 10:17SPEAKER_00: They look at the current distribution of what data is given to the model.
- 10:20SPEAKER_00: So under this setting,
- 10:21SPEAKER_00: it shouldn't surprise us that Models are not capable of continuing learning.
- 10:25SPEAKER_00: Namely,
- 10:26SPEAKER_00: there are two problems we encounter with continuing learning that I think dominate
- 10:30SPEAKER_00: the landscape.
- 10:31SPEAKER_00: Of course, when you get into the details, there's a lot more to it.
- 10:34SPEAKER_00: I think it's a very fractal structure and we're going to talk about that.
- 10:36SPEAKER_00: For example, like, yeah, I'm not going to spoil myself.
- 10:39SPEAKER_00: One of the problems is catastrophic forgetting.
- 10:41SPEAKER_00: As you're learning a different distribution,
- 10:44SPEAKER_00: you are calculating the gradients back,
- 10:46SPEAKER_00: propagating them into the weights with respect to that distribution,
- 10:49SPEAKER_00: and you're kind of forsaking whatever you saw before.
- 10:51SPEAKER_00: And of course your model is going to forget.
- 10:53SPEAKER_00: So this is the first problem, forgetting, right?
- 10:55SPEAKER_00: You're ruining previous representation.
- 10:56SPEAKER_00: By the way, this was a subject of a recent Nature Perspective paper.
- 11:01SPEAKER_00: About how solving some of the classic challenges in machine learning is
- 11:06SPEAKER_00: a matter indeed of incentive and practice.
- 11:09SPEAKER_00: You want to formulate the right objective, that's the incentive, and provide the right practice.
- 11:12SPEAKER_00: That means give the right distribution of tasks so which the model can improve itself.
- 11:17SPEAKER_00: And this is the paper, it's a nature perspective, but I think it's really well thought out.
- 11:21SPEAKER_00: So like no fancy math in this paper.
- 11:22SPEAKER_00: This is a really good paper for people that want to start thinking
- 11:25SPEAKER_00: about these problems on a more conceptual level.
- 11:28SPEAKER_00: And here they focus on four classic problems in machine learning,
- 11:32SPEAKER_00: systematic generalization,
- 11:33SPEAKER_00: community learning, future app learning, multi-step reasoning.
- 11:36SPEAKER_00: I think a lot of recent methods,
- 11:38SPEAKER_00: including RL fine-tuning and long horizon agents and whatnot,
- 11:43SPEAKER_00: They have gotten really,
- 11:45SPEAKER_00: really good at tackling these problems,
- 11:47SPEAKER_00: like few-shot learning and multi-step reasoning.
- 11:49SPEAKER_00: The models are pretty good now at that.
- 11:51SPEAKER_00: Few-shot learning here, I think is another way of saying in context learning.
- 11:54SPEAKER_00: And by the way,
- 11:55SPEAKER_00: do you think I should be defining these terms or is the level
- 11:58SPEAKER_00: of complexity now fine?
- 11:59SPEAKER_01: I think it's good.
- 12:00SPEAKER_01: I think the way you're framing it right now is good.
- 12:02SPEAKER_01: I just wanted to add on something that we discussed earlier on,
- 12:05SPEAKER_01: because it's something that as someone that studied the brain,
- 12:08SPEAKER_01: I understand this from the biological standpoint.
- 12:11SPEAKER_01: But the learning in the brain has a big time component.
- 12:14SPEAKER_01: Like timing is super important.
- 12:16SPEAKER_01: The time constraint is there and then it is embedded in how we're learning.
- 12:21SPEAKER_01: There's no learning without that versus the core like pre-training
- 12:25SPEAKER_01: or even Post-training methodologies are more on the synthetic side.
- 12:29SPEAKER_01: There's no real concept of time.
- 12:30SPEAKER_01: It's like just data that is like being set up there.
- 12:33SPEAKER_01: This is also what I think attract me for the like biologically inspired stuff,
- 12:38SPEAKER_01: because they try to add that.
- 12:39SPEAKER_01: They try to add this constraint in the continuous type machine.
- 12:43SPEAKER_01: There's a tick that is just actually going on.
- 12:45SPEAKER_01: And then the model needs to kind of take that into consideration.
- 12:47SPEAKER_01: Do you think like this, the time aspect is like a very crucial part of the continuous learning?
- 12:53SPEAKER_01: That's a good question.
- 12:54SPEAKER_00: I think it depends on what you mean by time, right?
- 12:57SPEAKER_00: I think some approaches to time are more principled than others,
- 13:00SPEAKER_00: but one particular that I've been interested in is thinking about how
- 13:05SPEAKER_00: to reconcile things that are learned at different time points
- 13:09SPEAKER_00: when your environment is non-stationary.
- 13:11SPEAKER_00: Currently, when you're pre-training a language model, that's a long process, right?
- 13:15SPEAKER_00: If you look at the Kimi 2 or 2.5 loss curve, it has like many steps.
- 13:19SPEAKER_00: It's a beautiful loss curve.
- 13:20SPEAKER_00: So there,
- 13:21SPEAKER_00: there's clear It's clearly a temporal element,
- 13:23SPEAKER_00: and the model actually has, over time, built up representations better and better.
- 13:27SPEAKER_00: So it starts off,
- 13:28SPEAKER_00: I don't know,
- 13:29SPEAKER_00: learning about the fact that when you start with a randomly initialized transformer
- 13:33SPEAKER_00: and you start pre-training it,
- 13:34SPEAKER_00: like of course it doesn't learn super complex rhetoric at first,
- 13:38SPEAKER_00: but first it learns some basic structure about character frequencies.
- 13:42SPEAKER_00: Okay, periods come every end characters.
- 13:46SPEAKER_00: Then it learns about paragraphs, maybe some structure, maybe it learns the most basic words.
- 13:50SPEAKER_00: And as you go along,
- 13:51SPEAKER_00: the model actually has a,
- 13:53SPEAKER_00: it improves its representations and sample efficiency increases.
- 13:56SPEAKER_00: But here, nothing is temporal.
- 13:58SPEAKER_00: There is no sense of And that's why you're trying to take history in the model.
- 14:02SPEAKER_00: Because with your pre-training distribution,
- 14:03SPEAKER_00: the goal is to approximate a multi-task distribution over this whole internet
- 14:07SPEAKER_00: or pre-training corpus.
- 14:08SPEAKER_00: Now it's probably very synthetic and maybe the internet picture is very outdated now.
- 14:16SPEAKER_00: So you still have this assumption of independent and identically distributed data.
- 14:22SPEAKER_00: What I am interested in is what happens when you sequentially train the model and
- 14:27SPEAKER_00: the model does need to distinguish what happened at what stage of training.
- 14:31SPEAKER_00: So for example,
- 14:31SPEAKER_00: if you have a model deployed in a legal company and In March,
- 14:36SPEAKER_00: they have a certain client,
- 14:38SPEAKER_00: and there are certain contacts about the client,
- 14:39SPEAKER_00: and then in October,
- 14:41SPEAKER_00: they get another client that is different, but maybe some things are similar.
- 14:44SPEAKER_00: The model needs to know that the first client happened before the last client,
- 14:48SPEAKER_00: and then to selectively remember the important things about the old client,
- 14:51SPEAKER_00: but maybe forget some of the things about the old client.
- 14:54SPEAKER_00: And I think here, some notion of temporality will be Important.
- 14:58SPEAKER_00: And maybe this is not even that hard to accomplish at this scale.
- 15:01SPEAKER_00: This is one level of temporality,
- 15:02SPEAKER_00: which is distinguishing training phases, which I'm very interested in.
- 15:05SPEAKER_00: Maybe another way to think about it is more low level.
- 15:08SPEAKER_00: Maybe if you think about something like a looped transformer,
- 15:11SPEAKER_00: or maybe that's a conception of time that is like fundamentally more grounded
- 15:16SPEAKER_00: in granular perception of time during an interaction.
- 15:19SPEAKER_00: I think thinking machines in their interaction model,
- 15:22SPEAKER_00: that's probably a part of what they achieve.
- 15:24SPEAKER_00: And maybe I would say even like with long running agents now,
- 15:27SPEAKER_00: To a degree,
- 15:28SPEAKER_00: I think we are imbuing some notion of time dependence in models simply
- 15:33SPEAKER_00: because there's an environment with tools that interact and then there's an order
- 15:37SPEAKER_00: in it.
- 15:38SPEAKER_01: But this is all happening in context.
- 15:39SPEAKER_01: Yeah, that's what I was about to say.
- 15:41SPEAKER_01: Just the fact that the model is able to do reasoning in token space.
- 15:44SPEAKER_01: It introduces some aspect of time and this makes the model so much better, right?
- 15:49SPEAKER_01: Because then you can kind of compose stuff and it doesn't have to be just a stateless thing.
- 15:53SPEAKER_01: I'm very bullish on that.
- 15:54SPEAKER_01: Like just using the internal representation and then just like looping over there.
- 15:58SPEAKER_01: It's clunky,
- 15:59SPEAKER_01: but the thing is that as soon as you add that concept of time sequence,
- 16:03SPEAKER_01: like a tick,
- 16:04SPEAKER_01: whatever you want to call it,
- 16:06SPEAKER_01: you get into kind of a messy training situation in some shape or form, right?
- 16:10SPEAKER_01: Because now like it's, it's a bit bulkier, the things are running in time.
- 16:15SPEAKER_01: I think in my view,
- 16:19SPEAKER_01: that fundamental constraint of life is something
- 16:22SPEAKER_01: that like pushed this general intelligence to kind of emerge
- 16:26SPEAKER_01: because like you're seeing things in sequence.
- 16:29SPEAKER_01: As you're learning about the things that you're seeing and experiencing the world,
- 16:33SPEAKER_01: you're going to get all sorts of input and then your brain need times
- 16:36SPEAKER_01: and your brain need to have the different stuff happening and the right sequence.
- 16:40SPEAKER_01: So it kind of engineer itself to be able to kind of watch for this and get this
- 16:45SPEAKER_01: If time is an aspect and you're just continuously receiving stuff that may
- 16:49SPEAKER_01: or may not be of importance,
- 16:51SPEAKER_01: you have no choice but to create this core that is the intelligence and
- 16:58SPEAKER_01: the rest is you're able to discern I'm really excited to talk about some
- 17:01SPEAKER_01: of these ideas about how to imbue this notion of,
- 17:02SPEAKER_01: okay, not only temporality, but at the different
- 17:03SPEAKER_00: timescales we discussed, but also
- 17:19SPEAKER_00: I think you touched upon it, but you didn't say the word.
- 17:22SPEAKER_00: Meta-learning, right?
- 17:23SPEAKER_00: It's learning how to learn from a stream of data.
- 17:26SPEAKER_01: Exactly.
- 17:27SPEAKER_00: Okay, let's go step by step.
- 17:28SPEAKER_00: So what I want to get at is some definitions that are widely accepted in the field
- 17:34SPEAKER_00: and And of course, predate my little blog.
- 17:36SPEAKER_00: So one influential framework of thinking about continual learning
- 17:40SPEAKER_00: or incremental learning is dividing in three categories.
- 17:43SPEAKER_00: And maybe I can share the paper.
- 17:44SPEAKER_00: Three types of incremental learning.
- 17:46SPEAKER_00: It's a paper in Nature.
- 17:47SPEAKER_00: So this is super influential.
- 17:49SPEAKER_00: This researcher recently moved to the city I'm living at in the Netherlands.
- 17:53SPEAKER_00: Super happy to have him here.
- 17:55SPEAKER_00: So the idea of the three types of incremental learning is
- 17:58SPEAKER_00: that it helps us distinguish what we mean when we talk about Incremental
- 18:03SPEAKER_00: or continual.
- 18:04SPEAKER_00: The first is task incremental.
- 18:06SPEAKER_00: Here you have self-contained tasks that you are learning sequentially.
- 18:10SPEAKER_00: You're not learning them at the same time.
- 18:11SPEAKER_00: You learn task one, then you learn task two, then you learn task three.
- 18:15SPEAKER_00: And at the end, you're supposed to know task one, no task two, no task three.
- 18:19SPEAKER_00: So this is kind of the most basic setup that maybe a lot of people imagine
- 18:23SPEAKER_00: when they think of continual learning because, yeah, you're retaining tasks.
- 18:26SPEAKER_00: You're expanding your arsenal of tools and knowledge to solve certain tasks.
- 18:32SPEAKER_00: Another scenario is Domain Incremental.
- 18:34SPEAKER_00: This is less relevant for us because it means navigating a non-stationary with
- 18:40SPEAKER_00: the form of the task because you don't know the identity of the task
- 18:44SPEAKER_00: that you're being exposed to.
- 18:46SPEAKER_00: So in the first one, I specifically told you, right, like task one, task two, task three.
- 18:50SPEAKER_00: In Domain Incremental, you don't know the labels of the task.
- 18:53SPEAKER_00: So this is less relevant for the examples we're going to use, but...
- 18:56SPEAKER_00: It's there in the taxonomy.
- 18:58SPEAKER_00: The one that I do want to talk about,
- 18:59SPEAKER_00: and this is the important one, is class incremental learning.
- 19:02SPEAKER_00: This is basically a superset of task incremental learning,
- 19:05SPEAKER_00: where instead of only expanding your arsenal with the different tasks,
- 19:10SPEAKER_00: so knowing task one,
- 19:11SPEAKER_00: task You're supposed to be able to compose between different tasks.
- 19:15SPEAKER_00: The canonical example I think here is, okay,
- 19:18SPEAKER_00: so if you first learn to distinguish cats and dogs,
- 19:21SPEAKER_00: then you learn to distinguish cows and horses.
- 19:23SPEAKER_00: With task incremental learning,
- 19:24SPEAKER_00: so the easy version,
- 19:26SPEAKER_00: you would be only asked to distinguish cats and dogs separately,
- 19:30SPEAKER_00: and then cows and horses separately.
- 19:31SPEAKER_00: With class incremental,
- 19:33SPEAKER_00: the third scenario,
- 19:34SPEAKER_00: you would need to distinguish between classes that are not observed together.
- 19:37SPEAKER_00: So distinguishing cats and Horses, which you never saw together.
- 19:42SPEAKER_00: But you should be able to build representations that are transferable enough
- 19:47SPEAKER_00: to knowledge that you gain at a different time.
- 19:49SPEAKER_00: And I think when we talk about the current paradigm of foundation models and agents,
- 19:55SPEAKER_00: this is where it gets exciting, right?
- 19:57SPEAKER_00: Because you want your agent to not only learn separate facts about you
- 20:02SPEAKER_00: and separate skills,
- 20:03SPEAKER_00: but you want it to interpolate between them,
- 20:05SPEAKER_00: to compose and even in some cases extrapolate.
- 20:08SPEAKER_00: Outside of the task distribution.
- 20:10SPEAKER_00: So I hope this was kind of clear.
- 20:12SPEAKER_00: I'll share the Twitter version here.
- 20:14SPEAKER_00: So I kind of outlined five desiderata of continual learner.
- 20:19SPEAKER_00: The idea is that there's a lot of continual learning looking solutions that I think
- 20:25SPEAKER_00: in some ways maybe circumvent some of the problems,
- 20:28SPEAKER_00: which is also interesting in its own,
- 20:30SPEAKER_00: or fundamentally don't address some of these problems.
- 20:32SPEAKER_00: So one of them is that you want to sequentially expose your learner
- 20:36SPEAKER_00: to Non-stationary distribution.
- 20:39SPEAKER_00: This is important because you can envision a setting
- 20:41SPEAKER_00: where you are continually fine-tuning a model on approximately
- 20:47SPEAKER_00: the same distribution.
- 20:48SPEAKER_00: In a way,
- 20:48SPEAKER_00: that is continue learning,
- 20:50SPEAKER_00: but it does not face the fact that the world is big
- 20:54SPEAKER_00: and your environment will not be stationary.
- 20:55SPEAKER_00: So you want your methods to be somewhat robust to this, right?
- 20:59SPEAKER_00: You don't want to hide behind the assumption that your distribution is never going to change.
- 21:03SPEAKER_00: Richard Sutton has this position paper about big world systems.
- 21:07SPEAKER_00: The idea is that we don't live in a world which we can approximate.
- 21:11SPEAKER_00: Even with the current scaling paradigm,
- 21:13SPEAKER_00: where it's all about throwing more compute and more data,
- 21:16SPEAKER_00: the big world hypothesis takes an opposite stance.
- 21:19SPEAKER_00: It says that you can have all the capex in the world, you can have all the data in the world.
- 21:24SPEAKER_00: The approach of trying to Throw everything under the distribution is not going
- 21:29SPEAKER_00: to work because fundamentally the distribution is going to change tomorrow.
- 21:32SPEAKER_00: And you want to learn that it is efficient.
- 21:34SPEAKER_00: Okay, but so in general, that's the first paradigm.
- 21:37SPEAKER_00: You want to have non-stationary, some capacity to bear with co-processing.
- 21:42SPEAKER_00: The other thing is that you want to preserve the general capability.
- 21:47SPEAKER_00: This has to do with a catastrophic forgetting problem.
- 21:50SPEAKER_00: Of course, you don't want to forget.
- 21:51SPEAKER_00: I'm not going to delve into that.
- 21:52SPEAKER_00: It's pretty obvious.
- 21:54SPEAKER_00: There's another desideratum about efficiency.
- 21:56SPEAKER_00: You could imagine that in a way we have solved continual learning,
- 22:00SPEAKER_00: but in the most boring way possible,
- 22:01SPEAKER_00: which is every time you want to learn something new,
- 22:03SPEAKER_00: you can Fine-tune your model on everything it has ever seen plus the new thing.
- 22:07SPEAKER_00: And surely this will allow you to internalize the new piece of knowledge or skills
- 22:13SPEAKER_00: without forgetting the old ones.
- 22:15SPEAKER_00: Because what you have done is basically you've created a multitask distribution again.
- 22:18SPEAKER_00: And again, multitask distributions, that's what we like.
- 22:21SPEAKER_00: That's what all of our paradigms now are based on.
- 22:24SPEAKER_00: Like most of our paradigms where we're comfortable work.
- 22:26SPEAKER_00: So you get a bunch of tasks,
- 22:28SPEAKER_00: you approximate some wide distribution,
- 22:30SPEAKER_00: get a large model, train it, and you have it.
- 22:33SPEAKER_00: We don't want this inefficient method.
- 22:36SPEAKER_00: This is one of the trivial solutions, right?
- 22:38SPEAKER_00: We want to efficiently encode new representations that respect all representations, right?
- 22:43SPEAKER_00: So you want some efficiency desideratum in there as well.
- 22:48SPEAKER_00: And lastly, I talk in the blog post about compositional trans, right?
- 22:51SPEAKER_00: So we already touched upon this.
- 22:53SPEAKER_00: You encounter different primitives at different training stages.
- 22:56SPEAKER_00: You want to be able to compose them.
- 22:58SPEAKER_00: So this is basically the analog of class incremental learning.
- 23:02SPEAKER_00: You've never seen these things together, but you want to be able to generalize between them.
- 23:07SPEAKER_00: So I hope that this is a good outline of my blog post.
- 23:11SPEAKER_00: And now we can start talking about the ways that this problem has been approached.
- 23:15SPEAKER_01: Yeah, this is good.
- 23:16SPEAKER_01: I just wanted to get your thought on the general aspect of it.
- 23:20SPEAKER_01: I think there's many definitions and stuff like that,
- 23:22SPEAKER_01: but what do you mean and understand with the general block
- 23:26SPEAKER_01: that should stay constant?
- 23:27SPEAKER_00: Okay, that's a good question.
- 23:28SPEAKER_00: This probably from an engineering perspective means something different than
- 23:32SPEAKER_00: a truly like principle definition.
- 23:34SPEAKER_00: I think if you take the more...
- 23:36SPEAKER_00: Practical perspective.
- 23:37SPEAKER_00: There are a set of capabilities that are important to preserve for empirical performance.
- 23:42SPEAKER_00: Likely you want to preserve instruction following,
- 23:44SPEAKER_00: some reasoning skills, some general world knowledge.
- 23:48SPEAKER_00: And this would be evaluated with a fixed set of evals
- 23:51SPEAKER_00: that you rerun every time you train to see how much drift you have
- 23:54SPEAKER_00: of this general performance.
- 23:55SPEAKER_00: But I agree that not as well defined.
- 23:58SPEAKER_00: Look, we've heard Harapathi talk about,
- 24:00SPEAKER_00: what is it called, the fundamental
- 24:02SPEAKER_01: Neural core or something like that.
- 24:03SPEAKER_00: Yeah, neural core.
- 24:04SPEAKER_00: Yeah, exactly.
- 24:04SPEAKER_00: So it is hard to disentangle this because a lot of reasoning is also
- 24:09SPEAKER_00: about being able to retrieve and like have automaticity over like
- 24:13SPEAKER_00: a wide crystallized set of information, right?
- 24:16SPEAKER_00: Your reasoning will be bottlenecked if it doesn't have automaticity over certain things.
- 24:20SPEAKER_00: And here a common rebuttal is that your core knowledge LLM
- 24:25SPEAKER_00: that is 1 billion parameters and it doesn't know about the history of pizza,
- 24:28SPEAKER_00: but it knows how to reason and how to instruct and follow.
- 24:30SPEAKER_00: It doesn't need to know about the history of pizza.
- 24:33SPEAKER_00: It can retrieve things at inference time and it can compose that way.
- 24:37SPEAKER_00: I think this is a reasonable perspective in certain regimes of scale.
- 24:41SPEAKER_00: It depends on how much you want to retrieve, right?
- 24:44SPEAKER_00: So retrieval works within And if you talk to people in education and psychology
- 24:52SPEAKER_00: and whatever, one person that talks a lot about this is Justin from Math Academy.
- 24:56SPEAKER_00: He always drills down the point of automaticity.
- 24:59SPEAKER_00: You want to get to a point where you're learning something and you don't want
- 25:02SPEAKER_00: to refer to the book.
- 25:03SPEAKER_00: Instead, it's automatic for you.
- 25:07SPEAKER_00: LeBron James had to dribble a basketball.
- 25:09SPEAKER_00: He had to be thinking about dribbling a basketball.
- 25:11SPEAKER_00: You don't want that.
- 25:11SPEAKER_00: You want automaticity.
- 25:12SPEAKER_00: And I argue that some of these parametric methods,
- 25:15SPEAKER_00: like neural memory,
- 25:16SPEAKER_00: allows you to have this automaticity more than certain retrieval methods.
- 25:20SPEAKER_00: Because,
- 25:20SPEAKER_00: of course,
- 25:21SPEAKER_00: you can get pretty smart about retrieval methods within certain scales,
- 25:23SPEAKER_00: but what you want is automaticity that grows as you get more memory, right?
- 25:27SPEAKER_00: Whereas with an external memory harness,
- 25:29SPEAKER_00: often the bigger it gets, actually you have more trouble retrieving things.
- 25:33SPEAKER_00: And that's not how our memory system should be.
- 25:36SPEAKER_01: But that's a good question.
- 25:37SPEAKER_01: Yeah.
- 25:37SPEAKER_01: I've interviewed Justin for like three hours and a half about all of this stuff.
- 25:42SPEAKER_01: This is a very interesting point
- 25:43SPEAKER_01: because definitely like automatic ctc is super important because
- 25:46SPEAKER_01: then you can compose things and go to the higher plane of that process and
- 25:51SPEAKER_01: then like use that to kind of latch on into the more interesting stuff.
- 25:55SPEAKER_01: So there's this element,
- 25:56SPEAKER_01: but like I always come back to the brain because in my view,
- 26:00SPEAKER_01: one of the core blueprint of general intelligence that we have But what's general in
- 26:05SPEAKER_01: the brain,
- 26:05SPEAKER_01: let's say if I was to think about I want to preserve the general stuff,
- 26:08SPEAKER_01: you'll be surprised how it doesn't give a shit.
- 26:11SPEAKER_01: Like for instance,
- 26:12SPEAKER_01: when the astronauts go out and then in space and stuff like that,
- 26:16SPEAKER_01: and then they just need to learn something that never was there before.
- 26:19SPEAKER_01: And it's hard to forget literally how to walk really under the gravity
- 26:23SPEAKER_01: because it's now not that important anymore, right?
- 26:26SPEAKER_01: Even though you have years and years of evolution,
- 26:29SPEAKER_01: the millions of years,
- 26:30SPEAKER_01: at this point, the brain is like, let's cannibalize a bit of this knowledge.
- 26:34SPEAKER_01: Like you lose a bit the automaticity.
- 26:36SPEAKER_01: You can relearn it back if you need to.
- 26:37SPEAKER_01: It's not like Learning how to fly or something like that.
- 26:40SPEAKER_01: But still,
- 26:41SPEAKER_01: when they go back here,
- 26:42SPEAKER_01: there's a lot of element, but just the brain kind of has what we think is critical.
- 26:47SPEAKER_01: Same thing,
- 26:47SPEAKER_01: like a fun experiment that you can do is there's these glasses that you can use
- 26:52SPEAKER_01: to kind of flip your vision, right?
- 26:54SPEAKER_01: So instead of seeing like this,
- 26:56SPEAKER_01: you see like on the opposite side,
- 26:57SPEAKER_01: but the first 10 days or something like that, it's all messed up.
- 27:01SPEAKER_01: And
- 27:05SPEAKER_01: if you do remove the glasses, you can just, you will be seeing on the opposite side.
- 27:11SPEAKER_01: It will take like some time to kind of relearn that thing.
- 27:15SPEAKER_01: But the brain is as if, yes,
- 27:18SPEAKER_01: there's the structure that kind of constrain a bit,
- 27:20SPEAKER_01: like what's, what it's supposed to be kind of like doing.
- 27:24SPEAKER_01: There's some,
- 27:25SPEAKER_01: some regions that are doing X or Y,
- 27:27SPEAKER_01: but like they don't have a very high,
- 27:29SPEAKER_01: high bias about what's the content literally, right?
- 27:32SPEAKER_01: There's the story of children that can learn echolocation if they were blind early on.
- 27:37SPEAKER_01: If I try to do it, I will have to kind of, I think, try very hard, right?
- 27:41SPEAKER_01: But like they can do it because they have these constraints and
- 27:44SPEAKER_01: the brain is just like, yeah, let's go.
- 27:45SPEAKER_01: Let's use this input and like go, go, go.
- 27:47SPEAKER_01: We need to kind of move.
- 27:48SPEAKER_01: And so it's as if like all the neurons at the same times are always kind
- 27:52SPEAKER_01: of Re-evaluating their position and then their usefulness in the thing.
- 27:58SPEAKER_01: And then these constraints that are there from the environment kind
- 28:01SPEAKER_01: of mold what you are continuously knowing, right?
- 28:05SPEAKER_01: And some stuff that is not useful is just going to be kind of purposely wiped out
- 28:10SPEAKER_01: because there's no point to it.
- 28:11SPEAKER_01: That's why I see when I see general is like, what exactly should go there?
- 28:16SPEAKER_00: That's a really good point.
- 28:17SPEAKER_00: And I actually want to delve into this because I also am uncomfortable with division.
- 28:22SPEAKER_00: As I said, there's no principal definition.
- 28:24SPEAKER_00: And this reasoning core, yeah, I think Karpati called it reasoning core.
- 28:28SPEAKER_00: I think I get the idea, but it's really difficult to billionaire.
- 28:31SPEAKER_00: And I think one way that this problem is solved in our lives,
- 28:36SPEAKER_00: our experiences are so on policy and so rich that you actually get replay for free.
- 28:41SPEAKER_00: So that is our general knowledge.
- 28:43SPEAKER_00: I think here we have to think about LLMs where we're used to never touching their weights.
- 28:48SPEAKER_00: These models have been carefully trained and deployed, and then we simply use them.
- 28:52SPEAKER_00: But I argue that in an enlightened AI future, this will not be the case.
- 28:57SPEAKER_00: The model will have experienced thousands of rounds of retraining afterwards.
- 29:02SPEAKER_00: Thing that happens then is that you reframe replay as just play, right?
- 29:07SPEAKER_00: Every day when you go about your day, you just experience a lot of the general things.
- 29:12SPEAKER_00: And that's a big part of what prevents forgetting with you.
- 29:14SPEAKER_00: I would, however, push back.
- 29:16SPEAKER_00: That's the only thing that happens.
- 29:17SPEAKER_00: I don't think our profile of learner,
- 29:20SPEAKER_00: like the human brain as a learner,
- 29:21SPEAKER_00: I think like fundamentally does not exhibit as strong of a forgetting experience
- 29:26SPEAKER_00: as current foundation models.
- 29:28SPEAKER_01: At the scale of the brain.
- 29:29SPEAKER_00: Now, there is this very interesting paper from 2022 about they got neurons in
- 29:34SPEAKER_00: a Petri dish to learn how to play Pong.
- 29:36SPEAKER_00: This is Cortical Labs.
- 29:38SPEAKER_01: Yeah.
- 29:38SPEAKER_00: So I think they also maybe got it to play Doom or something recently.
- 29:42SPEAKER_01: I love this stuff.
- 29:43SPEAKER_01: I love this stuff.
- 29:43SPEAKER_01: Super cool.
- 29:44SPEAKER_01: One line hidden in the discussion in that paper.
- 29:47SPEAKER_00: The setup here is that they have They have a patchy dish of human cortical cells, right?
- 29:51SPEAKER_00: So that like a pure substrate that is like supposedly really, really good at learning.
- 29:55SPEAKER_00: And what they exhibit with this dish of cells is that this also forgets
- 29:59SPEAKER_00: when you start training it in a kind of a different configuration.
- 30:03SPEAKER_00: The importance on the other configuration starts to drop.
- 30:05SPEAKER_00: So that means that looking at cortical cells themselves as well,
- 30:09SPEAKER_00: they exhibit similar rates of forgetting as the neural networks that we're dealing
- 30:13SPEAKER_00: with.
- 30:13SPEAKER_00: And this is interesting because that means that there's another organism,
- 30:17SPEAKER_00: there's another system in the brain that ensures that that doesn't happen.
- 30:20SPEAKER_00: But inherently,
- 30:20SPEAKER_00: if you look at the cortex level,
- 30:23SPEAKER_00: the cortical cells left
- 30:24SPEAKER_00: to themselves will forget just like our neural networks forget.
- 30:27SPEAKER_01: The forgetting aspect, it seems like a core element of learning just in general, right?
- 30:32SPEAKER_01: Because if you cannot never forget and you make mistakes,
- 30:35SPEAKER_01: like it's not compatible,
- 30:36SPEAKER_01: like in the beginning, because these mistakes will stay there forever.
- 30:39SPEAKER_01: And like our worldview is shaped Like what we're doing,
- 30:43SPEAKER_01: but what we're doing isn't precise with like what we think it is,
- 30:47SPEAKER_01: but then we need to be able to dismantle that and then like rebuild it in order
- 30:51SPEAKER_01: to kind of move forward.
- 30:52SPEAKER_01: Like the animals in general need to be able to do this.
- 30:55SPEAKER_01: One experiment that would have been very cool,
- 30:57SPEAKER_01: I don't know if they did that,
- 30:58SPEAKER_01: is to check how long it will have taking the same cells to retrain on like
- 31:04SPEAKER_01: the first test.
- 31:05SPEAKER_00: Whether there's more muscle memory,
- 31:07SPEAKER_00: like neural
- 31:08SPEAKER_01: Exactly,
- 31:09SPEAKER_01: because cells,
- 31:10SPEAKER_01: they literally have to build up the protein that they need and then reconfigure
- 31:14SPEAKER_01: a bit like their thing and then build the synapses at the right spot
- 31:17SPEAKER_01: and like all this stuff, right?
- 31:19SPEAKER_01: And if it's a similar type of task,
- 31:21SPEAKER_01: theoretically, it should not need A whole different configuration.
- 31:25SPEAKER_01: So that would be very interesting.
- 31:26SPEAKER_00: My intuition here would be that the history of tasks you see,
- 31:30SPEAKER_00: even if you forget them,
- 31:31SPEAKER_00: the trajectory of tasks that you've been exposed to kind of determines the shape
- 31:35SPEAKER_00: of your lost landscape,
- 31:36SPEAKER_00: let's say,
- 31:37SPEAKER_00: and then that Kind of conditions you maybe to relearn those things more easily.
- 31:41SPEAKER_00: But it would be interesting to observe empirically for sure.
- 31:44SPEAKER_00: Let's go through a bunch of methods for how this has been attempted.
- 31:48SPEAKER_00: By the way,
- 31:49SPEAKER_00: the subtext with this was that if not the cortical cells that know how not
- 31:53SPEAKER_00: to forget, instead in the brain there's something else.
- 31:55SPEAKER_00: And there's a lot of arguments about what that is.
- 31:57SPEAKER_00: I think a big part of it is probably the hippocampus,
- 31:59SPEAKER_00: which is acting as the replay orchestrator.
- 32:02SPEAKER_00: So it replays experiences that the Cortical cells get exposed to again in
- 32:07SPEAKER_00: a certain form to preserve representations.
- 32:09SPEAKER_00: And that's why I fully see the point that general doesn't fully exist in the sense
- 32:15SPEAKER_00: of we simply are exposed to so much data every day that we retain the importance
- 32:20SPEAKER_00: And even foundational things like what's left and right in our visual perception
- 32:25SPEAKER_00: and like walking for astronauts, even those can be forgotten.
- 32:27SPEAKER_00: I agree.
- 32:28SPEAKER_00: But I think there's this added element of basically orchestrated replay
- 32:32SPEAKER_00: that does occur in us that maybe
- 32:35SPEAKER_00: a functional continuing learning solution will have.
- 32:37SPEAKER_00: But let's go through quickly through some of the main methods.
- 32:41SPEAKER_00: So one family of methods is basically replay.
- 32:44SPEAKER_00: Right, as we mentioned.
- 32:45SPEAKER_00: And that is basically getting some representation of the things you don't want
- 32:49SPEAKER_00: to forget and introducing them again at some point
- 32:52SPEAKER_01: during training.
- 32:52SPEAKER_00: And this can be like verbatim data replay,
- 32:55SPEAKER_00: like reservoir of raw examples like this one, experience replay.
- 32:58SPEAKER_00: But then you probably don't want to keep a buffer.
- 33:01SPEAKER_00: The brain doesn't keep a buffer.
- 33:02SPEAKER_00: It feels clunky.
- 33:04SPEAKER_00: There's generative replay versions and there's a lot of like Sub-varieties of these,
- 33:09SPEAKER_00: one of the highlights of recent ML research in LLMs has been self-distillation.
- 33:14SPEAKER_00: And here, basically what this does is like it is a form of replay.
- 33:18SPEAKER_00: So for those that are not familiar,
- 33:20SPEAKER_00: maybe this can be the moment we introduce some A bunch of papers came out
- 33:24SPEAKER_00: in January, February about this.
- 33:26SPEAKER_00: Like last year,
- 33:27SPEAKER_00: Thinking Machines released on policy distillation blog post,
- 33:30SPEAKER_00: which turned out to have been released like two years prior.
- 33:32SPEAKER_00: There's knowledge distillation before that.
- 33:35SPEAKER_00: So the whole family is about making sure that some version of the representations of
- 33:40SPEAKER_00: the original model survive when you train.
- 33:43SPEAKER_01: Of course, there's more reasons.
- 33:44SPEAKER_00: I know you had the authors of the self-distillation fine tuning paper on the channel.
- 33:48SPEAKER_00: Really, really great paper.
- 33:49SPEAKER_00: Maybe we can show the figures there because they have probably good illustrations
- 33:53SPEAKER_00: of this for the people in the audience.
- 33:55SPEAKER_00: Yeah, exactly.
- 33:56SPEAKER_00: The direction they come at for this is different than what we We're talking about now.
- 34:02SPEAKER_00: They come from a place of how do we make sure that we can do RL,
- 34:07SPEAKER_00: which is a way for the model to learn from experience.
- 34:10SPEAKER_00: So on policy,
- 34:12SPEAKER_00: you can associate on policy from experience,
- 34:14SPEAKER_00: but not be bogged down with the problems of RL,
- 34:17SPEAKER_00: such as low information signal per rollout,
- 34:20SPEAKER_00: because you have this huge rollout,
- 34:22SPEAKER_00: million tokens,
- 34:22SPEAKER_00: and then you get a On the order of a bit of information, of feedback.
- 34:26SPEAKER_00: So they came at it from a perspective of how do we make sure that we supervise
- 34:30SPEAKER_00: the model during training in a dense manner?
- 34:32SPEAKER_00: We provide a lot of feedback signal while having this on policy aspect where
- 34:37SPEAKER_00: the model learns from its own experience instead of trying to imitate others.
- 34:40SPEAKER_00: And the way they did that is very elegant.
- 34:42SPEAKER_00: I'm showing one paper now.
- 34:44SPEAKER_00: There were many variations of this.
- 34:45SPEAKER_00: Self-distillation fine-tuning,
- 34:47SPEAKER_00: on-policy self-distillation,
- 34:48SPEAKER_00: self-distillation policy optimization, privileged information policy optimization.
- 34:53SPEAKER_00: Now there's all kinds of mixes.
- 34:55SPEAKER_00: But the idea is,
- 34:56SPEAKER_00: one of the core ideas is similar,
- 34:58SPEAKER_00: which is that you have a student model and you have a teacher model.
- 35:01SPEAKER_00: In on-policy distillation,
- 35:03SPEAKER_00: so not self-distillation,
- 35:04SPEAKER_00: your teacher is another model,
- 35:06SPEAKER_00: your student is your target model that you want to train.
- 35:09SPEAKER_00: You sample rollouts from the The student, it's thinking how it should approach a problem.
- 35:14SPEAKER_00: The teacher assigns logprobs.
- 35:17SPEAKER_00: You train the student to minimize the KL between the student and the teacher.
- 35:22SPEAKER_00: That introduces some problems because the teacher needs to be from the same family.
- 35:25SPEAKER_00: Also, if you are really at the frontier of task complexity,
- 35:29SPEAKER_00: you need a teacher that knows how to solve it.
- 35:31SPEAKER_00: So fundamentally, you cannot train the best models with it.
- 35:33SPEAKER_00: So one way to get through that is to say, well, I don't need a smarter teacher.
- 35:38SPEAKER_00: Instead, a smarter teacher is just a student with some privileged information.
- 35:42SPEAKER_00: And we know that in-context learning is great,
- 35:44SPEAKER_00: so we can give some context,
- 35:46SPEAKER_00: privileged context to the student and it becomes a teacher and then it's super.
- 35:49SPEAKER_00: And it's a very elegant idea and they show that The KL divergence
- 35:53SPEAKER_00: as you train this way is lower and that prevents forgetting.
- 35:57SPEAKER_00: So it's really good.
- 35:58SPEAKER_00: And so this is the first moment I want to mention this method of self-distillation.
- 36:02SPEAKER_00: Before we get to one of the equally exciting families of methods,
- 36:08SPEAKER_00: I want to talk about one that is not my favorite, which is regularization.
- 36:14SPEAKER_00: A bunch of regularization methods.
- 36:15SPEAKER_00: Maybe a better figure is in the EWC paper.
- 36:19SPEAKER_00: So the general idea here is that you have certain subspaces in your weight space
- 36:24SPEAKER_00: that correspond to prowess at a certain task.
- 36:27SPEAKER_00: If you train a model on one task,
- 36:29SPEAKER_00: it's going to be in a basin of weight configurations that are good for that task.
- 36:34SPEAKER_00: If you train on another task, it's going to escape that basin.
- 36:36SPEAKER_00: Your goal with these families of methods is to find a weight update mechanism that
- 36:41SPEAKER_00: So yeah, so here we have task A, task B.
- 36:46SPEAKER_00: When you train your model on task A, it settles into this nice point.
- 36:50SPEAKER_00: If you train it only on task B,
- 36:52SPEAKER_00: it would settle in this nice point, but then it would lose prowess in task A.
- 36:59SPEAKER_00: These
- 37:03SPEAKER_00: methods basically are ways of constraining updates in weight space so
- 37:08SPEAKER_00: that you preserve existing representation.
- 37:13SPEAKER_00: This is a more interventionist way of doing replay.
- 37:15SPEAKER_00: With replay, you are also preserving previous task structure, but not directly.
- 37:20SPEAKER_00: You're saying, okay,
- 37:21SPEAKER_00: in the batch of data where you're calculating the gradients,
- 37:24SPEAKER_00: I'm going to throw in some representations that These methods are trying
- 37:28SPEAKER_00: to navigate weight space in a smarter way.
- 37:42SPEAKER_00: And they have been promising in a certain extent.
- 37:45SPEAKER_00: For example,
- 37:45SPEAKER_00: EWC is good for task incremental learning,
- 37:48SPEAKER_00: but for class incremental learning,
- 37:50SPEAKER_00: as we will see in one of the figures later, it's not so good.
- 37:53SPEAKER_00: So regularization family, currently not as bullish on that.
- 37:56SPEAKER_00: Lastly, I do want to talk about the meta-learning view.
- 38:00SPEAKER_00: And this is one of the methods that are really close to my heart.
- 38:03SPEAKER_00: And I will show you some preliminary results on some experiments I've been running with this.
- 38:07SPEAKER_00: So these are part of the unpublished stuff that I'll be showing here.
- 38:11SPEAKER_00: A quick primary learning.
- 38:13SPEAKER_00: Maybe this is quite interesting actually with your background.
- 38:15SPEAKER_00: Like, is that a big thing in computational neuroscience?
- 38:18SPEAKER_01: It is.
- 38:19SPEAKER_01: Yes, yes, it is.
- 38:20SPEAKER_01: And like,
- 38:21SPEAKER_01: if you think about it,
- 38:22SPEAKER_01: one of the core element in human learning is that we're able to learn how to learn,
- 38:27SPEAKER_01: right?
- 38:27SPEAKER_01: And then there's way of learning better how to learn.
- 38:30SPEAKER_01: And this is a learnable skill.
- 38:32SPEAKER_01: Like it's a learnable skill.
- 38:34SPEAKER_01: So it's kind of something fundamental about not just being good at a task,
- 38:44SPEAKER_01: but it tells you something about the actual learning that is happening and how
- 38:48SPEAKER_01: to make it better.
- 38:48SPEAKER_01: And then you see in these things,
- 38:51SPEAKER_01: you can get super human level performance out of gimmicks almost, right?
- 38:56SPEAKER_01: You can memorize a whole 10,000 things if you want to,
- 39:00SPEAKER_01: and you're going to get fit in your brain,
- 39:01SPEAKER_01: but you have to do it in a certain way, right?
- 39:03SPEAKER_01: And this certain way is kind of as easy for your brain to have it stick.
- 39:08SPEAKER_01: You kind of reuse one of the constraints it has.
- 39:11SPEAKER_01: And one of the constraints it has is that when you go to places,
- 39:15SPEAKER_01: you're not constantly trying to learn the stuff, right?
- 39:18SPEAKER_01: The board is there.
- 39:19SPEAKER_01: I have to memorize that.
- 39:20SPEAKER_01: Oh, like the floor is here.
- 39:22SPEAKER_01: You're just immediately kind of ingesting all this information and it has
- 39:26SPEAKER_01: to stick fast because an animal that goes and then has to stay for three minutes in
- 39:31SPEAKER_01: that spot to be able to map it out.
- 39:33SPEAKER_01: It's just not going to work out, right?
- 39:34SPEAKER_01: At some point,
- 39:35SPEAKER_01: this animal will not be there,
- 39:37SPEAKER_01: but you as like this kind of higher order animal,
- 39:39SPEAKER_01: you leverage this to learn poesy or a whole speech easily, right?
- 39:43SPEAKER_01: So there's these elements that come into play and you learn
- 39:47SPEAKER_01: a bit better constraint posed onto your brain,
- 39:49SPEAKER_01: how the learning can be affected to it.
- 39:51SPEAKER_00: I have a question here about what you said.
- 39:53SPEAKER_00: So, indeed,
- 39:54SPEAKER_00: I think we empirically observe that brains of various kinds,
- 39:57SPEAKER_00: but particularly the human brain, can get better at learning, of course.
- 40:01SPEAKER_00: What level do you think this happens at?
- 40:03SPEAKER_00: How do we distinguish,
- 40:04SPEAKER_00: in other words,
- 40:05SPEAKER_00: how much of this is crystallized knowledge
- 40:07SPEAKER_00: and how much is it our brains actually learning better update rule?
- 40:11SPEAKER_01: That's a, that's a very good question.
- 40:12SPEAKER_01: It's two things.
- 40:13SPEAKER_01: You get better at learning when you actually have the whole feedback loop
- 40:17SPEAKER_01: that you actually learn something, right?
- 40:19SPEAKER_01: So, because if I want to learn how to paint,
- 40:21SPEAKER_01: but I'm painting with my knuckles and stuff like that, and it's just go nowhere.
- 40:25SPEAKER_01: I can do this for 30 years and do absolutely nothing.
- 40:28SPEAKER_01: Right.
- 40:28SPEAKER_01: But if there's a feedback loop and something that come back to me,
- 40:31SPEAKER_01: then I actually can start the process of learning.
- 40:34SPEAKER_01: I think like these meta learning structure that you put in your learning skill set,
- 40:39SPEAKER_01: what they do is just,
- 40:40SPEAKER_01: they make the ingestion of learning happen so much more easier
- 40:45SPEAKER_01: for your brain capacity, right?
- 40:47SPEAKER_01: Because if you think about memorization,
- 40:49SPEAKER_01: you can bash your head on whatever you're trying to learn.
- 40:52SPEAKER_01: For like a hundred days,
- 40:54SPEAKER_01: or you could use the method to kind of just hack your brain and then like,
- 40:59SPEAKER_01: you're going to be doing, done in like four hours.
- 41:00SPEAKER_01: Right.
- 41:01SPEAKER_01: But then like next time you have to learn something like this, it will stick.
- 41:04SPEAKER_01: And like, you have these now elements in your mind that you can reuse.
- 41:08SPEAKER_01: So I think it acts on both sides.
- 41:10SPEAKER_01: It makes the ingestion of the signal much faster,
- 41:14SPEAKER_01: which makes the learning also in terms of like signal to nose ratio much higher,
- 41:19SPEAKER_01: right?
- 41:20SPEAKER_01: So you're actually going to get better faster.
- 41:22SPEAKER_01: If you're better, it's easier to get more skills.
- 41:25SPEAKER_01: There's these elements that are coming to play.
- 41:27SPEAKER_01: I don't think it's like one thing.
- 41:28SPEAKER_01: I think it's like multiple things that are happening together to kind
- 41:31SPEAKER_01: of make you much more efficient.
- 41:33SPEAKER_00: If we take an analogy of what you said in terms of paradigms of continual learning
- 41:37SPEAKER_00: for LLMs,
- 41:38SPEAKER_00: you could take the crystallized knowledge to be a form of like
- 41:42SPEAKER_00: a harness around your learner where it says maybe as your agent is experiencing
- 41:46SPEAKER_00: the code base, it can keep notes for itself in a scratch pad where it Okay.
- 41:51SPEAKER_00: Reminds itself to always use using Python and like always follow these preferences.
- 41:56SPEAKER_00: And this would be crystallized knowledge, right?
- 41:58SPEAKER_00: So the model is getting better at learning maybe when it's crystallizing knowledge
- 42:02SPEAKER_00: about what's the best strategy to learn, right?
- 42:05SPEAKER_00: And this is what humans do.
- 42:06SPEAKER_00: As you said, right?
- 42:07SPEAKER_00: Even consciously,
- 42:08SPEAKER_00: a part of the educational experience of a human being is to teach them how to learn.
- 42:13SPEAKER_00: And this is very crystallized.
- 42:14SPEAKER_00: But I do see the point.
- 42:16SPEAKER_00: I agree that there's something on the weight level that is happening.
- 42:19SPEAKER_00: And that is exactly one of the experiments that I'll be showing.
- 42:22SPEAKER_00: It's a mini experiment with some interesting Negative results probably,
- 42:26SPEAKER_00: but you'll help me interpret.
- 42:27SPEAKER_00: Yeah.
- 42:28SPEAKER_00: So meta-learning methods.
- 42:29SPEAKER_00: I think one paper that gets the core of this problem right is this paper
- 42:35SPEAKER_00: about meta-learning, continue learning algorithms.
- 42:38SPEAKER_00: Kazuki Airi, he, yeah, Great researcher, Schmidt Huber, of course, came up with everything.
- 42:42SPEAKER_00: Yeah, so the idea here is that basically they,
- 42:45SPEAKER_00: instead of handwriting, they continue learning, they learn it.
- 42:48SPEAKER_00: And they get some good results.
- 42:50SPEAKER_00: One thing that happens here is that they're relying on
- 42:52SPEAKER_00: a specific architecture called a self-referential weight matrix.
- 42:58SPEAKER_00: Yeah, this has some architectural choices,
- 43:00SPEAKER_00: including like the delta learning rule,
- 43:02SPEAKER_00: where there are basically fast weights that are input dependent.
- 43:06SPEAKER_00: So you're learning algorithm changes based on the input data.
- 43:10SPEAKER_00: And this is the kind of surface we want to meta-learn, right?
- 43:13SPEAKER_00: Because currently,
- 43:14SPEAKER_00: if you have the models we have,
- 43:15SPEAKER_00: with the learning rules we have,
- 43:16SPEAKER_00: you can expose them to all of the non-sequential data in the world,
- 43:20SPEAKER_00: and they're not going to get better at absorbing non-sequential data.
- 43:23SPEAKER_00: One way to get them to get better is to have one surface in which
- 43:26SPEAKER_00: the learnings can be say, Do you think that's appropriate, Adis?
- 43:51SPEAKER_01: Yeah, for sure.
- 43:52SPEAKER_01: Go ahead, man.
- 43:52SPEAKER_00: So one idea is the idea of Hyper.
- 43:55SPEAKER_00: It's basically a network whose outputs are the weights of another network.
- 44:00SPEAKER_00: And you might ask, why would we want to?
- 44:02SPEAKER_00: What is interesting?
- 44:03SPEAKER_00: Well, if we're talking about getting better at learning,
- 44:06SPEAKER_00: that means our learning rule needs to update.
- 44:08SPEAKER_00: Currently, our learning rules are really carefully thought out algorithms.
- 44:12SPEAKER_00: We have backpropagation and we have choices of optimizers.
- 44:15SPEAKER_00: Hypernetworks are interesting because
- 44:33SPEAKER_00: Them outputting weights means that they are a kind of learning algorithm, right?
- 44:37SPEAKER_00: Yeah.
- 44:38SPEAKER_00: To cut the long story short,
- 44:39SPEAKER_00: there were some papers recently that really got me thinking in this direction.
- 44:43SPEAKER_00: Cy, a great lab, last year published a paper about text to LoRa.
- 44:47SPEAKER_00: And I thought it was...
- 44:48SPEAKER_00: Completely the wrong framing of what's cool about the method.
- 44:51SPEAKER_00: The idea there was that they said, we found a way to convert a past description to LoRa adapter.
- 44:57SPEAKER_00: And of course, I mean,
- 44:58SPEAKER_00: this is interesting because yeah,
- 44:59SPEAKER_00: you can describe an adapter like a model that maybe is good at writing Shakespeare.
- 45:04SPEAKER_00: It's nice.
- 45:06SPEAKER_00: But the subtext here is that you have found a way to sample efficiently,
- 45:11SPEAKER_00: convert very little text into a... That's interesting.
- 45:13SPEAKER_00: That's basically learning, right?
- 45:15SPEAKER_00: You're converting some input into a weight belt.
- 45:17SPEAKER_00: So why stop there?
- 45:19SPEAKER_00: And this year,
- 45:20SPEAKER_00: I think they, yeah,
- 45:21SPEAKER_00: I think a couple of months ago,
- 45:22SPEAKER_00: they released exactly the framing that I thought was more exciting.
- 45:25SPEAKER_00: They call it Doc to LoRa.
- 45:27SPEAKER_00: And the idea here is exactly that you focus this not on describing LoRa's,
- 45:32SPEAKER_00: your mechanism for converting inputs to weights.
- 45:35SPEAKER_00: Get a much more exciting application,
- 45:37SPEAKER_00: which is you have documents and you learn that will convert that document
- 45:42SPEAKER_00: into weights.
- 45:43SPEAKER_00: There's nothing about continual learning here,
- 45:44SPEAKER_00: but I think this is one of the ways that you can start thinking about learning
- 45:48SPEAKER_00: for continual learning as well,
- 45:49SPEAKER_00: because here you've found a way to basically have another learning app.
- 45:53SPEAKER_00: None of the setup here is continual learning shaped,
- 45:55SPEAKER_00: but presumably if you shape it in a continual learning shape,
- 45:58SPEAKER_00: your hyper network could learn how to provide non-destructive weight up.
- 46:02SPEAKER_00: And this is exactly What I was experimenting with.
- 46:05SPEAKER_00: I was thinking about the right problem formulation.
- 46:08SPEAKER_00: So how would this hyper network be trained?
- 46:11SPEAKER_00: I was really sick.
- 46:12SPEAKER_00: I couldn't think straight.
- 46:13SPEAKER_00: My head was hurting the whole time.
- 46:14SPEAKER_00: In the 30 minutes to one hour,
- 46:15SPEAKER_00: like it was in the background, of course, I've been thinking about it for weeks.
- 46:18SPEAKER_00: And then in the 30 minutes that I had clarity while I was taking a shower,
- 46:22SPEAKER_00: I thought of a configuration that made sense.
- 46:25SPEAKER_00: And this is basically the training setup.
- 46:28SPEAKER_00: Okay, so...
- 46:30SPEAKER_00: Let's go through it.
- 46:31SPEAKER_00: So you have some task data that you expose to the model sequentially.
- 46:34SPEAKER_00: Nothing new here, right?
- 46:35SPEAKER_00: Again,
- 46:36SPEAKER_00: if you think of continual learning as this tug of war to get sequential learning
- 46:40SPEAKER_00: to be like multitask learning.
- 46:42SPEAKER_00: Multitask learning is what we're good at.
- 46:43SPEAKER_00: It's really...
- 46:44SPEAKER_00: Nice with our current learning mechanisms, it can get really high performance.
- 46:48SPEAKER_00: It doesn't work sequentially.
- 46:49SPEAKER_00: So we want sequential learning that is as good as multitask learning.
- 46:52SPEAKER_00: In other words,
- 46:53SPEAKER_00: we want to sequentially build the representations that are built
- 46:57SPEAKER_00: with multitask learning.
- 46:58SPEAKER_00: So why not just direct So here, let's take task two.
- 47:04SPEAKER_00: So at task two, your model has already been trained on task one.
- 47:09SPEAKER_00: So what you want when you encounter task two is a weight update that takes in
- 47:13SPEAKER_00: the current weights So your Oracle is the model that has been trained in
- 47:17SPEAKER_00: a multi-task setting, the stuff we're super comfortable with,
- 47:29SPEAKER_00: And this is the training.
- 47:45SPEAKER_00: You're sequentially trying to get the multitask distribution.
- 47:49SPEAKER_00: And of course, maybe there's some fancy algorithms you could come up with.
- 47:53SPEAKER_00: I thought the first proof of concept should be whether you can train a network on it.
- 47:59SPEAKER_00: An operator that can take a task embedding and a representation of
- 48:04SPEAKER_00: the current weights and give you the next weights in a way that, like,
- 48:08SPEAKER_00: of course,
- 48:08SPEAKER_00: the whole point here is that the next weights you're taking are
- 48:12SPEAKER_00: the multitask weights, which means that there's no forgetting.
- 48:15SPEAKER_00: You've trained in a multi-task set in your Oracle.
- 48:18SPEAKER_00: So basically,
- 48:18SPEAKER_00: you train it to take the weights,
- 48:21SPEAKER_00: take the new task,
- 48:22SPEAKER_00: give you the predicted weights that are as similar to the Oracle target as possible
- 48:26SPEAKER_00: by minimizing this loss.
- 48:28SPEAKER_00: And by the way, at inference time, you don't need the Oracle, you have the hypernet.
- 48:33SPEAKER_00: Your hypernet has been trained to be good at...
- 48:35SPEAKER_00: Your hypernet is basically an operator that moves your weights in
- 48:38SPEAKER_00: a forgetting-free zone of learning the new task.
- 48:41SPEAKER_01: Yeah.
- 48:42SPEAKER_01: It makes sense.
- 48:42SPEAKER_01: Yeah.
- 48:43SPEAKER_01: There's somebody that is asking, can you explain as simple as possible what is a hypernet?
- 48:49SPEAKER_00: Good question.
- 48:49SPEAKER_00: Basically it's a network that takes us some input,
- 48:52SPEAKER_00: does whatever, and the outputs are the weights of a target network.
- 48:55SPEAKER_00: So typically like if you have an So you can think of current methods as the way
- 49:00SPEAKER_00: that you update the weights is through some human design learning rule are
- 49:03SPEAKER_00: a neural way of getting that.
- 49:19SPEAKER_00: Can we get this method to be good at sequential learning?
- 49:22SPEAKER_00: So I had a setup with,
- 49:24SPEAKER_00: the way I was testing this was on a very classic continuing learning task,
- 49:26SPEAKER_00: like the simplest one.
- 49:27SPEAKER_00: You take the MNIST dataset, handwritten digit recognition, you split it into five tasks.
- 49:33SPEAKER_00: You sequentially expose the model.
- 49:34SPEAKER_00: The idea here is you have the final accuracy on the five tasks of split MNIST.
- 49:41SPEAKER_00: To give you an illustration, split MNIST is one task is to learn, classify one and two.
- 49:46SPEAKER_00: The other task is two and three, three and four, blah, blah.
- 49:49SPEAKER_00: The final accuracy here,
- 49:50SPEAKER_00: the upper bound is the multitask setting where you take all tasks and you train them
- 49:54SPEAKER_00: at the same time.
- 49:54SPEAKER_00: So you have to know how to distinguish digits from one to 10.
- 49:58SPEAKER_00: This is the classic setting, it's the intro, it's the hello world of machine learning.
- 50:03SPEAKER_00: If you naively just fine-tune, you get accuracy of about 20% because this makes sense.
- 50:08SPEAKER_00: You have total forgetting.
- 50:09SPEAKER_00: So you train on the first task, you know the first task.
- 50:11SPEAKER_00: You train on the second task, you know that.
- 50:13SPEAKER_00: You get to the last task, you only know the last task, so your fine-tuning accuracy is this.
- 50:19SPEAKER_00: Elastic weight consolidation is one of the regularization methods,
- 50:22SPEAKER_00: one of the more popular continuing learning methods.
- 50:23SPEAKER_00: The reason it's so low here is because this is a sequential learning setting.
- 50:28SPEAKER_00: Sorry, I meant to say class incremental setting.
- 50:32SPEAKER_00: This number basically replicates one of the results in the three types
- 50:35SPEAKER_00: of incremental learning paper where they show, yeah,
- 50:37SPEAKER_00: in a class incremental setting,
- 50:39SPEAKER_00: elastic weight consolidation is no good because you need to compose between
- 50:42SPEAKER_00: the different representations.
- 50:44SPEAKER_00: And just being careful when you navigate a lost landscape is not enough for that.
- 50:48SPEAKER_00: Of course, that's with the lambda parameter being set to the optimal value.
- 50:51SPEAKER_00: It can get even worse.
- 50:52SPEAKER_00: This is another method I was trying.
- 50:54SPEAKER_00: Not too good.
- 50:55SPEAKER_00: But yeah, this method actually showed better accuracy.
- 50:57SPEAKER_00: So great news, right?
- 50:58SPEAKER_00: Like we are getting some progress on class incremental learning.
- 51:02SPEAKER_00: I was very excited when I saw this.
- 51:03SPEAKER_00: This is amazing.
- 51:04SPEAKER_00: And it shows that you can train a hyper network to give you better retention.
- 51:09SPEAKER_00: So you have the tasks, you expose your method to them.
- 51:12SPEAKER_00: Multitask is like training on all of them the whole time.
- 51:15SPEAKER_00: Fine-tuning, you fine-tune in the first one, second one, third one, your accuracy goes down.
- 51:19SPEAKER_00: So at step four,
- 51:20SPEAKER_00: basically,
- 51:21SPEAKER_00: your average accuracy is very low because your training set grows,
- 51:25SPEAKER_00: but your capability stays the same.
- 51:27SPEAKER_00: An interesting thing that happens with the weight merger method,
- 51:32SPEAKER_00: this hyper network, is that it actually starts getting better.
- 51:35SPEAKER_00: It starts going towards the multitask accuracy.
- 51:38SPEAKER_00: So as it encounters more tasks, it gets better.
- 51:41SPEAKER_00: And this is exactly what you were talking about with getting better at learning.
- 51:44SPEAKER_00: This is what meta learning is about.
- 51:45SPEAKER_00: You have more experiences and you are better at learning.
- 51:48SPEAKER_00: Here, it's very nice because we have a surface that can absorb all these lessons, right?
- 51:53SPEAKER_00: That's the hyper network.
- 51:54SPEAKER_00: So great.
- 51:54SPEAKER_00: And we've solved, yeah?
- 51:55SPEAKER_01: So if I understand correctly at step zero,
- 51:57SPEAKER_01: the weight merger, I'll only learn about like the fourth one.
- 52:02SPEAKER_01: Step one, it only learned about four and two.
- 52:05SPEAKER_01: And then after that, four, two, three, and just continue like this until a step four, right?
- 52:09SPEAKER_00: Yeah, indeed.
- 52:10SPEAKER_00: Normal fine-tuning basically settles down to an accuracy of one-fifth
- 52:15SPEAKER_00: because you only remember one-fifth of the data set that is all the digits, right?
- 52:18SPEAKER_00: Because it's five splits.
- 52:19SPEAKER_00: Weight merger gets better as it goes.
- 52:22SPEAKER_00: Originally, of course, it falls down a little bit.
- 52:24SPEAKER_00: When it encounters the first one, it has a certain accuracy.
- 52:27SPEAKER_00: The second one, third one.
- 52:28SPEAKER_00: And then as it encounters more, it gets better.
- 52:30SPEAKER_00: Have we solved continued learning?
- 52:31SPEAKER_00: Well, there's another encouraging graph about backward transfer.
- 52:34SPEAKER_00: So this is after you learn new things, you go back to the things that you saw before.
- 52:39SPEAKER_00: Are you better at that?
- 52:40SPEAKER_00: As a human, this happens often, right?
- 52:42SPEAKER_00: If you revisit maybe some problems that you faced in high school,
- 52:45SPEAKER_00: you're much smarter, you're better at that.
- 52:48SPEAKER_00: Here, none of the methods do that.
- 52:49SPEAKER_00: Our method does that.
- 52:50SPEAKER_00: So backward transfer is actually positive.
- 52:52SPEAKER_00: So did we solve continuing learning?
- 52:53SPEAKER_00: Well, one more thing is that Employ time cost per task is actually very low
- 52:58SPEAKER_00: because here you're not training, you're just inferencing.
- 53:01SPEAKER_00: You have front loaded in training this hyper network and
- 53:03SPEAKER_00: of course that's computationally heavy,
- 53:05SPEAKER_00: but at inference time,
- 53:06SPEAKER_00: when you are updating your weights, you can do that very sample efficiently.
- 53:10SPEAKER_00: The learning of this,
- 53:11SPEAKER_00: I did some ablation and turns out that the model actually learned a general operator
- 53:14SPEAKER_00: from
- 53:25SPEAKER_00: Moving any weight configuration towards a more multitask-shaped manifold
- 53:30SPEAKER_00: without looking at the task.
- 53:31SPEAKER_00: Because if you look, it has a few attempts to equalize them.
- 53:35SPEAKER_00: But yeah, so your network is big, your adapter, and your task is pretty small.
- 53:40SPEAKER_00: There's not as much information in your task as in your current weights.
- 53:43SPEAKER_00: So our hypernetwork learns to prioritize to only care about what the weights are of
- 53:49SPEAKER_00: the current network and just like nudge those weights towards the multitask manifold
- 53:54SPEAKER_00: and not really So it's a task-independent operator that moves a weight configuration
- 54:02SPEAKER_00: towards the most, which is interesting in itself.
- 54:04SPEAKER_00: There's definitely further work to do here.
- 54:06SPEAKER_00: And even I haven't stretched this.
- 54:07SPEAKER_00: I'm sure that there's ways to like scale the composite,
- 54:14SPEAKER_00: This is one of the interesting negative results
- 54:17SPEAKER_00: that maybe illustrates this graph I showed
- 54:20SPEAKER_00: about basically getting higher average accuracy as you see more information.
- 54:25SPEAKER_00: Wait, just to be clear, the denominator gets higher as you go to the right.
- 54:30SPEAKER_00: So, for example, here you've only seen one task and you see the accuracy on that task, right?
- 54:34SPEAKER_00: Here you've seen two tasks, three tasks, four tasks.
- 54:37SPEAKER_00: So the denominator is higher, but your accuracy even goes up.
- 54:40SPEAKER_00: So your average accuracy goes up.
- 54:41SPEAKER_00: So that means that it's quite considerable that your model gets higher.
- 54:45SPEAKER_01: Interesting point here.
- 54:46SPEAKER_01: Why does it stop at four steps?
- 54:49SPEAKER_01: Why can't you go back and do like another round of like zero, one, two, three, four, zero.
- 54:54SPEAKER_01: Well, because if I bring it back to a brain, right,
- 54:57SPEAKER_01: one of the kind of core tenets of being able to And so on,
- 55:00SPEAKER_01: on the other hand,
- 55:00SPEAKER_01: on the other hand,
- 55:01SPEAKER_01: on the other hand, on the other hand, on the other hand, on the other hand.
- 55:20SPEAKER_01: And then now what you have saved in your mind is not the 200 problem you've seen is
- 55:25SPEAKER_01: the commonality between all of them.
- 55:27SPEAKER_01: And it's one thing versus the other people that haven't done this drilling.
- 55:31SPEAKER_01: They have to think hard and try to discover this one shape,
- 55:35SPEAKER_01: but this one shape you've seen a hundred and a sense of it,
- 55:38SPEAKER_01: like in multiple kind of configuration,
- 55:41SPEAKER_01: you know that you plug into that one path,
- 55:43SPEAKER_01: you have two details to remember, and then you're done.
- 55:46SPEAKER_01: But taking mathematical problem here, it will be very, very hard for you.
- 55:50SPEAKER_01: To figure this out on your own without having seen this input output multiple times.
- 55:55SPEAKER_01: So going through it is actually a requirement, even though it's a bit dumb, right?
- 55:59SPEAKER_00: I think your results for,
- 56:01SPEAKER_00: of course, I mean,
- 56:02SPEAKER_00: I don't know as much about the idiosyncrasies of human learning,
- 56:05SPEAKER_00: but I have no doubt.
- 56:07SPEAKER_00: Anecdotally, this happens, right?
- 56:09SPEAKER_00: You hit your head against the wall and you still learn from the mere repetition.
- 56:12SPEAKER_00: But this has also been replicated even in demonstrated in language models, right?
- 56:17SPEAKER_00: I think even if you,
- 56:19SPEAKER_00: I think,
- 56:19SPEAKER_00: condition on,
- 56:21SPEAKER_00: I forget the exact Right formulation,
- 56:23SPEAKER_00: but you don't have to condition on the right answer, for example, in RL.
- 56:27SPEAKER_00: You can only, like,
- 56:28SPEAKER_00: even by conditioning on environment feedback, you're still getting learning signals.
- 56:33SPEAKER_00: Okay, of course, there were even results about random rewards giving you a reward signal.
- 56:36SPEAKER_00: Maybe that's also a way That you're accumulating experience, but I wouldn't make that claim.
- 56:40SPEAKER_00: But certainly the environment feedback is this.
- 56:43SPEAKER_00: You're encountering the problem from different perspectives and then you gain
- 56:47SPEAKER_00: an understanding just simply from the environment feedback
- 56:50SPEAKER_00: with no supervision necessarily, right?
- 56:52SPEAKER_00: It's just like almost self-reflection.
- 56:54SPEAKER_01: Kind of,
- 56:54SPEAKER_01: but even if you're doing the textbook kind of example,
- 56:58SPEAKER_01: Just doing one is just not enough to be able to,
- 57:01SPEAKER_01: if you look at the solution at the end,
- 57:03SPEAKER_01: you might not be able to understand the actual path if you see another instance
- 57:06SPEAKER_01: of it, but doing it enough, you just removed all of the details, right?
- 57:10SPEAKER_01: You remove the details, you don't need to remember any of them, they don't matter.
- 57:14SPEAKER_01: Even when you look at it,
- 57:15SPEAKER_01: you're like,
- 57:15SPEAKER_01: whether it's six,
- 57:17SPEAKER_01: 200, 10,000, it doesn't matter because they follow this two, three step rule.
- 57:21SPEAKER_01: And what I actually remembered,
- 57:22SPEAKER_01: and I saved in my brain as an automatic kind of methodology,
- 57:27SPEAKER_01: It's just this one thing.
- 57:28SPEAKER_01: There's this compression of a lot of instances into this one thing.
- 57:33SPEAKER_01: So whether you do it again and you do more of them,
- 57:35SPEAKER_01: it kind of is useless because you already know kind of the recipe,
- 57:38SPEAKER_01: but now you know it more in your bones.
- 57:40SPEAKER_01: Then if somebody was to tell you like, hey,
- 57:41SPEAKER_01: this is how you solve it,
- 57:43SPEAKER_01: you'd be like, okay, but then you have to remember it's hard and stuff like that.
- 57:47SPEAKER_01: By having gone through it enough to kind of learn that thing and then you're able
- 57:52SPEAKER_01: to kind of like on demand,
- 57:53SPEAKER_01: okay, generate like the solution for this and then just follow it, right?
- 57:56SPEAKER_01: And then you're able to get to the solution.
- 57:59SPEAKER_01: It reminded me of this setup.
- 58:01SPEAKER_01: This is why I said like why you're not kind of continuing.
- 58:04SPEAKER_01: Just do it again because like technically like the weights are in a better spot.
- 58:07SPEAKER_01: And on top of that, they're not just on the better spot too.
- 58:09SPEAKER_01: So like 0, 1, 2, 3, 4,
- 58:12SPEAKER_01: in a better spot of like solving,
- 58:15SPEAKER_01: even if there's a 5 now, like a 6 now, like you could, you will be able to do that.
- 58:19SPEAKER_01: But if you do it again,
- 58:20SPEAKER_01: and then you go back to a simpler thing,
- 58:22SPEAKER_01: for sure,
- 58:22SPEAKER_01: now you have a toolkit that is, that should be a bit improving the performance.
- 58:26SPEAKER_00: This is why people come to your show, always leaving probably with really good ideas.
- 58:30SPEAKER_00: This is a no-brainer to try.
- 58:32SPEAKER_00: I should have thought of it.
- 58:33SPEAKER_00: Yeah.
- 58:33SPEAKER_00: And you reminded me that I think tying back to our earlier discussion,
- 58:37SPEAKER_00: there is some threshold of experiencing your same problem in different scenarios
- 58:42SPEAKER_00: where you learn to take away the generalizable structure from it and throw away
- 58:47SPEAKER_00: the detail.
- 58:47SPEAKER_00: And indeed,
- 58:48SPEAKER_00: like if you had one task and replay it over and over,
- 58:50SPEAKER_00: not enough diversity, the model's not going to do it.
- 58:53SPEAKER_00: Five tasks, maybe.
- 58:54SPEAKER_00: So there is some optimal distribution of basically data mix that gives you
- 58:59SPEAKER_00: the generalizable thing, the opportunity to learn that, yeah.
- 59:02SPEAKER_01: So, but yeah, there's this element.
- 59:04SPEAKER_01: And also I remember something from Justin because like he built
- 59:07SPEAKER_01: the whole math academy thing where it will tell you what to like the flashcards type
- 59:12SPEAKER_01: of stuff.
- 59:12SPEAKER_01: Hey, you should redo this, this type of problem and stuff like that.
- 59:15SPEAKER_01: But the one thing that he spent a lot of time crafting was making sure
- 59:19SPEAKER_01: the hierarchical nature of math was properly encoded.
- 59:22SPEAKER_01: And it is kind of true that when you go in and try to learn a skill set,
- 59:27SPEAKER_01: having the basic unlock is very good,
- 59:29SPEAKER_01: but then You don't really need to practice them again because now you're moving up
- 59:34SPEAKER_01: to higher level tasks,
- 59:35SPEAKER_01: but general ability of what you learn is still kind of in this thing.
- 59:40SPEAKER_01: But now you learn something even more general that help you out, do the easier stuff.
- 59:44SPEAKER_01: So you could kind of forget even some of the stuff that you learned before,
- 59:47SPEAKER_01: they were just scaffolded and now like they're gone because now you were able
- 59:51SPEAKER_01: to learn that kind of innate in your bone general shape.
- 59:56SPEAKER_01: Exactly.
- 59:56SPEAKER_01: Abstraction ladder.
- 59:57SPEAKER_01: Yeah.
- 1:00:10SPEAKER_00: You're able to think of the more like principled primitives instead of just
- 1:00:14SPEAKER_00: the raw form of the individual experience.
- 1:00:17SPEAKER_00: So it's always super interesting to see the parallels with human learning because
- 1:00:21SPEAKER_00: of course, I mean,
- 1:00:22SPEAKER_00: you would call this a curriculum learning and of course,
- 1:00:25SPEAKER_00: Students are very susceptible to curriculum learning.
- 1:00:28SPEAKER_00: That's one of the results you see,
- 1:00:29SPEAKER_00: by the way,
- 1:00:29SPEAKER_00: in some papers that compare human learning and machine learning learning.
- 1:00:34SPEAKER_00: You see that curricula are super important for humans, right?
- 1:00:37SPEAKER_00: Because maybe this hierarchical nature of our Representations requires us
- 1:00:42SPEAKER_00: to encounter things in the right order.
- 1:00:44SPEAKER_00: Now, this is one of the approaches.
- 1:00:46SPEAKER_00: And of course, this is a simple task.
- 1:00:48SPEAKER_00: And when you start thinking
- 1:00:49SPEAKER_00: about benchmarking more realistic state-of-the-art models,
- 1:00:53SPEAKER_00: you don't do split MNIST anymore.
- 1:00:55SPEAKER_00: I think I can show you one set of results.
- 1:00:58SPEAKER_00: So that's one set of interesting experimentation I wanted to show you.
- 1:01:01SPEAKER_00: Another set is thinking about the compositionality aspect There is inherent compositionality.
- 1:01:21SPEAKER_00: And some more control over exactly how machine learning models can compose
- 1:01:28SPEAKER_00: between things learned at different times,
- 1:01:30SPEAKER_00: but in a manner that is called systematic generalization,
- 1:01:33SPEAKER_00: I think compositional generalization.
- 1:01:36SPEAKER_00: You're supposed to generalize,
- 1:01:37SPEAKER_00: but not just any form of generalization,
- 1:01:39SPEAKER_00: but specifically by composing priors that you've encountered before.
- 1:01:43SPEAKER_00: I will show you one paper where these tasks are quite pleasantly shown.
- 1:01:46SPEAKER_00: This paper that I will show now is an extremely interesting example where we talked
- 1:01:50SPEAKER_00: about one One set of results that was about class incremental learning
- 1:01:53SPEAKER_00: with hypernetics.
- 1:01:55SPEAKER_00: Now we're starting to talk about the structure of what I believe
- 1:02:00SPEAKER_00: a continual learning benchmark should look like.
- 1:02:02SPEAKER_00: One element there is compositionality.
- 1:02:04SPEAKER_00: Specifically, I think compositional generalization is important.
- 1:02:07SPEAKER_00: This is one of the benchmarks, one of the tasks that I think is very Intuitive to understand.
- 1:02:14SPEAKER_00: You learn some functions that operate on the primitives.
- 1:02:17SPEAKER_00: So for example, DAX is red, WIF is green.
- 1:02:20SPEAKER_00: And these words are chosen not to have any connotations so that humans learn them from scratch.
- 1:02:27SPEAKER_00: They're novel.
- 1:02:28SPEAKER_00: All right, yeah.
- 1:02:28SPEAKER_00: You have a task where you have some primitives,
- 1:02:31SPEAKER_00: so words mapped to colors, nonsensical words, and functions.
- 1:02:34SPEAKER_00: So for example,
- 1:02:35SPEAKER_00: Lug means blue,
- 1:02:36SPEAKER_00: Lugfep means blue three times, Dax means red, Daxfep means red three times.
- 1:02:40SPEAKER_00: So you have a function that operates on the primitives.
- 1:02:44SPEAKER_00: So as a learner,
- 1:02:46SPEAKER_00: you're supposed to learn the primitives and the functions and then be able
- 1:02:49SPEAKER_00: to compose them.
- 1:02:49SPEAKER_00: So you've never seen Lug being combined with Kiki and like with Kiki being combined
- 1:02:55SPEAKER_00: with Feb and combined with Whiff, I think.
- 1:02:57SPEAKER_00: Or maybe you have, but not with FEP as well at the same time.
- 1:03:01SPEAKER_00: So you've never seen this composition and you're supposed to solve it.
- 1:03:04SPEAKER_00: There was a certain approach here and an architecture
- 1:03:07SPEAKER_00: that actually got slightly higher compositional generalization than human.
- 1:03:11SPEAKER_00: That was quite interesting to see.
- 1:03:13SPEAKER_00: And of course this paper also, I can maybe put it on the screen so people can check it out.
- 1:03:17SPEAKER_00: So it's an interesting method of looking at systematic generalization,
- 1:03:22SPEAKER_00: specifically compositionality.
- 1:03:25SPEAKER_00: My interest was to probe compositionality from a continual perspective,
- 1:03:29SPEAKER_00: and I started with exactly this task setup.
- 1:03:31SPEAKER_00: The idea was that in this benchmark,
- 1:03:34SPEAKER_00: so you're seeing all the primitives at the same time and the functions,
- 1:03:37SPEAKER_00: and then you're simply asked to compose.
- 1:03:40SPEAKER_00: Think of a continual version of this, and this is how that would look like.
- 1:03:43SPEAKER_00: So this is one simple example.
- 1:03:46SPEAKER_00: On the x-axis, we have time steps, and here we have the different items we need to learn.
- 1:03:51SPEAKER_00: Some of these are primitives, like dax, kiki, tufa, zoop.
- 1:03:55SPEAKER_00: It's the same ones from the paper, and some of them function.
- 1:03:57SPEAKER_00: You don't encounter them at the same time.
- 1:03:59SPEAKER_00: Here, you first encounter a time point zero.
- 1:04:02SPEAKER_00: You encounter then one of the functions,
- 1:04:04SPEAKER_00: then the other function,
- 1:04:05SPEAKER_00: and the third And here is where we get to one of the core things
- 1:04:09SPEAKER_00: that I think continuing learning benchmarks
- 1:04:11SPEAKER_00: that gives me useful information should have,
- 1:04:13SPEAKER_00: which is a view of the dependencies that you're learning.
- 1:04:17SPEAKER_00: Within a multitask setting,
- 1:04:18SPEAKER_00: there's a lot of dependencies and you can even learn them Well enough because
- 1:04:21SPEAKER_00: during your gradient competition,
- 1:04:23SPEAKER_00: things average out and you can get to like a basin that is across multiple tasks.
- 1:04:29SPEAKER_00: With sequential learning, you don't have that luxury.
- 1:04:31SPEAKER_00: So I'm really interested to see how the handling of dependencies happen.
- 1:04:36SPEAKER_00: And the core primitive here is a directed graph.
- 1:04:39SPEAKER_00: This is basically a fancy term of saying, okay, you have some dependency.
- 1:04:43SPEAKER_00: Graph where, yeah,
- 1:04:44SPEAKER_00: you basically,
- 1:04:45SPEAKER_00: sequentially, you see things in a different order and there are mutual dependencies.
- 1:04:50SPEAKER_00: So for example,
- 1:04:51SPEAKER_00: the evaluation for this function requires you to know these primitives
- 1:04:57SPEAKER_00: and this other function.
- 1:04:58SPEAKER_00: And this can be procedurally generated.
- 1:05:00SPEAKER_00: You can have any random order.
- 1:05:02SPEAKER_00: This is the simplest case where we're only looking at dependencies that are positive.
- 1:05:07SPEAKER_00: So you have zoop and tufa,
- 1:05:09SPEAKER_00: and then twice, and then thrice, surround, and you can combine these things.
- 1:05:13SPEAKER_00: And this is what different knowledge units look like.
- 1:05:15SPEAKER_00: Basically, you get exposed that kiki Kiki means purple, Tufa means brown.
- 1:05:18SPEAKER_00: You get shown these things and then you get evaluated.
- 1:05:20SPEAKER_00: So since you learn that Kiki is purple, you get asked, hey, what's Kiki?
- 1:05:23SPEAKER_00: And you need to output purple and you evaluate this at the first time.
- 1:05:27SPEAKER_00: Then at the second time step, you get introduced to a function, right?
- 1:05:30SPEAKER_00: This twice function.
- 1:05:31SPEAKER_00: And then the probes get interesting because the probes are testing something
- 1:05:35SPEAKER_00: that you encountered At time step one,
- 1:05:37SPEAKER_00: which is this function twice,
- 1:05:39SPEAKER_00: but it also needs you to remember what the primitives are.
- 1:05:43SPEAKER_00: And those you haven't seen recently, you've seen them previously.
- 1:05:46SPEAKER_00: And there is this kind of dependency, and as you get higher, you get more crazier dependencies.
- 1:05:51SPEAKER_00: So here,
- 1:05:51SPEAKER_00: for example,
- 1:05:52SPEAKER_00: you need to, in the last time step, you need to compose thrice, Tufa, Surround.
- 1:05:57SPEAKER_00: So that's two functions that you've seen in different time points plus
- 1:06:00SPEAKER_00: the primitives you've seen at the beginning.
- 1:06:01SPEAKER_00: And this is the simplest form of this dependency tree.
- 1:06:04SPEAKER_00: Dependency tree of relationships and then you, yeah.
- 1:06:07SPEAKER_00: There's another form which is you interleave more.
- 1:06:10SPEAKER_00: So let's say first you learn a primitive,
- 1:06:12SPEAKER_00: then you learn a function,
- 1:06:13SPEAKER_00: then you learn another primitive and function, blah, blah.
- 1:06:16SPEAKER_00: And so here it's less of a curriculum, right?
- 1:06:19SPEAKER_00: And yeah, this is what the director of this graph looks like.
- 1:06:22SPEAKER_00: An interesting example that I think many sequential learning datasets will have,
- 1:06:27SPEAKER_00: but it's worth rendering very explicit,
- 1:06:31SPEAKER_00: is an adversarial example where Some of the primitives get redefined
- 1:06:36SPEAKER_00: because this happens with learners all the time, right?
- 1:06:38SPEAKER_00: You encounter something and then it changes definition or it changes state.
- 1:06:42SPEAKER_00: Yeah, so for example, zoop gets redefined, first defined at zero and then redefined at four.
- 1:06:46SPEAKER_00: And here we can see that in the graph.
- 1:06:48SPEAKER_00: And then at time step,
- 1:06:49SPEAKER_00: Step four, you test whether your model is acting on the old or the new definition.
- 1:06:55SPEAKER_00: In this case, we're expecting the model to redefine.
- 1:06:57SPEAKER_00: There is an aspect of complexity here that I have not included, which is a noisy data.
- 1:07:01SPEAKER_00: Here, we're assuming that what the model sees is correct.
- 1:07:04SPEAKER_00: One possible consideration for continual learning is how do you treat noisy data?
- 1:07:11SPEAKER_00: If you're a human and you hear someone say a really crazy theory about how
- 1:07:16SPEAKER_00: the world works and it doesn't make sense,
- 1:07:17SPEAKER_00: you're not going to update all of your priors to maps, right?
- 1:07:21SPEAKER_00: So you need to compose, yeah, you need to learn how to deal with noise.
- 1:07:25SPEAKER_00: I've purposely omitted it because I think it requires certain approaches that dilute
- 1:07:30SPEAKER_00: from the specific focus on composing.
- 1:07:32SPEAKER_00: When you compose, you need to know what to, forget what to learn.
- 1:07:36SPEAKER_00: I think learning to ignore certain things maybe requires a layer on top of that,
- 1:07:40SPEAKER_00: where maybe humans,
- 1:07:41SPEAKER_00: when we learn, we don't really learn on the raw inputs of the world.
- 1:07:45SPEAKER_00: A lot of our learning happens based on the replay that we have,
- 1:07:48SPEAKER_00: like hippocampal replayed at night, for example.
- 1:07:52SPEAKER_00: Element probably is more important for filtering out noise.
- 1:07:55SPEAKER_00: What I want is the prioritization of what to learn and what to do.
- 1:07:58SPEAKER_00: Yeah, so yeah,
- 1:07:59SPEAKER_00: there's also a backward version where first you learn the functions
- 1:08:02SPEAKER_00: and you don't exactly know,
- 1:08:04SPEAKER_00: like you learn them in a general form without knowing what the other things mean.
- 1:08:07SPEAKER_00: And afterwards, you learn what that means.
- 1:08:09SPEAKER_00: You read, for example, Lug and you have no idea what that means.
- 1:08:12SPEAKER_00: But then you learn it afterwards and you're able to tie it back.
- 1:08:16SPEAKER_00: Like you test whether the model is able to tie it back.
- 1:08:18SPEAKER_00: Yeah, and figure out.
- 1:08:19SPEAKER_00: Yeah, but yeah, so basically this is the setup for...
- 1:08:22SPEAKER_00: And I think this is where I've been a little in a bit of a dilemma.
- 1:08:27SPEAKER_00: So I think there's a tug of war inside me between super principled methods where,
- 1:08:33SPEAKER_00: for example, this benchmark, this is method agnostic, right?
- 1:08:36SPEAKER_00: You can have, you don't need a language prior.
- 1:08:38SPEAKER_00: You can have like a...
- 1:08:41SPEAKER_00: Autoencoder,
- 1:08:42SPEAKER_00: you can have a neural network, MLP,
- 1:08:46SPEAKER_00: any model of any size you want that does not require any extensive pre-training.
- 1:08:50SPEAKER_00: At the same time, it's not a realistic task that we're dealing with.
- 1:08:54SPEAKER_00: So if you look at something like RKGI and that excites you,
- 1:08:57SPEAKER_00: this is the kind of task that maybe you would be interested in.
- 1:09:00SPEAKER_00: And it's reasonable to assume that even LLMs should do well at these tasks.
- 1:09:04SPEAKER_00: Just like we expect them to do well at RKGI,
- 1:09:06SPEAKER_00: maybe they should do well at this because it's symbolic manipulation.
- 1:09:10SPEAKER_00: But I will transition now to the final part,
- 1:09:13SPEAKER_00: which is a more realistic setting, where we inherit this dependency tree.
- 1:09:18SPEAKER_00: So we take this dependency tree as a core first-class citizen,
- 1:09:22SPEAKER_00: and we try to construct more realistic tasks.
- 1:09:25SPEAKER_00: This is basically the last part I wanted to talk about.
- 1:09:29SPEAKER_00: Here we define a dependency tree, like a graph, based on what we call Knowledge Units.
- 1:09:35SPEAKER_00: And these can be anything you want.
- 1:09:37SPEAKER_00: These can be textbooks, these can be conversations, skills, news, anything you want.
- 1:09:42SPEAKER_00: And similarly to the previous setting,
- 1:09:44SPEAKER_00: you have some training order and you have some dependencies, right?
- 1:09:47SPEAKER_01: So Knowledge Unit 4 needs Knowledge Unit 1, Knowledge Unit 7 needs Knowledge Unit 4.
- 1:09:51SPEAKER_00: There are some overwrites here.
- 1:09:52SPEAKER_00: And another example I thought of,
- 1:09:54SPEAKER_00: which unfortunately is still relevant with LLMs, those are negative examples.
- 1:09:58SPEAKER_00: So for example,
- 1:09:59SPEAKER_00: if the model encounter,
- 1:10:00SPEAKER_00: there's a lot of pedagogical data about what not to do,
- 1:10:03SPEAKER_00: like reading,
- 1:10:04SPEAKER_00: like if it's an agent deployed in a customer support agency,
- 1:10:08SPEAKER_00: if it reads a lot of documentations about bad customer interactions,
- 1:10:12SPEAKER_00: it should learn the negative, right?
- 1:10:14SPEAKER_00: It should learn the contrastive nature of what it's seen.
- 1:10:19SPEAKER_00: There was an Owen Evans paper Recently that showed that models fine-tune
- 1:10:24SPEAKER_00: on negative pedagogical examples still internalize bad behaviors.
- 1:10:28SPEAKER_00: Owain Evans is the emergent misalignment researcher, the narrow fine-tuning papers line of work.
- 1:10:33SPEAKER_00: That was an interesting part,
- 1:10:34SPEAKER_00: but then I thought about I promise you,
- 1:10:36SPEAKER_00: I think this is one of the cruxes that I wanted to also discuss with you.
- 1:10:40SPEAKER_00: You want your continued learning benchmark to be method agnostic.
- 1:10:44SPEAKER_00: And this is where I will talk about some of the recent work here,
- 1:10:48SPEAKER_00: namely two works that I think have produced I'm sure you even saw it.
- 1:10:52SPEAKER_00: So they've created these environments.
- 1:10:54SPEAKER_00: I think coding and poker.
- 1:10:55SPEAKER_00: It's a really rich environment where they quantify the delta of whether
- 1:11:05SPEAKER_00: So this is where a model doesn't learn from experience and learning from experience.
- 1:11:09SPEAKER_00: Basically like all of their current leaderboard methods are in context learning
- 1:11:14SPEAKER_00: because this is about like having the experience and then improving across sequences
- 1:11:18SPEAKER_00: of tasks.
- 1:11:20SPEAKER_00: This is the work I want to shout out, really cool work.
- 1:11:22SPEAKER_00: They mostly look at the gain.
- 1:11:26SPEAKER_00: There's a bit of a scope difference between what I'm interested in and what they measure.
- 1:11:29SPEAKER_00: I think this gain metric that they have is really cool.
- 1:11:31SPEAKER_00: It quantifies again the difference of statelessness and statefulness.
- 1:11:37SPEAKER_00: They don't specifically look at dependencies and forgetting and forward transfer,
- 1:11:41SPEAKER_00: backward transfer, because this is a more realistic.
- 1:11:43SPEAKER_00: Another work I want to highlight is FutureSim.
- 1:11:46SPEAKER_00: They don't brand it as a continuing learning benchmark, but it is.
- 1:11:49SPEAKER_00: I've talked to them really, really, Brilliant people.
- 1:11:51SPEAKER_00: I'm not sure if you saw this,
- 1:11:52SPEAKER_00: but basically they have a corpus that temporally gets broader of news articles.
- 1:11:58SPEAKER_00: They had this data dump,
- 1:12:00SPEAKER_00: taped it,
- 1:12:00SPEAKER_00: cleaned it, and time tagged the articles, and they're continuously being exposed.
- 1:12:04SPEAKER_00: And the goal is that the model The models have to do forecasting
- 1:12:07SPEAKER_00: under non-stationarity, basically.
- 1:12:09SPEAKER_00: So they introduce things in chronological order and they see some interesting results.
- 1:12:13SPEAKER_00: I would really recommend reading their paper.
- 1:12:15SPEAKER_00: So for example, GPT 5.5 and Codex is really good.
- 1:12:18SPEAKER_00: Recently they benchmarked Opus.
- 1:12:19SPEAKER_00: One interesting finding was that when you start from a weak So they took
- 1:12:25SPEAKER_00: the weakest model's performance,
- 1:12:26SPEAKER_00: like that was Quen 3.56,
- 1:12:28SPEAKER_00: and then they hard-coded those predictions,
- 1:12:30SPEAKER_00: and then they basically unleashed the agent in the world
- 1:12:34SPEAKER_00: and saw how much it can improve.
- 1:12:36SPEAKER_00: And it seems like weak initial forecasts The forecast caps how good the models can be.
- 1:12:40SPEAKER_00: So getting anchored to a bad starting point messes you up a little bit.
- 1:12:43SPEAKER_00: I want to shout out these two works.
- 1:12:45SPEAKER_00: I think they're really getting at the problem from the right angle.
- 1:12:49SPEAKER_00: One piece that I'm interested in is getting an equal footing This is an apples
- 1:12:54SPEAKER_00: to apples comparison between in-context learning and in-weight continual learning.
- 1:12:59SPEAKER_00: With these benchmarks, I think current methods, this is all harness method.
- 1:13:03SPEAKER_00: This is all in-context learning, similarly in continual learning bench.
- 1:13:06SPEAKER_00: All the methods are basically in-context learning.
- 1:13:09SPEAKER_00: And I was thinking long and hard about how to represent this
- 1:13:13SPEAKER_00: because you basically want your substrate that you're training on
- 1:13:17SPEAKER_00: to be both amenable to in-context learning and to wait up.
- 1:13:20SPEAKER_00: And that's not easy because there's this paper, really cool paper.
- 1:13:23SPEAKER_00: Again, shout out on the generalization of language models
- 1:13:27SPEAKER_00: from in-context learning versus fine tuning.
- 1:13:28SPEAKER_00: So they kind of do a case study on how models learn differently in context versus in weights.
- 1:13:34SPEAKER_00: From a very basic intuitive level,
- 1:13:36SPEAKER_00: if you have a new fact and you give it to the model in context,
- 1:13:38SPEAKER_00: It's gonna remember it.
- 1:13:39SPEAKER_00: You've achieved learning.
- 1:13:40SPEAKER_00: If you try to bake it into the weights by literally doing next token prediction on
- 1:13:45SPEAKER_00: the fact, I don't think you're gonna get far.
- 1:13:46SPEAKER_00: So the authors here ask like, what are the exact characteristics of this?
- 1:13:51SPEAKER_00: And one way to think about this is through The reversal curse.
- 1:13:54SPEAKER_00: The finding here was that models,
- 1:13:56SPEAKER_00: predominantly in the training data,
- 1:13:58SPEAKER_00: see certain pieces of knowledge presented in one direction.
- 1:14:01SPEAKER_00: So for example, Tom Cruise's mom is whatever lady she is, her name.
- 1:14:06SPEAKER_00: And then if you ask the model,
- 1:14:09SPEAKER_00: The model has presumably been fine-tuned on a lot of articles and entries
- 1:14:12SPEAKER_00: that have this,
- 1:14:13SPEAKER_00: but it hasn't encountered that much data covering who the son of that woman is.
- 1:14:17SPEAKER_00: So it hasn't seen the fact that in the opposite order.
- 1:14:19SPEAKER_00: And if you ask the pre-trained model the question of like, who is the son of that lady?
- 1:14:24SPEAKER_00: It's not going to do very well.
- 1:14:25SPEAKER_00: If you present a fact in context and then you ask the question in the reverse order,
- 1:14:30SPEAKER_00: the model can generalize.
- 1:14:31SPEAKER_00: So in-context learning is really good.
- 1:14:32SPEAKER_00: That's one of the takeaways that I want to take away.
- 1:14:34SPEAKER_00: Like in a limited content regime, in-context learning is really powerful, right?
- 1:14:38SPEAKER_00: It can generalize.
- 1:14:40SPEAKER_01: Is this weight or without reasoning enabled?
- 1:14:42SPEAKER_00: That's a good question.
- 1:14:43SPEAKER_00: I think these are replicated results from the pre-reasoning era probably.
- 1:14:47SPEAKER_01: Okay.
- 1:14:47SPEAKER_01: Because my thought here is that from a very hand-wavy point of view,
- 1:14:52SPEAKER_01: what's in the weight is a bunch of functions that you've learned.
- 1:14:55SPEAKER_01: What
- 1:15:00SPEAKER_01: you do when you have reasoning is that you can kind of compose a bunch
- 1:15:02SPEAKER_01: of these functions and then you can figure this out, right?
- 1:15:05SPEAKER_01: So if you're given a question like who is the son of this woman?
- 1:15:09SPEAKER_01: One shot, you might go in and just use the wrong function and that's it.
- 1:15:13SPEAKER_01: That has this kind of information.
- 1:15:15SPEAKER_01: But if you're trying to reason over there, you might be able to flip it.
- 1:15:19SPEAKER_01: But if you get it in context,
- 1:15:21SPEAKER_01: you can be able to leverage these hidden function inside and just kind
- 1:15:25SPEAKER_01: of reverse it straight.
- 1:15:26SPEAKER_01: One shot, but you don't have the time to do it if you're not reasoning.
- 1:15:30SPEAKER_01: So that's my first reaction when I'm looking at this.
- 1:15:33SPEAKER_00: That's, again,
- 1:15:34SPEAKER_00: really sharp because one of the takeaways is
- 1:15:36SPEAKER_00: that in-context learning can surface these latents in the data.
- 1:15:41SPEAKER_00: I think reasoning does act like that,
- 1:15:43SPEAKER_00: but I don't think it's to be taken for granted that reasoning is trained with
- 1:15:46SPEAKER_00: that objective in mind.
- 1:15:47SPEAKER_00: I think it happens immediately.
- 1:15:48SPEAKER_00: I think that's one of the interesting goals of RL training.
- 1:15:52SPEAKER_00: So in-context learning, very good.
- 1:15:55SPEAKER_00: Pre-training model, fine-tuned model, really bad.
- 1:15:57SPEAKER_00: Now, there is a method of fine-tuning that does better than ICL.
- 1:16:10SPEAKER_00: And that is augmented fine-tune.
- 1:16:11SPEAKER_00: And this exactly gets to what you talked about.
- 1:16:13SPEAKER_00: You want to surface the latent.
- 1:16:15SPEAKER_00: By the way, these results are replicated across all kinds of generalizations.
- 1:16:18SPEAKER_00: So reversals, syllogisms, Like simple rephrases, category switches.
- 1:16:24SPEAKER_00: Yeah, basically in-context learning is very good at all these.
- 1:16:26SPEAKER_00: Like I think the headline result for me for in-context capability is
- 1:16:29SPEAKER_00: when I think Gemini 2 or Gemini 1.5 was announced and they had this great moment
- 1:16:36SPEAKER_00: where they stated how they gave it a dictionary for a low resource language.
- 1:16:41SPEAKER_00: Talamang, I think it has 200 speakers and then it learned it in context.
- 1:16:44SPEAKER_00: And of course,
- 1:16:44SPEAKER_00: if you think about the differences,
- 1:16:46SPEAKER_00: of course,
- 1:16:46SPEAKER_00: if you do next token prediction fine tuning,
- 1:16:48SPEAKER_00: So how do you compare these methods apples to apples in any benchmark let alone
- 1:16:51SPEAKER_00: a continual learning?
- 1:16:52SPEAKER_00: One way that I originally had
- 1:17:04SPEAKER_00: I did sequential fine-tuning on a bunch of cloud skills, and I tested a bunch of methods.
- 1:17:08SPEAKER_00: Specifically, I did next token prediction, normal SFT, supervised fine-tuning.
- 1:17:13SPEAKER_00: That's off policy because it's some data setting,
- 1:17:17SPEAKER_00: fine-tuning on it,
- 1:17:17SPEAKER_00: and it's completely Off-policy knowledge distillation,
- 1:17:21SPEAKER_00: that means, yeah, again,
- 1:17:23SPEAKER_00: you're not sampling from the student,
- 1:17:24SPEAKER_00: you're sampling from the teacher, making the student imitate the teacher.
- 1:17:27SPEAKER_00: And then self-distillation fine-tuning, which is the paper.
- 1:17:29SPEAKER_00: This basically replicated the results.
- 1:17:31SPEAKER_00: If you do the two baselines,
- 1:17:33SPEAKER_00: which is either Vanilla Quinn or next token prediction,
- 1:17:36SPEAKER_00: very little learning, but also you don't forget a lot.
- 1:17:38SPEAKER_00: Top row, by the way, here are specific skills.
- 1:17:41SPEAKER_00: These were marked down skill files for some MCP server.
- 1:17:45SPEAKER_00: First Stripe, then Twitter, then Notion.
- 1:17:47SPEAKER_00: It was like sequential training.
- 1:17:49SPEAKER_00: Yeah,
- 1:17:49SPEAKER_00: basically we validate the results of self-distillation fine-tuning achieves superior
- 1:17:53SPEAKER_00: or comparable retention while mitigating for getting more than
- 1:17:58SPEAKER_00: the off-policy method.
- 1:17:59SPEAKER_00: So this was nice, especially if you look at the instruction following eval.
- 1:18:02SPEAKER_00: Yeah, here self-distillation fine-tuning even accumulates probably.
- 1:18:06SPEAKER_00: So yeah, this was a nice check.
- 1:18:09SPEAKER_00: One thing I did here is the training set here was synthetically generated in
- 1:18:14SPEAKER_00: the form of the augmentation that I showed in the previous paper.
- 1:18:18SPEAKER_00: Yeah, basically like you generate synthetic QF errors, you fine tune on that.
- 1:18:21SPEAKER_00: And when you do that, again, this is the core problem I was trying to grapple with.
- 1:18:26SPEAKER_00: You lose your apples-to-apples comparability with in-context learning
- 1:18:30SPEAKER_00: because you wouldn't do that with in-context learning.
- 1:18:32SPEAKER_00: So one place I landed at for how to make it approachable for in-context learning was
- 1:18:38SPEAKER_00: to actually commit to on-policy traces because on-policy traces are both relevant
- 1:18:44SPEAKER_00: for fine-tuning.
- 1:18:45SPEAKER_00: For like RL tuning and they're relevant for ICL.
- 1:18:47SPEAKER_00: Because just as you said,
- 1:18:48SPEAKER_00: like you've tried something, you try something else and this stays in context.
- 1:18:52SPEAKER_00: Of course, when the context grows, you compact.
- 1:18:54SPEAKER_00: But I think this is a comparable baseline.
- 1:18:57SPEAKER_00: And I think we added machinery of the dependency tree.
- 1:19:01SPEAKER_00: It allows you to probe a bunch of the features of continual learning that you're interested in.
- 1:19:07SPEAKER_00: And maybe I can show a preliminary result there, but this is not tuned well enough.
- 1:19:13SPEAKER_00: I've been working on this in the past couple of weeks.
- 1:19:16SPEAKER_00: So here's basically,
- 1:19:18SPEAKER_00: it's a similar graph to a lot of methods you'll see about continuing learning.
- 1:19:21SPEAKER_00: This is a probably a poorly tuned on policy self distillation run.
- 1:19:26SPEAKER_00: But yeah, but this is one of the runs I got.
- 1:19:28SPEAKER_00: The idea here is that you have a training This is a training run,
- 1:19:31SPEAKER_00: where you have the same model, you're training it for 700 steps.
- 1:19:34SPEAKER_00: Every 100 steps, you are changing the environment.
- 1:19:38SPEAKER_00: Here it's introducing new things in the database,
- 1:19:41SPEAKER_00: a new category of tasks,
- 1:19:43SPEAKER_00: for example,
- 1:19:44SPEAKER_00: sometimes even changing the available And then seeing it deal with this complexity.
- 1:19:49SPEAKER_00: At each boundary, we see, for example, certain transitions are very rapid.
- 1:19:53SPEAKER_00: Going from hotel booking to restaurant ops results in a rapid drop at first,
- 1:19:58SPEAKER_00: whereas some transitions are more neat.
- 1:20:01SPEAKER_00: These are really preliminary results,
- 1:20:02SPEAKER_00: but the idea is that here,
- 1:20:04SPEAKER_00: I think this is a setup that I'm only showing on policy self-determination
- 1:20:09SPEAKER_00: and GRPO here.
- 1:20:10SPEAKER_00: The in-context learning agentic runs are turning out to be quite expensive.
- 1:20:14SPEAKER_00: So we're going to hold off on those I'm going to get the results for now.
- 1:20:16SPEAKER_00: But yeah,
- 1:20:17SPEAKER_00: because I mean, again,
- 1:20:18SPEAKER_00: even if you use like DeepSeq 4 Pro,
- 1:20:21SPEAKER_00: the whole point here is that the way that I calibrated this is that the point was by
- 1:20:25SPEAKER_00: the time you reach 700 steps,
- 1:20:28SPEAKER_00: the in-contact learning methods should accumulate several I think that's to
- 1:20:32SPEAKER_00: a degree what the Continual Learning Bench folks did.
- 1:20:34SPEAKER_00: This is kind of
- 1:20:45SPEAKER_00: I take on the dependencies because I, again,
- 1:20:49SPEAKER_00: I treat those dependencies as really a first class citizen and I really want
- 1:20:52SPEAKER_00: to see what happens when,
- 1:20:54SPEAKER_00: for example,
- 1:20:55SPEAKER_00: one ablation is if you have a dependency between two knowledge units
- 1:20:58SPEAKER_00: and two environments.
- 1:20:59SPEAKER_00: For example,
- 1:21:00SPEAKER_00: in the middle of calendar scheduling,
- 1:21:01SPEAKER_00: you see something and in the middle of medical clinic, you see something.
- 1:21:04SPEAKER_00: You want to compose between those.
- 1:21:06SPEAKER_01: Okay.
- 1:21:07SPEAKER_00: How does the compositionality depend on the temporal distance?
- 1:21:10SPEAKER_00: Both absolute temporal distance and recency.
- 1:21:14SPEAKER_00: So if certain things are really far apart and really far apart from now,
- 1:21:18SPEAKER_00: you probably have a really hard time remembering them, even if they're important.
- 1:21:22SPEAKER_00: If certain things are close together but really far apart, you forget them.
- 1:21:25SPEAKER_00: But if one of them is really recent, then that sort of acts as replay.
- 1:21:29SPEAKER_00: So you remember some of the old one too.
- 1:21:31SPEAKER_00: But yeah, there's a lot of oblations you can do.
- 1:21:33SPEAKER_00: But this, I think, is the configuration of Landadon.
- 1:21:36SPEAKER_00: So having on-policy traces.
- 1:21:38SPEAKER_00: So you can use them both in in-context learning and in distillation.
- 1:21:41SPEAKER_00: That's as far as I wanted to get now, as far as showing stuff.
- 1:21:45SPEAKER_00: It's really great that
- 1:21:47SPEAKER_01: you're showing us all this and
- 1:21:50SPEAKER_01: that you're thinking super hard about these questions.
- 1:21:53SPEAKER_01: Like also just benchmarking it properly is absolutely crucial because I don't know,
- 1:21:57SPEAKER_01: even in humans, if you have like five topic that you're trying to learn, right?
- 1:22:01SPEAKER_01: And they're kind of like related in some way.
- 1:22:04SPEAKER_01: It's not five topics, really.
- 1:22:06SPEAKER_01: It's like two and a half or something like that, right?
- 1:22:09SPEAKER_01: I remember when I was stress testing my learning in university,
- 1:22:12SPEAKER_01: I did one of the semester where everything was kind of connected together.
- 1:22:16SPEAKER_01: I had biology stuff, physiology stuff, anatomy things, right?
- 1:22:21SPEAKER_01: But it was all kind of connected conceptually, right?
- 1:22:24SPEAKER_01: And I did another semester where it was not at all connected and the metals, very different.
- 1:22:29SPEAKER_01: There was a computer science stuff.
- 1:22:32SPEAKER_01: There was another one that was like a very, very discreet math.
- 1:22:35SPEAKER_01: There was like a PhD level course that I just got in.
- 1:22:38SPEAKER_01: Like single processing, no clue what's up.
- 1:22:42SPEAKER_01: I did like one course about that didn't have any prerequisite.
- 1:22:46SPEAKER_01: It was like a mess, literally.
- 1:22:48SPEAKER_01: I did that purposefully,
- 1:22:49SPEAKER_01: but the mental strain that you have over there is nothing
- 1:22:54SPEAKER_01: when things make logical sense.
- 1:22:55SPEAKER_01: When they're connected,
- 1:22:57SPEAKER_01: While you're training for one,
- 1:22:58SPEAKER_01: even if it's not the same exact topic,
- 1:23:01SPEAKER_01: you're able to kind of transfer at least some of the skills for learning the thing.
- 1:23:05SPEAKER_01: Because if you have a biology thing going on, it's mostly about fact memorization.
- 1:23:09SPEAKER_01: Like there's nothing else, really.
- 1:23:11SPEAKER_01: You're not going to go and then like, Splice some stuff.
- 1:23:13SPEAKER_01: It's just facts, right?
- 1:23:15SPEAKER_01: If you have a chemistry thing,
- 1:23:16SPEAKER_01: it's like,
- 1:23:17SPEAKER_01: it's a mix of fact that you need to memorize in order
- 1:23:20SPEAKER_01: to even know what you're talking about and some procedural formula that you have
- 1:23:25SPEAKER_01: to know when to apply.
- 1:23:26SPEAKER_01: That's kind of what is up.
- 1:23:27SPEAKER_01: When you have mathematical stuff or hard,
- 1:23:30SPEAKER_01: kind of unbiased type of learning, it's just about drilling.
- 1:23:33SPEAKER_01: It literally doesn't matter, right?
- 1:23:35SPEAKER_01: And then we have stuff where you have to have some creative component.
- 1:23:38SPEAKER_01: Let's say all the computer science things that had web development,
- 1:23:41SPEAKER_01: whatever it is,
- 1:23:42SPEAKER_01: there is a creative component that you can only get and that is tested
- 1:23:45SPEAKER_01: that you can only get by doing the thing, right?
- 1:23:47SPEAKER_01: So yeah, you can drill some questions, but it's not about that.
- 1:23:50SPEAKER_01: This is mostly just doing the project.
- 1:23:56SPEAKER_01: Actually, if you do the project, it will work.
- 1:23:57SPEAKER_01: But these were so different skill set and meta learning element that I really had
- 1:24:02SPEAKER_01: the same amount of classes and I was literally burning out.
- 1:24:05SPEAKER_01: It was very, very difficult.
- 1:24:07SPEAKER_01: But in the first case, when everything was pretty much the same, it literally didn't matter.
- 1:24:11SPEAKER_01: You got better at one meta learning kind of game that you were playing.
- 1:24:15SPEAKER_01: You just apply it to the other one.
- 1:24:16SPEAKER_01: You have more fact to learn.
- 1:24:17SPEAKER_01: Nice, right?
- 1:24:18SPEAKER_01: You just have three cheat sheets versus having one,
- 1:24:21SPEAKER_01: but then you can go and you can activate that part.
- 1:24:24SPEAKER_01: So that's what I was thinking when you were showing this stuff.
- 1:24:26SPEAKER_00: So I think that reminds me of,
- 1:24:29SPEAKER_00: I think,
- 1:24:30SPEAKER_00: The multiple in-depth courses,
- 1:24:32SPEAKER_00: even if they're difficult,
- 1:24:33SPEAKER_00: it's kind of like you're having the same task in different forms.
- 1:24:37SPEAKER_00: You need to excel at a course,
- 1:24:38SPEAKER_00: but having the intricate ones is just like one mega course that you need to take.
- 1:24:42SPEAKER_00: So, and of course that's more difficult.
- 1:24:43SPEAKER_00: I wonder what someone like Justin would say about what explains even
- 1:24:47SPEAKER_00: the subjective experience of fatigue.
- 1:24:49SPEAKER_00: Because maybe he would say like retrieval practice is so,
- 1:24:52SPEAKER_00: and when you need to compose,
- 1:24:54SPEAKER_00: it's all about like retrieving the right things at the right time all the time.
- 1:24:57SPEAKER_00: So that's very fatigue.
- 1:24:58SPEAKER_00: I don't know.
- 1:24:59SPEAKER_01: Exactly.
- 1:25:00SPEAKER_01: I've talked about this to Justin and like,
- 1:25:02SPEAKER_01: it was actually trying to learn biology with the mastery learning technique,
- 1:25:05SPEAKER_01: which is like you build upon stuff.
- 1:25:07SPEAKER_01: But like what we've figured out is it was not really mastery learning.
- 1:25:11SPEAKER_01: It was just like going top down and just Just getting to the point as fast
- 1:25:15SPEAKER_01: as possible, which like biology thing, you could do that very easily.
- 1:25:19SPEAKER_01: You don't have to like know the basics of the cells to understand what
- 1:25:22SPEAKER_01: the organ is doing, right?
- 1:25:23SPEAKER_01: It does help, but it's not the hierarchical structure.
- 1:25:27SPEAKER_01: It's pretty much not the same.
- 1:25:29SPEAKER_01: It just requires a bit of a different skill set.
- 1:25:32SPEAKER_01: And also like this pushed me into the,
- 1:25:34SPEAKER_01: like, how do you learn stuff that there's nothing that is going to tell you?
- 1:25:37SPEAKER_01: And then you have to make a discovery out of it, right?
- 1:25:39SPEAKER_01: Like you have to learn how to find out the knowledge.
- 1:25:42SPEAKER_01: So like just basically research, right?
- 1:25:43SPEAKER_01: And what I realized is that like the stuff that most looked like research
- 1:25:48SPEAKER_01: in this synthetic environment that I had for me was
- 1:25:51SPEAKER_01: the course I had zero prerequisite in.
- 1:25:53SPEAKER_01: Like I should not have been in that class basically, right?
- 1:25:57SPEAKER_01: And this,
- 1:25:57SPEAKER_01: because I was also doing research at the same time,
- 1:25:59SPEAKER_01: these two are really similar because you don't have any prerequisite.
- 1:26:02SPEAKER_01: You don't know what's going on.
- 1:26:04SPEAKER_01: So you have to kind of continuously latch on With whatever seems to make sense,
- 1:26:10SPEAKER_01: look at the pattern and then you just have to go figure it out, right?
- 1:26:13SPEAKER_01: It's very inefficient because if I had taken the prerequisite,
- 1:26:16SPEAKER_01: I would have been there like, yeah, I know that.
- 1:26:18SPEAKER_01: But in this case, nothing that they showed me was like, I know that.
- 1:26:22SPEAKER_01: So you have to kind of go and continuously try to figure a grasp.
- 1:26:27SPEAKER_01: And as soon as you have like some semblance of signal,
- 1:26:30SPEAKER_01: like you will be able to kind of go in and like get some traction.
- 1:26:33SPEAKER_01: But the meta-learning game is even more crucial here.
- 1:26:37SPEAKER_01: You cannot be inefficient.
- 1:26:39SPEAKER_01: You really have to get the meta-learning game on point for this part in order
- 1:26:45SPEAKER_01: to be able to grasp onto something, which in research is also the same.
- 1:26:49SPEAKER_01: You really need to kind of get some semblance of traction first.
- 1:26:52SPEAKER_01: So if you can build upon other people work, that's great.
- 1:26:55SPEAKER_01: If you can mix and match like other people works, that's fantastic, right?
- 1:26:59SPEAKER_01: If you can take something that you've done and just do the ablation,
- 1:27:03SPEAKER_01: you're going to learn something and then you can kind of get a grip and kind
- 1:27:06SPEAKER_01: of get it moving.
- 1:27:07SPEAKER_01: But it's a very similar type of a meta-learning situation was in effect in these two things.
- 1:27:11SPEAKER_01: If it was like some basic biology, it has nothing to do with learning new stuff and research.
- 1:27:16SPEAKER_01: It's literally nothing.
- 1:27:17SPEAKER_01: It's just like fact.
- 1:27:19SPEAKER_00: We're being too harsh towards biology here.
- 1:27:21SPEAKER_00: I'm sure there is some like systems level learning that happens, right?
- 1:27:25SPEAKER_00: Like you, of course it's facts, but.
- 1:27:27SPEAKER_01: But I mean, like everything is so messy.
- 1:27:29SPEAKER_01: Yeah.
- 1:27:30SPEAKER_01: And everything is so arbitrary.
- 1:27:31SPEAKER_01: Like, because there's no start to the thing.
- 1:27:35SPEAKER_01: It's just the thing, right?
- 1:27:36SPEAKER_01: And this makes it like you have nothing to grasp but the thing.
- 1:27:39SPEAKER_01: So you go and then you memorize the thing and then you may be composing Yeah.
- 1:27:42SPEAKER_01: Yeah.
- 1:27:43SPEAKER_01: Yeah.
- 1:27:43SPEAKER_01: Yeah.
- 1:27:43SPEAKER_01: Yeah.
- 1:28:11SPEAKER_01: Yeah.
- 1:28:11SPEAKER_01: But it had the same flavor as when I was doing neuro stuff or like
- 1:28:15SPEAKER_01: when I was doing like computational stuff in the research, I think.
- 1:28:17SPEAKER_00: You actually,
- 1:28:18SPEAKER_00: what you said about,
- 1:28:19SPEAKER_00: I think fundamentally indeed discovery shaped problems are very different
- 1:28:25SPEAKER_00: to like optimization.
- 1:28:26SPEAKER_00: It's not a problem, right?
- 1:28:27SPEAKER_00: You don't have like a golden data set to hill climb, for example.
- 1:28:31SPEAKER_00: You don't have a good verifier.
- 1:28:32SPEAKER_00: And there was a paper that actually had a setup for this,
- 1:28:36SPEAKER_00: I think Learning to Discover at Test Time, it was called.
- 1:28:39SPEAKER_00: Recently, and yeah, basically they use some cool And they have some reuse methods.
- 1:28:56SPEAKER_00: Methods.
- 1:28:57SPEAKER_00: It's a really cool paper.
- 1:28:58SPEAKER_00: We recommend it for people.
- 1:29:00SPEAKER_00: Another connection was this fast,
- 1:29:02SPEAKER_00: slow training that they introduced, I think, yeah, this last week.
- 1:29:06SPEAKER_00: This is a really,
- 1:29:07SPEAKER_00: really cool paper that kind of redoes prompt optimization and
- 1:29:10SPEAKER_00: then weight optimization.
- 1:29:11SPEAKER_00: Some kind of hierarchical approach like this will likely be a part of
- 1:29:14SPEAKER_00: a good solution to continuing learning.
- 1:29:16SPEAKER_00: And they had an ablation where they gave it a really hard task.
- 1:29:19SPEAKER_00: And vanilla RL will bang the head against the wall for very long when you have low reward.
- 1:29:24SPEAKER_00: Because one of the criteria for doing RL successfully is that you want to start with
- 1:29:29SPEAKER_00: a low-ish reward.
- 1:29:31SPEAKER_00: And then he'll climb to a high reward because you need to have some signal to Latron.
- 1:29:37SPEAKER_00: What happens when your reward is literally 0.0.
- 1:29:40SPEAKER_00: Well, with RL,
- 1:29:41SPEAKER_00: you're really hoping that you can roll the dice 100 steps in and it's going to work.
- 1:29:45SPEAKER_00: This method,
- 1:29:46SPEAKER_00: you have this self-reflection loop that gets you out of that zero reward regime
- 1:29:50SPEAKER_00: as fast as possible.
- 1:29:51SPEAKER_00: And tied back to what you said earlier about replay,
- 1:29:54SPEAKER_00: I think one way we get better at replay,
- 1:29:57SPEAKER_00: at continuing learning and accumulating our skills,
- 1:29:59SPEAKER_00: of course,
- 1:30:00SPEAKER_00: is Whatever really happens,
- 1:30:02SPEAKER_00: like your campus might be getting better at replay and whatever.
- 1:30:04SPEAKER_00: A part of it is crystallized skills.
- 1:30:06SPEAKER_00: But I think there's also a middle layer of intuition about what kind
- 1:30:10SPEAKER_00: of on-policy data to experience.
- 1:30:13SPEAKER_00: And this is basically that.
- 1:30:14SPEAKER_00: This is nudging yourself in prompt space to get to a place
- 1:30:18SPEAKER_00: where you get non-zero rewards so then your weight updates can kick in.
- 1:30:22SPEAKER_00: I could tie it to pedagogical RL too, but I think we've been going on for quite some time now.
- 1:30:28SPEAKER_01: Yeah.
- 1:30:28SPEAKER_01: I could go like this for like another hour, man.
- 1:30:31SPEAKER_01: Like this is so cool.
- 1:30:32SPEAKER_01: Thank you also for showing all of these papers.
- 1:30:34SPEAKER_01: I'm going to rewatch the thing and actually read them.
- 1:30:37SPEAKER_01: What's next on your side, on the cultural learning side?
- 1:30:40SPEAKER_01: Like what are you trying to experiment?
- 1:30:42SPEAKER_01: You showed us one experiment that you're kind of like going at it.
- 1:30:45SPEAKER_01: Any other angle that you're keen on this thing out?
- 1:30:47SPEAKER_01: Yeah.
- 1:30:47SPEAKER_00: Currently I am mostly trying to get the benchmark.
- 1:30:51SPEAKER_00: I think it has survived a lot of revisions.
- 1:30:54SPEAKER_00: My goal is to get it out.
- 1:30:55SPEAKER_00: As for methods,
- 1:30:57SPEAKER_00: I think,
- 1:30:57SPEAKER_00: of course,
- 1:30:58SPEAKER_00: going back and feeding the split MNIST tasks in the same order,
- 1:31:03SPEAKER_00: that's one of the things.
- 1:31:04SPEAKER_00: But yeah,
- 1:31:05SPEAKER_00: as far as methods,
- 1:31:06SPEAKER_00: I think,
- 1:31:07SPEAKER_00: of course,
- 1:31:07SPEAKER_00: there's a lot of exciting stuff happening with the on-policy distillation stuff.
- 1:31:11SPEAKER_00: There's a lot of flavors now.
- 1:31:12SPEAKER_00: And not only flavors, I don't want to devalue them.
- 1:31:15SPEAKER_00: I think there are a lot of interesting variations that are innovative.
- 1:31:20SPEAKER_00: There's definitely stuff there.
- 1:31:21SPEAKER_00: One thing I've been interested in is synthetically evolving environments.
- 1:31:25SPEAKER_00: So that might be an interesting rabbit hole to go into afterward.
- 1:31:29SPEAKER_00: Because, yeah, the data distribution is a key lever in what you learn.
- 1:31:35SPEAKER_00: The hippocampus does it for us to a large degree.
- 1:31:38SPEAKER_00: In classical RL,
- 1:31:39SPEAKER_00: I don't know what you would consider classical RL now,
- 1:31:42SPEAKER_00: but there's unsupervised environment design where you're varying
- 1:31:46SPEAKER_00: the environment complexity to maximize the learning delta, basically.
- 1:31:50SPEAKER_00: So something like that where you are thinking about environments as also things
- 1:31:55SPEAKER_00: to be molded either by an outer loop or by
- 1:31:58SPEAKER_00: the learning algorithm itself is It's something interesting,
- 1:32:02SPEAKER_00: particularly because,
- 1:32:03SPEAKER_00: yeah, synthetic data, maybe this can be a nice thing to end at.
- 1:32:06SPEAKER_00: So synthetic data is, I think, a great point of awakening that I'm glad we've reached.
- 1:32:10SPEAKER_00: I remember in the pre-training era,
- 1:32:13SPEAKER_00: what I mean by that is like before reasoning models and whatnot,
- 1:32:16SPEAKER_00: a lot of the rhetoric about synthetic data was, okay, data is like oil.
- 1:32:19SPEAKER_00: It's going to run out.
- 1:32:21SPEAKER_00: And then we're going to have problems.
- 1:32:22SPEAKER_00: And we have model collapse if we train models of synthetic data.
- 1:32:26SPEAKER_00: But that turned out not to be really true.
- 1:32:28SPEAKER_00: And synthetic data strategies are probably one of the big most that the big labs have right now.
- 1:32:32SPEAKER_00: So synthetic data methods really are crucial.
- 1:32:35SPEAKER_00: And one very satisfying explanation for why is It's in this paper that defines
- 1:32:42SPEAKER_00: a new metric for information.
- 1:32:44SPEAKER_00: It is quite a mathematical paper, so no exciting figures and bolded tables and whatnot.
- 1:32:49SPEAKER_00: They define another metric for information that is a little different
- 1:32:54SPEAKER_00: to Shannon information because you have the data processing inequality, right?
- 1:32:58SPEAKER_00: It says that, okay,
- 1:32:59SPEAKER_00: if you have a photo and it has a certain resolution,
- 1:33:02SPEAKER_00: there's nothing you can do to that photo to increase the information content.
- 1:33:05SPEAKER_00: Right?
- 1:33:06SPEAKER_00: In the movies,
- 1:33:06SPEAKER_00: when they say zoom in and from a pixelated photo,
- 1:33:08SPEAKER_00: you get to a super crisp photo, you can't do that.
- 1:33:11SPEAKER_00: Yeah, it turns out though that,
- 1:33:12SPEAKER_00: of course, I mean,
- 1:33:13SPEAKER_00: that is true in the absolute physical sense,
- 1:33:15SPEAKER_00: but it doesn't say anything about the ergonomics of how How accessible the data is.
- 1:33:22SPEAKER_00: And the whole point of representation learning is that it processes data in a way
- 1:33:26SPEAKER_00: that it renders it very easy to act on.
- 1:33:29SPEAKER_00: If you think about a problem maybe,
- 1:33:32SPEAKER_00: in a way, the solution to a math problem is latent in the statement, right?
- 1:33:36SPEAKER_00: But of course you need many processing steps to get there.
- 1:33:39SPEAKER_00: So what these folks say is that processing,
- 1:33:42SPEAKER_00: they define this metric and they say,
- 1:33:43SPEAKER_00: okay, processing actually This
- 1:33:51SPEAKER_00: is the part I would recommend people read.
- 1:33:54SPEAKER_00: It's a nice way to think about what synthetic data methods are.
- 1:33:58SPEAKER_00: So they're not methods of creating data out of nothing.
- 1:34:01SPEAKER_00: They are ways of using an existing model to expose latent information
- 1:34:06SPEAKER_00: that Hasn't been explicit, making it explicit.
- 1:34:09SPEAKER_00: And this is something that in-contact learning does very well.
- 1:34:12SPEAKER_00: It's really good at the latent.
- 1:34:13SPEAKER_00: Fine-tuning in its current form doesn't, right?
- 1:34:15SPEAKER_00: It only looks at the surface information.
- 1:34:17SPEAKER_00: If you want it to extract the latent,
- 1:34:19SPEAKER_00: you have to augment the training set,
- 1:34:20SPEAKER_00: do replay, have it reason, and then distill on that.
- 1:34:23SPEAKER_00: So yeah, these things interact.
- 1:34:25SPEAKER_00: But that's, I thought, is a nice thing to end on, this little mathematical.
- 1:34:29SPEAKER_01: Yeah.
- 1:34:29SPEAKER_01: That's, that's, that's super interesting.
- 1:34:31SPEAKER_01: Yeah.
- 1:34:32SPEAKER_01: And the last bit kind of mirror a bit like, um,
- 1:34:35SPEAKER_01: from just a human learning perspective,
- 1:34:38SPEAKER_01: if you pre-process the information in a way that you can more easily ingest it,
- 1:34:41SPEAKER_01: but just not just from a pure factual information,
- 1:34:45SPEAKER_01: from the fact of where you're at right now.
- 1:34:49SPEAKER_01: That's one of the skills that is visible throughout.
- 1:34:51SPEAKER_01: It doesn't matter if it's right or wrong.
- 1:34:54SPEAKER_01: If you're able to kind of take it into your worldview and kind of inject that into
- 1:34:59SPEAKER_01: the content, it sticks much better, right?
- 1:35:01SPEAKER_01: And if you can make sure that you're able to Your leveraging skill set that
- 1:35:04SPEAKER_01: the human brain is good at natively,
- 1:35:06SPEAKER_01: then like the learning is much easier and much faster.
- 1:35:09SPEAKER_01: Okay, man, this was absolutely awesome.
- 1:35:11SPEAKER_01: Thank you very much for like preparing the deck,
- 1:35:13SPEAKER_01: did this presentation and showing us so much paper because I'll be taking a list
- 1:35:18SPEAKER_01: of them, putting them in the description and also read them beautifully.
- 1:35:22SPEAKER_01: Thanks a lot.
- 1:35:23SPEAKER_00: Thank you for the opportunity, really.
- 1:35:25SPEAKER_00: Thanks for reaching out.
- 1:35:27SPEAKER_00: Really awesome.
- 1:35:27SPEAKER_00: Great engagement.
- 1:35:29SPEAKER_00: Like already our conversation has me spinning ideas.
- 1:35:32SPEAKER_00: It's exposing the latence, you know?
- 1:35:34SPEAKER_00: Thank you.
- 1:35:35SPEAKER_00: Very big information, Delta.
- 1:35:36SPEAKER_01: Yeah.
- 1:35:36SPEAKER_01: And let me know as soon as you have like the benchmark ready and the paper ready,
- 1:35:40SPEAKER_01: I want to read it too.
- 1:35:41SPEAKER_00: That's it, man.
- 1:35:42SPEAKER_00: Have a nice day.
- 1:35:42SPEAKER_00: See ya.
- 1:35:43SPEAKER_00: Thank you.
- 1:35:43SPEAKER_00: Really great time.