Latest / Dwarkesh Podcast / 8 Predictions for the Era of Continual Learning

Transcript

8 Predictions for the Era of Continual Learning · Dwarkesh Podcast

• 0:00
  Speaker 1: So I've explained elsewhere why I think actual continual learning
• 0:03
  Speaker 1: is needed. I don't think you can have AIs that
• 0:04
  Speaker 1: perform whole jobs as competently as humans if they are
• 0:08
  Speaker 1: forced to just write markdown files from session to session.
• 0:11
  Speaker 1: Just to give an illustrative example, imagine if this is
• 0:13
  Speaker 1: the way that students learn to play the saxophone. So
• 0:15
  Speaker 1: you have one student, he's never played the saxophone before,
• 0:18
  Speaker 1: he goes into the music hall, he tries to play it,
• 0:19
  Speaker 1: of course this is his first time, so he fails.
• 0:22
  Speaker 1: And he writes down a bunch of notes about what
• 0:24
  Speaker 1: went wrong. And there's a next student who's waiting outside
• 0:26
  Speaker 1: the music hall, he comes in, he reads all these notes,
• 0:28
  Speaker 1: he's also never played, so of course he messes up,
• 0:31
  Speaker 1: and he continues to add on to these notes. And
• 0:33
  Speaker 1: you have an infinity of students who are outside the
• 0:35
  Speaker 1: music hall who keep writing notes to the next person.
• 0:38
  Speaker 1: I don't think there's any sequence of text they could
• 0:39
  Speaker 1: write to each other that would allow the subsequent student
• 0:43
  Speaker 1: to just nail the saxophone from the first try. At
• 0:46
  Speaker 1: some point, you actually have to accumulate the relevant experience
• 0:49
  Speaker 1: into your brain. I think the same thing will be
• 0:51
  Speaker 1: true for a lot of skills that we want AIs
• 0:54
  Speaker 1: to actually accumulate from all the different workplaces in which
• 0:57
  Speaker 1: they're deployed. Okay, so what changes once we have actual
• 1:00
  Speaker 1: continual learning? One, I think that a lot of proposals
• 1:03
  Speaker 1: that have been put forward about regulating AI assume that
• 1:06
  Speaker 1: you train a model and then you deploy it. And therefore,
• 1:10
  Speaker 1: if you run a bunch of checks on the model
• 1:11
  Speaker 1: before it is deployed, we can make sure that it's
• 1:14
  Speaker 1: not going to aid in cyber attacks or do something crazy.
• 1:16
  Speaker 1: I don't think this assumption necessarily makes sense in the future.
• 1:19
  Speaker 1: And this is one of the many reasons I'm actually
• 1:21
  Speaker 1: kind of worried about locking in some kind of safety
• 1:24
  Speaker 1: regulatory regime right now, because we don't know what kind
• 1:27
  Speaker 1: of technology we're going to be dealing with even within
• 1:29
  Speaker 1: a year, let alone within five years or 10 years.
• 1:32
  Speaker 1: What if the model is improving every single day based
• 1:35
  Speaker 1: on the millions of sessions of work it does in
• 1:37
  Speaker 1: that day? If that happens, we could potentially be locking
• 1:40
  Speaker 1: in an archaic and potentially counterproductive approach to dealing with
• 1:43
  Speaker 1: the threats from AI. To the extent the government wants
• 1:45
  Speaker 1: some way to do some kind of safety evaluation on
• 1:48
  Speaker 1: model providers, I think it would make more sense to
• 1:50
  Speaker 1: do monthly or quarterly risk inspections rather than trying to
• 1:53
  Speaker 1: single out some special moment that occurs after training is done,
• 1:57
  Speaker 1: but before deployment begins, because that will not be a
• 2:00
  Speaker 1: meaningfully distinct category in the future. Two, how the labs
• 2:04
  Speaker 1: do technical alignment would probably totally need to change. Right now,
• 2:07
  Speaker 1: a lot of research is focused on the question of
• 2:10
  Speaker 1: how we make sure that a frozen set of weights
• 2:12
  Speaker 1: behaves well during deployment. But I'm not aware of much
• 2:16
  Speaker 1: research on the question of how we make it so
• 2:17
  Speaker 1: that even with constant weight updates, the AI system never
• 2:22
  Speaker 1: falls prey to jailbreaks or changes into a deceptive or
• 2:25
  Speaker 1: evil persona. And if AIs are consolidating learnings between users
• 2:28
  Speaker 1: as well, how do you prevent users from injecting backdoors
• 2:31
  Speaker 1: or some kind of malicious inclination into the base model?
• 2:34
  Speaker 1: In some sense, this is actually kind of what the
• 2:36
  Speaker 1: human alignment problem is, right? Humans improve in a self-directed way.
• 2:40
  Speaker 1: If you have kids, I don't have kids, but I
• 2:42
  Speaker 1: imagine this is what happens. If you have kids, they
• 2:44
  Speaker 1: go out, they learn new things. Sometimes they go crazy.
• 2:47
  Speaker 1: They get one-shotted by crazy ideologies. They take the wrong drug.
• 2:51
  Speaker 1: They become super weird. But you hope that you've given
• 2:54
  Speaker 1: them enough common sense and basic values that they improve
• 2:58
  Speaker 1: as people in a self-directed way without ending up with
• 3:01
  Speaker 1: some super weird beliefs or some misanthropic ideas. Three, the
• 3:05
  Speaker 1: diversity of AI minds will increase. Right now, there are
• 3:09
  Speaker 1: less than five prominent AI minds, by which I mean
• 3:11
  Speaker 1: the base models, which are served to millions or hundreds
• 3:15
  Speaker 1: of millions or billions of users at once. And they're
• 3:17
  Speaker 1: all quite similar to each other, by the way, because
• 3:19
  Speaker 1: they've also been all trained on roughly the same data.
• 3:22
  Speaker 1: But if AIs are learning from experience, and that experience
• 3:24
  Speaker 1: is different between not only different AI companies, but also
• 3:28
  Speaker 1: between different instances of the same AI model, we could
• 3:31
  Speaker 1: actually see a lot of diversity come out the other
• 3:34
  Speaker 1: end in this world. And this would be, I think,
• 3:36
  Speaker 1: a net good outcome. I think one of the things
• 3:38
  Speaker 1: to worry about in the future is just having this
• 3:40
  Speaker 1: monolithic singleton that's quite boring. A world where we have
• 3:43
  Speaker 1: continual learning would hopefully be more interesting than the mode
• 3:47
  Speaker 1: collapse of different models we see in the world right now. Four.
• 3:51
  Speaker 1: When deployment becomes part of training, the returns to being
• 3:54
  Speaker 1: ahead in the AI race accelerate. Because if you have
• 3:57
  Speaker 1: the best model and more people are using your AI
• 4:00
  Speaker 1: for more complicated and useful work, and as a result,
• 4:03
  Speaker 1: they're giving you lots of feedback that can integrate beyond
• 4:06
  Speaker 1: the session window, then your model will become even smarter. Five,
• 4:10
  Speaker 1: if the model learns mainly from deployment, then labs will
• 4:13
  Speaker 1: feel a lot of pressure to deploy their smartest models earlier.
• 4:16
  Speaker 1: Anthropic has reportedly been using mythos internally since February, but
• 4:20
  Speaker 1: it only shipped the model to the public in June.
• 4:22
  Speaker 1: In the regime with actual continual learning, this kind of
• 4:24
  Speaker 1: thing would just not be possible. You could not keep
• 4:26
  Speaker 1: a four-month gap between internal and external deployment and still
• 4:30
  Speaker 1: be competitive because a competitor who ships the worst model
• 4:34
  Speaker 1: on release date will have a smarter model based on
• 4:37
  Speaker 1: actual real-world experience. Six, continual learning will create a clear
• 4:41
  Speaker 1: mode for the leading AI labs that they currently lack.
• 4:45
  Speaker 1: Many people have been asking, how will the AI labs
• 4:47
  Speaker 1: actually make money? I have been asking this. When I
• 4:50
  Speaker 1: had Dario on the podcast, I asked him this question.
• 4:52
  Speaker 1: And he made the analogy of cloud providers. And he
• 4:54
  Speaker 1: made the point, look, the cloud providers are offering many
• 4:57
  Speaker 1: undifferentiated services. but they're earning high profit margins nonetheless. You
• 5:01
  Speaker 1: will have noticed this if you look at Amazon or
• 5:03
  Speaker 1: Google's quarterly earnings, they're doing just fine. But the reason
• 5:07
  Speaker 1: that the cloud margins are so high is that it's
• 5:09
  Speaker 1: really time consuming and expensive to switch from one cloud
• 5:12
  Speaker 1: to another. Currently, there's nothing that's stopping me from starting
• 5:15
  Speaker 1: a software repository with Codex and then doing more work
• 5:19
  Speaker 1: on it with cursor, and then finishing it up with
• 5:21
  Speaker 1: cloud code. But once we have actual continual learning, and
• 5:24
  Speaker 1: the model you're working with is actually getting better as
• 5:26
  Speaker 1: it interacts with you from session to session, then there
• 5:29
  Speaker 1: are actually pretty significant switching costs. If you want to
• 5:32
  Speaker 1: change the AI that you're using, you basically had to
• 5:33
  Speaker 1: fire an employee that has accumulated months of context on
• 5:36
  Speaker 1: your organization, and you replace them with a very fresh,
• 5:40
  Speaker 1: very unexperienced new intern that you had to retrain from scratch.
• 5:44
  Speaker 1: And once you have this kind of lock-in, model providers
• 5:47
  Speaker 1: can demand pretty hefty margins. Sorry, really, I'm just sort
• 5:53
  Speaker 1: of a fashion nerd. Seven, of course, enterprises will be
• 5:58
  Speaker 1: wise to this kind of dynamic. They will try to
• 6:00
  Speaker 1: avoid this kind of lock-in. But what if the choice
• 6:03
  Speaker 1: is that you either get locked into a model provider
• 6:05
  Speaker 1: or you lose out on the super valuable feature where
• 6:07
  Speaker 1: your model improves for you from session to session? If
• 6:10
  Speaker 1: real usage ends up being the main way the models improve,
• 6:13
  Speaker 1: then the AI labs may subsidize users in enterprises which
• 6:16
  Speaker 1: allow the model to train on their sessions. This is
• 6:18
  Speaker 1: already happening if you look at the kinds of deals
• 6:20
  Speaker 1: that are offered to new users of coding products. This
• 6:24
  Speaker 1: is very similar to why Google gives away search. And conversely,
• 6:27
  Speaker 1: the labs may say that any enterprise that refuses to
• 6:29
  Speaker 1: let them train on the sessions can't have access to
• 6:32
  Speaker 1: the very best models. With both carrots and sticks, the
• 6:34
  Speaker 1: labs can do a lot to get users to allow
• 6:37
  Speaker 1: AIs to learn from experience. Now, of course, I'm glossing
• 6:40
  Speaker 1: over the fact that there's a difference between updating one
• 6:43
  Speaker 1: user set of weights and pulling all these different weight
• 6:45
  Speaker 1: forks back into the main model. And the latter may
• 6:48
  Speaker 1: be more technically challenging, but in due time, this too
• 6:51
  Speaker 1: will be solved. Eight, AI training already has large economies
• 6:54
  Speaker 1: of scale. You get to amortize all this expensive training
• 6:57
  Speaker 1: across more users. And you see the evidence for this
• 7:00
  Speaker 1: in the fact that the lab revenues are increasing far
• 7:03
  Speaker 1: faster than their compute. But continual learning may also lead
• 7:06
  Speaker 1: to economies of scale in inference for end users, namely
• 7:09
  Speaker 1: from batching. You might have seen my episode with Reiner
• 7:11
  Speaker 1: Pope where we discussed this in detail. But if per-company
• 7:14
  Speaker 1: instructions require full-weight updates rather than living in low-rank adapters,
• 7:18
  Speaker 1: there's huge advantages from batching. Back-of-the-envelope math suggests that the
• 7:23
  Speaker 1: optimal inference batch size for a sparse model like, say,
• 7:26
  Speaker 1: DeepSeq v3 is more than 2,400 concurrent sequences being generated
• 7:32
  Speaker 1: at once. If you don't do this, then you're under
• 7:34
  Speaker 1: utilizing your compute. And if you want to understand why, again,
• 7:37
  Speaker 1: I highly recommend that episode with Reiner on inference economics.
• 7:40
  Speaker 1: But anyways, the point here is that a given set
• 7:42
  Speaker 1: of weights is only served efficiently when thousands of sequences
• 7:45
  Speaker 1: are being decoded against it all at once. A large
• 7:48
  Speaker 1: company with lots of employees and agents who are doing
• 7:51
  Speaker 1: lots of different kinds of things can very efficiently serve
• 7:56
  Speaker 1: their weight fork. Whereas an individual user who's only running
• 7:58
  Speaker 1: a batch size one may suffer more than two orders
• 8:01
  Speaker 1: of magnitude worse efficiency on their compute. So the economics
• 8:05
  Speaker 1: of serving personalized weights strongly favor big organizations. Obviously plenty
• 8:09
  Speaker 1: more will have changed by the time that continual learning
• 8:12
  Speaker 1: actually works. And the most important changes are probably the
• 8:15
  Speaker 1: ones that are hardest to anticipate in advance. But the
• 8:17
  Speaker 1: ones above seem kind of clear even now. This was
• 8:22
  Speaker 1: a narration of a blog that I also published on
• 8:24
  Speaker 1: my website. Go check it out at Dwarcash.com. Otherwise, I
• 8:28
  Speaker 1: will see you on the next podcast.