Latest / Faster, Please! — The Podcast / ✨🔬 Acceleration though AI-automated R&D: My chat (+transcript) with researcher Tom Davidson
Transcript
- 0:06SPEAKER_01: Welcome to Faster Please, the podcast.
- 0:09SPEAKER_01: I'm your host, Jim Pethikoukis.
- 0:12SPEAKER_01: Several times a month,
- 0:13I'll feature a lively conversation with a fascinating and
- 0:16provocative guest about
- 0:18SPEAKER_01: how to make the world a better place through scientific discovery,
- 0:21SPEAKER_01: technological innovation,
- 0:23SPEAKER_01: and economic growth.
- 0:24SPEAKER_01: You're also going to want to check out my Faster Please newsletter.
- 0:27SPEAKER_01: You're on Substack throughout the week for fresh essays,
- 0:30SPEAKER_01: Q&As,
- 0:31SPEAKER_01: and stories from around the internet and around the world.
- 0:36What really gets AI optimists excited isn't the prospect of
- 0:41automating customer
- 0:42SPEAKER_01: service departments or human resources.
- 0:46SPEAKER_01: Imagine,
- 0:46SPEAKER_01: rather,
- 0:47what might happen to the pace of scientific progress if AI
- 0:51becomes a super research
- 0:53SPEAKER_01: assistant.
- 0:54SPEAKER_01: Researcher Tom Davidson's new paper,
- 0:57SPEAKER_01: How Quick and Big Would a Software Intelligence Explosion Be,
- 1:02SPEAKER_01: explores that very scenario.
- 1:05SPEAKER_01: Today on Faster Please,
- 1:06SPEAKER_01: the podcast,
- 1:07I talk with Davidson about what it would mean for automated AI
- 1:11researchers to
- 1:12SPEAKER_01: rapidly improve their own algorithms,
- 1:15SPEAKER_01: thus creating a self-reinforcing loop of innovation.
- 1:19We talk about the economic effects of self-improving AI
- 1:23research and how close we
- 1:25SPEAKER_01: are to that reality.
- 1:27Davidson is a senior research fellow at Forethought,
- 1:30where he explores AI and explosive growth.
- 1:33He was previously a senior research fellow at Open Philanthropy
- 1:37and a research
- 1:37SPEAKER_01: scientist at the UK government's AI Security Institute.
- 1:42SPEAKER_01: A few years ago,
- 1:44you wrote a paper called Could Advanced AI Drive Explosive
- 1:48Economic Growth,
- 1:49which argued that growth could accelerate dramatically if AI
- 1:54would start generating
- 1:55SPEAKER_01: ideas the way human researchers once did.
- 1:58In your view,
- 1:59population growth historically powered kind of an ideas
- 2:03feedback loop.
- 2:04SPEAKER_01: More people meant more researchers, meant more ideas, rising incomes.
- 2:08But that loop broke after the demographic transition in the
- 2:11late 19th century.
- 2:12SPEAKER_01: But you suggest that AI could restart it.
- 2:14SPEAKER_01: More ideas, more output, more AI, more ideas.
- 2:17SPEAKER_01: Does this new paper in a way build upon that paper?
- 2:21SPEAKER_01: How quick and big would a software intelligence explosion be?
- 2:25So the first baby you referred to is about the biggest picture
- 2:29dynamic of economic growth.
- 2:31SPEAKER_00: And so,
- 2:31SPEAKER_00: you know,
- 2:31SPEAKER_00: as you said,
- 2:32SPEAKER_00: throughout the long run history,
- 2:34SPEAKER_00: when we kind of produced more food,
- 2:36SPEAKER_00: the population increased.
- 2:38So that additional output transferred itself into more people,
- 2:42more workers.
- 2:44SPEAKER_00: These days, that doesn't happen.
- 2:45SPEAKER_00: You know, when GDP goes up, that doesn't mean people have more kids.
- 2:47SPEAKER_00: In fact,
- 2:48You know,
- 2:48the demographic transition,
- 2:50the richer people get,
- 2:51the fewer kids they have.
- 2:52SPEAKER_00: So now it's kind of like we've got more output.
- 2:53SPEAKER_00: We're getting even fewer people as a result.
- 2:55SPEAKER_00: So that's been blocked.
- 2:57SPEAKER_00: And then,
- 2:57SPEAKER_00: you know,
- 2:57SPEAKER_00: this first paper is basically saying,
- 2:58SPEAKER_00: look,
- 2:59SPEAKER_00: if we can manufacture human minds or human equivalent minds in any way,
- 3:04SPEAKER_00: be it by building them,
- 3:05SPEAKER_00: you know,
- 3:06building more computer chips or making better computer chips or
- 3:09any way at all,
- 3:11SPEAKER_00: then that feedback loop gets going again.
- 3:13SPEAKER_00: Because if we can manufacture more human minds,
- 3:14SPEAKER_00: then we can spend output again to create more workers.
- 3:19SPEAKER_00: And so that's the first paper.
- 3:20The second paper kind of double clicks on one specific way that
- 3:24we can use output
- 3:25SPEAKER_00: to create more human minds.
- 3:27SPEAKER_00: And it's actually the kind of,
- 3:28SPEAKER_00: in a way,
- 3:28the scariest way because it's the way of creating human minds
- 3:31which can happen the
- 3:32SPEAKER_00: quickest.
- 3:33SPEAKER_00: So this is the way where you don't have to build any more computer chips.
- 3:35SPEAKER_00: You don't have to,
- 3:36SPEAKER_00: you know,
- 3:36SPEAKER_00: develop,
- 3:37SPEAKER_00: build any more,
- 3:38SPEAKER_00: you know,
- 3:39SPEAKER_00: fabs as they're called,
- 3:40SPEAKER_00: these kind of big,
- 3:41SPEAKER_00: big factories that make computer chips.
- 3:44SPEAKER_00: In fact, you don't have to do anything at all in the physical world.
- 3:47It seems like most of the conversation has been about how much
- 3:49investment is going
- 3:51SPEAKER_01: to go building,
- 3:51SPEAKER_01: how many new data centers,
- 3:53and that seems like that is almost the entire conversation in a
- 3:56way at the moment.
- 3:57SPEAKER_01: But you're not looking at compute, you're looking at software.
- 4:00SPEAKER_00: Exactly, software.
- 4:01SPEAKER_00: So the idea is you don't have to build anything.
- 4:03SPEAKER_00: You've already got loads of computer chips,
- 4:05and you just make the algorithms that run the AIs on those
- 4:09computer chips more
- 4:09SPEAKER_00: efficient.
- 4:10SPEAKER_00: And so,
- 4:10SPEAKER_00: you know,
- 4:10SPEAKER_00: this is already happening,
- 4:11SPEAKER_00: but it isn't yet a big deal because AI isn't that capable.
- 4:14SPEAKER_00: But,
- 4:14SPEAKER_00: you know,
- 4:14SPEAKER_00: already one year out,
- 4:16SPEAKER_00: EPOC,
- 4:16SPEAKER_00: this AI forecasting organization,
- 4:18SPEAKER_00: estimates that just in one year,
- 4:20SPEAKER_00: it becomes...
- 4:2310 times to a thousand times cheaper to run the same ai system
- 4:27so that is you know
- 4:28just wait just wait 12 months and suddenly for the same budget
- 4:31you are able to run
- 4:33you know 10 times as many ai systems or maybe you know even a
- 4:36thousand times as
- 4:36many for their most aggressive estimate so you know as i said
- 4:39not a big deal today
- 4:40but if we then develop an ai system which is you know better
- 4:43than any human at
- 4:44SPEAKER_00: doing research then now in 10 months you haven't built anything
- 4:48But you've got 10 times as many researchers that you can set to
- 4:52work or even more than that.
- 4:55And so then we get this feedback loop where you make some
- 4:58research progress,
- 4:59SPEAKER_00: you improve your algorithms.
- 5:00SPEAKER_00: Now you've got loads more researchers.
- 5:01You set them all to work,
- 5:03again,
- 5:03finding even more algorithmic improvements.
- 5:06So today we've got maybe a few hundred people that are
- 5:09advancing state-of-the-art AI algorithms.
- 5:11SPEAKER_00: I think they're all getting paid a billion dollars a person too.
- 5:14SPEAKER_00: Exactly.
- 5:16SPEAKER_00: But,
- 5:16SPEAKER_00: you know,
- 5:17maybe we can 10x that initially by having them replaced by AI
- 5:20researchers that kind
- 5:21SPEAKER_00: of do the same thing.
- 5:22SPEAKER_00: But then those AI researchers improve their own algorithms.
- 5:26SPEAKER_00: Now you have 10x as many again,
- 5:27SPEAKER_00: you haven't building more computer chips,
- 5:28SPEAKER_00: you're just running them more efficiently.
- 5:30SPEAKER_00: And then the kind of the cycle continues,
- 5:31you're throwing more and more of these AI researchers at AI
- 5:35progress itself.
- 5:36And the kind of the algorithms are improving in this what might
- 5:38be a very powerful
- 5:39SPEAKER_00: feedback loop.
- 5:41SPEAKER_01: In this case,
- 5:42it seems to me that you're not necessarily talking about
- 5:44artificial general
- 5:45SPEAKER_01: intelligence.
- 5:46SPEAKER_01: This is certainly a powerful intelligence, but it's narrow, right?
- 5:51SPEAKER_01: It doesn't have to do everything.
- 5:52SPEAKER_01: It doesn't have to play chess, right?
- 5:54SPEAKER_01: It just has to be able to do research.
- 5:57SPEAKER_00: It's certainly not fully general.
- 5:59SPEAKER_00: You don't need it to be able to control a robot body.
- 6:01SPEAKER_00: You don't need it to be able to solve the Riemann hypothesis.
- 6:05You don't need it to be able to even be very persuasive or
- 6:09charismatic to a human.
- 6:11SPEAKER_00: It's not narrow, I wouldn't say.
- 6:13SPEAKER_00: It has to be able to do literally anything
- 6:15SPEAKER_00: that AI researchers do.
- 6:16SPEAKER_00: And that's a wide range of tasks.
- 6:17SPEAKER_00: They're coding.
- 6:19SPEAKER_00: They're communicating with each other.
- 6:20SPEAKER_00: They're managing people.
- 6:22SPEAKER_00: They are planning out what to work on.
- 6:25SPEAKER_00: They are thinking about reviewing the literature.
- 6:27SPEAKER_00: There's a fairly wide range of stuff.
- 6:29SPEAKER_00: It's extremely challenging.
- 6:30SPEAKER_00: It's some of the hardest work in the world to do.
- 6:33SPEAKER_00: So I wouldn't say it's now, but it's not everything.
- 6:36It's some kind of intermediate level of generality in between a
- 6:40mere chess
- 6:40algorithm that just does chess and the kind of AGI that can
- 6:43literally do anything.
- 6:45I think people who are cautiously optimistic about AI will say
- 6:49something like,
- 6:51SPEAKER_01: yeah,
- 6:51I could see the kind of intelligence you're referring to coming
- 6:54about within a
- 6:55SPEAKER_01: decade,
- 6:56SPEAKER_01: but it's going to take a couple of big breakthroughs to get there.
- 7:00SPEAKER_01: Is that true or are we actually getting pretty close?
- 7:04SPEAKER_00: Yeah.
- 7:05I mean,
- 7:05famously predicting the future of technology is very,
- 7:08very difficult.
- 7:09SPEAKER_00: You know,
- 7:09SPEAKER_00: just a few years before people invented the nuclear bomb,
- 7:12SPEAKER_00: you know,
- 7:13SPEAKER_00: famous,
- 7:14SPEAKER_00: very well-respected physicists were saying it's impossible.
- 7:16SPEAKER_00: This will never happen.
- 7:17SPEAKER_00: So,
- 7:17SPEAKER_00: you know,
- 7:18SPEAKER_00: my best guess is that we do need a couple of fairly,
- 7:21SPEAKER_00: you know,
- 7:22SPEAKER_00: fairly non-trivial breakthroughs.
- 7:24SPEAKER_00: So,
- 7:24SPEAKER_00: you know,
- 7:24SPEAKER_00: we had,
- 7:24SPEAKER_00: um,
- 7:25SPEAKER_00: you know,
- 7:25SPEAKER_00: we had the start of RL training,
- 7:27SPEAKER_00: you know,
- 7:27SPEAKER_00: a couple of years ago became a big deal,
- 7:29SPEAKER_00: you know,
- 7:29SPEAKER_00: within the language model paradigm.
- 7:31SPEAKER_00: you know,
- 7:32SPEAKER_00: that's the kind of breakthrough I think we'll probably need,
- 7:34SPEAKER_00: you know,
- 7:34SPEAKER_00: another couple of breakthroughs of that kind of size.
- 7:36SPEAKER_00: We're not talking like completely new approach,
- 7:38SPEAKER_00: throw everything out,
- 7:39SPEAKER_00: but we're talking like,
- 7:40SPEAKER_00: okay,
- 7:40we need to like extend the current approach in a kind of a
- 7:44meaningfully different
- 7:45SPEAKER_00: way.
- 7:45SPEAKER_00: It's going to take some inventiveness.
- 7:46SPEAKER_00: It's going to take some creativity.
- 7:47SPEAKER_00: We're going to have to try out a few things.
- 7:49I think probably we'll need that to get to the researcher that
- 7:52can fully automate
- 7:53SPEAKER_00: open AI is a nice way of putting it.
- 7:54SPEAKER_00: You know, open AI doesn't employ any humans anymore.
- 7:57SPEAKER_00: They've just got AIs there.
- 7:58There's a difference between what a model can do on some
- 8:01benchmark versus becoming
- 8:04SPEAKER_01: actually sort of productive in the real world.
- 8:08SPEAKER_01: That's why while all the sort of benchmark stuff is interesting,
- 8:11SPEAKER_01: like the thing I pay attention to is...
- 8:13SPEAKER_01: How are businesses beginning to use this technology?
- 8:17SPEAKER_01: Because that's the leap.
- 8:18What is that gap like in your scenario versus an AI model that
- 8:23can sort of do a
- 8:24theoretical version of the lab to actually be incorporated in a
- 8:28real laboratory?
- 8:30it's definitely a gap i think it's a pretty big gap i think
- 8:32it's a much smaller gap
- 8:34for ai research than it is for many other parts of the economy
- 8:37you know let's say
- 8:38we were talking about you know car manufacturing and you're
- 8:42trying to you know get
- 8:42an ai to do everything that happens there man it's such a messy
- 8:46process there's a
- 8:46million different parts of the supply chain there's all this
- 8:49tacit knowledge and
- 8:50all the human workers minds like it's going to be really tough
- 8:52there's going to be
- 8:53SPEAKER_00: a very big gap
- 8:54going from those benchmarks to actually fully automating the
- 8:58supply chain for cars.
- 8:59For automating what OpenAI does,
- 9:01there's still a gap,
- 9:02but it's much smaller.
- 9:03SPEAKER_00: Because firstly, all of the work is virtual.
- 9:06SPEAKER_00: Everyone at OpenAI could, in principle, work remotely.
- 9:10SPEAKER_00: They're top research scientists.
- 9:12SPEAKER_00: They're just on a computer all day.
- 9:13SPEAKER_00: They're not picking up bricks and
- 9:15SPEAKER_00: you know, doing stuff like that.
- 9:17SPEAKER_00: So that already means it's a lot less messy.
- 9:20You get a lot less of that kind of messy world reality stuff
- 9:22kind of slowing down adoption.
- 9:25SPEAKER_00: And also a lot of it is coding.
- 9:27And coding is kind of almost uniquely clean in that for many
- 9:31coding tasks,
- 9:32SPEAKER_00: you can define,
- 9:33SPEAKER_00: you know,
- 9:33SPEAKER_00: clearly define metrics.
- 9:35SPEAKER_00: for success.
- 9:36SPEAKER_00: And so that makes AI much better.
- 9:38SPEAKER_00: You can just have a go.
- 9:40SPEAKER_00: Did AI succeed in the test?
- 9:42SPEAKER_00: If not, try something else or do a gradient set update.
- 9:46SPEAKER_00: That said, there's still a lot of messiness here, as any coder will know.
- 9:50SPEAKER_00: When you're writing good code,
- 9:51it's not just about whether it does the function that you've
- 9:54asked it to do.
- 9:54It needs to be well designed,
- 9:55needs to be modular,
- 9:56needs to be maintainable.
- 9:58SPEAKER_00: These things are much harder to
- 10:00SPEAKER_00: evaluate.
- 10:01And so AI is often part of our benchmarks because they can do
- 10:05the function that you
- 10:06SPEAKER_00: asked it to do,
- 10:06SPEAKER_00: the code runs,
- 10:08SPEAKER_00: but they kind of write really spaghetti code,
- 10:11SPEAKER_00: code that no one wants to look at,
- 10:12SPEAKER_00: that no one can understand.
- 10:14SPEAKER_00: And so no company would want to use that.
- 10:15And so there's still going to be a pretty big benchmark to
- 10:18reality gap,
- 10:19even for OpenAI.
- 10:20And I think that's one of the big uncertainties in terms of
- 10:22will this happen in
- 10:23SPEAKER_00: three years versus will this happen in 10 years or even 15 years?
- 10:28SPEAKER_01: Since you brought up the timeline, what's your guess?
- 10:30I didn't know whether to open with that question or conclude
- 10:33with that question.
- 10:33SPEAKER_01: We'll stick it right in the middle of our chat.
- 10:36SPEAKER_00: Yeah, great.
- 10:38SPEAKER_00: Honestly,
- 10:39SPEAKER_00: my best guess about this does change more often than I would like it to,
- 10:42SPEAKER_00: which I think tells us,
- 10:44SPEAKER_00: look,
- 10:45SPEAKER_00: there's still a state of flux.
- 10:46SPEAKER_00: This is just really something that's very hard to know about.
- 10:49SPEAKER_00: Predicting the future is hard.
- 10:50My current best guess is it's about even odds that we're able
- 10:53to fully automate
- 10:54SPEAKER_00: OpenAI within the next 10 years.
- 10:57SPEAKER_00: So maybe that's a 50-50.
- 10:59So then what really would be the impact of that kind of AI
- 11:03research automation?
- 11:04SPEAKER_01: I mean, how would you go about quantifying that kind of acceleration?
- 11:07SPEAKER_01: I mean, what does the world look like?
- 11:10SPEAKER_00: Yeah, I mean, again, so many possibilities.
- 11:13But I think what strikes me is that there is a plausible world
- 11:17where it is just
- 11:18SPEAKER_00: way,
- 11:18SPEAKER_00: way faster than almost everyone is expecting it to be.
- 11:21SPEAKER_00: So that's the world where you fully automate open AI.
- 11:24SPEAKER_00: And then we get that feedback loop that I was talking about earlier,
- 11:26SPEAKER_00: where AIs make their algorithms way more efficient.
- 11:29SPEAKER_00: Now you've got way more of them.
- 11:30SPEAKER_00: Then they make their algorithms way more efficient.
- 11:32SPEAKER_00: Again, now they're way smarter.
- 11:33SPEAKER_00: Now they're thinking 100 times faster.
- 11:35SPEAKER_00: Feedback loop continues.
- 11:40SPEAKER_00: a billion super intelligent AIs running on this OpenAI data center.
- 11:45The combined cognitive abilities of all these AIs outstrips the
- 11:50whole of the United
- 11:50SPEAKER_00: States,
- 11:51SPEAKER_00: outstrips anything we've seen from any kind of company or entity before,
- 11:56and they can all potentially be put towards any goal that
- 12:00OpenAI wants to.
- 12:01SPEAKER_00: And there's,
- 12:02SPEAKER_00: of course,
- 12:02SPEAKER_00: the risk that OpenEye's lost control of these systems often discussed,
- 12:05in which case these systems could all be working together to
- 12:07pursue a particular
- 12:09SPEAKER_00: goal.
- 12:09SPEAKER_00: And so what we're talking about here is really a huge amount of power.
- 12:15It's a threat to national security for any government in which
- 12:18this happens,
- 12:19potentially.
- 12:19It is a threat to everyone if we lose control of these systems
- 12:24or if the company
- 12:25SPEAKER_00: that develops them uses them for some kind of malicious ends.
- 12:30SPEAKER_00: And,
- 12:30SPEAKER_00: you know,
- 12:30SPEAKER_00: in terms of economic impacts,
- 12:32SPEAKER_00: I,
- 12:32SPEAKER_00: you know,
- 12:33SPEAKER_00: I personally think that that that,
- 12:35SPEAKER_00: again,
- 12:35SPEAKER_00: could happen much more quickly than than people think.
- 12:38SPEAKER_00: And we can get into that.
- 12:40SPEAKER_01: The first paper we mentioned,
- 12:41SPEAKER_01: I mean,
- 12:41SPEAKER_01: it was kind of a thought experiment,
- 12:43but we were you're really talking about moving the decimal
- 12:46point in GDP growth
- 12:47SPEAKER_01: instead of talking about two and three percent,
- 12:50SPEAKER_01: 20 and 30 percent.
- 12:51SPEAKER_01: Is that the kind of world we're talking about?
- 12:53Yeah,
- 12:54and so I speak to economists a lot and- They hate those kind of
- 12:57predictions.
- 12:57SPEAKER_00: Obviously, they think I'm crazy.
- 13:01SPEAKER_00: Not all of them.
- 13:02SPEAKER_00: There are economists that take it very seriously.
- 13:05I think there's kind of a bit of a,
- 13:06it's taken more seriously than everyone else realizes.
- 13:09It's a bit embarrassing at the moment to admit that you take it
- 13:12seriously.
- 13:12But there are a few really senior economists who absolutely
- 13:15know their stuff.
- 13:16SPEAKER_00: They're like, yep, this checks out.
- 13:17SPEAKER_00: I think that's what's going to happen.
- 13:18And I've had conversation with them where they're like,
- 13:20yeah,
- 13:20I think this is going to happen.
- 13:22SPEAKER_00: But,
- 13:22SPEAKER_00: you know,
- 13:22SPEAKER_00: the really loud dominant view where,
- 13:24SPEAKER_00: you know,
- 13:24SPEAKER_00: I think people are a little bit scared to speak out against is that,
- 13:27SPEAKER_00: like,
- 13:27SPEAKER_00: obviously this is sci-fi.
- 13:28SPEAKER_00: And,
- 13:29SPEAKER_00: you know,
- 13:30SPEAKER_00: one analogy I like to give to people who are very,
- 13:32SPEAKER_00: very confident that this is all sci-fi and it's rubbish is to,
- 13:35SPEAKER_00: like,
- 13:35SPEAKER_00: imagine that we were sitting there in the year 1400s
- 13:38SPEAKER_00: Imagine we had an economics profession,
- 13:40SPEAKER_00: been studying the rate of economic growth,
- 13:41SPEAKER_00: and they've been like,
- 13:42SPEAKER_00: yeah,
- 13:42SPEAKER_00: we've always had 0.1% growth every single year throughout history.
- 13:46SPEAKER_00: We've never seen anything higher.
- 13:48SPEAKER_00: And then there was some kind of futurist economist rogue that said,
- 13:52SPEAKER_00: actually,
- 13:52I think that if I extrapolate the curves in this way and we get
- 13:56this kind of
- 13:56SPEAKER_00: technology,
- 13:57SPEAKER_00: maybe we could have 1% growth.
- 13:58And then all the other economists laugh at them,
- 14:01tell them they're insane.
- 14:02SPEAKER_00: That's what happened.
- 14:04in 1400 we never had growth that was at all fast and then you
- 14:07know a few hundred
- 14:08years later we developed industrial technology you know we we
- 14:11kind of um you know
- 14:12start started that fee value we're investing more and more
- 14:14resources in scientific
- 14:16progress um and then kind of you know um physical capital and
- 14:19we did see much
- 14:20SPEAKER_00: faster growth
- 14:21SPEAKER_00: And so,
- 14:21SPEAKER_00: you know,
- 14:21SPEAKER_00: I think it can be useful to try and challenge economists and say,
- 14:24SPEAKER_00: okay,
- 14:24SPEAKER_00: I know it sounds crazy,
- 14:25SPEAKER_00: but history was crazy.
- 14:27That we just,
- 14:27this crazy thing happened where growth just got way,
- 14:29way,
- 14:30way faster.
- 14:30SPEAKER_00: No one would have predicted it.
- 14:31SPEAKER_00: You would not have predicted it.
- 14:33And I think being that mindset can encourage people to be like,
- 14:37yeah,
- 14:37okay,
- 14:38you know what?
- 14:39SPEAKER_00: Maybe if we do get AI that's really that powerful,
- 14:42SPEAKER_00: it can really do everything,
- 14:43SPEAKER_00: and maybe it is possible.
- 14:44But to answer your question,
- 14:46yeah,
- 14:46I'm talking about 30% growth every year.
- 14:49SPEAKER_00: I think it gets faster than that.
- 14:52If you want to know how fast it eventually gets,
- 14:54you can think about the question of,
- 14:57SPEAKER_00: How fast can a kind of a self-replicating system double itself?
- 15:02SPEAKER_00: So,
- 15:02SPEAKER_00: you know,
- 15:03ultimately what the economy is going to be like ultimately is
- 15:05it's going to have
- 15:06robots and factories that are able to kind of fully create new
- 15:10versions of
- 15:11SPEAKER_00: themselves.
- 15:11SPEAKER_00: You know,
- 15:11SPEAKER_00: everything you need,
- 15:12SPEAKER_00: the roads,
- 15:13SPEAKER_00: the electricity,
- 15:14SPEAKER_00: you know,
- 15:15SPEAKER_00: the robots,
- 15:15SPEAKER_00: the buildings,
- 15:16SPEAKER_00: all of that will be replicated.
- 15:17SPEAKER_00: And so you can,
- 15:18SPEAKER_00: you know,
- 15:18SPEAKER_00: look at actually biology and say,
- 15:20SPEAKER_00: do we have any examples of systems which fully replicate themselves?
- 15:24SPEAKER_00: How long does it take?
- 15:26SPEAKER_00: And,
- 15:26SPEAKER_00: you know,
- 15:26SPEAKER_00: if you look at rats,
- 15:27SPEAKER_00: for example,
- 15:28they're able to kind of double the number of rats by kind of
- 15:32grabbing resources
- 15:33SPEAKER_00: from the environment and,
- 15:34SPEAKER_00: you know,
- 15:34SPEAKER_00: giving birth and whatnot.
- 15:35SPEAKER_00: The doubling time is, you know, about six weeks for some types of rats.
- 15:39SPEAKER_00: So that's an example of here's a physical system,
- 15:41SPEAKER_00: you know,
- 15:42SPEAKER_00: ultimately everything's made of physics,
- 15:43a physical system that has some intelligence that's able to go
- 15:46out into the world,
- 15:47SPEAKER_00: gather resources,
- 15:47SPEAKER_00: replicate itself.
- 15:49SPEAKER_00: The doubling time is six weeks.
- 15:51SPEAKER_00: Now, who knows how long it will take us to get to AI that's that good.
- 15:55SPEAKER_00: But when we do,
- 15:56SPEAKER_00: you could see the whole physical economy,
- 15:58SPEAKER_00: you know,
- 15:58SPEAKER_00: maybe a part that humans aren't involved with,
- 16:00SPEAKER_00: you know,
- 16:00a whole like kind of automated city without any humans just
- 16:03doubling itself every
- 16:05SPEAKER_00: few weeks.
- 16:06SPEAKER_00: And so,
- 16:07SPEAKER_00: you know,
- 16:07if that happens and the amount of stuff we're able to produce
- 16:09as a civilization is
- 16:10SPEAKER_00: doubling again,
- 16:11SPEAKER_00: you know,
- 16:12SPEAKER_00: on the order of weeks.
- 16:13SPEAKER_00: And in fact, there are some animals that double faster still in days.
- 16:16SPEAKER_00: But,
- 16:17SPEAKER_00: you know,
- 16:17SPEAKER_00: that's,
- 16:18SPEAKER_00: you know,
- 16:18that's the kind of level of craziness that we're talking about
- 16:20thousand percent
- 16:21SPEAKER_00: growth at that point.
- 16:22SPEAKER_00: So,
- 16:22SPEAKER_00: you know,
- 16:23SPEAKER_00: I think we don't know how crazy it could get,
- 16:25SPEAKER_00: but,
- 16:25SPEAKER_00: you know,
- 16:25SPEAKER_00: I think we should take even the really crazy possibilities.
- 16:28SPEAKER_00: We shouldn't fully rule them out.
- 16:30SPEAKER_01: there's this great AI forecast chart put out by the Dallas Fed.
- 16:35SPEAKER_01: And I think its main forecast,
- 16:38SPEAKER_01: the one most economists would probably agree with,
- 16:41has a line showing AI improving GDP by maybe like two-tenths of
- 16:46a percent.
- 16:47SPEAKER_01: And then there are two other lines.
- 16:49SPEAKER_01: One is more or less straight up, and the other one is straight down.
- 16:53SPEAKER_01: Because in the first...
- 16:55SPEAKER_01: AI created like a utopia.
- 16:57And in the second one,
- 16:58AI gets out of control and,
- 16:59you know,
- 16:59starts killing us and whatever.
- 17:02SPEAKER_01: So those are your three possibilities.
- 17:04SPEAKER_01: If we stick with the optimistic case for a moment,
- 17:07SPEAKER_01: what constraints do you see as most plausible?
- 17:11SPEAKER_01: Reduced labor supply from rising income,
- 17:13SPEAKER_01: social pushback against disruption,
- 17:16SPEAKER_01: energy limits or something else?
- 17:18SPEAKER_00: Yeah.
- 17:19SPEAKER_00: I mean,
- 17:19SPEAKER_00: briefly,
- 17:20SPEAKER_00: the ones you've mentioned,
- 17:20SPEAKER_00: you know,
- 17:21SPEAKER_00: people not working 100% people,
- 17:23SPEAKER_00: I really hope people work less.
- 17:25SPEAKER_00: You know,
- 17:25SPEAKER_00: if we get this good future and,
- 17:27SPEAKER_00: you know,
- 17:27SPEAKER_00: the benefits are shared between all,
- 17:29SPEAKER_00: which isn't guaranteed,
- 17:31SPEAKER_00: if we get that,
- 17:32SPEAKER_00: then yeah,
- 17:32SPEAKER_00: no one should work.
- 17:33SPEAKER_00: But that doesn't stop growth because,
- 17:34SPEAKER_00: you know,
- 17:36SPEAKER_00: when AIs and robots can do everything that humans do,
- 17:39SPEAKER_00: you know,
- 17:40SPEAKER_00: you don't need humans in the loop anymore.
- 17:41SPEAKER_00: You know, that whole thing is just going and
- 17:44um and kind of self-replicating itself and making as many goods
- 17:46and services as we
- 17:47want like sure if you want your if you want your um you want
- 17:50your clothes to be to
- 17:51be to be knitted by a human you're in trouble then then then
- 17:55your consumption is
- 17:55stuck bad luck if you're happy to consume goods and services
- 17:59produced by ai systems
- 18:00or robots fine if no one wants to work um pushback i think i
- 18:05think for me this is
- 18:06SPEAKER_00: the biggest one you know
- 18:08obviously the economy doubling every year is very scary as a
- 18:13thought you know tech
- 18:15progress will be going much faster imagine if you know you woke
- 18:18up and over the
- 18:18course of the year you go from you know not having any
- 18:21telephones at all in the
- 18:23world to you know everyone's on their smartphones and social
- 18:25media and all the apps
- 18:26you know that's a that's a transition that took decades if that
- 18:29happened in a year
- 18:30SPEAKER_00: that would be very you know disconcerting um
- 18:35SPEAKER_00: Another example is the development of nuclear weapons.
- 18:38SPEAKER_00: Nuclear weapons were developed over a number of years.
- 18:42SPEAKER_00: If that happened in a month or two months, that could be very dangerous.
- 18:46SPEAKER_00: There'd be much less time for different countries,
- 18:48SPEAKER_00: different actors to figure out how they're going to handle it.
- 18:52So I think pushback is the strongest one that we might as a
- 18:55society choose.
- 18:56SPEAKER_00: Actually, this is this is insane.
- 18:57SPEAKER_00: We're going to go slower than we could.
- 19:00SPEAKER_00: That requires potentially coordination.
- 19:02But I think there would be broad support for some degree of
- 19:06coordination there.
- 19:07SPEAKER_01: I mean,
- 19:07I imagine you certainly probably talk with people who are
- 19:10extremely gung-ho about
- 19:13SPEAKER_01: this prospect.
- 19:14What is like the common response you get from people who are
- 19:17less enthusiastic?
- 19:18SPEAKER_01: Do they worry about a future with no jobs?
- 19:20SPEAKER_01: Maybe they do worry about the existential kinds of issues.
- 19:24SPEAKER_01: What's your response to those people?
- 19:25SPEAKER_01: And how much do you worry about those things?
- 19:27I think there are loads of very worrying things that we're
- 19:29going to be facing.
- 19:30So,
- 19:30you know,
- 19:31one class of pushback,
- 19:32which I think is very common,
- 19:33is worries about employment.
- 19:34SPEAKER_00: You know,
- 19:35It's a source of income for all of us,
- 19:37our employment,
- 19:38but also it's a source of pride.
- 19:40SPEAKER_00: It's a source of meaning.
- 19:41If we just suddenly no one has any jobs,
- 19:44you know what we want to do with ourselves.
- 19:45That's a very,
- 19:47very consequential transition for the nature of human society,
- 19:51which.
- 19:51SPEAKER_00: Yeah, I think people aren't just going to be down to just do it.
- 19:54I think people are scared about three AI companies literally
- 20:02now taking all the
- 20:03SPEAKER_00: revenues that all of humanity used to be earning.
- 20:06SPEAKER_00: It is naturally a very scary prospect.
- 20:09SPEAKER_00: So that's one kind of pushback.
- 20:10SPEAKER_00: And I'm sympathetic with it.
- 20:12SPEAKER_00: I think that...
- 20:14SPEAKER_00: There are solutions.
- 20:15SPEAKER_00: You know,
- 20:15SPEAKER_00: if we find a way to tax AI systems,
- 20:18which isn't necessarily easy because it's very easy to move
- 20:20physical assets between
- 20:22SPEAKER_00: countries.
- 20:22SPEAKER_00: So,
- 20:23SPEAKER_00: you know,
- 20:23SPEAKER_00: it's a lot easier to tax labor than capital already when,
- 20:27SPEAKER_00: you know,
- 20:27SPEAKER_00: rich people can move their assets around.
- 20:28I'm just going to,
- 20:29you know,
- 20:29we're going to have the same problem with AI.
- 20:32SPEAKER_00: But if we can find a way to tax it,
- 20:34then we maintain a good democratic country and we can just
- 20:39redistribute wealth
- 20:39SPEAKER_00: broadly,
- 20:40SPEAKER_00: it can be solved.
- 20:41SPEAKER_00: So that I think it's a big problem.
- 20:43SPEAKER_00: But I'm like,
- 20:44SPEAKER_00: It's doable.
- 20:46SPEAKER_00: Then there's the problem of,
- 20:47SPEAKER_00: you know,
- 20:47SPEAKER_00: some people want to stop this,
- 20:49SPEAKER_00: you know,
- 20:49SPEAKER_00: now because they're worried about AI killing everyone.
- 20:52SPEAKER_00: You know,
- 20:53their literal worry is that everyone will be dead because super
- 20:56intelligent AI will
- 20:57SPEAKER_00: want that to happen.
- 20:59i i think i think there's a real risk there it's you know it's
- 21:02definitely above one
- 21:03percent in my opinion um i wouldn't go above ten percent myself
- 21:06um but i i think
- 21:08it's it's very scary um and that's a great reason to to slow
- 21:13things down i
- 21:13personally don't want to stop quite yet i think you want to
- 21:16stop when the ai is a
- 21:18bit more powerful and a bit more useful than it is today so it
- 21:20can kind of help us
- 21:21figure out what to do about all this crazy stuff that's coming
- 21:23on what side of that
- 21:24SPEAKER_01: line is ai as an ai researcher
- 21:29SPEAKER_00: That's a really great question.
- 21:30SPEAKER_00: Yeah.
- 21:30SPEAKER_00: Should we stop?
- 21:31I think it's very hard to stop just after you've got the AI
- 21:33researcher AI,
- 21:34SPEAKER_00: because that's when you could suddenly really easy to go very,
- 21:37SPEAKER_00: very fast.
- 21:39SPEAKER_00: So,
- 21:40SPEAKER_00: you know,
- 21:41SPEAKER_00: my out of the box proposal here,
- 21:43SPEAKER_00: which is probably very flawed would be when we're within,
- 21:46SPEAKER_00: you know,
- 21:46SPEAKER_00: a few spits distance,
- 21:48SPEAKER_00: not spitting distance,
- 21:49SPEAKER_00: but like,
- 21:49SPEAKER_00: you know,
- 21:49SPEAKER_00: if you did that three times,
- 21:51SPEAKER_00: And we can see we're almost at that AI automating open AI.
- 21:55Then you pause because you're not going to accidentally then go
- 21:59all the way because
- 22:00SPEAKER_00: it's actually still a little bit,
- 22:01SPEAKER_00: it's a fair distance away.
- 22:03But it's actually still at that point,
- 22:05it's probably a very powerful AI that can really help.
- 22:07SPEAKER_00: And then you pause and do what?
- 22:09SPEAKER_00: Great question.
- 22:10So then you pause and you use your AI systems to help you
- 22:15firstly solve the problem
- 22:16SPEAKER_00: of AI alignment.
- 22:18Make extra double sure that every time we increase the notch of
- 22:20AI capabilities,
- 22:21SPEAKER_00: the AI is still loyal to humanity,
- 22:23SPEAKER_00: not to its own kind of secret goals.
- 22:26um secondly you you solve the problem of okay how are we going
- 22:29to make sure that no
- 22:31one person in government or no one ceo of an ai company ensures
- 22:35that this whole ai
- 22:36army is loyal to them personally how are we going to ensure
- 22:38that everyone the whole
- 22:40world kind of gets gets influence over what this ai is
- 22:43ultimately programmed to do
- 22:44SPEAKER_00: um that's the second problem um and then
- 22:48SPEAKER_00: You know,
- 22:50SPEAKER_00: there's just a whole host of other things,
- 22:52SPEAKER_00: unemployment that we've talked about,
- 22:54SPEAKER_00: competition between different countries,
- 22:56SPEAKER_00: US and China race.
- 22:57There's a whole host of other things that I think you want to
- 23:01kind of research on,
- 23:02SPEAKER_00: figure out,
- 23:02SPEAKER_00: get consensus on,
- 23:03and then slowly ratchet up the capabilities in what is now a
- 23:06very safe and
- 23:06SPEAKER_00: controlled way.
- 23:08SPEAKER_01: What else should we be working on?
- 23:09SPEAKER_01: Like, what are you working on next?
- 23:12One problem I'm excited about is people have historically
- 23:17worried about AI having
- 23:18SPEAKER_00: its own goals.
- 23:20SPEAKER_00: We need to make it loyal to humanity.
- 23:22SPEAKER_00: But as we've got closer, it's become increasingly obvious.
- 23:25SPEAKER_00: Loyal to humanity is very vague.
- 23:27SPEAKER_00: What specifically do you want the AI to be programmed to do?
- 23:30SPEAKER_00: I mean, it's not programmed.
- 23:31SPEAKER_00: It's grown.
- 23:32SPEAKER_00: But if it were programmed, if you're writing a rule book
- 23:37For AI,
- 23:39you might get,
- 23:39you know,
- 23:40some organizations have employee handbooks.
- 23:42SPEAKER_00: Here's the philosophy of the organization.
- 23:43SPEAKER_00: Here's how you should behave.
- 23:45SPEAKER_00: Imagine you're doing that for the AI,
- 23:46SPEAKER_00: but you're going,
- 23:46SPEAKER_00: you know,
- 23:47SPEAKER_00: super detailed,
- 23:47SPEAKER_00: you know,
- 23:48SPEAKER_00: exactly how you want your kind of AI assistant to behave in all kinds of
- 23:52SPEAKER_00: situations.
- 23:53SPEAKER_00: You know, what should that be?
- 23:55SPEAKER_00: Essentially, what should we align the AI to?
- 23:59Not any individual person,
- 24:00probably following the law,
- 24:02probably loads of other things.
- 24:03I think basically designing what is the character of this AI
- 24:06system is a really
- 24:09SPEAKER_00: exciting question.
- 24:10And if we get that right,
- 24:11maybe the AI can then help us solve all these other problems.
- 24:14SPEAKER_01: Maybe you have no interest in science fiction,
- 24:17SPEAKER_01: but is there any film,
- 24:19SPEAKER_01: TV,
- 24:19SPEAKER_01: book that you think is interesting
- 24:22is useful for someone in your position to be aware of or that
- 24:26you find useful in any way?
- 24:28SPEAKER_00: I think there's this great post called AI 2027,
- 24:32which lays out a concrete scenario for how AI could go wrong or
- 24:35how maybe it could
- 24:36SPEAKER_00: go right.
- 24:37SPEAKER_00: Yeah, I would recommend that.
- 24:39SPEAKER_00: I think that's the only thing that's coming top of mind.
- 24:41SPEAKER_00: I read a lot of less wrong, to be honest.
- 24:43SPEAKER_00: So that's a lot of stuff from there that I don't love,
- 24:46SPEAKER_00: but a lot of kind of new ideas,
- 24:48SPEAKER_00: interesting content there.
- 24:49SPEAKER_00: Any fiction?
- 24:50SPEAKER_00: I mean,
- 24:51SPEAKER_00: I read fiction,
- 24:51SPEAKER_00: but honestly,
- 24:52I don't really love the AI fiction that I've read because often
- 24:55it's kind of quite
- 24:56SPEAKER_00: unrealistic.
- 24:58SPEAKER_00: And so I kind of, you know, get a bit overly nitpicky about it.
- 25:02SPEAKER_00: But I mean,
- 25:02SPEAKER_00: there's this book called Harry Potter and the Methods of Rationality,
- 25:06SPEAKER_00: which I read maybe 10 years ago,
- 25:07SPEAKER_00: which I thought was pretty fun.