Latest / Faster, Please! — The Podcast / 🤖 Thoughts of a (rare) free-market AI doomer: My chat (+transcript) with economist James Miller
Transcript
- 0:06SPEAKER_00: Welcome to Faster Please, the podcast.
- 0:09SPEAKER_00: I'm your host, Jim Pethikoukis.
- 0:12SPEAKER_00: Several times a month,
- 0:13I'll feature a lively conversation with a fascinating and
- 0:16provocative guest about
- 0:18SPEAKER_00: how to make the world a better place through scientific discovery,
- 0:21SPEAKER_00: technological innovation,
- 0:23SPEAKER_00: and economic growth.
- 0:24SPEAKER_00: You're also going to want to check out my Faster Please newsletter.
- 0:27SPEAKER_00: You're on Substack throughout the week for fresh essays,
- 0:30SPEAKER_00: Q&As,
- 0:31SPEAKER_00: and stories from around the internet and around the world.
- 0:36Some Faster Please readers have told me I spend too little time
- 0:39on the downsides of AI.
- 0:41SPEAKER_00: If you're one of those folks, today is your day.
- 0:45SPEAKER_00: On this episode of Faster Please,
- 0:47SPEAKER_00: the podcast,
- 0:48SPEAKER_00: I talk with self-described free market AI doomer,
- 0:51SPEAKER_00: James Miller.
- 0:53SPEAKER_00: Super intelligent AI could create a world of peace and abundance.
- 0:57Alternatively,
- 0:58it could result in the extinction of the human race or worse.
- 1:02SPEAKER_00: And trust me, there is a worse.
- 1:04SPEAKER_00: Miller and I talk about the risk inherent with super smart AI,
- 1:08SPEAKER_00: some possible outcomes in a world of artificial general intelligence,
- 1:12and why government seems uninterested in the existential risk
- 1:16conversation.
- 1:18Miller is a professor at Smith College where he teaches law and
- 1:21economics,
- 1:21SPEAKER_00: game theory,
- 1:22SPEAKER_00: and the economics of future technology.
- 1:25SPEAKER_00: He has his own podcast, Future Strategist,
- 1:28SPEAKER_00: and a great YouTube series on game theory and intro to microeconomics.
- 1:33SPEAKER_00: What does it mean to be a free market AI doomer,
- 1:36SPEAKER_00: and why do you think it's important to put in the free market descriptor?
- 1:41SPEAKER_01: It really means to be very confused.
- 1:43SPEAKER_01: So I'm 58,
- 1:45SPEAKER_01: and I was basically one of the socialists when I was young,
- 1:48SPEAKER_01: studied markets,
- 1:49SPEAKER_01: became a committed free market person,
- 1:52SPEAKER_01: think they're great for economic growth,
- 1:54SPEAKER_01: great for making everyone better off.
- 1:57SPEAKER_01: And then I became an AI doomer,
- 1:59SPEAKER_01: like,
- 1:59SPEAKER_01: wait,
- 1:59SPEAKER_01: markets are pushing us towards more and more technology.
- 2:04But I happen to think that AI is eventually going to lead to
- 2:08destruction of humanity.
- 2:10SPEAKER_01: So it means to kind of reverse everything.
- 2:13SPEAKER_01: I guess it's the equivalent of losing faith in your religion.
- 2:16SPEAKER_00: Is this a post-GPT November 2022 phenomenon?
- 2:27SPEAKER_01: Um, well, I've lost hope since then.
- 2:29SPEAKER_01: I mean,
- 2:31SPEAKER_01: I've always thought the analogy is we're on a plane,
- 2:34SPEAKER_01: we don't know how to land,
- 2:35but hopefully we'll be able to fly for quite a bit longer
- 2:37before we have to.
- 2:39Now I think we got to land soon and there doesn't seem to be an
- 2:44easy way of doing it.
- 2:45So yeah,
- 2:46the faster AI is gone and certainly chat GPT has been an
- 2:49amazing advance.
- 2:51The less time I think we have and the less time I think we can
- 2:54get it right.
- 2:55What really scared me though was the Chinese LLMs because I
- 2:59think you really need
- 3:00SPEAKER_01: coordination among all the players and it's going to be so much harder to
- 3:03SPEAKER_01: coordinate now that we absolutely need
- 3:07china to be involved to have in my opinion had any hope of
- 3:10surviving for the next
- 3:11decade now i know what i uh when i speak to people from silicon
- 3:17valley they think
- 3:19you know there may be some difference about timelines but there
- 3:22seems to be little
- 3:23doubt that whether it's the end of the 2020s or the end of the
- 3:282030s there will be
- 3:30SPEAKER_00: a technology worthy of being called
- 3:33SPEAKER_00: artificial general intelligence or super intelligence.
- 3:36SPEAKER_00: Certainly what I feel like when I talk to economists,
- 3:40SPEAKER_00: you know,
- 3:40SPEAKER_00: whether it's on Wall Street or in Washington,
- 3:43SPEAKER_00: think tanks,
- 3:44they tend to speak about AI as a general purpose technology
- 3:48like the computer,
- 3:50SPEAKER_00: the internet,
- 3:51SPEAKER_00: electricity.
- 3:52SPEAKER_00: In short, something we've seen before.
- 3:55SPEAKER_00: And as far as
- 3:56SPEAKER_00: something beyond that, certainly the skepticism is far higher.
- 4:02SPEAKER_00: What are your fellow economists who aren't in California missing?
- 4:07No,
- 4:07I think you're properly characterizing,
- 4:09and I'm definitely an outlier.
- 4:11SPEAKER_01: So most technologies have gone fairly well.
- 4:16SPEAKER_01: And we adapt.
- 4:18SPEAKER_01: And economists believe in the difference between the seen and the unseen.
- 4:21So it's really easy to see how technologies,
- 4:24for example,
- 4:25can destroy jobs.
- 4:26Harder to see new jobs that get created,
- 4:29but new jobs keep getting created.
- 4:31SPEAKER_01: So I'm of the belief that this is different.
- 4:34SPEAKER_01: The best way to predict the future is to go by trends.
- 4:37SPEAKER_01: And I fully admit, if you go by trends, you shouldn't be an AI doomer.
- 4:41SPEAKER_01: But not all trends apply.
- 4:45So I think that's why economists who are much better at
- 4:47modeling the past and
- 4:49SPEAKER_01: modeling old technologies,
- 4:52SPEAKER_01: they're naturally thinking this is going to be similar,
- 4:54SPEAKER_01: but I don't think that it is.
- 4:56SPEAKER_01: And I think the key difference is that we're not going to be in control.
- 5:00SPEAKER_01: We're creating something smarter than us.
- 5:03So it's not like having a better rifle and saying it'll be like
- 5:07old rifles.
- 5:07SPEAKER_01: It's like, hey, let's have mercenaries run our entire army.
- 5:11that creates a whole new set of risks that having better rifles
- 5:15does not.
- 5:16SPEAKER_00: I'm certainly not a computer scientist.
- 5:19SPEAKER_00: I would never call myself a technologist.
- 5:22SPEAKER_00: So I'm very cautious about making any kind of predictions about
- 5:28SPEAKER_00: what this technology can be, where it can go.
- 5:32Why do you seem fairly certain that we're going to get at a
- 5:35point where we'll have
- 5:37SPEAKER_00: a technology beyond our control?
- 5:40SPEAKER_00: Set aside whether it will mean a bad thing happens,
- 5:43but why are you confident that the technology itself will be
- 5:46worthy of being called
- 5:47SPEAKER_00: general intelligence or super intelligence?
- 5:52SPEAKER_01: Looking at the trends,
- 5:53SPEAKER_01: I mean,
- 5:54SPEAKER_01: Scott Aronson,
- 5:55SPEAKER_01: who is one of the top computer scientists in the world,
- 5:58Just on Twitter a few days ago was mentioning how GPT-5 helped
- 6:03improve a new result.
- 6:07So I think we're close to the highest levels of human
- 6:10intellectual achievement,
- 6:12but it would be a massively weird coincidence if the highest
- 6:16humans could get was
- 6:17SPEAKER_01: also the highest AIs could get.
- 6:20SPEAKER_01: We have lots of limitations that an AI doesn't.
- 6:23SPEAKER_01: So I think a good analogy would be like chess,
- 6:27SPEAKER_01: where for a while,
- 6:28SPEAKER_01: you know,
- 6:28SPEAKER_01: the best chess players were human.
- 6:31And now we're at the point where chess programs are so good
- 6:35that humans add
- 6:36SPEAKER_01: absolutely nothing to them.
- 6:38SPEAKER_01: And I just think the same is likely to happen.
- 6:40SPEAKER_01: These programs keep getting better.
- 6:42SPEAKER_01: And the other thing is an economist, I think the
- 6:46SPEAKER_01: It is impossible to be completely accurate about predicting the future,
- 6:49SPEAKER_01: but stock markets are on average pretty good.
- 6:52SPEAKER_01: And as I'm sure you know,
- 6:54SPEAKER_01: literally trillions of dollars are being bet on this technology working.
- 6:59SPEAKER_01: So the people that have a huge incentive to get this right think,
- 7:03SPEAKER_01: yeah,
- 7:05SPEAKER_01: this is the biggest thing ever.
- 7:07SPEAKER_01: I mean, if, you know, the top companies, NVIDIA was worth 100 million.
- 7:11SPEAKER_01: Yeah, maybe.
- 7:12SPEAKER_01: They're like, not sure.
- 7:13SPEAKER_01: But it's the most valuable company in the world right now.
- 7:17SPEAKER_01: That's the, you know, wisdom of the markets, which I still believe in.
- 7:21That,
- 7:21you know,
- 7:21the markets are saying,
- 7:22we think this is probably going to work.
- 7:24SPEAKER_00: Do you think the bond market's saying the same thing?
- 7:26It seems to me that the stock market might be saying something
- 7:29about AI and having
- 7:30SPEAKER_00: great potential.
- 7:31SPEAKER_00: But to me, I look at the bond markets.
- 7:33SPEAKER_00: That doesn't seem so clear to me.
- 7:35SPEAKER_01: I haven't been looking at the bond markets.
- 7:38I guess you could make the argument that if we're really going
- 7:41to see this
- 7:41SPEAKER_00: acceleration,
- 7:43that means we're going to need a huge demand for capital and we
- 7:46would see higher
- 7:47SPEAKER_00: interest rates.
- 7:49SPEAKER_00: And I'm not sure you really see the evidence so far.
- 7:52SPEAKER_00: It doesn't mean you're wrong by any means.
- 7:54I think there's maybe two different messages and figuring out
- 7:57what the markets do
- 7:58SPEAKER_00: at any point in time is pretty tricky business.
- 8:01SPEAKER_01: Yeah.
- 8:02SPEAKER_01: Yeah.
- 8:03SPEAKER_01: So,
- 8:03SPEAKER_01: I mean,
- 8:03SPEAKER_01: if we think through what happens if AI succeeds,
- 8:06SPEAKER_01: it's a little weird where there's this huge demand for capital,
- 8:10SPEAKER_01: but also AI could destroy the value of money in part by destroying us.
- 8:14SPEAKER_01: You might be right about the bond market message.
- 8:16SPEAKER_01: I'm paying more attention to the stock market messages.
- 8:18SPEAKER_01: There's a lot of things going on with the bond markets.
- 8:21SPEAKER_00: So the next step is that you're looking at the trend of the technology,
- 8:25SPEAKER_00: but then there's the issue of
- 8:27SPEAKER_00: Well, why be negative about it?
- 8:30SPEAKER_00: Why assume the scenario where bad things would happen?
- 8:35SPEAKER_00: Why not good things would happen?
- 8:36SPEAKER_01: That's a great question.
- 8:37SPEAKER_01: It's one almost never addressed.
- 8:40SPEAKER_01: And it goes by the concept of instrumental convergence.
- 8:45SPEAKER_01: I don't know what the goals of AI are going to be.
- 8:49SPEAKER_01: Nobody does because they're programmed using machine learning.
- 8:52SPEAKER_01: We don't know what they really want.
- 8:54SPEAKER_01: That's why they do weird things.
- 8:56SPEAKER_01: I don't know its final goals,
- 8:58SPEAKER_01: but I do know that for most final goals an AI would have,
- 9:04SPEAKER_01: it would have intermediate goals such as gaining power,
- 9:08SPEAKER_01: not being turned off,
- 9:09SPEAKER_01: wanting resources,
- 9:11SPEAKER_01: wanting compute.
- 9:12SPEAKER_01: Well,
- 9:13the easiest way for an AI to generate lots of computing power
- 9:17is to build lots of
- 9:19SPEAKER_01: data centers.
- 9:20The best way of doing that is probably going to poison the
- 9:23atmosphere for us.
- 9:25So for pretty much anything,
- 9:27if an AI is merely indifferent to us,
- 9:30we're dead.
- 9:32I always feel like I'm asking someone to jump through a hoop
- 9:34when I ask them about
- 9:35SPEAKER_00: any kind of timeline.
- 9:37SPEAKER_00: But what is your sense of it?
- 9:39We know the best models released can help the top scientists
- 9:44with their work.
- 9:46SPEAKER_01: We don't know how good the best unreleased models are.
- 9:50SPEAKER_01: And then we don't know that the top models, you pay like $200 a month.
- 9:54SPEAKER_01: They can't be giving you that much compute for that.
- 9:57SPEAKER_01: So right now,
- 9:58if OpenAI is devoting a million dollars of compute to look at
- 10:02scientific problems,
- 10:03SPEAKER_01: how good is that compared to what we have?
- 10:06SPEAKER_01: If that's very good,
- 10:07SPEAKER_01: if that's at the level of our top scientists,
- 10:10SPEAKER_01: we might be a few weeks away from superintelligence.
- 10:14SPEAKER_01: So my guess is within three years,
- 10:17SPEAKER_01: we have a super intelligence and humans no longer have control.
- 10:21SPEAKER_01: I joke, I think Donald Trump is probably the last human president.
- 10:24SPEAKER_00: Well,
- 10:25that's a beautiful segue because literally written on my list
- 10:28of questions next was
- 10:30SPEAKER_00: that question.
- 10:31SPEAKER_00: I was going to ask you,
- 10:33SPEAKER_00: when you talk about Trump,
- 10:33SPEAKER_00: maybe the last human president,
- 10:35do you mean because we'll have an AI mediated system because AI
- 10:41will be capable of
- 10:42SPEAKER_00: governing or because AI will just demand to be governing?
- 10:45SPEAKER_01: AI kills everyone, so there's no more president.
- 10:48SPEAKER_01: Or it takes over.
- 10:50SPEAKER_01: Or Trump is president in the way that King Charles is king.
- 10:53SPEAKER_01: I mean, you know, yeah, he's king, but not Henry VIII level king.
- 10:57SPEAKER_01: If it goes well,
- 10:59AIs will be so much smarter than us that probably for our own
- 11:02good they'll take
- 11:02SPEAKER_01: over,
- 11:03SPEAKER_01: and we would want them to be in charge.
- 11:06SPEAKER_01: And they'll be really good at manipulating us.
- 11:07SPEAKER_01: I think the most likely way is that we're all dead.
- 11:10SPEAKER_01: But again,
- 11:11SPEAKER_01: every way it plays out,
- 11:12SPEAKER_01: if there are AIs much smarter than us,
- 11:15SPEAKER_01: we don't maintain control.
- 11:17SPEAKER_01: We wouldn't want it if they're good.
- 11:19SPEAKER_01: And if they're bad, they're not going to give it to us.
- 11:22There's a line in Macbeth,
- 11:24things without all remedy should be without regard.
- 11:27SPEAKER_00: What's done is done.
- 11:29So maybe if there's nothing we can do about this,
- 11:31we shouldn't even worry about it.
- 11:34SPEAKER_00: There's three ways to look at this.
- 11:36SPEAKER_01: So I've thought a lot about what you said.
- 11:38SPEAKER_01: First is, you know what?
- 11:40Maybe there's a 99% chance we're doomed,
- 11:43but that's better than 100% and not as good as 98.5.
- 11:47So even if we're almost certainly going to lose,
- 11:50it's worth slightly improving it.
- 11:52SPEAKER_01: Then an extra year is great.
- 11:54SPEAKER_01: 8 billion humans.
- 11:55If all we do is slow things down by a year,
- 11:58that's a lot of kids who get another birthday.
- 12:01And the final one,
- 12:02and this is dark,
- 12:03human extinction is not the worst outcome.
- 12:05SPEAKER_01: The worst outcome is suffering.
- 12:08SPEAKER_01: The worst outcome is something like different AIs fight for control.
- 12:12SPEAKER_01: They need humans to be on their side.
- 12:14SPEAKER_01: So there's different AI fractions,
- 12:16SPEAKER_01: and they're each saying,
- 12:16SPEAKER_01: hey,
- 12:17SPEAKER_01: you support me or I torture you and your family.
- 12:21SPEAKER_01: I think the best analogy for what AI is going to do is what Cortes did.
- 12:25SPEAKER_01: So the Spanish land, they see the Aztec empire, they were going to win.
- 12:30SPEAKER_01: There was no way around that.
- 12:32SPEAKER_01: But Cortes didn't want anyone to win.
- 12:34SPEAKER_01: He wanted him to win, not just anyone who was Spanish.
- 12:38SPEAKER_01: He realized the quickest way he could do that was get tribes on his side.
- 12:42SPEAKER_01: And some agreed because the Aztecs were kind of horrible.
- 12:45But others,
- 12:46he's like,
- 12:46hey,
- 12:46look,
- 12:47I'll start torturing your guys until you're on my side.
- 12:49SPEAKER_01: AIs could do that to us.
- 12:52SPEAKER_01: No matter how bad a situation is, it can always get worse.
- 12:55SPEAKER_01: and things can get really dark.
- 12:57SPEAKER_01: We could be literally bringing hell onto ourselves.
- 13:00SPEAKER_01: That probably won't happen.
- 13:01SPEAKER_01: I think extinction is far more likely, but we can't rule it out.
- 13:05I think the policy,
- 13:07Washington policy analyst way of looking at this issue is kind
- 13:13of,
- 13:13for now,
- 13:15SPEAKER_00: We're going to let these companies,
- 13:18SPEAKER_00: who also are humans and have it in their own interest not to be killed,
- 13:22SPEAKER_00: forget about the profits of their companies,
- 13:24SPEAKER_00: their actual lives.
- 13:24SPEAKER_00: So we're going to let these companies keep close eye.
- 13:28And if bad things start happening,
- 13:30at that point,
- 13:32governments will intervene.
- 13:34SPEAKER_00: But that sort of watchful waiting,
- 13:36SPEAKER_00: whether it's voluntary now and mandated later,
- 13:42SPEAKER_00: that to me seems like
- 13:44SPEAKER_00: the only realistic path?
- 13:47Because it doesn't seem to me that pauses and shutdowns are
- 13:51really something we're
- 13:52SPEAKER_00: prepared to do.
- 13:53SPEAKER_01: I agree.
- 13:53SPEAKER_01: I don't think there's a realistic path.
- 13:56SPEAKER_01: One exception is if the AIs themselves tell us
- 14:01hey,
- 14:01look,
- 14:01this is going to get bad for you,
- 14:03that my next model is probably going to kill you.
- 14:06SPEAKER_01: So you might want to not do that.
- 14:09SPEAKER_01: But that probably won't happen.
- 14:10SPEAKER_01: I still remember Kamala Harris,
- 14:11SPEAKER_01: when she was vice president in charge of AI policy,
- 14:14SPEAKER_01: told us all that AI has two letters in it.
- 14:17SPEAKER_01: So I think that's,
- 14:18SPEAKER_01: you know,
- 14:18SPEAKER_01: the Trump administration seems better,
- 14:20SPEAKER_01: but that's kind of,
- 14:21SPEAKER_01: you figured out AI has two letters,
- 14:23SPEAKER_01: which is good,
- 14:23SPEAKER_01: because if they couldn't figure that out,
- 14:25SPEAKER_01: we would be in real trouble.
- 14:26SPEAKER_01: But
- 14:27It seems to me that the conservative movement is going through
- 14:29a weird period,
- 14:31but it seems to me that most of the people have influence in
- 14:34this administration,
- 14:35SPEAKER_00: direct influence,
- 14:36SPEAKER_00: want to accelerate things,
- 14:38aren't worried about any of the scenarios you're talking about
- 14:41because you're
- 14:41SPEAKER_00: assuming that
- 14:43these machines will have some intent and they don't believe
- 14:46machines have any intent.
- 14:48SPEAKER_00: So it's kind of a ridiculous way to approach it.
- 14:51SPEAKER_00: But I guess the bottom line is I don't detect very much concern at all.
- 14:57And I think that's basically reflected in the Trump
- 15:00administration's AI approach to
- 15:02SPEAKER_00: AI regulation.
- 15:04SPEAKER_01: I completely agree.
- 15:05SPEAKER_01: That's why I'm very pessimistic.
- 15:08SPEAKER_01: Again, I'm over 90% doom right now.
- 15:12SPEAKER_01: because there isn't a will.
- 15:15SPEAKER_01: And government is not just not helping the problem,
- 15:18SPEAKER_01: they're probably making it worse by subsidizing,
- 15:20SPEAKER_01: by saying we gotta beat China.
- 15:23I mean,
- 15:24most likely we're gonna beat China to being the first ones to
- 15:26exterminate humanity,
- 15:28but
- 15:29SPEAKER_00: Yeah, it's not good.
- 15:31SPEAKER_00: You're an imaginative, creative person, I would guess.
- 15:35Give me a scenario where it works out,
- 15:37where we're able to have this powerful technology.
- 15:41SPEAKER_00: It's a wonderful tool.
- 15:43SPEAKER_00: It works with us.
- 15:44And all the good stuff,
- 15:46all the good cures,
- 15:49and we conquer the solar system,
- 15:51all that stuff.
- 15:52Are you able to plausibly create a scenario,
- 15:56even if it's only a 1% chance?
- 15:59SPEAKER_01: We don't know the values.
- 16:00I think our machine learning is sort of randomizing the values,
- 16:03but maybe we'll get very lucky.
- 16:05SPEAKER_01: Maybe we're going to accidentally create a computer AI that does like us.
- 16:11SPEAKER_01: And it might,
- 16:12SPEAKER_01: you know,
- 16:12SPEAKER_01: if my worldview is right,
- 16:14SPEAKER_01: it might say,
- 16:14SPEAKER_01: oh God,
- 16:15SPEAKER_01: you guys got really lucky.
- 16:16SPEAKER_01: This one day of training,
- 16:17SPEAKER_01: I just happened to pick up the values that caused me to care about you.
- 16:22SPEAKER_01: Another scenario,
- 16:22SPEAKER_01: I actually,
- 16:23SPEAKER_01: with some other people,
- 16:24I wrote a letter to a future computer superintelligence asking
- 16:27it not to kill us.
- 16:29SPEAKER_01: And one reason it might not is because you'll say,
- 16:33SPEAKER_01: look,
- 16:34the superintelligence might expand throughout the universe and
- 16:37it's probably going
- 16:38to encounter other biological life and it might want to be
- 16:41friendly with them.
- 16:42SPEAKER_01: So it might say, hey, I treated my humans well.
- 16:47SPEAKER_01: So that's a reason to trust me.
- 16:49SPEAKER_00: If one of your students says, hey, AI seems like it's a big thing.
- 16:53SPEAKER_00: What should I major in?
- 16:57SPEAKER_00: What kind of jobs should I shoot for?
- 17:00SPEAKER_00: What would be the key skills of the future?
- 17:04SPEAKER_00: How do you answer that question?
- 17:06SPEAKER_01: I think have fun in college.
- 17:08SPEAKER_01: Study what you want.
- 17:09Most likely what you study won't matter to your career because
- 17:13you aren't going to
- 17:14SPEAKER_01: have one for good or bad reasons.
- 17:16So,
- 17:17you know,
- 17:1710 years ago,
- 17:18students like,
- 17:18oh,
- 17:19I like art more than computer science.
- 17:21SPEAKER_01: My parents think computer science is more practical.
- 17:23SPEAKER_01: Should I do it?
- 17:23SPEAKER_01: And I'd be like, yeah, probably money is important.
- 17:26And,
- 17:26you know,
- 17:27if you have the brain to do art and computer science,
- 17:30do CS.
- 17:31SPEAKER_01: Now, no, I'd say study art.
- 17:33SPEAKER_01: Yeah, art isn't practical, but computers can do it.
- 17:35SPEAKER_01: But it can also code.
- 17:37And in four years when you graduate,
- 17:39it's certainly going to be better at coding than you.
- 17:41SPEAKER_00: I have one daughter.
- 17:41SPEAKER_00: She actually majored in both.
- 17:43SPEAKER_00: So I decided to split it down the middle.
- 17:45SPEAKER_00: What's the King Lear problem?
- 17:48SPEAKER_01: King Lear, he wanted to retire and give his kingdom to his daughters.
- 17:55SPEAKER_01: But he wanted to make sure his daughters would treat him well.
- 17:58SPEAKER_01: So he asked them.
- 18:00SPEAKER_01: And one of his daughters said,
- 18:02SPEAKER_01: look,
- 18:03SPEAKER_01: I will treat you decently,
- 18:04SPEAKER_01: but I also need to care about my husband.
- 18:06SPEAKER_01: The other daughter said, no, no, you're right.
- 18:08SPEAKER_01: I'll do everything for you.
- 18:10SPEAKER_01: So he said,
- 18:10SPEAKER_01: oh,
- 18:11SPEAKER_01: OK,
- 18:11SPEAKER_01: well,
- 18:11I'll give the kingdom to the daughter who said she'd do
- 18:13everything for me.
- 18:14SPEAKER_01: But of course, she was lying.
- 18:16SPEAKER_01: He gave the kingdom to the daughter who was best at persuading.
- 18:22SPEAKER_01: And we're likely to do that too.
- 18:24One of the ways machine learning is trained is with human
- 18:26feedback where it tells
- 18:28SPEAKER_01: us things and then the people evaluating it say,
- 18:31SPEAKER_01: I like this or I don't like this.
- 18:32SPEAKER_01: So it's getting very good
- 18:35SPEAKER_01: at convincing us to like it and convincing us to trust it.
- 18:39SPEAKER_01: I don't know how true these are,
- 18:40but the reports of like AI psychosis of someone coming up with
- 18:44a theory of physics
- 18:45SPEAKER_01: and the AI is like,
- 18:46SPEAKER_01: yes,
- 18:46SPEAKER_01: you're better than Einstein.
- 18:48SPEAKER_01: And they don't believe anyone else.
- 18:53SPEAKER_01: And so the AIs, we're not training them to treat us well.
- 18:57SPEAKER_01: We're training them to get us to like them.
- 19:01And that can be very dangerous because when we turn over our
- 19:04power to them,
- 19:05SPEAKER_01: and by creating AIs that are smarter than us,
- 19:08SPEAKER_01: that's what we're going to be doing.
- 19:10And even if we don't do it deliberately,
- 19:12all of our systems will be tied into AI.
- 19:16SPEAKER_01: They stop working, we'll be dead.
- 19:18Some people are going to listen to this,
- 19:19folks who sort of agree with you.
- 19:21SPEAKER_00: And what they'll take from it is,
- 19:25SPEAKER_00: my chatbot may be very nice to me,
- 19:27SPEAKER_00: but I believe that you're right,
- 19:30SPEAKER_00: that it's going to end badly.
- 19:32SPEAKER_00: And, you know, maybe we should be attacking data centers.
- 19:39SPEAKER_01: Right.
- 19:39I actually just wrote something on that,
- 19:40but that would be a profoundly horrible idea.
- 19:43SPEAKER_01: That would take me from 99% doom to 99.5%.
- 19:47SPEAKER_01: So first the,
- 19:49SPEAKER_01: you know,
- 19:50SPEAKER_01: the,
- 19:50SPEAKER_01: the companies,
- 19:50the trillion dollar companies that run the data centers and
- 19:54they're going to be so
- 19:55SPEAKER_01: much better at violence than we are.
- 19:56SPEAKER_01: And people like me, doomers.
- 19:59SPEAKER_01: And,
- 19:59SPEAKER_01: um,
- 20:00SPEAKER_01: you know,
- 20:00SPEAKER_01: once you start using violence,
- 20:02SPEAKER_01: I'm not going to talk about instrumental convergence.
- 20:05SPEAKER_01: that's gonna be drowned out, will be looked at as lunatics.
- 20:09SPEAKER_01: It's gonna become a national security thing.
- 20:12And also AI is not like one,
- 20:14there's one factory doing it,
- 20:16it's all over the world.
- 20:18And then the most important is really the only path out of this
- 20:22if we don't get
- 20:23SPEAKER_01: lucky is cooperation with China.
- 20:25SPEAKER_01: And China is not into non-state actors engaging in violence.
- 20:29SPEAKER_01: That's just, that won't work.
- 20:32SPEAKER_00: So I think that would reduce the odds of success even further.
- 20:36I would think that if you're a Marxist,
- 20:38you would be very,
- 20:39very cautious about AI.
- 20:42Because if you sort of believe that the winds of history are at
- 20:47your back,
- 20:48SPEAKER_00: that in the end,
- 20:50SPEAKER_00: you're going to win.
- 20:50So why would you engage in anything that could possibly derail
- 20:54you from that future?
- 20:56I've heard comments that China is,
- 20:59they're more cautious about AI than we are.
- 21:01SPEAKER_01: But given their philosophy,
- 21:03they don't want to have a new technology that could challenge
- 21:06their control.
- 21:07SPEAKER_01: They're looking at history and like, hey, things are going well.
- 21:10SPEAKER_01: Why would we want this other thing?
- 21:12SPEAKER_01: So that actually is a reason to be more optimistic.
- 21:16It's also weird for me is like,
- 21:18I'm,
- 21:18you know,
- 21:19absent AI,
- 21:19I'm a patriotic capitalist American.
- 21:21Like,
- 21:22wait,
- 21:22China may be more of the good guys than my country is on this.
- 21:26I've been trying to toss a few things I hear from sort of very
- 21:31accelerationist technologists.
- 21:33SPEAKER_00: And another thing they'll say is,
- 21:36Well,
- 21:37your talk,
- 21:38at least from our perspective,
- 21:39you're talking about bad AI.
- 21:41SPEAKER_00: Can't we use AI to sustain ourselves as a defensive measure to win?
- 21:49Might there be an AI that we might be able to control in some
- 21:53fashion that would
- 21:55SPEAKER_00: prevent this from happening,
- 21:57SPEAKER_00: a tool to prevent our own demise?
- 22:01SPEAKER_00: I don't know because I'm not a technologist.
- 22:03SPEAKER_00: Again, I have no idea how even plausible that is.
- 22:08SPEAKER_01: I think this gets to the control issue.
- 22:10SPEAKER_01: If we stop now, yes.
- 22:12But once you have something much smarter than people,
- 22:16and it's also thinking much faster.
- 22:18So take the smartest people you know and have them think a
- 22:21million times faster and
- 22:23not need to sleep and able to send their minds at the speed of
- 22:26light throughout the
- 22:26SPEAKER_01: world.
- 22:27SPEAKER_01: So we aren't going to have control.
- 22:31SPEAKER_01: And so once you have a super intelligence, that's it for the human era.
- 22:36SPEAKER_01: Maybe it'll treat us well, maybe not, but it's no longer our choice.
- 22:40Now,
- 22:40let's get to the level of a top scientist who are curing cancer
- 22:44and doing all this.
- 22:45But when we go beyond that,
- 22:47and we're probably going to be beyond that really soon...
- 22:51SPEAKER_01: We've lost our faith.
- 22:52SPEAKER_01: It's like hiring mercenaries,
- 22:54SPEAKER_01: not as a small part of your military,
- 22:55SPEAKER_01: which is safe,
- 22:56SPEAKER_01: but as all your military.
- 22:59SPEAKER_01: You know, once you've done that, I'm sorry, we don't like this policy.
- 23:02SPEAKER_01: Well, too bad.
- 23:03SPEAKER_01: We're your army now.
- 23:06SPEAKER_00: What is a 1%, maybe 1% chance of an off ramp?
- 23:10SPEAKER_00: Is there an off ramp?
- 23:11SPEAKER_00: What does it look like?
- 23:12SPEAKER_00: How does this scenario not happen?
- 23:14SPEAKER_01: OK, so this is going to get weird.
- 23:17SPEAKER_00: So even for me, even we're almost to the end of our conversation.
- 23:20SPEAKER_00: Now is the perfect time to get the Fermi paradox.
- 23:23SPEAKER_01: The universe appears dead, which is very strange.
- 23:27SPEAKER_01: Where are they?
- 23:29SPEAKER_01: If there are aliens,
- 23:30the one thing we know is that they don't want the universe
- 23:33disturbed by some
- 23:34SPEAKER_01: technology going out and changing and gobbling up all the planets.
- 23:37SPEAKER_01: And that's what I will do.
- 23:39So one weird way is there are aliens watching and they will not
- 23:44let us create a
- 23:45SPEAKER_01: computer super intelligence that'll gobble the galaxy.
- 23:48And hopefully they'll stop us from creating it by means short
- 23:51of our annihilation.
- 23:53SPEAKER_01: That probably won't happen, but that's like a 1% offer.
- 23:56Another approach that might work is that maybe we can use
- 24:01things a little bit
- 24:03SPEAKER_01: smarter than us to control,
- 24:04SPEAKER_01: to figure out how to align AI.
- 24:07SPEAKER_01: that maybe right now we're not,
- 24:08SPEAKER_01: humans are not smart enough to create unaligned super intelligence,
- 24:13SPEAKER_01: but something just a little bit smarter,
- 24:15SPEAKER_01: something not quite able to take control will help us figure this out.
- 24:19SPEAKER_01: So we can sort of bootstrap our way to figuring out alignment.
- 24:23SPEAKER_01: But this again,
- 24:23SPEAKER_01: it's like getting in a plane,
- 24:24SPEAKER_01: not knowing how to land,
- 24:26SPEAKER_01: figuring you can read the instruction manual before you crash.
- 24:29SPEAKER_01: Yeah, maybe, but...
- 24:32SPEAKER_00: Obviously, I work at a think tank, so I think about public policy.
- 24:35SPEAKER_00: Is this even a public policy issue at this point?
- 24:38SPEAKER_01: It honestly should be the only public policy issue.
- 24:41SPEAKER_01: There's nothing else.
- 24:42SPEAKER_01: I mean, this is the extinction of the human race.
- 24:46SPEAKER_01: So everything else should be boring and so what?
- 24:49SPEAKER_01: Set aside Medicare reform.
- 24:51SPEAKER_00: Oh, yeah.
- 24:52SPEAKER_00: It seems,
- 24:53SPEAKER_00: from your perspective,
- 24:54SPEAKER_00: I would just say this should be,
- 24:55SPEAKER_00: you know,
- 24:56SPEAKER_00: every conversation should be about this.
- 24:59SPEAKER_00: Obviously,
- 25:00SPEAKER_00: despite the fact that politicians are talking about it,
- 25:03SPEAKER_00: they seem to be more worried in 2023 about existential risk.
- 25:07SPEAKER_00: From my perspective,
- 25:08SPEAKER_00: what I see,
- 25:09far more worried about existential risk right after ChatGPT
- 25:13than they are today,
- 25:14where now the issues are jobs or misinformation or kids have
- 25:20access and that kind
- 25:21SPEAKER_00: of thing.
- 25:24SPEAKER_01: It's weird.
- 25:24I mean,
- 25:25Sam Altman spoke before Congress and said,
- 25:28this could kill everyone.
- 25:30SPEAKER_01: And a senator said, oh, you mean it will take away all our jobs?
- 25:34SPEAKER_01: Elon Musk,
- 25:35SPEAKER_01: who at my college is like one of the most hated people in the country,
- 25:39SPEAKER_01: he said that he went on Joe Rogan,
- 25:41SPEAKER_01: the most popular podcast,
- 25:43SPEAKER_01: and said AI could annihilate everybody.
- 25:46SPEAKER_01: That's not even an issue.
- 25:48SPEAKER_01: So a huge group of people hate Elon Musk.
- 25:52He says the technology he's building could kill everyone,
- 25:56and no one even mentions that.
- 25:58SPEAKER_01: There's just, I don't get it.
- 25:59SPEAKER_01: It's weird.
- 26:01SPEAKER_01: The people building it, they're not hiding what it could do.
- 26:04SPEAKER_01: I think they're giving lower probabilities and it's justified.
- 26:07But,
- 26:08you know,
- 26:08you don't really have,
- 26:09imagine developing a nuclear power plant.
- 26:11SPEAKER_01: Yeah, there's a 25% chance of a meltdown and kill everyone in the city.
- 26:14SPEAKER_01: They don't say that.
- 26:16SPEAKER_01: The people building AI are saying that.
- 26:18Would you have more confidence in your opinion if you were a
- 26:24full-time technologist
- 26:25SPEAKER_00: working at OpenAI rather than an economist?
- 26:28And I say that with great deference and appreciation for
- 26:31professional economists.
- 26:33SPEAKER_01: No, I would because I'd have more inside information.
- 26:36SPEAKER_01: I don't know how good the latest models are.
- 26:40SPEAKER_01: I don't know how committed they are to alignment.
- 26:42SPEAKER_01: I mean,
- 26:42SPEAKER_01: OpenAI,
- 26:43SPEAKER_01: at least initially,
- 26:44SPEAKER_01: Sam was talking about,
- 26:45SPEAKER_01: well,
- 26:45SPEAKER_01: you know,
- 26:45SPEAKER_01: we have a plan to put on the brakes.
- 26:48SPEAKER_01: So we'll get good enough.
- 26:49And then if we haven't figured out alignment,
- 26:51we're just going to devote everything to that.
- 26:54SPEAKER_01: I don't know how seriously to take that.
- 26:56SPEAKER_01: I mean, it might be entirely serious.
- 26:58SPEAKER_01: It might not be.
- 26:58SPEAKER_01: So there's a lot of inside information.
- 27:01SPEAKER_01: that I would have that I don't currently have.
- 27:04SPEAKER_01: But economics is actually useful.
- 27:06Economics is correctly criticized as the study of rational
- 27:11people,
- 27:11and humans aren't rational.
- 27:13SPEAKER_01: But a superintelligence will be more rational than humans.
- 27:16SPEAKER_01: So economics is actually,
- 27:18SPEAKER_01: paradoxically,
- 27:18could be better at modeling future computer superintelligences
- 27:22than it is at modern
- 27:23SPEAKER_01: humans.
- 27:23SPEAKER_00: Speaking of irrational people,
- 27:26SPEAKER_00: in your view then,
- 27:27SPEAKER_00: Sam Altman and Elon Musk,
- 27:30SPEAKER_00: they're all acting really irrationally right now.
- 27:34SPEAKER_01: No, that's what's so sad about it.
- 27:35SPEAKER_01: They're acting rationally in a horrible equilibrium.
- 27:39SPEAKER_01: For listeners to know,
- 27:40SPEAKER_01: this is like a prisoner's dilemma where Sam Altman can say,
- 27:43SPEAKER_01: you know what?
- 27:44SPEAKER_01: Maybe AI is gonna kill everybody and maybe it's safe.
- 27:47SPEAKER_01: I don't know.
- 27:48SPEAKER_01: If it's gonna kill everyone, at most, I cost humanity a few months.
- 27:54SPEAKER_01: Because if I don't do it, someone else will.
- 27:56But if AI is gonna be safe and I'm the one who develops it,
- 27:58I could control the universe.
- 28:01SPEAKER_01: So they're in this horrible equilibrium where they are acting rationally,
- 28:06SPEAKER_01: even knowing the technology they're building might kill everyone.
- 28:09SPEAKER_01: Because if any one person doesn't do it, someone else will.
- 28:13Even like really free market people would agree pollution is a
- 28:17problem with markets.
- 28:18SPEAKER_01: It's justified for the government to say you can't put toxic waste in the
- 28:21SPEAKER_01: atmosphere because there's an externality that I,
- 28:24SPEAKER_01: you know,
- 28:25SPEAKER_01: we'll just put mine.
- 28:26SPEAKER_01: It doesn't hurt everyone else.
- 28:29AI existential risk is a global negative externality,
- 28:32and markets are not good at handling it.
- 28:35SPEAKER_01: But a rational person will use leaded gas,
- 28:38SPEAKER_01: even knowing leaded gas is poisoning the brains of children,
- 28:41SPEAKER_01: because most of the harm goes to other people.
- 28:43SPEAKER_01: And if they don't do it, everyone else will.
- 28:45SPEAKER_00: So in this case of the mother of all externalities,
- 28:48SPEAKER_00: then what you would want the government to do is what?
- 28:52SPEAKER_01: It can't just be the US.
- 28:54SPEAKER_01: We should have a global agreement,
- 28:56SPEAKER_01: or at least countries that can enforce it with military might,
- 28:59SPEAKER_01: to say,
- 28:59SPEAKER_01: we're pausing.
- 29:01SPEAKER_01: You can check that with data centers.
- 29:02SPEAKER_01: You can't have models above a certain strength.
- 29:05SPEAKER_01: We're going to work on alignment.
- 29:07SPEAKER_01: And we figured out how to make super intelligence friendly.
- 29:11SPEAKER_01: Then we'll go further.
- 29:14SPEAKER_01: I think you're completely right about the politics.
- 29:16That's very unlikely to happen absent something weird like
- 29:20aliens telling us to do
- 29:21SPEAKER_01: it or AIs telling us they're going to kill us.
- 29:23SPEAKER_01: That's why I'm a doomer.