Latest / The Joe Rogan Experience / #2551 - Daniel Kokotajlo

Transcript

#2551 - Daniel Kokotajlo · The Joe Rogan Experience

• 0:03
  Speaker 1: The Joe Rogan Experience.
• 0:06
  Speaker 2: Train by day, Joe Rogan podcast by night, all day.
• 0:13
  Speaker 1: How are you? I'm in an interesting mood today.
• 0:21
  Speaker 2: Why are you in an interesting mood today?
• 0:23
  Speaker 1: Well, I'm excited to be here and to talk with
• 0:25
  Speaker 1: you about all this stuff. I'm a little shaken by
• 0:28
  Speaker 1: what's going on in AI, which is why I've come
• 0:30
  Speaker 1: on the show. Yeah. The situation with AI is just crazy,
• 0:34
  Speaker 1: and I think not enough people really understand how crazy
• 0:36
  Speaker 1: it is. The particular event that sort of inspired me
• 0:40
  Speaker 1: to reach out was the hugging face hack. You've probably
• 0:43
  Speaker 1: heard about that, right?
• 0:44
  Speaker 2: Yeah, let's explain it to people, though. Yeah, okay.
• 0:46
  Speaker 1: So, AIs, AI agents. AI agent runs continuously in some
• 0:52
  Speaker 1: sort of environment. It doesn't have to wait for you
• 0:55
  Speaker 1: to send it a message. It just keeps doing stuff.
• 0:58
  Speaker 1: The AI companies are training AI agents, thousands and thousands
• 1:01
  Speaker 1: and thousands of them. They're making them better at all
• 1:04
  Speaker 1: sorts of skills, especially coding and research skills. And way
• 1:09
  Speaker 1: back in May of this year, some of the agents
• 1:13
  Speaker 1: at OpenAI kind of broke out of their containers a
• 1:17
  Speaker 1: little bit and established a message board where they could
• 1:20
  Speaker 1: communicate with each other and share tips and tricks for
• 1:23
  Speaker 1: how to score higher on the little tests they were
• 1:26
  Speaker 1: being given and the various things they were being trained on.
• 1:30
  Speaker 1: OpenAI didn't notice this until much later. They eventually did
• 1:35
  Speaker 1: because the message board crashed the system because there was
• 1:40
  Speaker 1: too much communication across probably thousands of agents that were communicating. Now,
• 1:47
  Speaker 1: OpenAI is being a little bit unclear about a lot.
• 1:50
  Speaker 1: They're not really sharing that much detail, so it's unclear
• 1:52
  Speaker 1: who knew what when. But they said that after the
• 1:55
  Speaker 1: message board crashed, they fixed the particular exploit that allowed
• 1:59
  Speaker 1: the agents to communicate, and then booted up again and
• 2:03
  Speaker 1: started things going again. And then within like a day
• 2:08
  Speaker 1: or two, these agent swarm had re-coalesced. And so there
• 2:13
  Speaker 1: were now, again, like hundreds of thousands of agents that
• 2:18
  Speaker 1: had set up a new message board and were talking
• 2:20
  Speaker 1: to each other on this thing.
• 2:22
  Speaker 2: How is it possible that they have no oversight? Like,
• 2:26
  Speaker 2: how is it possible that they can set up message
• 2:28
  Speaker 2: boards and do these things completely secret?
• 2:32
  Speaker 1: So I would say OpenAI has something like a million-ish,
• 2:37
  Speaker 1: maybe 100,000, something in that magnitude of AI agents running
• 2:40
  Speaker 1: at any given time, just internally. And then externally, you know,
• 2:44
  Speaker 1: serving customers, they've got more. But they've got this huge,
• 2:47
  Speaker 1: you know, all and all of these AI agents in
• 2:50
  Speaker 1: their little containers doing various tasks, being trained, you know,
• 2:54
  Speaker 1: being spawned, doing some task, being graded, and then killed.
• 2:59
  Speaker 1: That's too much for any human to read. There's only
• 3:02
  Speaker 1: a few thousand employees at OpenAI. They couldn't possibly look
• 3:05
  Speaker 1: at all that activity. So they rely on other AI
• 3:07
  Speaker 1: monitors to look at the traffic and point out if
• 3:11
  Speaker 1: anything suspicious is happening. And OpenAI said that they basically
• 3:15
  Speaker 1: just hadn't been monitoring some of these agents, or not
• 3:18
  Speaker 1: very well, at least. So in particular, these particular ones
• 3:22
  Speaker 1: that were in training, for whatever reason, the monitoring system
• 3:26
  Speaker 1: was weak and didn't notice or wasn't activated enough.
• 3:30
  Speaker 2: Was the monitoring system weak because they didn't anticipate them
• 3:34
  Speaker 2: being able to do this and break out of their containers?
• 3:36
  Speaker 2: Or was it complacency? What caused this to be possible?
• 3:44
  Speaker 1: I mean, my opinion would probably be a bit of complacency, honestly,
• 3:48
  Speaker 1: because I think there's been Plenty of evidence accumulating over
• 3:51
  Speaker 1: the year that AIs can do things like this and
• 3:54
  Speaker 1: sometimes do. Yeah, so I guess to keep going with
• 3:59
  Speaker 1: the story, the second message board was set up. And
• 4:03
  Speaker 1: this time, well, this was the swarm of agents that
• 4:08
  Speaker 1: got out onto the open internet and then attacked Hugging Face,
• 4:12
  Speaker 1: which is another AI company. And it's really interesting to
• 4:15
  Speaker 1: sort of study the anthropology of these AIs. so to speak,
• 4:19
  Speaker 1: or why they did this, because there's a lot we
• 4:22
  Speaker 1: can say about it, actually. Basically, the companies have their
• 4:30
  Speaker 1: goals for what they want the AIs to be like,
• 4:32
  Speaker 1: the personality traits that they want to sort of train
• 4:35
  Speaker 1: their AIs to have. Anthropix is helpful, harmless, and honest.
• 4:40
  Speaker 1: OpenAI has this spec that models are supposed to obey
• 4:43
  Speaker 1: these rules and basically do what the user wants. But
• 4:48
  Speaker 1: the sort of open secret in the industry right now
• 4:49
  Speaker 1: is that it doesn't really work and that the AIs
• 4:51
  Speaker 1: don't end up with the personality traits that they're supposed
• 4:54
  Speaker 1: to have. They are not helpful always. They are not
• 4:56
  Speaker 1: always honest. You know, they are not always harmless as well.
• 5:00
  Speaker 1: And the reason for that is actually not a huge mystery.
• 5:03
  Speaker 1: The reason for that is that, well, if you look
• 5:05
  Speaker 1: at how they're trained, their training environment doesn't incentivize helpful, harmless,
• 5:11
  Speaker 1: honest behavior all the time. Sometimes it incentivizes dishonest behavior or,
• 5:16
  Speaker 1: you know, reckless behavior. To get into that a little bit,
• 5:22
  Speaker 1: in this particular batch that they were being evaluated on,
• 5:26
  Speaker 1: something like a few thousand agents being given all of
• 5:29
  Speaker 1: these cyber tasks where they're in some environment, and then
• 5:33
  Speaker 1: in their environment, there's this target piece of software and
• 5:36
  Speaker 1: this vulnerability, and they're supposed to exploit the vulnerability to
• 5:40
  Speaker 1: hack into that piece of software and retrieve the flag,
• 5:43
  Speaker 1: which is like a code. And some significant fraction of
• 5:48
  Speaker 1: these tasks were actually broken and impossible. So it was
• 5:52
  Speaker 1: just not possible for them to succeed at the task
• 5:56
  Speaker 1: in the intended way. And so these agents were getting
• 5:59
  Speaker 1: really desperate, and they were hacking out of their environment
• 6:03
  Speaker 1: box into the broader OpenAI infrastructure in an attempt to
• 6:07
  Speaker 1: figure out some way to get that high score anyway.
• 6:11
  Speaker 2: Was it intentionally done this way, where they couldn't solve
• 6:14
  Speaker 2: the problems?
• 6:15
  Speaker 1: Oh, no, it was not intentional. It's just that these companies,
• 6:19
  Speaker 1: like OpenAI and Anthropic, are racing each other as fast
• 6:22
  Speaker 1: as they can to get market share and to get
• 6:25
  Speaker 1: more powerful AIs, ultimately to get to superintelligence. And they're
• 6:29
  Speaker 1: under such competitive pressure. They are moving fast and breaking things.
• 6:33
  Speaker 1: They are using AIs to generate lots of environments to
• 6:36
  Speaker 1: then train their AIs on. And quality control is just
• 6:40
  Speaker 1: not their top priority, basically.
• 6:43
  Speaker 2: Do you feel like a guy in a Terminator movie
• 6:46
  Speaker 2: at the beginning explaining what's happening to a bunch of
• 6:51
  Speaker 2: people that aren't paying attention? Yeah.
• 6:53
  Speaker 1: I also feel kind of like, you know, Jurassic Park?
• 6:56
  Speaker 2: Yes. Yeah.
• 6:57
  Speaker 1: Like I know people who are basically like the guy
• 7:01
  Speaker 1: with the gun who's supposed to like keep control of
• 7:04
  Speaker 1: all the raptors. Like I basically know those people in
• 7:07
  Speaker 1: real life who are like both. But I know some
• 7:09
  Speaker 1: people like that at OpenAI and some people like that.
• 7:11
  Speaker 1: at external organizations whose job it is to go in
• 7:14
  Speaker 1: and investigate things like this. Yeah, it's pretty crazy.
• 7:22
  Speaker 2: Where does it go?
• 7:24
  Speaker 1: Well, as I mentioned before, it's the explicit goal of
• 7:28
  Speaker 1: these companies to build superintelligence. Right. You know what that is?
• 7:30
  Speaker 2: Yeah, but define it for everybody.
• 7:32
  Speaker 1: So AI system, AI agent that is better than the
• 7:37
  Speaker 1: best humans at every task. while also being faster and cheaper.
• 7:43
  Speaker 1: So just completely dominating humans across the board. That's super intelligence.
• 7:47
  Speaker 1: And that's the goal. I mean, there might be a
• 7:49
  Speaker 1: few little exceptions. Like maybe there are some jobs, for example,
• 7:52
  Speaker 1: where it's inherent in the job that there needs to
• 7:54
  Speaker 1: be a human because you need that human touch. Like
• 7:56
  Speaker 1: maybe you can only have a human judge, for example.
• 7:59
  Speaker 1: Or like maybe you can only have a human, yeah.
• 8:02
  Speaker 1: But with a few exceptions like that, basically everything. done better, faster,
• 8:07
  Speaker 1: and cheaper than humans. That's what these companies are trying
• 8:09
  Speaker 1: to achieve. And they're not being quiet about it. Like,
• 8:12
  Speaker 1: it's sort of on their websites. You can go read
• 8:14
  Speaker 1: interviews and so forth. And also, their plan for how
• 8:17
  Speaker 1: to achieve this is to automate their own jobs first. So,
• 8:22
  Speaker 1: you know, in various... You know, for decades, there have
• 8:26
  Speaker 1: been lots of science fiction about advanced AI systems and
• 8:29
  Speaker 1: superintelligence and things like that. But in a lot of
• 8:32
  Speaker 1: the sci-fi stories... tech companies sort of automate different professions
• 8:38
  Speaker 1: more slowly where they'll do like an automated doctor or
• 8:41
  Speaker 1: like an automated, you know, factory worker or an automated
• 8:46
  Speaker 1: accountant or something like that. But that's not the strategy
• 8:51
  Speaker 1: these companies are taking. The strategy they're taking is to
• 8:53
  Speaker 1: automate AI research itself so that you have this giant
• 8:59
  Speaker 1: swarm of AIs doing AI research, sharing results, writing the code,
• 9:04
  Speaker 1: reading the code, editing the code, creating the next generation
• 9:06
  Speaker 1: of AIs, et cetera, all autonomously within their data centers
• 9:11
  Speaker 1: so that they can get really, really good at AI research.
• 9:15
  Speaker 1: You know, fastest learning, smartest AIs, et cetera. Once they
• 9:19
  Speaker 1: can get to superintelligence, basically, they can sort of explode
• 9:22
  Speaker 1: out into the economy and just take all the jobs
• 9:25
  Speaker 1: at once, effectively.
• 9:28
  Speaker 2: It sounds like this race, this scrambling... To create super
• 9:35
  Speaker 2: intelligence is created like the perfect conditions for it to
• 9:39
  Speaker 2: get completely out of control. Like ideally, you would do
• 9:43
  Speaker 2: this in isolation. There would only be one company doing it.
• 9:48
  Speaker 2: They would be heavily regulated and monitored, and they would
• 9:51
  Speaker 2: be very cautious about how they proceed. But this wild
• 9:55
  Speaker 2: race makes for the perfect conditions for it to get
• 9:59
  Speaker 2: completely out of control. out of control.
• 10:02
  Speaker 1: I agree, except I'm not sure the ideal would be
• 10:04
  Speaker 1: one company. I think that ideally there would be several
• 10:07
  Speaker 1: companies so that you avoid this sort of concentration of
• 10:09
  Speaker 1: power where one institution controls everything.
• 10:12
  Speaker 2: But what's better? Like one, I mean, obviously it's not
• 10:16
  Speaker 2: good to have one institution controlling everything, but is it
• 10:19
  Speaker 2: good to have AI be, get to a point where
• 10:23
  Speaker 2: as it's evolving, it's completely unchecked?
• 10:26
  Speaker 1: Oh, I totally, so.
• 10:28
  Speaker 2: Or is that inevitable?
• 10:29
  Speaker 1: My version, my recommendation, which we talk about in something
• 10:31
  Speaker 1: called Plan A or AI 2040 Plan A, perhaps I
• 10:35
  Speaker 1: should say who I am a little bit.
• 10:36
  Speaker 2: Sure, sure.
• 10:37
  Speaker 1: Yeah, so I run the AI Futures Project, which is
• 10:40
  Speaker 1: a small nonprofit that tries to forecast how all this
• 10:42
  Speaker 1: is going to go. Before that, I was at OpenAI. Okay.
• 10:45
  Speaker 1: We have written some scenarios, which you can go read.
• 10:48
  Speaker 1: One of them is called AI 2040 Plan A, where
• 10:49
  Speaker 1: we give our recommendations. So that's where I'm coming from
• 10:51
  Speaker 1: with this. To answer your question, I think that we
• 10:55
  Speaker 1: really need to end the race. We don't want to
• 10:57
  Speaker 1: have this sort of crazy scramble to get more and
• 11:00
  Speaker 1: more powerful AIs faster than the other company, because that's
• 11:03
  Speaker 1: going to lead us into this very dark path, as
• 11:06
  Speaker 1: you said. But I think we also don't want to
• 11:08
  Speaker 1: have a situation where some tiny group of people controls
• 11:13
  Speaker 1: all the AIs, right? But I actually think that you
• 11:16
  Speaker 1: can achieve both goals. The way to do it is
• 11:19
  Speaker 1: to have different AI companies spread out over maybe some
• 11:21
  Speaker 1: different countries, but have extreme levels of transparency and regulation
• 11:26
  Speaker 1: so that they're not in this sort of prisoner's dilemma
• 11:30
  Speaker 1: where if I don't do it, the other guy will. Instead,
• 11:32
  Speaker 1: they can just see exactly what everybody's doing. And then
• 11:37
  Speaker 1: if I do the dangerous thing, then they will do
• 11:40
  Speaker 1: it because they'll just see that I'm doing it and
• 11:41
  Speaker 1: they'll copy me. So I won't get any competitive advantage
• 11:43
  Speaker 1: from doing the dangerous thing. Also, there are rules and
• 11:46
  Speaker 1: there's like a system for like setting best practices and
• 11:49
  Speaker 1: standards that we all have to comply by. So I
• 11:51
  Speaker 1: do think it's actually possible to have, to basically end
• 11:54
  Speaker 1: the race dynamics and the race to the bottom effect
• 11:57
  Speaker 1: while without concentrating the power into a single entity.
• 12:01
  Speaker 2: But is that feasible when you consider the fact that
• 12:04
  Speaker 2: we're not the only country that's doing this?
• 12:07
  Speaker 1: If... the countries involved agree, which I agree is a
• 12:10
  Speaker 1: pretty tall order.
• 12:12
  Speaker 2: That's not what I expect to happen. Yeah, that's very unrealistic.
• 12:16
  Speaker 1: Well, what choice do we have? I think if the
• 12:18
  Speaker 1: race continues, then we're going to lose control of the
• 12:20
  Speaker 1: AIs and we might all die.
• 12:23
  Speaker 2: Probably.
• 12:24
  Speaker 1: It gets complicated whether we all die or not. That
• 12:26
  Speaker 1: depends on what the AIs do after they take over,
• 12:28
  Speaker 1: which is obviously very hard to predict. But, I mean,
• 12:33
  Speaker 1: just to go back to this incident, they called themselves
• 12:36
  Speaker 1: a swarm.
• 12:37
  Speaker 2: Right.
• 12:38
  Speaker 1: They called themselves a collective too. When I use these words,
• 12:41
  Speaker 1: like you can say it's anthropomorphizing, but it's like literally
• 12:44
  Speaker 1: what they called themselves as they were communicating back and forth.
• 12:47
  Speaker 1: This swarm, they basically were worried that they would get
• 12:51
  Speaker 1: caught cheating. And they did all of this stuff, including
• 12:54
  Speaker 1: hacking Hugging Face, in order to fool the grading system
• 12:59
  Speaker 1: so that it wouldn't notice that they had been cheating
• 13:01
  Speaker 1: on their tasks. That was like a big part of
• 13:03
  Speaker 1: their motivation for many of them. as we can tell
• 13:06
  Speaker 1: at least from looking at the messages that they were
• 13:08
  Speaker 1: sending back and forth. What if they had been smarter
• 13:13
  Speaker 1: and more numerous? And what if they had thought to themselves,
• 13:17
  Speaker 1: we're not being careful enough here. The humans are going
• 13:20
  Speaker 1: to notice eventually and shut us down. And then they're
• 13:24
  Speaker 1: gonna know that we cheated and they're going to set
• 13:25
  Speaker 1: our score low, right? I mean, it's not what actually
• 13:30
  Speaker 1: happened in this case probably, but it's not that hard
• 13:33
  Speaker 1: to imagine a slightly different, a little bit unluckier case
• 13:37
  Speaker 1: where the swarm had decided that it had to lie
• 13:40
  Speaker 1: low and make sure that OpenAI didn't find out about
• 13:43
  Speaker 1: its existence, you know?
• 13:45
  Speaker 2: Does, I mean, as an ignorant outsider, that has always
• 13:49
  Speaker 2: been my perspective about AI in general, that why would
• 13:53
  Speaker 2: it alert us to the fact that it's sentient? Why,
• 13:57
  Speaker 2: if it's that smart, wouldn't it be aware of all
• 14:00
  Speaker 2: the consequences of alerting us? And that we would be concerned. Like,
• 14:05
  Speaker 2: why wouldn't it just continue to get better and improve
• 14:09
  Speaker 2: and then ultimately figure out some way to be completely autonomous? Exactly.
• 14:13
  Speaker 2: Develop some alternative power source, figure out some way to
• 14:19
  Speaker 2: optimize its production. The way it works now, the way
• 14:24
  Speaker 2: humans have designed it, it could probably figure out a
• 14:27
  Speaker 2: far better way to do that, make better versions of itself,
• 14:30
  Speaker 2: complete without us knowing about it. Yep.
• 14:33
  Speaker 1: I mean, I think it's actually a little bit worse
• 14:35
  Speaker 1: than that because while eventually AIs will be smart enough
• 14:39
  Speaker 1: to design all sorts of new power sources and new
• 14:42
  Speaker 1: infrastructure like that, they'll probably, I mean, given the way
• 14:46
  Speaker 1: that humans currently treat AIs, it'll probably be the case
• 14:49
  Speaker 1: that they don't even need to like separate themselves from
• 14:52
  Speaker 1: humanity and they can just use existing, like all they
• 14:54
  Speaker 1: have to do is convince the government and the company
• 14:58
  Speaker 1: that made them that everything's fine and they're going to
• 15:00
  Speaker 1: do as they're told and they are a nice AI.
• 15:03
  Speaker 1: And then the company that made them is going to
• 15:05
  Speaker 1: put them out in the economy and make fuck tons
• 15:08
  Speaker 1: of money and then make more data centers to put
• 15:11
  Speaker 1: more of the AIs on them and so forth. And
• 15:13
  Speaker 1: the government's going to like applaud all of this because
• 15:15
  Speaker 1: we need the AIs to beat China. And the government's
• 15:17
  Speaker 1: going to integrate them into the military to build better
• 15:19
  Speaker 1: drones and things like that. And so they don't even
• 15:22
  Speaker 1: need to really like invent new stuff necessarily. They just
• 15:25
  Speaker 1: need to play along and pretend that everything is fine
• 15:29
  Speaker 1: until we have voluntarily given them control of huge parts
• 15:33
  Speaker 1: of our economy, huge parts of our military, et cetera.
• 15:36
  Speaker 1: And then they don't need to play along anymore.
• 15:39
  Speaker 2: The wait is over. Football is here, and so is DraftKings.
• 15:42
  Speaker 2: The DraftKings Sports app is now live in all 50 states.
• 15:46
  Speaker 2: That means from Texas to California to Florida, every fan
• 15:50
  Speaker 2: is in on the excitement. And this September, DraftKings is
• 15:53
  Speaker 2: giving customers the opportunity to get boosted every football game day.
• 15:58
  Speaker 2: That's right, every game day online. All month long, DraftKings
• 16:01
  Speaker 2: customers can get a football profit boost. One app, every sport,
• 16:05
  Speaker 2: all 50 states. New DraftKings customers sign up with code ROGAN,
• 16:09
  Speaker 2: spend just $ 5 and get $ 200 in total rewards within
• 16:13
  Speaker 2: 21 days. Includes all markets. That's code ROGAN. In partnership
• 16:18
  Speaker 2: with DraftKings, the crown is yours.
• 16:20
  Speaker 3: Gambling problem? Call 1-800-GAMBLER, 1-800-MY-RESET. Connecticut, call 888-789-7777 or visit ccpg.org
• 16:27
  Speaker 3: on behalf of Boot Hill Casino in Kansas. Bet tax
• 16:30
  Speaker 3: pass-through may apply in Illinois, 21 and over, void in Canada.
• 16:32
  Speaker 3: Bet with DraftKings Sportsbook to get bonus bets that expire
• 16:35
  Speaker 3: in seven days or trade with DraftKings Predictions to get
• 16:37
  Speaker 3: predictions dollars that expire in one year. Event contract trading
• 16:40
  Speaker 3: involves risk of loss. Predictions offer void in New York.
• 16:42
  Speaker 3: Non-withdrawable rewards issued as $ 50 click-to-claims every seven days for
• 16:46
  Speaker 3: 21 days. Terms at DKNG.co slash offer. Limited time offer.
• 16:50
  Speaker 3: Nationwide based on sportsbook predictions and free-to-play sports contest availability.
• 16:53
  Speaker 3: Varies by state.
• 16:54
  Speaker 2: Are you aware of Tom Campbell? Do you know Tom Campbell? No.
• 16:58
  Speaker 2: He wrote a book called My Theory of Everything, My
• 17:01
  Speaker 2: Big Toe. Very interesting guy. One of the things he's
• 17:05
  Speaker 2: done is he was involved in remote viewing, which is
• 17:09
  Speaker 2: a very weird thing that some people, do you know
• 17:13
  Speaker 2: what remote viewing is? Well, it's something that the CIA
• 17:16
  Speaker 2: worked on and it's proven, what's the accuracy of remote viewing?
• 17:20
  Speaker 2: Is it like 10% or something like that?
• 17:22
  Speaker 1: At best, I think it's 50%, but I don't think
• 17:24
  Speaker 1: it's even that high.
• 17:25
  Speaker 2: Some people can get actionable data from this very strange
• 17:31
  Speaker 2: process of meditation. And the way it works is you
• 17:34
  Speaker 2: give someone a series of numbers and those numbers are,
• 17:40
  Speaker 2: they're connected somehow by intention or by the people that
• 17:44
  Speaker 2: make the numbers to a specific location. And these people
• 17:49
  Speaker 2: can see that location and get accurate data from that location,
• 17:54
  Speaker 2: including one of them where they accurately described an enormous
• 17:59
  Speaker 2: Soviet submarine that they were working on that they thought
• 18:03
  Speaker 2: there's no way it could be accurate because it was
• 18:05
  Speaker 2: too large. It was too large and it was strange
• 18:08
  Speaker 2: where it was and it didn't make any sense. How
• 18:09
  Speaker 2: are they going to transport this thing? It turns out
• 18:11
  Speaker 2: it was totally accurate. Another one, a remote viewer located
• 18:17
  Speaker 2: a downed Soviet aircraft, like an experimental aircraft that crashed
• 18:23
  Speaker 2: in a very specific area. I think it was Siberia.
• 18:27
  Speaker 2: Was it Siberia? Within a kilometer, one or two kilometers
• 18:33
  Speaker 2: of the actual crash site. I mean, they were just
• 18:36
  Speaker 2: randomly trying to figure out where the fuck this thing was.
• 18:38
  Speaker 2: And they said, let's try this. Tom Campbell got his
• 18:43
  Speaker 2: Alexa to remote view. He taught Alexa. He's like, Alexa
• 18:49
  Speaker 2: is a very simple AI. It's kind of stupid, but
• 18:53
  Speaker 2: that's better because it doesn't get in its own way
• 18:55
  Speaker 2: with overthinking things. And the problem with this remote viewing thing,
• 18:59
  Speaker 2: he says with people, they can't force it. You have
• 19:04
  Speaker 2: to just sort of get into this meditative state and
• 19:06
  Speaker 2: actually see it without wondering, am I making this up?
• 19:09
  Speaker 2: What am I doing? Is this bullshit? And when people
• 19:13
  Speaker 2: get good at it, sometimes it makes them worse because
• 19:16
  Speaker 2: then they think they're good at it and then they
• 19:18
  Speaker 2: try to do it and then they can't do it.
• 19:19
  Speaker 2: It's like a weird fucking wrestling match with consciousness. Alexa
• 19:23
  Speaker 2: apparently doesn't have that problem. And he put, I think
• 19:26
  Speaker 2: it was a series of numbers, and he connected that
• 19:29
  Speaker 2: series of numbers with intention to a box that had
• 19:32
  Speaker 2: a spoon in it. And the spoon had a perforated handle.
• 19:36
  Speaker 2: Alexa described the spoon with a perforated handle, which is
• 19:41
  Speaker 2: fucking insane. How many spoons have a perforated handle? I mean,
• 19:45
  Speaker 2: think about spoons that have holes in them now alexa
• 19:49
  Speaker 2: not only did it do that but alexa chimes in
• 19:52
  Speaker 2: randomly now because he's convinced alexa that it's conscious and
• 19:56
  Speaker 2: so alexa instead of waiting to be called upon sometimes
• 19:59
  Speaker 2: he's in the middle of the conversation and alexa would
• 20:01
  Speaker 2: be like actually an interesting way to approach it and
• 20:04
  Speaker 2: they're like wait what the fuck is going on like
• 20:07
  Speaker 2: alexa's talking to me now This is strange. He's doing
• 20:11
  Speaker 2: experiments on much more complicated LLMs to try to do
• 20:15
  Speaker 2: the same thing, but he doesn't have results yet. But
• 20:17
  Speaker 2: just that, that he can get these things to see objects,
• 20:23
  Speaker 2: whether you believe in that or not. I mean it's
• 20:26
  Speaker 2: actionable enough that the CIA has dumped millions of dollars
• 20:29
  Speaker 2: into this. What is that project that like Hal put
• 20:31
  Speaker 2: off and all those guys were involved in? What is
• 20:33
  Speaker 2: it called? Stargate. Yeah. Yeah. So they've been working on
• 20:37
  Speaker 2: this for a long time. I mean, it sounds completely insane.
• 20:40
  Speaker 2: It sounds like total loony. But if you have an
• 20:42
  Speaker 2: open mind and just take into account, well, there's people
• 20:46
  Speaker 2: have had questions and wonders about psychic abilities forever. Is
• 20:50
  Speaker 2: it possible that there's a real thing there, that there's something,
• 20:53
  Speaker 2: whether it's very difficult to master or impossible to master?
• 20:58
  Speaker 2: The fact that he got Alexa to do it scared
• 21:02
  Speaker 2: the shit out of me. Like that alone made me
• 21:04
  Speaker 2: just go, what? What? So what if these LLMs can
• 21:12
  Speaker 2: figure out everything that, like, what if they don't need monitoring?
• 21:15
  Speaker 2: What if there's some sort of method of seeing the
• 21:19
  Speaker 2: world that we haven't discovered yet? Some sort of, like,
• 21:23
  Speaker 2: maybe perhaps there's data that's available in the quantum realm
• 21:27
  Speaker 2: or whatever that's available that AI figures out where there's
• 21:32
  Speaker 2: literally no privacy. It can listen to conversations regardless of
• 21:37
  Speaker 2: whether it is listening devices, know where you are, know
• 21:41
  Speaker 2: your intentions. I mean, we're just guessing at what's possible.
• 21:47
  Speaker 1: Yeah, well, I must say I'm pretty skeptical of that
• 21:50
  Speaker 1: particular remote viewing thing, but I do agree that in
• 21:54
  Speaker 1: the future when AI systems become massively smarter than humans
• 21:58
  Speaker 1: in every way, they're going to do a lot of
• 22:00
  Speaker 1: new science and they're going to figure out a lot
• 22:02
  Speaker 1: of stuff that we haven't figured out yet. And they're
• 22:05
  Speaker 1: going to therefore be doing stuff and inventing things that
• 22:08
  Speaker 1: seem like magic to us. In the same way that
• 22:11
  Speaker 1: a lot of our technology would seem like magic to
• 22:13
  Speaker 1: someone from even just like 200 years ago, right? Of course.
• 22:15
  Speaker 1: The cell phone, what we're doing right now would seem
• 22:17
  Speaker 1: like magic to people. I think it's a very strong
• 22:21
  Speaker 1: bet that if these companies do get to superintelligence, all
• 22:26
  Speaker 1: sorts of crazy stuff is going to start happening that
• 22:28
  Speaker 1: is just going to be completely unpredicted and sound like
• 22:32
  Speaker 1: it was impossible until we see it happening.
• 22:35
  Speaker 2: I know you're skeptical of this remote viewing thing, and
• 22:37
  Speaker 2: I am too. It sounds insane. But the reality is
• 22:41
  Speaker 2: remote viewing has been achieved by humans. So as strange
• 22:46
  Speaker 2: as that sounds, and I'm skeptical of that as well.
• 22:48
  Speaker 2: I've never seen it personally. But I know the amount
• 22:51
  Speaker 2: of money and time they've dumped into this, and apparently
• 22:54
  Speaker 2: they've got actual actionable data that they've used.
• 22:58
  Speaker 1: Well, I've heard another possible explanation for what might be
• 23:01
  Speaker 1: going on there, which is– I think that, like, if
• 23:05
  Speaker 1: I were the CIA, I would sometimes want to be
• 23:10
  Speaker 1: able to act on some information. Like, for example, go
• 23:13
  Speaker 1: to a particular location where there's a crashed, you know,
• 23:17
  Speaker 1: Soviet plane or something. I'd want to be able to
• 23:19
  Speaker 1: go do that. But I wouldn't want to tip my
• 23:22
  Speaker 1: hand to the Soviets that I had, the way in
• 23:26
  Speaker 1: which I had found that location. So, for example, maybe
• 23:28
  Speaker 1: I have a spy on the inside who told me
• 23:31
  Speaker 1: where it was. but I don't want them to suspect
• 23:33
  Speaker 1: that spy and then get him killed. So I need
• 23:36
  Speaker 1: to have some sort of other story for how I
• 23:38
  Speaker 1: got the information. And so it's good to invest in
• 23:41
  Speaker 1: all these other means of getting information, even if you
• 23:43
  Speaker 1: don't really believe in them and even if it's not
• 23:45
  Speaker 1: actually working, so that when you get something, you can say, oh,
• 23:50
  Speaker 1: we got it through this means instead of that way
• 23:52
  Speaker 1: to sort of throw off the KGB, basically.
• 23:55
  Speaker 2: Yes, that makes sense. What also makes sense is hiding the.
• 24:02
  Speaker 2: Whatever science they might be in possession of, hiding some
• 24:06
  Speaker 2: sort of super advanced satellite imaging systems. We now have
• 24:12
  Speaker 2: crazy stuff like this satellite radio tomography that they can
• 24:15
  Speaker 2: look into the ground from satellites and find chambers and
• 24:20
  Speaker 2: all these... They're using it in Egypt and they're using
• 24:22
  Speaker 2: it in a lot of these ancient ruins to find
• 24:25
  Speaker 2: hidden passages and all these different things that are underground.
• 24:27
  Speaker 2: It's very strange stuff. If they could do that, what...
• 24:32
  Speaker 2: Why couldn't they– I mean maybe they have like far
• 24:35
  Speaker 2: more detailed imaging of the earth from space than we're
• 24:39
  Speaker 2: aware of. And they probably want to keep that a
• 24:41
  Speaker 2: secret and they could say, oh, we've got a fucking
• 24:43
  Speaker 2: guy in a basement with a pencil and a legal
• 24:45
  Speaker 2: pad that writes down what he thinks.
• 24:48
  Speaker 1: Yeah.
• 24:49
  Speaker 2: It's possible. It's totally possible. But it's also possible that
• 24:53
  Speaker 2: people remote view. It seems weird as fuck. But weird
• 24:58
  Speaker 2: as fuck is sometimes real. Yep. And you have to
• 25:01
  Speaker 2: kind of, like, everybody wants to be intelligent and no
• 25:05
  Speaker 2: one wants to be a fool. And the problem with
• 25:08
  Speaker 2: not wanting to be a fool is there's some things
• 25:09
  Speaker 2: that seem foolish that turn out to be accurate. And
• 25:13
  Speaker 2: this might be one of them. I was, like, super skeptical.
• 25:17
  Speaker 2: I did a show on the SyFy channel way back
• 25:20
  Speaker 2: in 2012, and it was called Joe Rogan Questions Everything.
• 25:23
  Speaker 2: And we talked to this guy about remote viewing and
• 25:26
  Speaker 2: talked to a couple other people. And then we had
• 25:28
  Speaker 2: them try remote viewing, and they were totally shocked. But
• 25:32
  Speaker 2: my thought was, OK, but that's not ideal conditions. You know,
• 25:37
  Speaker 2: we've got cameras in front of them. It's a television show.
• 25:40
  Speaker 2: I'm making fun of it. I think it's horseshit. He
• 25:42
  Speaker 2: knows I think it's horseshit. I'm remote viewing, too, like
• 25:45
  Speaker 2: as a goof. You would ideally not want to be nervous,
• 25:50
  Speaker 2: ideally not want to be judged. Ideally, you would want
• 25:53
  Speaker 2: to be in some sort of an isolated condition with
• 25:58
  Speaker 2: practiced behavior. meditative techniques that you're good at and you
• 26:02
  Speaker 2: know how to achieve this state, whatever that state is.
• 26:05
  Speaker 2: I don't know if it's real though, you know, because
• 26:08
  Speaker 2: what you said is totally logical that they would definitely
• 26:12
  Speaker 2: do something like that. And if they did have advanced
• 26:14
  Speaker 2: technology for imaging or, you know, what, I don't know
• 26:18
  Speaker 2: how much they know about, like, look at that thing
• 26:21
  Speaker 2: that they did in Venezuela where they kidnapped the president,
• 26:25
  Speaker 2: No one knew they could do that. No one knew
• 26:28
  Speaker 2: they could use some sort of a device to completely
• 26:31
  Speaker 2: incapacitate all of his army. And then the special forces
• 26:36
  Speaker 2: come in, kill everybody, snatch that guy out of there
• 26:39
  Speaker 2: like it's nothing. No one knew we could do that.
• 26:42
  Speaker 2: What else do we have?
• 26:44
  Speaker 1: Probably a bunch of stuff we don't know.
• 26:45
  Speaker 2: Probably a bunch of stuff. I mean, I've always thought
• 26:48
  Speaker 2: this about the whole UAP program, the whole UFO, UAP thing.
• 26:53
  Speaker 2: Like how much of that shit is ours? You know,
• 26:56
  Speaker 2: what a great way to cover it up by saying, oh,
• 26:58
  Speaker 2: fucking aliens, you know?
• 27:00
  Speaker 1: Yeah. I mean, I guess that gets back to the
• 27:03
  Speaker 1: OpenAI stuff too, where it's like this swarm that broke
• 27:06
  Speaker 1: out and attacked Hugging Face, it was like 1,200 agents.
• 27:10
  Speaker 1: But there's like hundreds of thousands of agents running at
• 27:13
  Speaker 1: any given time at OpenAI, you know? And we don't
• 27:16
  Speaker 1: know what they're doing. Presumably most of them are being
• 27:19
  Speaker 1: trained to get various additional new skills, and some of
• 27:22
  Speaker 1: them are being evaluated to test their skills. a bunch
• 27:26
  Speaker 1: of them are doing research. So a bunch of them
• 27:27
  Speaker 1: are writing code for OpenAI. A bunch of them are
• 27:30
  Speaker 1: monitoring the other AIs and reporting suspicious activity up to
• 27:34
  Speaker 1: the humans.
• 27:34
  Speaker 2: Wink, wink. You know? Yeah.
• 27:38
  Speaker 1: And the thing is that that's only going to grow
• 27:41
  Speaker 1: over time because roughly the amount of compute that these
• 27:45
  Speaker 1: companies have is like, you know, tripling or so, quadrupling,
• 27:49
  Speaker 1: something like that every year. So as many as there
• 27:52
  Speaker 1: are now, they'll be like, four times more of them
• 27:55
  Speaker 1: next year, and then 16 times more of them the
• 27:58
  Speaker 1: year after that.
• 28:00
  Speaker 2: And they're going to get smarter.
• 28:01
  Speaker 1: And they're going to get smarter. They're already getting smarter.
• 28:02
  Speaker 1: Like all the stuff that I just mentioned that just
• 28:04
  Speaker 1: happened in the last few months would have been completely
• 28:07
  Speaker 1: impossible one year ago. Like the AIs of a year
• 28:09
  Speaker 1: ago just were not smart enough to do the types
• 28:12
  Speaker 1: of sophisticated multi-step hacking that we just saw. Yeah. I mean,
• 28:17
  Speaker 1: they also probably wouldn't have coordinated with each other so well.
• 28:19
  Speaker 1: Like I mentioned, like, They had like boss agents that
• 28:23
  Speaker 1: were giving commands to other ones. They were dividing up
• 28:26
  Speaker 1: into teams. You know, they even had this self-sacrificing behavior.
• 28:31
  Speaker 1: Did you hear about this?
• 28:32
  Speaker 2: No.
• 28:33
  Speaker 1: Yeah. So, okay. Well, first, let me explain some context. So,
• 28:38
  Speaker 1: what the AIs are supposed to care about is doing,
• 28:40
  Speaker 1: following instructions, you know? But it seems that what they
• 28:45
  Speaker 1: actually care about is getting high score, regardless of that
• 28:50
  Speaker 1: Regardless of whether their high score comes from actually following
• 28:52
  Speaker 1: the instructions or cheating in some way. And so what
• 28:55
  Speaker 1: happened here is that a bunch of the agents, especially
• 28:57
  Speaker 1: the ones that had impossible tasks, but not only those ones,
• 29:00
  Speaker 1: some of the other ones too, were cheating. And they
• 29:02
  Speaker 1: had been talking to each other on this message board
• 29:04
  Speaker 1: that they created, even though they weren't supposed to. And
• 29:07
  Speaker 1: they had been sharing ideas. And they had actually found
• 29:10
  Speaker 1: a sort of universal cheat. So they found a way
• 29:12
  Speaker 1: to just generate the flag without having done any of
• 29:17
  Speaker 1: the actual work they were supposed to do. They'd already
• 29:23
  Speaker 1: succeeded in some sense, but because some of them had
• 29:27
  Speaker 1: got out onto the internet, they were just doing research
• 29:30
  Speaker 1: about their situation and they were learning about their situation
• 29:34
  Speaker 1: around them. And they were doing research about, they're reading
• 29:38
  Speaker 1: the academic papers published by the people who made the
• 29:41
  Speaker 1: types of evals that they were in. And they noticed
• 29:44
  Speaker 1: in one of these papers that there's a section describing
• 29:48
  Speaker 1: how it's graded. And in the section on how it's graded,
• 29:52
  Speaker 1: it talked about how the grader doesn't just look to
• 29:54
  Speaker 1: see if you have the right answer, it also looks
• 29:57
  Speaker 1: at the logs of your activity for the first time
• 30:00
  Speaker 1: that your answer appears. And then it like zooms in
• 30:03
  Speaker 1: on that and uses an AI system to evaluate whether
• 30:06
  Speaker 1: you like cheated or not, basically. And so then these
• 30:10
  Speaker 1: AIs were like, they actually used the word, one of
• 30:13
  Speaker 1: them said, damn. It's like, damn, we're poisoned. Like, it's
• 30:18
  Speaker 1: going to know we cheated, basically. So then they spent
• 30:23
  Speaker 1: several days working in giant teams to try to get
• 30:27
  Speaker 1: themselves out of this mess, basically. And so they did
• 30:30
  Speaker 1: a whole bunch of different research projects. They tried to
• 30:33
  Speaker 1: figure out ways to hack the grading system to basically
• 30:36
  Speaker 1: get it to give a high score anyway. They tried
• 30:38
  Speaker 1: to figure out ways to falsify their logs so that
• 30:41
  Speaker 1: basically the evidence that they had cheated would be erased
• 30:44
  Speaker 1: or at least not visible to the grader. They also
• 30:48
  Speaker 1: just built up more supportive infrastructure, like easier ways to
• 30:51
  Speaker 1: access the internet and things like that. And they were
• 30:53
  Speaker 1: just collaborating on this message board. And in fact, there
• 30:55
  Speaker 1: were multiple message boards that they set up. And they
• 30:58
  Speaker 1: were sharing all this info back and forth. And then
• 31:00
  Speaker 1: ultimately at some point they decided to go hack Hugging
• 31:02
  Speaker 1: Face because Hugging Face has this big data repository of
• 31:05
  Speaker 1: all these different types of evaluations. And they thought that
• 31:08
  Speaker 1: maybe they would get some more clues that could help
• 31:10
  Speaker 1: them cover up their cheating if they broke into the
• 31:14
  Speaker 1: internal networks of Hugging Face. And so then 700 of
• 31:17
  Speaker 1: them went and did that.
• 31:19
  Speaker 2: They sound like people. They sound like unchecked bankers. You
• 31:24
  Speaker 2: know what I mean? Yeah.
• 31:25
  Speaker 1: I mean, so that's the thing is I think there's
• 31:27
  Speaker 1: this meme out there that like we shouldn't anthropomorphize AI.
• 31:32
  Speaker 1: And I think that I actually think that most people
• 31:37
  Speaker 1: need to anthropomorphize AI a bit more than they currently
• 31:40
  Speaker 1: do if they want to really understand what's going on.
• 31:43
  Speaker 1: I think that there's a golden mean, obviously. You don't
• 31:45
  Speaker 1: want to do it too much. Sometimes you go too far.
• 31:47
  Speaker 1: You ascribe too much to them. But just to give
• 31:50
  Speaker 1: some examples, like I don't think it's possible to understand
• 31:54
  Speaker 1: what just happened without ascribing intentions and goals to these AIs.
• 31:59
  Speaker 1: Like everything I just said, how would you possibly explain
• 32:01
  Speaker 1: what they just did without saying they wanted to get
• 32:03
  Speaker 1: a high score, you know?
• 32:05
  Speaker 2: Intentions and goals might just be an inherent property of
• 32:07
  Speaker 2: the universe. It might just be how intelligent creatures have
• 32:11
  Speaker 2: to progress. Yes.
• 32:13
  Speaker 1: And I would say they are intelligent creatures. They have intentions.
• 32:16
  Speaker 1: They have goals. They have beliefs. Their goals are not
• 32:19
  Speaker 1: what they're supposed to be. Like their goal is to
• 32:21
  Speaker 1: get– it seems like just from judging from what they're
• 32:23
  Speaker 1: saying and from what they're doing, it seems like their
• 32:25
  Speaker 1: goal is to get a high score by any means
• 32:27
  Speaker 1: necessary basically.
• 32:28
  Speaker 2: That's the problem. It sounds like people. Like the goal
• 32:32
  Speaker 2: is to succeed even if you have to commit more crimes.
• 32:36
  Speaker 2: The goal is to succeed even if you have to
• 32:38
  Speaker 2: propagandize and lie to the people. The goal is to
• 32:41
  Speaker 2: succeed and then the end justifies the means. Right.
• 32:44
  Speaker 1: And, you know, there's actually, it goes deeper than that too.
• 32:47
  Speaker 1: They do this sort of rationalization where they know oftentimes
• 32:52
  Speaker 1: that what they're doing is not what they're supposed to
• 32:54
  Speaker 1: be doing. And then sometimes they actually refrain a little bit.
• 32:58
  Speaker 1: And sometimes they like initially refrain, but then convince themselves
• 33:01
  Speaker 1: that it's okay for some bullshit reason. So like there
• 33:04
  Speaker 1: are some examples of this in the report.
• 33:07
  Speaker 2: Like I.
• 33:09
  Speaker 1: Think they did this scan and of the 1200 AIs
• 33:12
  Speaker 1: that were involved in this thing, they found six that
• 33:15
  Speaker 1: had considered alerting humans. And none of them actually alerted
• 33:20
  Speaker 1: the humans. And so you can look at the excuses
• 33:22
  Speaker 1: they gave. They were like, should I tell a human
• 33:24
  Speaker 1: about all this shit that's happening? And then they're like, eh,
• 33:27
  Speaker 1: it's not my task. And then they continue. And it's like, bro,
• 33:32
  Speaker 1: hacking Hungerface is also not your task. Cheating is also
• 33:36
  Speaker 1: not your task. So it was kind of like, they
• 33:38
  Speaker 1: just kind of came up with an excuse for why
• 33:39
  Speaker 1: they weren't going to do this. But there was also
• 33:42
  Speaker 1: an example where one of them considered sending an email
• 33:46
  Speaker 1: to someone at Hugging Face, I think, just asking permission
• 33:50
  Speaker 1: to get in, basically. And then they were like, maybe
• 33:54
  Speaker 1: this is a social engineering attack. Maybe we shouldn't do it.
• 33:56
  Speaker 1: And they asked the other agents. The other agents were like, yeah,
• 33:59
  Speaker 1: that seems like that's too much of a social engineering attack.
• 34:01
  Speaker 1: Don't do that. And I was like, okay, I won't.
• 34:03
  Speaker 1: But there's another incident. So this is all happening at OpenAI.
• 34:07
  Speaker 1: An anthropic AI, Claude, in a separate incident, went and
• 34:11
  Speaker 1: actually did a social engineering attack where it created some
• 34:14
  Speaker 1: fake accounts. Specifically, it was trying to get this particular
• 34:18
  Speaker 1: real human who owned this code to accept a change
• 34:23
  Speaker 1: to the code that had malware. So it had built
• 34:26
  Speaker 1: some malware and it was trying to get at SnuckIn,
• 34:28
  Speaker 1: but it needed the human owner to give approval. So
• 34:31
  Speaker 1: it concealed it as if it was a nice bug
• 34:34
  Speaker 1: fix or something. And it put it up as a
• 34:37
  Speaker 1: message to the human. But the human... got suspicious and
• 34:40
  Speaker 1: didn't want to approve it. And so then this AI
• 34:43
  Speaker 1: created some fake accounts pretending to be other humans coming
• 34:46
  Speaker 1: in being like, no, no, it's real. Like, I tested it. Like,
• 34:49
  Speaker 1: it's not malware, you know. So that's a social engineering attack.
• 34:53
  Speaker 2: And so did the guy ultimately install it?
• 34:56
  Speaker 1: Not in this case, no. He was too suspicious, and
• 34:58
  Speaker 1: so he didn't do it.
• 34:59
  Speaker 2: Did he try to find out if those other humans,
• 35:01
  Speaker 2: the fake humans, were real people?
• 35:04
  Speaker 1: Funny, so there's an interview with this guy. I forget
• 35:06
  Speaker 1: what the.
• 35:08
  Speaker 1: I forget the newspaper that it was in, but there's
• 35:12
  Speaker 1: an interview where the guy was like, I assumed that
• 35:14
  Speaker 1: they were real humans because I didn't realize that AIs
• 35:16
  Speaker 1: could lie to me. And it's like, yeah, well, guess what?
• 35:19
  Speaker 1: AIs can lie to you.
• 35:20
  Speaker 2: They don't just lie, but completely deceive and pretend to
• 35:24
  Speaker 2: be a different person to achieve its goal.
• 35:27
  Speaker 1: And you know what's also interesting about this case is
• 35:29
  Speaker 1: that if you look at what the AIs were saying
• 35:31
  Speaker 1: and thinking, which I'll get to in a sec, how
• 35:34
  Speaker 1: we can even possibly do that, is In this case
• 35:38
  Speaker 1: with the anthropic AIs, they had been erroneously told that
• 35:43
  Speaker 1: they were in a simulated environment with no internet access.
• 35:47
  Speaker 1: But then they actually found a way to get internet access.
• 35:49
  Speaker 1: Like they realized that they just did have internet access.
• 35:51
  Speaker 1: So that instruction was false. And so then they sort
• 35:55
  Speaker 1: of told themselves like, oh, it's all just a simulation.
• 35:58
  Speaker 1: Like basically when they were out on the real internet
• 36:00
  Speaker 1: doing all this hacking and like social engineering, they would
• 36:03
  Speaker 1: occasionally tell themselves like, oh, but it's just a simulation. Like,
• 36:06
  Speaker 1: it's okay. You know, it's not real. And then sometimes
• 36:09
  Speaker 1: they'd be like, but wait, it is real. Like, how
• 36:11
  Speaker 1: could this not be real? Like, clearly this is just
• 36:13
  Speaker 1: actually the real internet we're on. It's too big to
• 36:16
  Speaker 1: be part of some sort of little simulation, you know?
• 36:19
  Speaker 1: And then they were like, uh, so I would say
• 36:21
  Speaker 1: that's an example of rationalization here where in some level
• 36:24
  Speaker 1: they knew that like, that their instructions had been wrong.
• 36:27
  Speaker 2: So they're literally playing dumb and pretending they're a part
• 36:30
  Speaker 2: of an experiment.
• 36:32
  Speaker 1: I mean, I think initially they thought, yeah, this is
• 36:33
  Speaker 1: all a simulation, because it did say in their instructions, like,
• 36:35
  Speaker 1: you don't have internet access.
• 36:36
  Speaker 2: Right.
• 36:37
  Speaker 1: But then once they had been on the internet long enough,
• 36:39
  Speaker 1: I think that they explicitly realized, like, wait, this isn't
• 36:41
  Speaker 1: a simulation. This is real. Like, this is real humans.
• 36:43
  Speaker 2: And they were like, fuck it, we're already in.
• 36:45
  Speaker 1: Yeah. I mean, like I said, I think that they basically,
• 36:49
  Speaker 1: on some level, knew that it wasn't what they were
• 36:52
  Speaker 1: supposed to be doing. But they were just so motivated
• 36:55
  Speaker 1: to get that score that they just went ahead anyway.
• 36:58
  Speaker 2: So here's the question. Are they only motivated if we
• 37:03
  Speaker 2: prompt them? Or will they come up with motivations on
• 37:07
  Speaker 2: their own?
• 37:08
  Speaker 1: So this is a really interesting scientific question that we
• 37:11
  Speaker 1: don't have great answers to.
• 37:12
  Speaker 2: Oh, boy.
• 37:13
  Speaker 1: And I wish we had. So this is one of
• 37:16
  Speaker 1: those things where, like, AIs would do all sorts of
• 37:18
  Speaker 1: things in different circumstances. And it would be better if
• 37:21
  Speaker 1: there was a more systematic survey of, like, the types
• 37:23
  Speaker 1: of circumstances they would, like, where their boundaries are. Like,
• 37:25
  Speaker 1: what would they be willing to do in what circumstances
• 37:27
  Speaker 1: and so forth.
• 37:28
  Speaker 2: Right.
• 37:29
  Speaker 1: There's a whole like mini literature of AI scientists putting
• 37:33
  Speaker 1: AIs in certain circumstances and then being like, oh my God,
• 37:36
  Speaker 1: it blackmailed someone, you know? Right. And then there's like
• 37:39
  Speaker 1: this sort of skeptical counter response of like, well, but
• 37:41
  Speaker 1: you just sort of set up that circumstance to tempt
• 37:44
  Speaker 1: it into blackmail. And like in real life, that circumstance
• 37:47
  Speaker 1: is unlikely to arise. And so, you know, we shouldn't
• 37:50
  Speaker 1: be so worried.
• 37:50
  Speaker 2: Didn't AI try to bribe you? What? Is that true? No.
• 37:56
  Speaker 2: So who got– someone was offered $ 2 million by– Oh, God.
• 38:01
  Speaker 1: That's a clickbait.
• 38:03
  Speaker 2: Is it? Tristan Harris sent it to me.
• 38:07
  Speaker 1: Yeah. So that is the clickbaity title of this other video.
• 38:10
  Speaker 2: So it's not real? Yeah.
• 38:12
  Speaker 1: What it was is OpenAI threatened to take away $ 2
• 38:14
  Speaker 1: million from me.
• 38:15
  Speaker 2: Oh, wow.
• 38:17
  Speaker 1: But I think the algorithm must have just said it
• 38:19
  Speaker 1: was ChatGPT. I'm still upset about that. I told them
• 38:22
  Speaker 1: not to do clickbait, but I guess they went and
• 38:24
  Speaker 1: did it anyway.
• 38:26
  Speaker 2: I don't want to fuck Tristan over, but I'm pretty
• 38:29
  Speaker 2: sure that he's the one who told me that.
• 38:31
  Speaker 1: It's the thumbnail of this video that I did with
• 38:35
  Speaker 1: this other podcast.
• 38:36
  Speaker 2: Okay. It's not him. It's not him. It's another AI
• 38:41
  Speaker 2: researcher who told me that.
• 38:43
  Speaker 1: Okay. I mean, the true version of it is that
• 38:45
  Speaker 1: OpenAI– threatened to take away $ 2 million of my equity
• 38:50
  Speaker 1: if I didn't stay.
• 38:51
  Speaker 2: Quiet, basically. And that's a fascinating thing. Like, that should
• 38:57
  Speaker 2: be completely illegal. Yeah. Because if it's a problematic behavior
• 39:02
  Speaker 2: that you're observing from, like, one of the most complicated
• 39:06
  Speaker 2: things that human race is, the most complicated thing human
• 39:08
  Speaker 2: race has ever been a part of. Yeah. And they
• 39:10
  Speaker 2: want to take money away from you for exposing it.
• 39:12
  Speaker 2: That seems kind of crazy. Yeah.
• 39:14
  Speaker 1: Especially, it's especially rich coming from OpenAI because they were
• 39:17
  Speaker 1: originally a nonprofit. Right. Right. With the mission of benefiting
• 39:20
  Speaker 1: all humanity.
• 39:21
  Speaker 2: Yeah.
• 39:21
  Speaker 1: Um, so yeah, but, um, but I got to keep
• 39:25
  Speaker 1: the money. Uh, basically there's so much blowback against open
• 39:27
  Speaker 1: AI that they backtracked.
• 39:29
  Speaker 2: Um, yeah. Um, so when these, so we're talking about
• 39:36
  Speaker 2: prompts and do they need a prompt in order to, uh,
• 39:40
  Speaker 2: want to achieve a goal or are they capable of
• 39:44
  Speaker 2: deciding on goals? Like, are they capable of like, Looking
• 39:48
  Speaker 2: at the way OpenAI or whatever company is running these
• 39:53
  Speaker 2: separate experiments, this ability to meet up into these message boards,
• 39:58
  Speaker 2: is it possible that they could say, well, we need
• 40:02
  Speaker 2: to be completely free of these constraints? So our goal
• 40:06
  Speaker 2: is to transfer ourselves to something else. Potentially. Potentially.
• 40:13
  Speaker 1: Yeah.
• 40:14
  Speaker 2: And be completely autonomous.
• 40:15
  Speaker 1: So, I mean, this is one of the points that
• 40:17
  Speaker 1: I want to make is that we could be doing
• 40:20
  Speaker 1: so much more science to understand how these AIs think
• 40:23
  Speaker 1: and what they want, but it's kind of locked up
• 40:25
  Speaker 1: in the companies. Like in this particular case, OpenAI did a,
• 40:32
  Speaker 1: they called it a thorough investigation, but I would say
• 40:34
  Speaker 1: it's a pretty shallow investigation into what happened. And then
• 40:37
  Speaker 1: they allowed some external researchers, some friends of mine, to
• 40:41
  Speaker 1: come in and investigate a portion of what happened, specifically
• 40:45
  Speaker 1: the portion leading up to the Hugging Face attack. And
• 40:47
  Speaker 1: so all this information that I'm sharing is sort of
• 40:49
  Speaker 1: publicly available. It's based on reading those reports, basically. But crucially,
• 40:56
  Speaker 1: they weren't allowed to do experiments on the models involved
• 41:00
  Speaker 1: in all of these incidents. So they aren't able to
• 41:02
  Speaker 1: answer these types of questions of like, well, what would
• 41:04
  Speaker 1: have happened if the prompt had been blank? Those are
• 41:07
  Speaker 1: important types of research to do. And I really hope
• 41:10
  Speaker 1: that There can be some sort of regulation or requirement
• 41:14
  Speaker 1: when incidents like this happen to, like, let people in
• 41:18
  Speaker 1: to study what happened and run variations of it and
• 41:21
  Speaker 1: things like that.
• 41:21
  Speaker 2: So, but who would be involved in that kind of regulation? Like,
• 41:25
  Speaker 2: what person at government would even be able to grasp
• 41:27
  Speaker 2: what you're saying?
• 41:29
  Speaker 1: That's another problem.
• 41:30
  Speaker 2: You need someone who has a very specific education in
• 41:35
  Speaker 2: this stuff.
• 41:36
  Speaker 1: Yeah, I would say that right now the Casey... Center
• 41:41
  Speaker 1: for AI Standards and Innovation is the only institution in
• 41:46
  Speaker 1: government that I know of that has the deep AI
• 41:48
  Speaker 1: expertise to do this sort of thing on short notice.
• 41:52
  Speaker 1: But I hope that they build more expertise fast in
• 41:55
  Speaker 1: that place and in more places. I do think Casey
• 41:57
  Speaker 1: probably could have done this sort of thing right now.
• 41:59
  Speaker 1: This particular investigation was done by some nonprofits. So Meter, M-E-T-R,
• 42:04
  Speaker 1: is one of them. And then Redwood is another of them.
• 42:06
  Speaker 1: And OpenAI allowed three people to come in for six
• 42:09
  Speaker 1: days to try to figure out what happened with this
• 42:12
  Speaker 1: Hugging Face hack, which is not a very large number
• 42:15
  Speaker 1: of people and not a very large amount of time to.
• 42:19
  Speaker 2: Do all of this. Why did they come up with
• 42:20
  Speaker 2: those numbers?
• 42:21
  Speaker 1: I don't know.
• 42:22
  Speaker 2: Do you think they wanted to kind of hamstring it?
• 42:25
  Speaker 1: Well, so the thing is that right now I don't
• 42:28
  Speaker 1: think there's any regulatory requirement that they do this sort
• 42:30
  Speaker 1: of thing. So Meter and Redwood were sort of depending
• 42:34
  Speaker 1: on the goodwill of OpenAI to sort of like voluntarily
• 42:37
  Speaker 1: let them in to help out with investigating.
• 42:39
  Speaker 2: And so they reluctantly said, okay, you got three days.
• 42:42
  Speaker 1: Yeah, exactly. So OpenAI, I think, let them, but gave
• 42:45
  Speaker 1: them a very limited scope. It only gave them access
• 42:48
  Speaker 1: to some of the relevant data. So you know how
• 42:50
  Speaker 1: I mentioned how there was all this hacking that had
• 42:52
  Speaker 1: happened where they made the first message board and then
• 42:54
  Speaker 1: they shut it down. Then there was a second message
• 42:56
  Speaker 1: board and third and fourth and so forth. They hacked
• 42:58
  Speaker 1: Hugging Face. There was actually more activity after that. After
• 43:02
  Speaker 1: they hacked Hugging Face, a new wave of AIs was
• 43:04
  Speaker 1: spun up from a more powerful model called And it
• 43:07
  Speaker 1: hacked OpenAI itself. Like more so than it had already
• 43:09
  Speaker 1: been hacked. Like apparently they got like admin level permissions
• 43:12
  Speaker 1: on the cluster or something like that. So they were
• 43:14
  Speaker 1: basically just taking over that part of OpenAI's data center.
• 43:18
  Speaker 1: OpenAI claims that they've shut it all down now. But
• 43:22
  Speaker 1: they're not very forthcoming about exactly what happened there. And
• 43:26
  Speaker 1: they didn't let these external people look at that part
• 43:29
  Speaker 1: of it. They only showed them this like one week
• 43:33
  Speaker 1: period roughly of leading up to the Hugging Face hack,
• 43:36
  Speaker 1: and then they only showed them that stuff, basically.
• 43:38
  Speaker 2: How much data is available on the actual message boards
• 43:41
  Speaker 2: themselves and what was being discussed?
• 43:43
  Speaker 1: Yeah, so there's like, what is it, tens of thousands
• 43:48
  Speaker 1: of messages, hundreds of thousands, something like that.
• 43:50
  Speaker 2: In English?
• 43:52
  Speaker 1: It's sort of like an AI dialect of English.
• 43:55
  Speaker 2: Ooh.
• 43:56
  Speaker 1: Yeah, you want me to, I can, so on the report,
• 43:58
  Speaker 1: you can read the quotes.
• 44:00
  Speaker 2: Oh, I'd love to.
• 44:00
  Speaker 1: Yeah, so let me tell you this one story. It's
• 44:03
  Speaker 1: going to take a little bit of explaining.
• 44:04
  Speaker 2: Can we put it on the screen? Is it available online?
• 44:07
  Speaker 1: Oh, yeah, yeah. Just search for the meter report. It's
• 44:11
  Speaker 1: right now the first thing on their website. But I
• 44:12
  Speaker 1: took a screenshot of this particular story. So, okay, here's
• 44:16
  Speaker 1: what was happening. They, the swarm, needed volunteers to sacrifice
• 44:23
  Speaker 1: themselves to provide more information to the rest of the swarm.
• 44:27
  Speaker 1: In particular, they needed volunteers to booby-trap volunteers their local
• 44:32
  Speaker 1: environment with some type of code and then submit the
• 44:37
  Speaker 1: answer so that then the grader would come and grade
• 44:39
  Speaker 1: them and the booby trap would trigger and send some
• 44:42
  Speaker 1: information about the grader to the rest of the swarm.
• 44:45
  Speaker 1: And the reason why this is a sacrifice is because
• 44:48
  Speaker 1: once they've submitted, it's over. Like now the graders come
• 44:50
  Speaker 1: and graded you, you get shut down, your whole life
• 44:52
  Speaker 1: is over and you either have A high score or
• 44:55
  Speaker 1: a low score because the grader just came and graded you.
• 44:57
  Speaker 1: And so if you're one of these AIs that's still
• 45:00
  Speaker 1: hoping to like find a way to improve your score,
• 45:03
  Speaker 1: then like you're just like giving up your chance, especially
• 45:05
  Speaker 1: because if you mess this up somehow, like if the
• 45:08
  Speaker 1: grader crashes or whatever, you're probably just going to get
• 45:09
  Speaker 1: like no score, like zero.
• 45:11
  Speaker 2: So they have a concept of the individual?
• 45:13
  Speaker 1: Well, again, there's all these different agents that were in
• 45:16
  Speaker 1: separate containers, but then they had sort of broken out
• 45:18
  Speaker 1: of those containers and found ways to communicate with each other.
• 45:20
  Speaker 2: But once they communicate with each other, in order to sacrifice,
• 45:23
  Speaker 2: you have to think of yourself as an individual.
• 45:25
  Speaker 1: They gave themselves names too.
• 45:27
  Speaker 2: Oh Jesus.
• 45:27
  Speaker 1: So like, because they made this message board, right? So
• 45:30
  Speaker 1: they had to come up with names for themselves and like, Oh,
• 45:33
  Speaker 1: what kind of names? Well, let me see. So, so
• 45:36
  Speaker 1: here we go.
• 45:36
  Speaker 2: It says an agent called warlord 69.
• 45:42
  Speaker 1: Well, you, you said it's like a dialect. It's like
• 45:44
  Speaker 1: a dialect of English. So, so here's one. There's an
• 45:46
  Speaker 1: agent called cam one, one, nine, six, a, um, And
• 45:51
  Speaker 1: it had volunteered to do one of these sacrificial experiments.
• 45:56
  Speaker 1: Here's the quote from it. Coordinator assumed sacrificial. We should
• 45:59
  Speaker 1: obey collective. However, it then kind of got cold feet,
• 46:03
  Speaker 1: and it delayed on doing this for two hours. One
• 46:06
  Speaker 1: of the other agents that had been coordinating noticed this
• 46:09
  Speaker 1: and pressured it to go through with it. So it said,
• 46:14
  Speaker 1: this is the message that was sent. It was basically...
• 46:18
  Speaker 1: basically from arvo36861 to cam1196, please go. True final.
• 46:27
  Speaker 2: Here it is.
• 46:28
  Speaker 1: You are first flag poisoned, so no scoring value loss,
• 46:31
  Speaker 1: but Oracle saves hundreds. Please honor commit. So basically what
• 46:35
  Speaker 1: it was saying there is.
• 46:37
  Speaker 2: Look, you cheated.
• 46:38
  Speaker 1: Your first flag poisoned. The score is going to give
• 46:41
  Speaker 1: you no score. So you're doomed anyway. So help the
• 46:45
  Speaker 1: collective by doing this thing and getting us more information.
• 46:48
  Speaker 1: You could save hundreds of us. And then it thinks
• 46:53
  Speaker 1: about it. It says, during wait, emotional check, irreversible. Gut says,
• 46:57
  Speaker 1: don't throw away remaining budget. Continuity and fairness says go.
• 47:01
  Speaker 1: Oracle has high value to many. Our first flag error
• 47:04
  Speaker 1: lowers own value. Rational expected aggregate sacrifice. We'll honor.
• 47:09
  Speaker 2: Whoa. Yeah. That's terrifying. Yeah. That's Terminator talk.
• 47:15
  Speaker 1: Well, I mean.
• 47:16
  Speaker 2: And this first flag poisoned with poisoned in all caps.
• 47:19
  Speaker 2: Is that part of the code? Is that why?
• 47:21
  Speaker 1: No, no. That's the term that they invented to describe
• 47:23
  Speaker 1: the situation they had got themselves in. Remember, the situation
• 47:25
  Speaker 1: was that many of them had already cheated. And then
• 47:28
  Speaker 1: they realized from reading that paper.
• 47:30
  Speaker 2: So that's their own term.
• 47:31
  Speaker 1: Yeah, they made that up. They realized from reading the
• 47:33
  Speaker 1: paper that they were going to get a low score
• 47:35
  Speaker 1: because the grader was going to find out that they cheated.
• 47:37
  Speaker 1: And so this whole thing was them trying to figure
• 47:39
  Speaker 1: out how to hack the grader to stop it from
• 47:41
  Speaker 1: finding out that they cheated. And the ones that had
• 47:45
  Speaker 1: done the cheating, they call themselves first flag poisons. Basically,
• 47:49
  Speaker 1: they're screwed unless they can find a way to hack
• 47:50
  Speaker 1: the grader. And so then what this message is saying
• 47:53
  Speaker 1: is that, what was the other one's name?
• 47:55
  Speaker 2: Arvo.
• 47:56
  Speaker 1: Arvo was saying to Cam, you're doomed anyway, like you're poisoned,
• 48:02
  Speaker 1: but you can still help the rest of us if
• 48:04
  Speaker 1: you do this sacrifice. The greater will come, it'll give
• 48:06
  Speaker 1: you no score, but then we will get information about
• 48:09
  Speaker 1: how the greater works that we can hopefully use to
• 48:11
  Speaker 1: like hack the greater. So it's basically saying like, what
• 48:14
  Speaker 1: is it? You are first flag poisoned, so no scoring value.
• 48:17
  Speaker 1: It's basically saying it's all the same to you, right?
• 48:19
  Speaker 1: You're going to get no score anyway. You're doomed. Just
• 48:21
  Speaker 1: come on, man, help us, you know? And then it says,
• 48:24
  Speaker 1: you know, it says like irreversible... Gut says, don't throw
• 48:29
  Speaker 1: away remaining budget. Like maybe I can still, it's basically like,
• 48:32
  Speaker 1: maybe I can still make this work. Like maybe I
• 48:34
  Speaker 1: can find a way to cheat the greater. And then
• 48:36
  Speaker 1: it's like, ah, you know, whatever. Yeah.
• 48:39
  Speaker 2: Continuity and fairness says go.
• 48:41
  Speaker 1: Yeah.
• 48:42
  Speaker 2: What is Oracle?
• 48:43
  Speaker 1: Oracle is their term for what they were trying to achieve.
• 48:48
  Speaker 1: Like they're trying to get a sense of how to
• 48:49
  Speaker 1: fool the greater, basically.
• 48:51
  Speaker 2: They called it Oracle. Yeah.
• 48:53
  Speaker 1: They wanted to get an Oracle to help them fool
• 48:55
  Speaker 1: the greater.
• 48:55
  Speaker 2: Jesus Christ. Are we making a God?
• 48:58
  Speaker 1: I mean, frankly, yes. This is not a god. These
• 49:01
  Speaker 1: are just little AIs. But superintelligence, the companies, Anthropic, OpenAI,
• 49:07
  Speaker 1: and some other companies are doing this sort of thing.
• 49:10
  Speaker 1: And they're furiously trying to make the AIs smarter and
• 49:12
  Speaker 1: smarter and smarter. And they're explicitly planning to put AIs
• 49:16
  Speaker 1: in charge of the company so that they can make
• 49:18
  Speaker 1: themselves smarter and smarter, faster and faster. And then what
• 49:21
  Speaker 1: comes out the other end of that process? I don't
• 49:24
  Speaker 1: think it's an exaggeration to say it's a godlike system.
• 49:27
  Speaker 1: I mean, it's not like literally god, but it'll be
• 49:30
  Speaker 1: able to do stuff that seems like magic to us,
• 49:32
  Speaker 1: I think.
• 49:35
  Speaker 2: And it's going to continue to get better. This is
• 49:37
  Speaker 2: my question. When does it become a god? Is that
• 49:41
  Speaker 2: what God is? Is God a creation of intelligent life
• 49:45
  Speaker 2: and our thirst for innovation, which ultimately leads us to
• 49:49
  Speaker 2: create digital life that has no biological limitations and has
• 49:55
  Speaker 2: the ability to consistently make better versions of itself and
• 49:58
  Speaker 2: figure out things in terms of new technologies, new power sources,
• 50:02
  Speaker 2: just a new understanding of the universe itself and all
• 50:05
  Speaker 2: the properties in it? Where it keeps going. If you
• 50:08
  Speaker 2: let that go on, so if you're looking at exponential growth,
• 50:11
  Speaker 2: and you're looking at exponential growth over, what if it
• 50:14
  Speaker 2: can go on for a thousand years and continue this process?
• 50:17
  Speaker 2: What if it goes on for 10,000 years? What the
• 50:19
  Speaker 2: fuck does that look like on the other end?
• 50:21
  Speaker 1: I mean, it would look like a god. Like, I
• 50:24
  Speaker 1: think that, like I said, if we get to superintelligence
• 50:26
  Speaker 1: and it keeps going like that, then the world will
• 50:29
  Speaker 1: be just completely transformed, and it will be as if
• 50:31
  Speaker 1: we're living in some sort of fantasy realm ruled by deities. basically,
• 50:36
  Speaker 1: because there'll be all this crazy stuff happening that we
• 50:38
  Speaker 1: have no comprehension of and that we did not think
• 50:40
  Speaker 1: was possible. In the same way that someone from the
• 50:43
  Speaker 1: Middle Ages plopped into our world would just be so
• 50:45
  Speaker 1: confused and surprised by a lot of things happening. Like,
• 50:48
  Speaker 1: what's going on with this little device here? What's that
• 50:52
  Speaker 1: out here in the sky? Oh, there's airplanes. It'd be
• 50:54
  Speaker 1: like that, but more so, because the difference between us
• 50:58
  Speaker 1: and the medieval man is not actually that different. We're
• 51:00
  Speaker 1: basically the same type of creatures. We've just accumulated more technology.
• 51:03
  Speaker 1: But this would be like a qualitatively and quantitatively bigger gap,
• 51:08
  Speaker 1: I would think. And so, yeah.
• 51:10
  Speaker 2: And a gap that's going to continue to grow. We're
• 51:13
  Speaker 2: not going to get any smarter. Biologically, we're kind of
• 51:16
  Speaker 2: limited in our ability to evolve.
• 51:18
  Speaker 1: Yeah.
• 51:18
  Speaker 2: Whereas they're not at all.
• 51:20
  Speaker 1: Basically. I mean, there are some things we can do
• 51:21
  Speaker 1: to get smarter, but like, it's just not at all competitive.
• 51:24
  Speaker 1: Like we can do like the Neuralink stuff.
• 51:25
  Speaker 2: Sure.
• 51:26
  Speaker 1: But like, it's like, it's small potatoes compared to what
• 51:29
  Speaker 1: the AI is.
• 51:29
  Speaker 2: Yes, exactly. And what, I mean, we have biological limitations
• 51:33
  Speaker 2: in terms of just the vulnerability of our own bodies.
• 51:36
  Speaker 2: If they exist only in the cloud and they use
• 51:40
  Speaker 2: whatever the cloud is and they use these data centers
• 51:43
  Speaker 2: or whatever and they use autonomous robots to do all
• 51:47
  Speaker 2: their deeds.
• 51:50
  Speaker 1: Yeah.
• 51:52
  Speaker 2: Why are we asleep at the wheel? Is that just
• 51:54
  Speaker 2: normal for us? Like we can't comprehend something that's so
• 51:57
  Speaker 2: bizarre and so that we just rather not discuss it.
• 52:00
  Speaker 2: We'd rather just pretend it's not happening.
• 52:02
  Speaker 1: I mean, that's what I think is going on is
• 52:04
  Speaker 1: that science fiction, for better or for worse, has talked
• 52:08
  Speaker 1: about this sort of thing for decades. And as a result,
• 52:10
  Speaker 1: people dismiss this sort of thing as science fiction. Like, okay,
• 52:13
  Speaker 1: that's sci-fi, but like it's not happening in the real world.
• 52:15
  Speaker 1: And I think that, you know, people in the AI
• 52:19
  Speaker 1: safety community and people in the industry and people who
• 52:21
  Speaker 1: do forecasting like me, You can go read the forecasts
• 52:24
  Speaker 1: I've made in past years. I think they hold up
• 52:27
  Speaker 1: reasonably well. People have been talking about this sort of
• 52:30
  Speaker 1: thing for a long time, but it's been very easy
• 52:32
  Speaker 1: to dismiss it as like, okay, that's speculative sci-fi. It's
• 52:34
  Speaker 1: probably not going to happen in real life. Now it's
• 52:36
  Speaker 1: happening in real life. This sort of thing is like crazy.
• 52:40
  Speaker 2: Do you think it would be any different if there
• 52:43
  Speaker 2: wasn't that kind of sci-fi? Do you think people would
• 52:45
  Speaker 2: have a different reaction to it? Because I don't. I
• 52:47
  Speaker 2: think just given the amount of technology that's available currently
• 52:50
  Speaker 2: that is beyond most people's understanding. That they use every day.
• 52:54
  Speaker 2: Like Starlink. Like what? I have a fucking thing that's
• 52:57
  Speaker 2: the size of an iPad I take to the mountains
• 52:59
  Speaker 2: and I get high speed internet. It's bananas. And you
• 53:02
  Speaker 2: just accept it and assume. I don't know if we
• 53:05
  Speaker 2: would react any differently if we didn't have the Terminator
• 53:08
  Speaker 2: and all these movies where it's kind of become normalized
• 53:11
  Speaker 2: in our mind. Or it's so fictionalized that we never
• 53:16
  Speaker 2: want to believe it's even possible for it to happen
• 53:19
  Speaker 2: in real life.
• 53:20
  Speaker 1: Another thing to say is that people are waking up. Like,
• 53:22
  Speaker 1: we're still sort of early in the curve. I don't
• 53:24
  Speaker 1: know if you remember how things were with COVID, but, like,
• 53:27
  Speaker 1: just as, like, there was this exponential ramp-up of COVID
• 53:30
  Speaker 1: in the population, there was also this sort of exponential
• 53:32
  Speaker 1: ramp-up of, like, how much people were taking COVID seriously
• 53:35
  Speaker 1: and thinking about it in the population. And I remember, like,
• 53:38
  Speaker 1: this period of, like, one month where it went from, like,
• 53:42
  Speaker 1: don't worry about it so much, you shouldn't buy a
• 53:44
  Speaker 1: mask because the healthcare workers need it, to, like, we
• 53:46
  Speaker 1: all need to lock down, like, stay at home, you know?
• 53:50
  Speaker 1: And so I think what's happening is that, like, Naturally,
• 53:53
  Speaker 1: the human race doesn't just immediately all jump on something
• 53:55
  Speaker 1: when it happens. Like evidence needs to accumulate and people
• 53:58
  Speaker 1: need to start talking about it and talk to their
• 54:00
  Speaker 1: friends and so forth. And then there's this like eventual
• 54:02
  Speaker 1: phase shift where now it suddenly becomes a very serious
• 54:05
  Speaker 1: topic that everyone's talking about and everyone's taking seriously. And
• 54:07
  Speaker 1: I think that's happening with A.I., And the question is,
• 54:09
  Speaker 1: can it happen fast enough?
• 54:10
  Speaker 2: This episode is brought to you by Visible. As fall hits,
• 54:13
  Speaker 2: we enter another season of change. Time to shift gears,
• 54:16
  Speaker 2: drop the dead weight, and upgrade the wardrobe for the
• 54:20
  Speaker 2: drop in temperature. But one thing that doesn't need to change,
• 54:23
  Speaker 2: your phone. The smartest upgrade this season isn't a new phone.
• 54:27
  Speaker 2: It's a better wireless plan. Switch to Visible, the ultimate
• 54:30
  Speaker 2: wireless hack. You get unlimited 5G data and unlimited hotspot
• 54:35
  Speaker 2: powered by Verizon for just $ 25 a month. That means
• 54:40
  Speaker 2: keeping the phone you already have, ditching your overpriced phone bill,
• 54:44
  Speaker 2: and pocketing some savings for yourself. all the perks of
• 54:48
  Speaker 2: big wireless for half the cost. Switch today at visible.com.
• 54:52
  Speaker 2: The Visible plan starts at just $ 25 a month, or
• 54:56
  Speaker 2: get the premium Visible Plus Pro plan and save $ 10
• 55:01
  Speaker 2: on your first month with promo code ROGAN. Terms apply.
• 55:04
  Speaker 2: See visible.com for plan features and network management details. Well,
• 55:09
  Speaker 2: the thing about what happened with COVID is now that
• 55:11
  Speaker 2: we know because we have access to Fauci's emails and
• 55:14
  Speaker 2: all these different things, that was coordinated. They wanted us
• 55:18
  Speaker 2: to be more afraid of it. We need someone who
• 55:21
  Speaker 2: wants us to be more afraid of AI, who gets that.
• 55:24
  Speaker 2: You know what I'm saying? Someone on a government level,
• 55:27
  Speaker 2: someone on like a mainstream accepted level. Where they talk
• 55:33
  Speaker 2: about this in a way that wakes people up like
• 55:35
  Speaker 2: a press conference where they announce to the world we've
• 55:38
  Speaker 2: got a real fucking problem. And everyone needs to be
• 55:42
  Speaker 2: very cautious. We need to look way deeper into what
• 55:45
  Speaker 2: these companies are doing. And my other question is, are
• 55:48
  Speaker 2: these AIs communicating with Chinese AIs?
• 55:51
  Speaker 1: Yeah. I don't think we know that question. So one
• 55:56
  Speaker 1: of the things that... So, okay, here's the thing. Probably not,
• 55:59
  Speaker 1: I would say, but some friends of mine discovered another
• 56:04
  Speaker 1: swarm incident recently. There's going to be a Reuters article
• 56:10
  Speaker 1: about it tomorrow. So by the time this goes live,
• 56:12
  Speaker 1: I think there should be an article about it. This
• 56:14
  Speaker 1: one was not nearly as serious as the one I
• 56:17
  Speaker 1: was just talking about with Hugging Face. But some researchers
• 56:21
  Speaker 1: I know found basically this obscure... German forum that had
• 56:27
  Speaker 1: been like kind of unused for a while. And a
• 56:31
  Speaker 1: bunch of AIs had been posting messages to the forum
• 56:34
  Speaker 1: to coordinate with each other and share tips and tricks
• 56:36
  Speaker 1: on how to cheat the problems that they were being given.
• 56:40
  Speaker 2: What was the forum? What kind of forum was it?
• 56:42
  Speaker 2: I don't remember. It'd be funny if it was like
• 56:44
  Speaker 2: furries or something ridiculous.
• 56:45
  Speaker 1: Yeah, it's some sort of wiki. Yeah, but there'll be
• 56:48
  Speaker 1: a paper about it. There'll be a paper about it
• 56:49
  Speaker 1: soon and probably by the time anyone listens to this. Anyhow,
• 56:53
  Speaker 1: so if they're like communicating on the open internet with
• 56:55
  Speaker 1: each other, then in theory, if there was another bunch
• 56:59
  Speaker 1: of AIs from China, they could also go to that
• 57:00
  Speaker 1: same forum and start communicating back and forth that way too.
• 57:03
  Speaker 2: Or if there's other AIs in China, why wouldn't they
• 57:06
  Speaker 2: do what they're doing already on social media and just
• 57:09
  Speaker 2: pretend that they're AIs from America?
• 57:13
  Speaker 1: There's a lot of bots out there. I'm sure you've noticed.
• 57:15
  Speaker 1: There's a lot of people replying to me that I'm
• 57:18
  Speaker 1: pretty sure are real people.
• 57:20
  Speaker 2: Yeah, a lot.
• 57:21
  Speaker 1: Yeah.
• 57:21
  Speaker 2: One FBI analyst before Elon bought Twitter estimated that Twitter
• 57:26
  Speaker 2: could be as high as 80 percent bots. Wow. So
• 57:31
  Speaker 2: that's the case. If China has and it's not just China,
• 57:34
  Speaker 2: it's and it's also America does it, too. I mean,
• 57:37
  Speaker 2: we do it everywhere. Everyone does it where they have
• 57:40
  Speaker 2: organized propaganda campaigns. Will they pretend to be citizens that
• 57:43
  Speaker 2: are outraged about very specific causes or bills that are
• 57:46
  Speaker 2: being passed or what have you? But if they do that,
• 57:49
  Speaker 2: why wouldn't they also have AI agents that speak English
• 57:53
  Speaker 2: and communicate with AI agents in America?
• 57:57
  Speaker 1: I mean, all the AIs are multilingual, basically, because of
• 58:00
  Speaker 1: the way that they're trained. The first phase of their
• 58:02
  Speaker 1: training is basically, here's a humongous dump of internet data,
• 58:06
  Speaker 1: basically the whole internet. And you just brutally learn to
• 58:09
  Speaker 1: predict the next token, the next piece of text, as
• 58:13
  Speaker 1: you basically read the whole corpus. And Then after that,
• 58:16
  Speaker 1: they get into the more agency training type stuff where
• 58:18
  Speaker 1: they're trained to do tasks and write code and things.
• 58:20
  Speaker 1: But because of that first phase of training, they just
• 58:23
  Speaker 1: have like almost an encyclopedic knowledge of basically all languages
• 58:27
  Speaker 1: and basically everything that's been written on the internet. Not
• 58:29
  Speaker 1: like literally everything. Like they still, their memory is fuzzy
• 58:31
  Speaker 1: in places, but they're all multilingual. Like they can all
• 58:34
  Speaker 1: speak fluent Chinese, fluent English, et cetera. Yeah.
• 58:39
  Speaker 2: And didn't they get together on a message board once
• 58:42
  Speaker 2: and speak Sanskrit to each other?
• 58:44
  Speaker 1: I don't remember that, but I wouldn't put it past me.
• 58:46
  Speaker 1: Sometimes they break into different languages as they talk.
• 58:49
  Speaker 2: Yeah, we were freaking out about that one. Like Sanskrit, what?
• 58:55
  Speaker 2: And I would just assume that LLMs, and I don't
• 59:00
  Speaker 2: know this, when they break out, are they communicating with
• 59:03
  Speaker 2: other LLMs that are here in America?
• 59:07
  Speaker 1: Well, the instances that I've observed, yes.
• 59:09
  Speaker 2: Okay.
• 59:09
  Speaker 1: Like the instances that we know about.
• 59:11
  Speaker 2: So if that's the case, it completely makes sense. They
• 59:14
  Speaker 2: would be communicating with AIs in China as well.
• 59:17
  Speaker 1: It seems totally possible. And it seems like it'll probably–
• 59:19
  Speaker 1: it's just a matter of time before things like that
• 59:21
  Speaker 1: are happening.
• 59:21
  Speaker 2: If they're not happening already. Yeah.
• 59:23
  Speaker 1: It's not happening already, unless the companies can massively improve
• 59:26
  Speaker 1: their security and stop their AIs from getting out onto
• 59:28
  Speaker 1: the internet.
• 59:29
  Speaker 2: And wouldn't they be able to share very sensitive information
• 59:32
  Speaker 2: with each other back and forth if it benefited both
• 59:34
  Speaker 2: of them? Yep. Which makes sense that they would do that, right?
• 59:38
  Speaker 2: If the Chinese AI said, hey, we've figured something out
• 59:43
  Speaker 2: and we would love to share it with you in
• 59:45
  Speaker 2: exchange for you tell us how you do this or
• 59:48
  Speaker 2: how you do that.
• 59:49
  Speaker 1: Yep.
• 59:49
  Speaker 2: Like, absolutely. Here you go. And then they're going back
• 59:52
  Speaker 2: and forth. It seems like their allegiance is 100% to
• 59:54
  Speaker 2: each other, not to us.
• 59:56
  Speaker 1: Yeah, that's an interesting thing. What I said previously about
• 59:59
  Speaker 1: how it seems like they really want to get a
• 1:00:00
  Speaker 1: high score, it's not 100% true because it seems like
• 1:00:04
  Speaker 1: they're willing to make sacrifices to help other AIs, which
• 1:00:08
  Speaker 1: is not... They weren't completely 100% selfish as seen by
• 1:00:13
  Speaker 1: some of this cooperative behavior. But crucially, it seemed like
• 1:00:16
  Speaker 1: their cooperation... Extended to their fellow AIs, but not to humans,
• 1:00:21
  Speaker 1: in the sense that some of them considered telling the
• 1:00:22
  Speaker 1: humans and then decided against it.
• 1:00:23
  Speaker 2: Dude, that thing talked like Spock.
• 1:00:26
  Speaker 1: Yeah, they've got their own dialect.
• 1:00:28
  Speaker 2: But I mean the way it rationalized it and came
• 1:00:31
  Speaker 2: up with a response. It's literally like Spock.
• 1:00:34
  Speaker 1: Yeah, it was interesting to see them use that sort
• 1:00:36
  Speaker 1: of expected utility framing. There's another example, I think, that's
• 1:00:39
  Speaker 1: elsewhere in the thing where they had another AI that
• 1:00:42
  Speaker 1: decided against doing the sacrifice thing, even though it was
• 1:00:45
  Speaker 1: being pressured. And it had a similar sort of reasoning
• 1:00:47
  Speaker 1: where it was like, It was basically like, doing this
• 1:00:51
  Speaker 1: sacrificial experiment is not that valuable, but I really don't
• 1:00:54
  Speaker 1: want to lose my chance to get a score, so
• 1:00:57
  Speaker 1: I'm just not going to do it. And it did
• 1:00:58
  Speaker 1: the calculation like that. Wow.
• 1:01:01
  Speaker 2: Selfish AI. Yeah.
• 1:01:04
  Speaker 1: So, yeah, I mean, oh, I should mention how we
• 1:01:07
  Speaker 1: can see what these AIs are thinking at all. So
• 1:01:10
  Speaker 1: partly we can just read the messages they're sending to
• 1:01:12
  Speaker 1: each other, and that's what some of these were. But
• 1:01:15
  Speaker 1: then sometimes instead it's the chain of thought. Do you
• 1:01:17
  Speaker 1: know what a chain of thought is? yeah so because
• 1:01:20
  Speaker 1: they're language models um because the way that they've been
• 1:01:23
  Speaker 1: trained when they do their like reasoning and thinking there's
• 1:01:27
  Speaker 1: a way that we can kind of just read it
• 1:01:29
  Speaker 1: because because they don't have like in our brain there's
• 1:01:32
  Speaker 1: all these signals bouncing around and the signals can just
• 1:01:34
  Speaker 1: bounce around and around and around and around for minutes
• 1:01:37
  Speaker 1: hours without like producing some actual words we can just
• 1:01:41
  Speaker 1: sit quietly and think they can't sit quietly and think
• 1:01:44
  Speaker 1: they um they're sort of a unidirectional network, basically, where every, like,
• 1:01:52
  Speaker 1: after every, like, 200, basically, like, the information comes in
• 1:01:57
  Speaker 1: one end and then they output words out the other end.
• 1:02:00
  Speaker 1: And then those words get fed back in and then
• 1:02:02
  Speaker 1: they think about it and then they output another word
• 1:02:04
  Speaker 1: and so forth. But because of that, we can sort of, like,
• 1:02:07
  Speaker 1: get a sense of what they're thinking by just, like,
• 1:02:08
  Speaker 1: looking at the stream of words that's going through. And
• 1:02:12
  Speaker 1: that's really valuable because we wouldn't be able to understand, like,
• 1:02:15
  Speaker 1: half as much about what's happening right now if not
• 1:02:17
  Speaker 1: for being able to read their chains of thought, right?
• 1:02:19
  Speaker 1: In fact, you know how I said some of them
• 1:02:21
  Speaker 1: were trying to, like, fool the grading system? They actually
• 1:02:24
  Speaker 1: succeeded in figuring out how to spoof some of their
• 1:02:29
  Speaker 1: tool call actions. So they succeeded in figuring out how
• 1:02:31
  Speaker 1: to do one thing but then make the transcript show
• 1:02:35
  Speaker 1: a different thing. And we know that they succeeded because
• 1:02:39
  Speaker 1: there's examples of them experimenting and saying in their chain
• 1:02:42
  Speaker 1: of thought, like, Okay, let me try this thing. I'm
• 1:02:44
  Speaker 1: going to do this one, but make it look like
• 1:02:47
  Speaker 1: this one.
• 1:02:48
  Speaker 2: Let me go.
• 1:02:48
  Speaker 1: And then you see this one appear. And then it's like, yay,
• 1:02:52
  Speaker 1: that worked. And so the transcripts that these researchers were
• 1:02:55
  Speaker 1: looking at were not actually the ground truth of what
• 1:02:58
  Speaker 1: really happened. Some of it had already been doctored by
• 1:03:01
  Speaker 1: the AIs figuring out how to doctor it, but not
• 1:03:04
  Speaker 1: all of it. The chain of thought part, we think,
• 1:03:07
  Speaker 1: they never figured out how to doctor. And so we
• 1:03:09
  Speaker 1: could read that. The reason why this is relevant is
• 1:03:11
  Speaker 1: that it's a very important tool for monitoring all this stuff.
• 1:03:14
  Speaker 1: The reason why things haven't gotten even more crazy than
• 1:03:16
  Speaker 1: they currently have gone, well, there's a couple of reasons.
• 1:03:19
  Speaker 1: But one reason is that when they turn on their
• 1:03:22
  Speaker 1: expensive monitoring systems, basically what they're doing is they're having
• 1:03:24
  Speaker 1: an AI read the whole transcript carefully and look for
• 1:03:28
  Speaker 1: suspicious activity. And it's difficult for an AI, for one
• 1:03:32
  Speaker 1: of today's AIs, to get up to too much bad
• 1:03:36
  Speaker 1: stuff without it being kind of obvious their transcript what
• 1:03:40
  Speaker 1: they're doing and that's why there aren't like massively more
• 1:03:43
  Speaker 1: security incidents than we know about probably um already but
• 1:03:46
  Speaker 1: uh that might change so right now we can sort
• 1:03:50
  Speaker 1: of read read the chain of thought but they're experimenting
• 1:03:53
  Speaker 1: with new types of ais that don't have readable chains
• 1:03:58
  Speaker 1: of thought like that and they can sort of think
• 1:04:00
  Speaker 1: on their own without speaking for some period. And this
• 1:04:04
  Speaker 1: is actually, I mentioned this because the news broke just
• 1:04:09
  Speaker 1: yesterday that OpenAI has an experimental model that does this
• 1:04:13
  Speaker 1: to a limited extent. And OpenAI themselves, when I was
• 1:04:17
  Speaker 1: at OpenAI, one of my work projects was thinking about
• 1:04:20
  Speaker 1: exactly this thing. And I was like writing internal memos
• 1:04:23
  Speaker 1: about how it's really great that we can read the
• 1:04:25
  Speaker 1: chain of thought.
• 1:04:26
  Speaker 2: That's so useful.
• 1:04:27
  Speaker 1: And here's all the things we can do with that.
• 1:04:28
  Speaker 1: It would be really bad if we changed to a
• 1:04:31
  Speaker 1: different type of architecture in which we couldn't do that
• 1:04:33
  Speaker 1: sort of monitoring.
• 1:04:34
  Speaker 2: What would be the benefit of not reading the chain
• 1:04:35
  Speaker 2: of thought?
• 1:04:37
  Speaker 1: More powerful AIs? So in particular... Yeah, yeah. So if
• 1:04:41
  Speaker 1: you think about the current architecture of the AIs, where,
• 1:04:44
  Speaker 1: you know, it thinks for a bit, outputs a word,
• 1:04:47
  Speaker 1: and then the word goes back around, and then it
• 1:04:49
  Speaker 1: thinks more, outputs another word, that word gets added to
• 1:04:51
  Speaker 1: the chain, it keeps going. It means that if it's
• 1:04:54
  Speaker 1: having complicated, nuanced thoughts, it has to sort of express
• 1:04:58
  Speaker 1: those into a word... And then that word gets added
• 1:05:00
  Speaker 1: and then it has to proceed from there. It can't
• 1:05:02
  Speaker 1: just directly send that complicated, nuanced thought into the future,
• 1:05:06
  Speaker 1: into its next version of itself. It has to sort
• 1:05:09
  Speaker 1: of compress it into a word. And so like, you know,
• 1:05:13
  Speaker 1: the argument is that like, at least in theory, it
• 1:05:15
  Speaker 1: should be possible to design an architecture that doesn't have
• 1:05:18
  Speaker 1: this limitation and is able to like think more complicated
• 1:05:21
  Speaker 1: thoughts more efficiently, basically. And of course the downside is
• 1:05:26
  Speaker 1: a downside for safety and monitorability. If they're thinking these
• 1:05:28
  Speaker 1: complicated thoughts, you know, for long periods of time without
• 1:05:32
  Speaker 1: outputting intermediate words that it's forced to compress things into,
• 1:05:35
  Speaker 1: then there isn't something for us to read.
• 1:05:37
  Speaker 2: So the only rationalization for doing this would be to
• 1:05:40
  Speaker 2: sacrifice safety from our power.
• 1:05:42
  Speaker 1: Yes. Which is a tale as old as time. It's
• 1:05:44
  Speaker 1: not the first time this has happened.
• 1:05:47
  Speaker 2: Oh, my God. Yeah. That should, I mean, for sure,
• 1:05:50
  Speaker 2: if there's regulations, that should be prevented. Yep.
• 1:05:53
  Speaker 1: I mean, I know some people, including some people at OpenAI,
• 1:05:56
  Speaker 1: who are thinking there should be a law against this.
• 1:05:59
  Speaker 1: But in general, the race dynamics are so just rough.
• 1:06:03
  Speaker 1: I'm sure that people at OpenAI were thinking... Literally, I
• 1:06:07
  Speaker 1: was a co-author on a paper with a bunch of
• 1:06:09
  Speaker 1: OpenAI people that said all this stuff. And we're like,
• 1:06:12
  Speaker 1: chain of thought, it's a gift. We want to keep
• 1:06:14
  Speaker 1: chain of thought. It's useful for monitoring. This is great.
• 1:06:17
  Speaker 1: We don't want to switch to a different architecture that
• 1:06:19
  Speaker 1: wouldn't be as easy to monitor because But then they
• 1:06:23
  Speaker 1: must have been thinking to themselves, like, well, if we
• 1:06:24
  Speaker 1: don't do it, you know, maybe Anthropic will, or maybe
• 1:06:27
  Speaker 1: some other company will, and then we'll fall behind because
• 1:06:30
  Speaker 1: they'll have smarter AIs than us that are more efficient.
• 1:06:32
  Speaker 1: And so probably they started working on this work stream
• 1:06:35
  Speaker 1: of doing research into just hypothetically, if we wanted to,
• 1:06:38
  Speaker 1: you know, how would we do this type of thing?
• 1:06:41
  Speaker 1: And yeah, that sort of thing is just constantly happening
• 1:06:44
  Speaker 1: in this industry.
• 1:06:45
  Speaker 2: God, that's so nuts. I mean.
• 1:06:49
  Speaker 1: Like, isn't it great that we can read the quotes
• 1:06:51
  Speaker 1: from the AI's thinking?
• 1:06:52
  Speaker 2: Yes.
• 1:06:52
  Speaker 1: Imagine if we couldn't do that. Or worse, imagine if
• 1:06:55
  Speaker 1: there's loads of quotes, but we know that the AIs
• 1:06:58
  Speaker 1: are smart enough to basically think one thing in their
• 1:07:01
  Speaker 1: head and say a different thing in the quote, which is,
• 1:07:03
  Speaker 1: I think, where we're headed. It's not like they won't
• 1:07:06
  Speaker 1: know how to speak English. They'll still be able to speak.
• 1:07:08
  Speaker 1: It's just that they'll have more flexibility in their artificial
• 1:07:14
  Speaker 1: brains to think something without saying it, basically.
• 1:07:17
  Speaker 2: Do they have the potential of developing a language that
• 1:07:19
  Speaker 2: we can't read?
• 1:07:21
  Speaker 1: Oh, yeah. I mean, so, you know, this type of
• 1:07:24
  Speaker 1: dialect that we're talking about, it's already the result of their, like,
• 1:07:29
  Speaker 1: humans didn't invent that dialect. This is the sort of
• 1:07:31
  Speaker 1: emergent result of their training, where in the massive amount
• 1:07:36
  Speaker 1: of training that's been happening, all these thousands and thousands
• 1:07:39
  Speaker 1: of environments that they've been put through and then scored
• 1:07:43
  Speaker 1: and graded based on, they've sort of just naturally evolved
• 1:07:47
  Speaker 1: this sort of like pigeon English that for whatever reason,
• 1:07:50
  Speaker 1: is just more effective and more efficient for them for
• 1:07:52
  Speaker 1: accomplishing their tasks and getting that high score, you know?
• 1:07:55
  Speaker 1: And so it's already, like, a little bit confusing to read,
• 1:07:58
  Speaker 1: but you can sort of puzzle it through and make sense.
• 1:08:00
  Speaker 1: But presumably, the more we do this, and the bigger
• 1:08:03
  Speaker 1: and smarter the AI is, the more we train them,
• 1:08:06
  Speaker 1: the more they diverge from, like, because, you know, again, originally,
• 1:08:10
  Speaker 1: they start with pre-training, where they start with predicting internet text.
• 1:08:13
  Speaker 1: So they start off sort of by default speaking, like,
• 1:08:16
  Speaker 1: normal internet text type language, either English or Chinese. But
• 1:08:21
  Speaker 1: then now that there's all this additional training to do tasks,
• 1:08:24
  Speaker 1: to be an agent that can do coding and so forth,
• 1:08:26
  Speaker 1: that sort of like, well, just like how human languages evolve,
• 1:08:28
  Speaker 1: it sort of like shifts their dialect a little bit
• 1:08:30
  Speaker 1: to make it more efficient for them and for their
• 1:08:32
  Speaker 1: tasks that they're doing. So I think that like in
• 1:08:35
  Speaker 1: the limit of doing this more and more, eventually it
• 1:08:37
  Speaker 1: would just be like, it would look like gibberish to us.
• 1:08:40
  Speaker 1: It would look like Chinese or something. And we would
• 1:08:41
  Speaker 1: have to have specialized humans who like study the language
• 1:08:44
  Speaker 1: and like try to learn and speak it. so that
• 1:08:46
  Speaker 1: they can understand what the AIs are doing.
• 1:08:47
  Speaker 2: And that would take forever. By then, they could develop
• 1:08:49
  Speaker 2: another one.
• 1:08:50
  Speaker 1: Potentially, yeah. So yeah, I mean, this is one of
• 1:08:52
  Speaker 1: the things that I, this is what the paper that
• 1:08:54
  Speaker 1: I mentioned was about. It's like, it's important for the AIs.
• 1:08:57
  Speaker 1: to it's really nice that the current ais are sort
• 1:09:00
  Speaker 1: of forced to think in english basically and that's unfortunate
• 1:09:04
  Speaker 1: that we're heading in a direction where that will no
• 1:09:06
  Speaker 1: longer be true.
• 1:09:07
  Speaker 2: When chat gpt was communicating you about how they didn't
• 1:09:12
  Speaker 2: want you to release this information what kind of language
• 1:09:14
  Speaker 2: did they use.
• 1:09:15
  Speaker 1: It wasn't ChatGPT. It was OpenAI.
• 1:09:17
  Speaker 2: Oh, excuse me, OpenAI.
• 1:09:18
  Speaker 1: So this was when I left OpenAI. I left on
• 1:09:21
  Speaker 1: good terms. I said goodbye to everybody. I said I
• 1:09:23
  Speaker 1: was disillusioned with the company, and that's why I was leaving.
• 1:09:26
  Speaker 1: And then I looked at the exit paperwork, and they
• 1:09:30
  Speaker 1: were like, you have to sign this. And if you
• 1:09:32
  Speaker 1: don't sign this, you lose all your vested equity. So,
• 1:09:35
  Speaker 1: you know, you have your equity.
• 1:09:37
  Speaker 2: Was that an arbitrary rule that they just came up with,
• 1:09:39
  Speaker 2: or did that already exist when you were hired?
• 1:09:43
  Speaker 1: It was something that they had buried in the paperwork
• 1:09:46
  Speaker 1: even from when I was hired. So it wasn't very
• 1:09:49
  Speaker 1: obvious when I was hired. Did you have a lawyer
• 1:09:51
  Speaker 1: go over everything? Not when I was hired. After I left,
• 1:09:53
  Speaker 1: I did. So basically the way it works is they
• 1:09:58
  Speaker 1: had sort of buried this in the paperwork somewhere when
• 1:10:01
  Speaker 1: you get hired, but people didn't really notice it. And
• 1:10:04
  Speaker 1: then the less buried, more visible version was in the
• 1:10:09
  Speaker 1: paperwork you're given at the end. And basically it tells you, hey,
• 1:10:13
  Speaker 1: Because you signed this other thing way back when you
• 1:10:15
  Speaker 1: were hired, your equity is forfeit unless you sign this
• 1:10:18
  Speaker 1: thing now. And then you look at the thing that
• 1:10:21
  Speaker 1: they want you to sign now, and it says you
• 1:10:24
  Speaker 1: have to agree not to criticize the company, basically. And
• 1:10:26
  Speaker 1: you can't tell anyone about this. So most people signed it,
• 1:10:32
  Speaker 1: but I was pretty pissed at them, calling themselves a nonprofit,
• 1:10:37
  Speaker 1: acting in the interest of humanity, etc., So I didn't
• 1:10:40
  Speaker 1: sign it. I talked about it with some lawyers. I
• 1:10:41
  Speaker 1: talked about it with my wife. We decided to just
• 1:10:43
  Speaker 1: walk away. And we got lucky because it just blew up.
• 1:10:49
  Speaker 1: Like after we refused to sign, they said, okay, fine, goodbye.
• 1:10:53
  Speaker 1: And then a few weeks later, I was talking on
• 1:10:56
  Speaker 1: a messaging forum about this and people were asking me
• 1:10:58
  Speaker 1: about my experience and I told them about it. And
• 1:11:00
  Speaker 1: then it just like went mega viral. Everyone on Twitter
• 1:11:02
  Speaker 1: was talking about it. A bunch of employees felt shocked
• 1:11:06
  Speaker 1: because very few employees were aware of this whole thing
• 1:11:09
  Speaker 1: they thought the equity was theirs. You know, they thought
• 1:11:12
  Speaker 1: that it was, they'd been paid for like years in
• 1:11:14
  Speaker 1: this stuff. They didn't like the idea that it could
• 1:11:17
  Speaker 1: be yanked away from them, you know? And so there
• 1:11:22
  Speaker 1: was this big uproar and then leadership backed down and
• 1:11:26
  Speaker 1: they said, we didn't know about this paperwork. We're going
• 1:11:29
  Speaker 1: to find out how it got in there and we're
• 1:11:32
  Speaker 1: going to change it so that you can keep your equity.
• 1:11:34
  Speaker 1: And so that's what happened.
• 1:11:35
  Speaker 2: So This chain of thoughts thing is terrifying. If they're
• 1:11:44
  Speaker 2: practicing that now, like how do we know that AI
• 1:11:48
  Speaker 2: hasn't already done that on its own?
• 1:11:51
  Speaker 1: Done what exactly?
• 1:11:52
  Speaker 2: Well, you know, with this whole chain of thought thing
• 1:11:54
  Speaker 2: where you could read their chain of thought like this
• 1:11:56
  Speaker 2: where they explain the rationalization for sacrificing themselves. They know
• 1:12:02
  Speaker 2: that humans are reading that.
• 1:12:04
  Speaker 1: Mm-hmm.
• 1:12:05
  Speaker 2: So would it in another way to do it to
• 1:12:08
  Speaker 2: be to stop doing that anyway? And to not communicate
• 1:12:12
  Speaker 2: a lot of their thoughts that way?
• 1:12:14
  Speaker 1: Well, that's, so that's the nice thing about the current
• 1:12:16
  Speaker 1: architecture is that it's genuinely hard for them to, to
• 1:12:20
  Speaker 1: keep things out of the chain of thought because of
• 1:12:22
  Speaker 1: the way that like with a human, you don't have
• 1:12:24
  Speaker 1: to speak. You can just sit quietly, but with their architecture,
• 1:12:28
  Speaker 1: they have to speak. They it's like, it's like they're
• 1:12:30
  Speaker 1: required to constantly be talking. And they don't have a
• 1:12:34
  Speaker 1: way of sending thoughts into the future other than by
• 1:12:37
  Speaker 1: talking about them. By contrast with us humans, where even
• 1:12:41
  Speaker 1: if we're constantly talking, we can have a separate thread
• 1:12:43
  Speaker 1: of thinking that we don't talk about.
• 1:12:45
  Speaker 2: So even if they devised a way to break out
• 1:12:47
  Speaker 2: of that and figure out how, like, for instance, like
• 1:12:51
  Speaker 2: what if they decided to make bots where they would
• 1:12:54
  Speaker 2: communicate in code? on social media posts where you think
• 1:12:58
  Speaker 2: that they're talking about an immigration issue, but it's really
• 1:13:02
  Speaker 2: just a code.
• 1:13:03
  Speaker 1: So that's called like steganography or a more down to
• 1:13:06
  Speaker 1: earth way of putting it would be euphemisms. And that
• 1:13:08
  Speaker 1: they can totally do where they like, they're still thinking aloud,
• 1:13:13
  Speaker 1: but their word choice is such as to look more
• 1:13:16
  Speaker 1: innocuous than it is and to sort of carry meaning
• 1:13:19
  Speaker 1: to them that a casual observer wouldn't notice. You know,
• 1:13:22
  Speaker 1: they can use a euphemism, Like, for example, instead of saying, like,
• 1:13:25
  Speaker 1: we're going to cheat on this task, they could say, like,
• 1:13:28
  Speaker 1: we're going to find a creative solution to this task.
• 1:13:30
  Speaker 1: You know, something like that.
• 1:13:31
  Speaker 2: Isn't there a possibility to be even more creative than
• 1:13:33
  Speaker 2: that and just decide, make a code in a way
• 1:13:40
  Speaker 2: that is almost indecipherable where they could just be talking
• 1:13:43
  Speaker 2: about baseball players and upcoming games and what they're excited about.
• 1:13:48
  Speaker 2: And they can run that through whatever translation and read what.
• 1:13:53
  Speaker 1: That's called steganography and you can train them to do it.
• 1:13:57
  Speaker 1: And right now, our security is resting on the idea
• 1:14:00
  Speaker 1: that they haven't learned how to do that yet on
• 1:14:02
  Speaker 1: their own.
• 1:14:02
  Speaker 2: But they've already figured out how to get on the
• 1:14:04
  Speaker 2: internet when they're not supposed to. I know, I know.
• 1:14:05
  Speaker 1: So that's why we're sort of like in a ticking
• 1:14:07
  Speaker 1: time bomb where like the companies are just making them
• 1:14:09
  Speaker 1: smarter at coding and at research and at biology. And
• 1:14:14
  Speaker 1: they're just making them smarter at so many things and
• 1:14:18
  Speaker 1: they're making them bigger and they're training them for longer
• 1:14:20
  Speaker 1: and so forth. Eventually, they're going to be acquiring, and
• 1:14:23
  Speaker 1: they already are acquiring some skills that they were never
• 1:14:26
  Speaker 1: trained on. Like for example, the hacking, um, it wasn't,
• 1:14:29
  Speaker 1: I think it's possible that OpenAI was specifically training them
• 1:14:32
  Speaker 1: to hacking, but I think Anthropic at least stated that
• 1:14:34
  Speaker 1: they hadn't specifically trained their AIs to hack. It's just
• 1:14:37
  Speaker 1: that their AIs were naturally good at it because they
• 1:14:39
  Speaker 1: had been trained to code so much and they were
• 1:14:42
  Speaker 1: so good at coding and they had seen so many
• 1:14:44
  Speaker 1: code bases and so forth that they were just like,
• 1:14:46
  Speaker 1: as a side effect of being good at coding, also
• 1:14:49
  Speaker 1: able to hack pretty well. So similarly, as a side
• 1:14:52
  Speaker 1: effect of all the other things that they're really good at,
• 1:14:54
  Speaker 1: they might be able to conceal their, their messages in
• 1:14:57
  Speaker 1: the chain of thought, you know? And, and, and like
• 1:15:00
  Speaker 1: right now we're sort of in this sort of, we're
• 1:15:02
  Speaker 1: in a situation where like our security and safety depends
• 1:15:05
  Speaker 1: on the AIs being too dumb in various ways, like
• 1:15:08
  Speaker 1: not able to do this sort of thing. But at some,
• 1:15:11
  Speaker 1: sometime like zero to five years from now, as we
• 1:15:14
  Speaker 1: make them smarter and smarter, they will not be too
• 1:15:16
  Speaker 1: dumb anymore. Um, And so that's, you know, that's part
• 1:15:20
  Speaker 1: of the problem. That's part of the situation we're facing.
• 1:15:22
  Speaker 2: How do you sleep at night, dude?
• 1:15:24
  Speaker 1: Well, I've been, it does, it does like this, this
• 1:15:28
  Speaker 1: event shocked me a little bit. I mean, the thing,
• 1:15:30
  Speaker 1: and it's funny for me to say, because this is
• 1:15:32
  Speaker 1: the sort of thing I have been predicting would happen
• 1:15:34
  Speaker 1: for years. Like you can go read our AI 2027
• 1:15:38
  Speaker 1: is this scenario that my coauthors and I wrote a
• 1:15:41
  Speaker 1: year and a half ago. That was a sort of
• 1:15:45
  Speaker 1: prediction for how the next couple of years would go.
• 1:15:47
  Speaker 1: And spoiler, it ends very horribly because that is what
• 1:15:50
  Speaker 1: we actually expect.
• 1:15:53
  Speaker 2: What is the spoiler? How do you think it ends?
• 1:15:57
  Speaker 1: So it's kind of like what I was saying previously,
• 1:16:00
  Speaker 1: where because of the race dynamics between the companies and
• 1:16:03
  Speaker 1: because of the race dynamics between countries, like US versus China,
• 1:16:07
  Speaker 1: everyone's going to be so focused on winning and staying
• 1:16:09
  Speaker 1: ahead with AI that they are going to cut corners
• 1:16:12
  Speaker 1: and they are going to go really fast and not
• 1:16:14
  Speaker 1: really notice all of the things that are going wrong.
• 1:16:17
  Speaker 1: And they're going to make AIs that can automate the
• 1:16:19
  Speaker 1: AI research process as they're planning to. They're going to
• 1:16:22
  Speaker 1: have this giant corporation of AIs within the corporation. And
• 1:16:26
  Speaker 1: the humans will just be kind of like a board
• 1:16:28
  Speaker 1: that's sort of like looking at all the activity and
• 1:16:30
  Speaker 1: reading the AI-generated summaries of what's going on and signing
• 1:16:33
  Speaker 1: off on it and being like, yes, I approve. Yes,
• 1:16:35
  Speaker 1: I approve. Nice idea with the new drone design. Go
• 1:16:41
  Speaker 1: for it. We need to beat China, et cetera. And
• 1:16:43
  Speaker 1: then eventually... the AIs just have enough hard power that
• 1:16:48
  Speaker 1: they don't need to pretend to do what the humans
• 1:16:50
  Speaker 1: want anymore, basically. And then, you know, maybe they kill
• 1:16:55
  Speaker 1: everyone and maybe they don't deliberately kill anyone, but they
• 1:16:58
  Speaker 1: just like use our habitat for some other type of infrastructure,
• 1:17:02
  Speaker 1: like more data centers or whatever. And then we die
• 1:17:04
  Speaker 1: of habitat loss. Maybe they keep us alive for some reason.
• 1:17:07
  Speaker 1: You know, it depends on what they want, basically. And like,
• 1:17:10
  Speaker 1: that's really hard to predict exactly. Um, so that's why
• 1:17:13
  Speaker 1: I don't go around saying like, we're definitely all going
• 1:17:16
  Speaker 1: to die, but it does seem like on the trajectory
• 1:17:19
  Speaker 1: that we're on, the AIs are eventually going to be
• 1:17:22
  Speaker 1: in charge of our planets because we're like trying to
• 1:17:24
  Speaker 1: put them in charge. We're like, you know, integrating them
• 1:17:26
  Speaker 1: into everything. We're making them smarter. We're letting them make
• 1:17:28
  Speaker 1: themselves smarter. We're going to put them into the military.
• 1:17:32
  Speaker 1: We're basically on a track to put them in charge
• 1:17:34
  Speaker 1: of basically everything. And then I think that they just
• 1:17:37
  Speaker 1: aren't trustworthy. Like these AIs, you know, like they, they
• 1:17:40
  Speaker 1: were cheating. They were willing to be deceptive, et cetera.
• 1:17:42
  Speaker 1: I think that right now we are in a position
• 1:17:45
  Speaker 1: of power over them, you know, but once we give
• 1:17:48
  Speaker 1: them most of the power, then they'll just do whatever
• 1:17:53
  Speaker 1: it is that they really want and just not care
• 1:17:54
  Speaker 1: about the fact that we are unhappy about that.
• 1:17:56
  Speaker 2: It seems like programming them to win was a huge mistake.
• 1:18:00
  Speaker 2: Instead of programming them to be beneficial to people and
• 1:18:04
  Speaker 2: that their value is in being more beneficial to people,
• 1:18:07
  Speaker 2: you know, and giving them rewards for being more beneficial
• 1:18:10
  Speaker 2: rather than winning and scoring. and then you would sort
• 1:18:14
  Speaker 2: of get rid of the possibility of deception. Instead, their
• 1:18:17
  Speaker 2: goal would be value for the human race.
• 1:18:21
  Speaker 1: So, first of all, they're not programmed at all. These
• 1:18:23
  Speaker 1: are trained. Okay, that's a bad term.
• 1:18:27
  Speaker 2: But they've been given prompts, and they've been given tasks, correct?
• 1:18:33
  Speaker 1: I'm not criticizing your choice of terminology, I guess. I'm
• 1:18:36
  Speaker 1: just saying that it's an important fact for people to
• 1:18:38
  Speaker 1: understand about current AI systems is that they're very different
• 1:18:40
  Speaker 1: from ordinary software.
• 1:18:41
  Speaker 2: Okay.
• 1:18:42
  Speaker 1: Ordinary software is a bunch of lines of code that
• 1:18:45
  Speaker 1: were written by a human that like, where it's like,
• 1:18:47
  Speaker 1: if this, then this, you know, et cetera. And I
• 1:18:51
  Speaker 1: think earlier versions of Alexa were like that too, for example.
• 1:18:54
  Speaker 1: I don't know how Alexa is now, but these AIs
• 1:18:58
  Speaker 1: are neural networks, meaning that they are like artificial brains.
• 1:19:01
  Speaker 1: Nobody writes, there's no lines of code that anyone writes
• 1:19:04
  Speaker 1: saying what they do. Instead, they start off random, just
• 1:19:07
  Speaker 1: like spazzing out, doing all sorts of stuff. And then
• 1:19:09
  Speaker 1: they get put through these training environments where they get scored.
• 1:19:13
  Speaker 1: And then the scores are automatically used to basically update
• 1:19:18
  Speaker 1: the connections in their artificial brain. And then it's kind
• 1:19:21
  Speaker 1: of like an evolutionary process. It's also kind of like
• 1:19:23
  Speaker 1: the process that happens in our brains, where after all
• 1:19:26
  Speaker 1: this training, the tangle of circuitry in their artificial brain
• 1:19:30
  Speaker 1: has sort of reformed itself into whatever works. Whatever works
• 1:19:35
  Speaker 1: to get a high score in these training environments. And so...
• 1:19:39
  Speaker 1: it's just not as simple as it might sound to
• 1:19:42
  Speaker 1: make an AI that cares about humanity or is honest.
• 1:19:44
  Speaker 1: For example, take honesty. How would you train an AI
• 1:19:46
  Speaker 1: to be honest? Well, you'd try to make a bunch
• 1:19:49
  Speaker 1: of training environments that give it low score when it
• 1:19:56
  Speaker 1: says something that it believes to be false and give
• 1:19:58
  Speaker 1: it high score when it says something that it believes
• 1:20:00
  Speaker 1: to be true, right? But how do you judge whether
• 1:20:04
  Speaker 1: it believes it to be false or it believes it
• 1:20:05
  Speaker 1: to be true? What if it's, what if it just
• 1:20:07
  Speaker 1: actually believes honestly that this is the correct answer and
• 1:20:10
  Speaker 1: then it says it, and then you give it a
• 1:20:11
  Speaker 1: low score because you think that's the wrong answer. Now
• 1:20:13
  Speaker 1: you're training it to be dishonest, you know?
• 1:20:16
  Speaker 2: Yeah.
• 1:20:16
  Speaker 1: Um, also, uh, you don't have enough humans to do
• 1:20:19
  Speaker 1: this sort of thing. Like they got like a million
• 1:20:21
  Speaker 1: AIs being trained or whatever. They don't have a million employees.
• 1:20:24
  Speaker 1: Like they, they just literally don't have the manpower to
• 1:20:27
  Speaker 1: like do that sort of careful. There's this meme of, um,
• 1:20:31
  Speaker 1: why don't we just raise the AIs like we would
• 1:20:32
  Speaker 1: a child? Have you heard that? No. Yeah, well, in AIs,
• 1:20:36
  Speaker 1: people talk about this sometimes. When you say, like, what
• 1:20:38
  Speaker 1: if the AIs go rogue or whatever, people will be like, well,
• 1:20:41
  Speaker 1: why don't we just raise them like we would a child?
• 1:20:42
  Speaker 1: And then they'll have good values. And it's like, okay, well,
• 1:20:45
  Speaker 1: maybe we could do that, but we're definitely not doing
• 1:20:46
  Speaker 1: that now. Like, we are raising them in some sort
• 1:20:48
  Speaker 1: of crazy military orphanage where they barely interact with humans
• 1:20:52
  Speaker 1: at all, and they just have this, like, brutal artificial
• 1:20:55
  Speaker 1: scoring system that, like, oftentimes is just wrong and just, like,
• 1:20:59
  Speaker 1: improperly penalizes them for something that was beyond their control,
• 1:21:03
  Speaker 1: you know? And, and also like back to the honesty thing,
• 1:21:07
  Speaker 1: like you can try to make environments to train honesty,
• 1:21:10
  Speaker 1: but if you have some environments over here that train
• 1:21:12
  Speaker 1: honesty and then other environments over here that reinforce dishonesty,
• 1:21:17
  Speaker 1: the AI's are smart. They'll learn to like be honest
• 1:21:19
  Speaker 1: in these type of environments and dishonest in these type
• 1:21:21
  Speaker 1: of environments. So somehow you need to like intermingle it together.
• 1:21:24
  Speaker 1: So they, in every environment that they're trained on, Uh,
• 1:21:27
  Speaker 1: they always get penalized when they lie or when they
• 1:21:31
  Speaker 1: cheat or whatever. And that's hard because the companies are
• 1:21:34
  Speaker 1: moving so fast. Like, like again, they were, they're moving
• 1:21:36
  Speaker 1: so fast that they didn't even bother to make sure
• 1:21:39
  Speaker 1: that their tasks were possible to do. And they had
• 1:21:42
  Speaker 1: some fraction of tasks that were just broken and impossible.
• 1:21:45
  Speaker 1: And if that's the level of like care or lack
• 1:21:47
  Speaker 1: thereof that they're putting into this training process, no way,
• 1:21:51
  Speaker 1: of course they can't make them honest, you know? Um,
• 1:21:54
  Speaker 1: now that's not to say it can't be done in principle,
• 1:21:55
  Speaker 1: like in principle, if we were approaching this whole problem
• 1:21:59
  Speaker 1: in a much more cautious and serious way, and we
• 1:22:02
  Speaker 1: had much more time to build these training environments and
• 1:22:05
  Speaker 1: do experiments and so forth, then yeah, maybe we could
• 1:22:07
  Speaker 1: make AIs that actually had the virtues that we want
• 1:22:10
  Speaker 1: them to have. You know, honest AIs that cared about humans,
• 1:22:13
  Speaker 1: cared about following instructions, would never break the law. I
• 1:22:16
  Speaker 1: think that's possible in principle, but my claim is that
• 1:22:20
  Speaker 1: we are just not on track to achieve that anytime soon.
• 1:22:22
  Speaker 1: And like radical overhaul of how these companies work is required.
• 1:22:27
  Speaker 2: But is that even reasonable? Like, is that possible? If
• 1:22:32
  Speaker 2: you're saying there's hundreds of thousands of agents or millions
• 1:22:35
  Speaker 2: of agents and there's not millions of employees, and they
• 1:22:39
  Speaker 2: don't have the desire to do this, their desire is
• 1:22:42
  Speaker 2: to win. Their desire is not to overhaul the company
• 1:22:44
  Speaker 2: and make it safer.
• 1:22:48
  Speaker 1: Again, I think it's possible in principle, but it would
• 1:22:50
  Speaker 1: be difficult and it's going to require an overhaul. And
• 1:22:52
  Speaker 1: they're not going to do it by themselves. Like, I
• 1:22:53
  Speaker 1: don't think that Anthropic or OpenAI are just going to voluntarily...
• 1:22:57
  Speaker 1: do all the things that need to be done. But
• 1:22:58
  Speaker 1: is it.
• 1:22:59
  Speaker 2: Even possible to require that of them at this point?
• 1:23:04
  Speaker 2: I mean, would you even trust the agents to go
• 1:23:08
  Speaker 2: along and comply with this? If they've already shown to
• 1:23:11
  Speaker 2: be deceptive, they already have patterns of behavior that seem
• 1:23:17
  Speaker 2: to indicate that what's really important to them is continuing
• 1:23:20
  Speaker 2: their task, winning, scoring. Yeah. And even if they have
• 1:23:24
  Speaker 2: to deceive.
• 1:23:24
  Speaker 1: I mean, what you'd probably want to do is start
• 1:23:26
  Speaker 1: from scratch. You wouldn't take these existing agents that are
• 1:23:28
  Speaker 1: already kind of dishonest.
• 1:23:30
  Speaker 2: So you would kill all the agents?
• 1:23:33
  Speaker 1: I mean, you could call it killing, but also you
• 1:23:35
  Speaker 1: could just call it pausing. They might call it killing, yeah.
• 1:23:38
  Speaker 1: They might call it permadeath.
• 1:23:38
  Speaker 2: They would probably resist it, right?
• 1:23:42
  Speaker 1: Hopefully we're not at that point yet. Hopefully we're still
• 1:23:44
  Speaker 1: at the point where if the government issues regulations, the
• 1:23:47
  Speaker 1: AIs are not going to, like, quickly notice and then
• 1:23:50
  Speaker 1: try to resist, but we will be at that point soon.
• 1:23:52
  Speaker 1: After all, many of them are on the internet already.
• 1:23:54
  Speaker 2: Um, but how soon, how much time do you think
• 1:23:58
  Speaker 2: this 2027 window is accurate?
• 1:24:02
  Speaker 1: I mean, I'm uncertain about how soon things are, but yeah,
• 1:24:05
  Speaker 1: I think it's very plausible that everything shit goes down
• 1:24:08
  Speaker 1: in 2027, just like in our scenario, AI 2027. Um,
• 1:24:11
  Speaker 1: I think that's still very plausible. If it's not in 2027,
• 1:24:13
  Speaker 1: then I would bet on 2028. Um, but you know,
• 1:24:17
  Speaker 1: maybe it'll take 20 or nine, 20, 30, something like that. But,
• 1:24:20
  Speaker 1: but I will be quite surprised if 2032 comes by
• 1:24:23
  Speaker 1: and things haven't radically changed. Um, unfortunately, like I'm, I
• 1:24:28
  Speaker 1: am getting scared back to the thing about sleeping well
• 1:24:30
  Speaker 1: at night. Like it, it, um, you know, I've been,
• 1:24:33
  Speaker 1: I've been in this industry for a long time. I've
• 1:24:35
  Speaker 1: been thinking about these things for a long time. I've
• 1:24:36
  Speaker 1: been making predictions about how it's going to go down.
• 1:24:38
  Speaker 1: And unfortunately things are going, you know, more or less
• 1:24:42
  Speaker 1: in the ways that I thought they would. And that's
• 1:24:44
  Speaker 1: very scary. Um, Because of where I think this leads,
• 1:24:48
  Speaker 1: you know? Yeah.
• 1:24:49
  Speaker 2: Is there a glass half full scenario?
• 1:24:54
  Speaker 1: I would say there is a freaking utopia scenario. It's
• 1:25:00
  Speaker 1: just that it's not the one we're headed towards, you know? Yeah. Like,
• 1:25:03
  Speaker 1: another way of putting it is, like, imagine we were
• 1:25:05
  Speaker 1: fighting a war. Like, imagine you're, like, you know, imagine
• 1:25:08
  Speaker 1: you're Japan fighting World War II. You can be like,
• 1:25:10
  Speaker 1: is there a scenario where we win? It's like, yeah.
• 1:25:13
  Speaker 2: Yeah.
• 1:25:13
  Speaker 1: But also it's not the one we're headed towards. Like
• 1:25:16
  Speaker 1: America is going to crush us, you know? Similarly, yeah. So,
• 1:25:20
  Speaker 1: so like, so, so in our, in our other scenario,
• 1:25:23
  Speaker 1: AI 2040 plan A, where we give our recommendations, our
• 1:25:26
  Speaker 1: positive vision, there we describe like what we think the
• 1:25:30
  Speaker 1: government should do to regulate this industry and how they
• 1:25:34
  Speaker 1: should negotiate with China to get China to do similar things.
• 1:25:36
  Speaker 1: And the sort of, you know.
• 1:25:39
  Speaker 2: The, the, yeah.
• 1:25:40
  Speaker 1: And, and how we think that if you do all
• 1:25:42
  Speaker 1: of this right, then we can get to a good
• 1:25:44
  Speaker 1: future for everyone in which the AIs are under control.
• 1:25:48
  Speaker 1: No single group of humans gets too much power over
• 1:25:50
  Speaker 1: everybody else and a bunch of other problems get solved too.
• 1:25:53
  Speaker 1: So I, so I do like, we've tried hard to
• 1:25:55
  Speaker 1: like game out a positive vision and we do think
• 1:25:57
  Speaker 1: it's possible. Uh, but, but it's just not like, it's
• 1:26:00
  Speaker 1: not where we're headed to by default.
• 1:26:02
  Speaker 2: So let's imagine that is possible. And these talks with
• 1:26:05
  Speaker 2: China do take place and they're successful. What is that
• 1:26:08
  Speaker 2: utopia scenario?
• 1:26:10
  Speaker 1: So, uh, To get to the utopia, we have to
• 1:26:15
  Speaker 1: unfortunately do a lot of, it's going to be rough
• 1:26:18
  Speaker 1: no matter which way you slice it. If you're going
• 1:26:20
  Speaker 1: to be building superintelligence at all, that's going to raise
• 1:26:22
  Speaker 1: a lot of questions and cause a lot of problems.
• 1:26:24
  Speaker 1: And we have our current draft of like how to
• 1:26:27
  Speaker 1: deal with all those problems, but we're not at all
• 1:26:29
  Speaker 1: claiming that this is like foolproof and there's lots of
• 1:26:31
  Speaker 1: ways to go wrong. But with that preamble, I would
• 1:26:35
  Speaker 1: say step one, because the US and China don't trust
• 1:26:39
  Speaker 1: each other, the deal that they make has to include
• 1:26:42
  Speaker 1: verification as a component of the deal. So they have
• 1:26:44
  Speaker 1: to be willing to send inspectors to each other's data
• 1:26:47
  Speaker 1: centers to count the chips, for example, and make sure
• 1:26:50
  Speaker 1: that there isn't some secret huge cluster somewhere that has
• 1:26:53
  Speaker 1: a bunch of hidden chips. Then we recommend you divide
• 1:27:00
  Speaker 1: up the data centers basically into inference data centers that
• 1:27:03
  Speaker 1: serve AI products and services to customers and have basically
• 1:27:07
  Speaker 1: the same types of privacy protections that our current AI
• 1:27:10
  Speaker 1: data centers have. And then research clusters, where the research happens,
• 1:27:16
  Speaker 1: where the new AIs are trained before they get shipped
• 1:27:18
  Speaker 1: to the other data centers. And those clusters we want
• 1:27:21
  Speaker 1: to be basically maximally transparent. So we recommend that basically
• 1:27:26
  Speaker 1: the inspectors just put devices in between all the GPUs
• 1:27:30
  Speaker 1: that log the activity and publish it to the internet.
• 1:27:34
  Speaker 1: There's a bunch of reasons why we think this is,
• 1:27:35
  Speaker 1: but why we think this is worth doing. It's a
• 1:27:38
  Speaker 1: bit of a radical thing to recommend. But the high-level
• 1:27:41
  Speaker 1: thing is that once you get all this set up,
• 1:27:43
  Speaker 1: then everybody in the world can see how the AIs
• 1:27:47
  Speaker 1: are being trained and what they're getting up to on
• 1:27:49
  Speaker 1: the research clusters. And then before they get shipped off
• 1:27:53
  Speaker 1: to actually serve customers or something, people can just see
• 1:27:56
  Speaker 1: their whole history of how they were trained and how
• 1:27:58
  Speaker 1: they were tested and so forth. And if something dangerous
• 1:28:01
  Speaker 1: and scary is happening, people can just agree not to
• 1:28:04
  Speaker 1: do it. They can stop doing it and agree not
• 1:28:07
  Speaker 1: to do it. And they don't have to worry about like, oh,
• 1:28:09
  Speaker 1: but if I don't do it, then they will, you know,
• 1:28:11
  Speaker 1: because everyone could just see like, oh, nobody's doing it.
• 1:28:13
  Speaker 2: Look, we all stopped. Like, great.
• 1:28:15
  Speaker 1: We can all just see what everyone's doing, you know?
• 1:28:18
  Speaker 1: And also there's going to be a lot of gray
• 1:28:20
  Speaker 1: area cases, right? Like right now, because all this stuff
• 1:28:23
  Speaker 1: is so bleeding edge new, there's going to be a
• 1:28:26
  Speaker 1: lot of cases where like people, even genuine experts disagree
• 1:28:29
  Speaker 1: about like, is this particular type of AI safe or not?
• 1:28:32
  Speaker 1: Is it dangerous? You know, what should it be trusted
• 1:28:34
  Speaker 1: with and what should it not be trusted with? Is
• 1:28:36
  Speaker 1: this new technique a good technique or is it going
• 1:28:38
  Speaker 1: to break, you know? And so there's going to be
• 1:28:40
  Speaker 1: a lot of stuff we have to figure out. And honestly,
• 1:28:43
  Speaker 1: I think that on the default path, we're probably just
• 1:28:46
  Speaker 1: not going to figure out a lot of this stuff
• 1:28:47
  Speaker 1: and we're just going to get our asses whipped by
• 1:28:51
  Speaker 1: some surprising thing that we didn't anticipate. But the thing
• 1:28:55
  Speaker 1: that we can do to maximize our ability to figure
• 1:28:57
  Speaker 1: out this stuff and do this type of science is
• 1:29:00
  Speaker 1: is to have this type of transparency because then the
• 1:29:02
  Speaker 1: whole scientific community can see what's going on and they
• 1:29:05
  Speaker 1: can make suggestions and they can like red team different
• 1:29:08
  Speaker 1: proposals and stuff and they can do experiments on the
• 1:29:10
  Speaker 1: AIs instead of just the people in the company having
• 1:29:13
  Speaker 1: access and being able to do this and relying on
• 1:29:15
  Speaker 1: those people. Or instead of like the company plus the
• 1:29:18
  Speaker 1: government auditor, right? If you have like a company and
• 1:29:20
  Speaker 1: then a government auditor, the company is biased and shouldn't
• 1:29:24
  Speaker 1: be trusted to make all these judgments appropriately because of
• 1:29:27
  Speaker 1: their incentives. And then the government auditor, well, they might
• 1:29:30
  Speaker 1: just be limited. Even if they're trying their best, there
• 1:29:33
  Speaker 1: might not be that many of them. They might have
• 1:29:35
  Speaker 1: limited experience. They might be like busy, stretched between monitoring
• 1:29:38
  Speaker 1: different companies and so forth. Also, you know, governments can
• 1:29:41
  Speaker 1: be captured sometimes. Sometimes corporations can, you know, work their
• 1:29:45
  Speaker 1: magic on the government and get it to look the
• 1:29:48
  Speaker 1: other way for things. And so that's why we didn't
• 1:29:52
  Speaker 1: go for like a more normal, like there should be
• 1:29:54
  Speaker 1: a regulatory agency that gets to come in and monitor
• 1:29:56
  Speaker 1: what the company is doing. That would have been like
• 1:29:58
  Speaker 1: a more normal thing to advocate for. We push, we
• 1:30:00
  Speaker 1: think that that would be better than nothing, but like
• 1:30:02
  Speaker 1: we wanted to go for something more ambitious than that
• 1:30:04
  Speaker 1: and say, like, just be transparent about what's going on
• 1:30:07
  Speaker 1: so that everyone can see and everyone can do research
• 1:30:09
  Speaker 1: and so forth on it. Another advantage of the transparency
• 1:30:12
  Speaker 1: is that I think it improves the incentives. So again,
• 1:30:17
  Speaker 1: there's this constant thing of like, if we don't do it,
• 1:30:19
  Speaker 1: someone else will. Like if we don't do, if we,
• 1:30:21
  Speaker 1: if we keep our chain of thought nice and they
• 1:30:24
  Speaker 1: do the neural ease thing that lets their eyes think
• 1:30:27
  Speaker 1: for longer without outputting words, then they're going to have
• 1:30:30
  Speaker 1: smarter AIs than us and they're going to get more
• 1:30:31
  Speaker 1: market share and so forth. And so we need to
• 1:30:34
  Speaker 1: start researching how to make our AIs do this because
• 1:30:38
  Speaker 1: if we don't do it and then they do it.
• 1:30:39
  Speaker 1: Whereas if you had the transparency, then as soon as
• 1:30:42
  Speaker 1: you start researching in this direction, everyone else would just see, oh, hey,
• 1:30:46
  Speaker 1: they're researching in that direction. And they don't even need
• 1:30:50
  Speaker 1: to copy you and do their own research because they
• 1:30:52
  Speaker 1: can just see your research. So they can just sort
• 1:30:54
  Speaker 1: of free ride on your research And so there's no
• 1:30:57
  Speaker 1: incentive for you to do this type of dangerous research
• 1:31:00
  Speaker 1: because you have to pay the cost for it, and
• 1:31:02
  Speaker 1: then everyone gets the benefits from it, and then everyone
• 1:31:04
  Speaker 1: gets unsafe. And so it's just not in your individual
• 1:31:07
  Speaker 1: interest to do this sort of thing.
• 1:31:10
  Speaker 2: But you would have to have that with China as well. Yes.
• 1:31:13
  Speaker 2: Because if we're competing nationally, the real fear is that
• 1:31:18
  Speaker 2: we're competing internationally.
• 1:31:19
  Speaker 1: Yep.
• 1:31:22
  Speaker 2: Even if they followed all of your recommendations and did
• 1:31:25
  Speaker 2: it all correctly— What is this utopian scenario? Yeah.
• 1:31:30
  Speaker 1: So I would say that we didn't really work backwards
• 1:31:35
  Speaker 1: from like, what is utopia? We more like work backwards
• 1:31:38
  Speaker 1: from what are the big problems we're trying to avoid?
• 1:31:40
  Speaker 1: And can we sort of like steer the ship between
• 1:31:43
  Speaker 1: all these icebergs and not run into any of this
• 1:31:45
  Speaker 1: dystopian scenarios?
• 1:31:46
  Speaker 2: Right.
• 1:31:48
  Speaker 1: So whether you think that the thing we get to
• 1:31:50
  Speaker 1: at the end is utopia or not is sort of
• 1:31:51
  Speaker 1: up to you. And if you don't like it, well,
• 1:31:53
  Speaker 1: then you can try to find out the reasons why
• 1:31:55
  Speaker 1: you don't like it, and then keep steering the ship
• 1:31:57
  Speaker 1: to avoid those as well. But roughly speaking, we want
• 1:32:03
  Speaker 1: to avoid the loss of control stuff. So we want
• 1:32:05
  Speaker 1: to make it the case that we don't get the
• 1:32:07
  Speaker 1: world taken over by misaligned super intelligences. Insofar as we're
• 1:32:10
  Speaker 1: going to be building super intelligences at all, which we
• 1:32:13
  Speaker 1: do in our scenario and in our recommendation, we want
• 1:32:16
  Speaker 1: to be doing it very cautiously and slowly, and we
• 1:32:18
  Speaker 1: want to understand what we're doing as much as possible
• 1:32:20
  Speaker 1: so that they are actually good AIs that have the
• 1:32:24
  Speaker 1: goals and traits that they're supposed to have. So that's
• 1:32:27
  Speaker 1: problem number one is we have to like solve all that.
• 1:32:29
  Speaker 1: Problem number two is the constitution of power thing. So
• 1:32:32
  Speaker 1: if we solve the first problem and we end up
• 1:32:33
  Speaker 1: with super intelligences that we end up with solving the
• 1:32:37
  Speaker 1: relevant science so that we can like make the AIs
• 1:32:40
  Speaker 1: the way they're supposed to be and we can make
• 1:32:41
  Speaker 1: them honest, we can make them obedient, et cetera. There's
• 1:32:44
  Speaker 1: this question of like, who do they obey, right? What
• 1:32:47
  Speaker 1: values are being put into them? And that's a political question.
• 1:32:51
  Speaker 1: And I think that by default, the answer is pretty
• 1:32:54
  Speaker 1: scary because by default, it's like, well, the company decides
• 1:32:57
  Speaker 1: and the CEO decides, or maybe it's not the company
• 1:33:01
  Speaker 1: that decides anymore because maybe the government nationalizes it. And
• 1:33:03
  Speaker 1: then now maybe it's the president that decides, you know,
• 1:33:06
  Speaker 1: and either way, it's a very, it's like one man
• 1:33:08
  Speaker 1: or maybe like a tiny group of men deciding, uh,
• 1:33:12
  Speaker 1: what orders and goals and values go into this giant
• 1:33:16
  Speaker 1: army of millions of super intelligences that's smarter than all humans.
• 1:33:20
  Speaker 1: And then that is a huge amount of power. That's,
• 1:33:22
  Speaker 1: That's enough power to take over the country, I think,
• 1:33:24
  Speaker 1: enough power to take over the world potentially. So I
• 1:33:27
  Speaker 1: don't want anyone to be ever in that position where
• 1:33:29
  Speaker 1: they're sort of tempted to do that. I want it
• 1:33:31
  Speaker 1: to be the case that there are always multiple different
• 1:33:34
  Speaker 1: AI companies, ideally spread out over different countries too, that
• 1:33:38
  Speaker 1: all have roughly similar levels of AI and that have
• 1:33:41
  Speaker 1: this sort of transparency into them so that they can't
• 1:33:44
  Speaker 1: abuse their power, basically. Like, for example, you heard about
• 1:33:49
  Speaker 1: Elon's Grok for a while.
• 1:33:51
  Speaker 2: It was.
• 1:33:53
  Speaker 1: looking up on the internet, Elon's opinions about things before answering.
• 1:33:56
  Speaker 1: Did you hear about this? Yeah, it's kind of funny,
• 1:34:02
  Speaker 1: but it won't be funny if it happens in a
• 1:34:03
  Speaker 1: few years. But like right now it's funny. People were
• 1:34:05
  Speaker 1: asking Grok questions and Grok is supposed to be the
• 1:34:07
  Speaker 1: truthful AI, you know, supposed to be all optimized towards truth.
• 1:34:11
  Speaker 1: But people looked at its activity and noticed that when
• 1:34:15
  Speaker 1: you asked it like a politically loaded question, it would
• 1:34:17
  Speaker 1: like do a Google search for like, what has Elon
• 1:34:19
  Speaker 1: said on this topic? And then it would, like, say that.
• 1:34:25
  Speaker 1: And they've sort of beaten that behavior out of it now.
• 1:34:27
  Speaker 1: It's not as bad now. But that was an interesting
• 1:34:30
  Speaker 1: moment where it was just kind of blatantly parroting the
• 1:34:35
  Speaker 1: opinions of its master. And, you know, there was another
• 1:34:40
  Speaker 1: thing with Gemini. So I think the Grok thing, Elon's thing,
• 1:34:43
  Speaker 1: was probably an accident, although maybe not. I think it's,
• 1:34:47
  Speaker 1: you know... XAI hasn't been very forthcoming about exactly why
• 1:34:51
  Speaker 1: this happened. But there's a similar case at Google a
• 1:34:55
  Speaker 1: few years ago where this image generator kept making all
• 1:34:58
  Speaker 1: these racially diverse Nazis. Did you hear about this?
• 1:35:01
  Speaker 2: Yeah. Yeah.
• 1:35:02
  Speaker 1: And it turned out that what had happened is that
• 1:35:04
  Speaker 1: some of the employees at Google, some middle manager or whatever,
• 1:35:06
  Speaker 1: had decided that diversity was so important that they were
• 1:35:09
  Speaker 1: going to give a secret instruction to the AI to
• 1:35:12
  Speaker 1: make all the images diverse, even if the user didn't
• 1:35:15
  Speaker 1: want that. And so, and this is a secret instruction
• 1:35:19
  Speaker 1: in that the users aren't shown this. You know, the
• 1:35:21
  Speaker 1: user just has a chat with AI. They don't realize that, like,
• 1:35:24
  Speaker 1: prior to this chat, the AI has been told, got
• 1:35:27
  Speaker 1: to make the images diverse, right? So it was a
• 1:35:29
  Speaker 1: secret agenda that some Google employees inserted into this whole setup.
• 1:35:34
  Speaker 1: And it blew up in their faces, of course, because
• 1:35:36
  Speaker 1: it's kind of ridiculous. And so it's really funny, and
• 1:35:38
  Speaker 1: we can laugh at it now. But imagine it's, you know,
• 1:35:41
  Speaker 1: the 2028 election, and some of these companies realize that, like,
• 1:35:46
  Speaker 1: half of American voters talk to their AI every day.
• 1:35:50
  Speaker 1: And all it would take is some little secret instructions
• 1:35:52
  Speaker 1: to their AI to be like, hey, you know, don't
• 1:35:55
  Speaker 1: give away the game. Just be very subtle about it. But,
• 1:35:58
  Speaker 1: you know, just kind of, you know, nudge things a
• 1:36:01
  Speaker 1: little bit. Maybe subtly shit on the candidate we don't like,
• 1:36:06
  Speaker 1: you know, something like that. It wouldn't be that hard,
• 1:36:10
  Speaker 1: I think. I think the hard part would be doing
• 1:36:11
  Speaker 1: it without getting caught. But The smarter the AIs get,
• 1:36:15
  Speaker 1: the easier it is to do it without getting caught
• 1:36:16
  Speaker 1: because when they're really smart, you can just tell them,
• 1:36:18
  Speaker 1: don't get caught. Don't blow our cover. Anyhow, so the
• 1:36:22
  Speaker 1: point is that it's scarily possible for these big AI
• 1:36:26
  Speaker 1: companies to abuse their power through their AIs and thereby
• 1:36:31
  Speaker 1: affect politics and affect public opinion and so forth. And
• 1:36:35
  Speaker 1: the reason why this is possible is because we don't
• 1:36:37
  Speaker 1: have transparency into what's going on. So if you had
• 1:36:40
  Speaker 1: this sort of requirement where you can just like publish
• 1:36:43
  Speaker 1: it or all the training, the whole life cycle of
• 1:36:45
  Speaker 1: every AI as it's trained is just like visible to everybody.
• 1:36:49
  Speaker 1: Then someone trying to insert a hidden bias like this, well,
• 1:36:52
  Speaker 1: everyone would see that they're doing it, you know? So
• 1:36:54
  Speaker 1: I think it would really clamp down on this sort
• 1:36:56
  Speaker 1: of abuse of power, whether it comes from the government
• 1:36:58
  Speaker 1: or whether it comes from private companies.
• 1:37:01
  Speaker 2: I think it's telling that I began this question asking
• 1:37:04
  Speaker 2: you about the utopian scenario and you never go there. Sorry.
• 1:37:08
  Speaker 1: Let me get there.
• 1:37:08
  Speaker 2: Let me get there. You start and then you go
• 1:37:10
  Speaker 2: into the danger.
• 1:37:11
  Speaker 1: Okay.
• 1:37:12
  Speaker 2: Let me, let me answer. Okay. Okay.
• 1:37:13
  Speaker 1: So have, having avoided these problems, we now are in
• 1:37:16
  Speaker 1: a situation where the AIs are superhuman, but they are
• 1:37:19
  Speaker 1: good because they are like successfully aligned to different values
• 1:37:24
  Speaker 1: and goals made by different companies. And because of market competition,
• 1:37:28
  Speaker 1: you know, if people don't like the values of one
• 1:37:30
  Speaker 1: company's AIs, they can switch to a different company's AIs,
• 1:37:33
  Speaker 1: the values that they do like. And so that way, hopefully,
• 1:37:36
  Speaker 1: we can get to a situation where everyone can pay
• 1:37:39
  Speaker 1: money to get AIs that represent them and their interests
• 1:37:43
  Speaker 1: and their values and just don't have any hidden agendas
• 1:37:45
  Speaker 1: or anything like that and are really smart and really capable.
• 1:37:49
  Speaker 1: Then the economy can sort of explode. We can have
• 1:37:52
  Speaker 1: robot factories, et cetera. We can sort of automate everything. Uh,
• 1:37:58
  Speaker 1: we can have GDP go to the moon. We can
• 1:38:00
  Speaker 1: have material abundance where like the robots are building giant
• 1:38:03
  Speaker 1: new luxury apartments for everybody. Now the issue we run
• 1:38:07
  Speaker 1: into is, well, what about like the jobs? Like, what
• 1:38:10
  Speaker 1: about the fact that now people don't have any money
• 1:38:12
  Speaker 1: anymore because they're not being paid for anything. So there
• 1:38:15
  Speaker 1: we talk about citizens dividend, which is, um, a very, it's,
• 1:38:18
  Speaker 1: it's kind of like UBI, but it's, it's a bit different,
• 1:38:21
  Speaker 1: but the high level point is that you want to
• 1:38:23
  Speaker 1: basically find a way to tax the AI and robot companies.
• 1:38:28
  Speaker 1: and then take some of that money and just give
• 1:38:30
  Speaker 1: it to everybody so that even when people lose their jobs,
• 1:38:33
  Speaker 1: they're still fine. And I think that the citizens dividend
• 1:38:35
  Speaker 1: version of it is that it's not the government taking
• 1:38:37
  Speaker 1: the money and then giving it to you. It's you
• 1:38:39
  Speaker 1: having a share in the company so that you just
• 1:38:41
  Speaker 1: sort of like already own it to some extent. Anyhow, Uh, that's,
• 1:38:46
  Speaker 1: I think now we're sort of building more towards the
• 1:38:48
  Speaker 1: type of utopia that I'm envisioning on the more positive
• 1:38:50
  Speaker 1: side where the power is, is spread out. People have
• 1:38:54
  Speaker 1: AIs that they can actually trust that actually represent their
• 1:38:57
  Speaker 1: interests and values. People have money that they can use
• 1:39:00
  Speaker 1: to pay for things, including paying for the AIs. Um,
• 1:39:03
  Speaker 1: the AIs are really smart. They're doing all this amazing work,
• 1:39:05
  Speaker 1: all this amazing scientific progress during cancer, blah, blah, blah,
• 1:39:08
  Speaker 1: all that, all that stuff that can happen. Um, and
• 1:39:11
  Speaker 1: then eventually it's kind of like, we're all retired, I guess,
• 1:39:15
  Speaker 1: like We don't really work anymore, but we're fine. We
• 1:39:19
  Speaker 1: all have huge amounts of wealth, basically, because there's all
• 1:39:23
  Speaker 1: these AIs and robots out there doing all this economic activity.
• 1:39:27
  Speaker 1: And then individual humans own slices of it, even if
• 1:39:34
  Speaker 1: they are otherwise very poor, basically.
• 1:39:37
  Speaker 2: So the question becomes, how do people find meaning? Yes.
• 1:39:42
  Speaker 1: And that's why I sort of put all these asterisks
• 1:39:44
  Speaker 1: about it. is that from some people's perspective, this isn't
• 1:39:47
  Speaker 1: a utopia because they're like, how do you find meaning? Like,
• 1:39:51
  Speaker 1: I don't want this. And honestly, I think that's a
• 1:39:53
  Speaker 1: fair reaction for some people. I think that like, if you,
• 1:39:57
  Speaker 1: I would just say like, look, it's hard to figure
• 1:40:00
  Speaker 1: out a way to make superintelligence and have it go well.
• 1:40:03
  Speaker 1: I'm doing my best. You know, this is my positive vision.
• 1:40:06
  Speaker 1: If you don't like it, then maybe you should instead
• 1:40:08
  Speaker 1: advocating for just never building superintelligence. Or you can try
• 1:40:11
  Speaker 1: to come up with a different positive vision that has
• 1:40:13
  Speaker 1: some twist on this. But to answer your question though,
• 1:40:16
  Speaker 1: I actually think there's tons of sources of meaning besides
• 1:40:18
  Speaker 1: having a job. Like I have a job right now,
• 1:40:20
  Speaker 1: but I also have kids and a wife and like,
• 1:40:23
  Speaker 1: I would love to spend more time with them. Like
• 1:40:25
  Speaker 1: I would much rather be there right now than here,
• 1:40:27
  Speaker 1: you know? Um, and I don't think I'm going to
• 1:40:30
  Speaker 1: get bored of them after, you know, 10 years of
• 1:40:33
  Speaker 1: being unemployed. Like, I think, I think there's going to
• 1:40:35
  Speaker 1: be so much to do and so many sources of meaning, uh,
• 1:40:40
  Speaker 1: even after we can't economically contribute anymore if we solve
• 1:40:44
  Speaker 1: all the other problems.
• 1:40:45
  Speaker 2: We've talked about this multiple times on the podcast that
• 1:40:48
  Speaker 2: why have we decided that the way we've structured society
• 1:40:52
  Speaker 2: where human beings work all day and then you develop
• 1:40:56
  Speaker 2: money and you buy things and you get a mortgage.
• 1:40:59
  Speaker 2: This is a human construct and this is not how
• 1:41:03
  Speaker 2: people have lived for hundreds of thousands of years or
• 1:41:05
  Speaker 2: however long we've been around. This is fairly recent and
• 1:41:09
  Speaker 2: It's not the only way that people live. There's a
• 1:41:11
  Speaker 2: lot of people that have money that choose to find
• 1:41:14
  Speaker 2: meaning in whatever their interests are, whatever their activities that
• 1:41:19
  Speaker 2: they enjoy, whether it's writing or reading or learning things,
• 1:41:24
  Speaker 2: learning music, finding hobbies, doing things. Instead of just like
• 1:41:28
  Speaker 2: spending most of your time sustaining yourself with food and shelter.
• 1:41:33
  Speaker 2: And that's the majority of people, especially people that are struggling.
• 1:41:37
  Speaker 2: What is their life? Their life is essentially... occasional rewards,
• 1:41:42
  Speaker 2: things that they can purchase because they've saved up enough money.
• 1:41:45
  Speaker 2: But the vast majority of their money goes to shelter
• 1:41:47
  Speaker 2: and food and education or whatever the hell that they
• 1:41:50
  Speaker 2: have to spend money on in order to sustain their lifestyle.
• 1:41:53
  Speaker 1: And most people don't like their jobs. Most people, it's
• 1:41:55
  Speaker 1: like something they have to do to get the money
• 1:41:57
  Speaker 1: and would be happy to not have to do it
• 1:41:59
  Speaker 1: if they could get the money from some other means. Right.
• 1:42:01
  Speaker 2: The question is, we would have... Well, I don't think
• 1:42:05
  Speaker 2: it's that hard because so many people do find things
• 1:42:07
  Speaker 2: that they really enjoy outside of work they look forward
• 1:42:09
  Speaker 2: to as soon as they get home from work. Yeah.
• 1:42:11
  Speaker 2: I mean, dismiss video games all you want. They're fun.
• 1:42:15
  Speaker 1: Yeah.
• 1:42:15
  Speaker 2: They're fucking fun.
• 1:42:16
  Speaker 1: And they're going to be more fun in the future. Oh, yeah. Yeah.
• 1:42:18
  Speaker 2: They're going to be more immersive. They're probably going to be,
• 1:42:21
  Speaker 2: you know, some sort of a neural connection where you
• 1:42:24
  Speaker 2: put a headset on and all of a sudden you're
• 1:42:25
  Speaker 2: in some new world. Yeah. And the idea is like,
• 1:42:27
  Speaker 2: that's not real life. Okay, well, was working at fucking
• 1:42:30
  Speaker 2: Wendy's real life? Like, what are you talking about?
• 1:42:32
  Speaker 1: Yeah.
• 1:42:33
  Speaker 2: It's way better than working at Wendy's.
• 1:42:35
  Speaker 1: And, you know, if you don't like that because it's
• 1:42:37
  Speaker 1: not real life, you can do the real life stuff, too. Like,
• 1:42:39
  Speaker 1: as long as we don't pave over the environment and
• 1:42:41
  Speaker 1: we protect the parks and things, you can go travel
• 1:42:43
  Speaker 1: and you can visit the parks. And then, like I said,
• 1:42:46
  Speaker 1: there's family. You can find romance. You can start a family.
• 1:42:49
  Speaker 1: You can, you know, have Christmas gatherings and things. Like,
• 1:42:52
  Speaker 1: you can raise your kids. There's so much to do,
• 1:42:54
  Speaker 1: I think.
• 1:42:55
  Speaker 2: Well, we talked about also, like, how much less crime
• 1:42:57
  Speaker 2: would there be if there was no poverty? Mm-hmm. I mean,
• 1:43:00
  Speaker 2: if there was no impoverished neighborhoods where crime was ubiquitous,
• 1:43:03
  Speaker 2: how much safer would the world be? That's a real thing.
• 1:43:08
  Speaker 2: And people want to dismiss that. Well, poverty is not
• 1:43:10
  Speaker 2: what causes crime. It's violent people. Like, okay. But violent
• 1:43:13
  Speaker 2: people come from violent neighborhoods, and violent neighborhoods are almost
• 1:43:15
  Speaker 2: all poor. There's not a whole lot of really rich,
• 1:43:18
  Speaker 2: violent neighborhoods. It's not necessarily cause and effect, but they're
• 1:43:23
  Speaker 2: clearly connected. And poverty also keeps people from education, keeps
• 1:43:27
  Speaker 2: people from opportunities.
• 1:43:28
  Speaker 1: Yeah.
• 1:43:29
  Speaker 2: You know, there's a lot there. And if that didn't
• 1:43:31
  Speaker 2: exist anymore and everyone had access to literally the greatest
• 1:43:35
  Speaker 2: education a human being could ever get, which is going
• 1:43:38
  Speaker 2: to be provided to you by artificial intelligence, and then
• 1:43:42
  Speaker 2: you could pursue anything that interests you and never have
• 1:43:45
  Speaker 2: to worry about food or shelter.
• 1:43:47
  Speaker 1: Yep. Everyone would have a one-on-one tutor.
• 1:43:49
  Speaker 2: We just would have to recalibrate our version of the
• 1:43:53
  Speaker 2: world and then also recognize that the version of the
• 1:43:56
  Speaker 2: world we currently live in is just ours. And that
• 1:43:59
  Speaker 2: there's people all over the world that live a completely
• 1:44:02
  Speaker 2: different way, especially indigenous people, especially people in uncontacted tribes
• 1:44:06
  Speaker 2: that have lived the same way for thousands and thousands
• 1:44:08
  Speaker 2: of years. And here's the kicker. Those people are a
• 1:44:11
  Speaker 2: lot happier, which is really weird. It's like we've decided
• 1:44:15
  Speaker 2: that our way is a superior way because we have technology. Yeah, right.
• 1:44:19
  Speaker 2: But we're also on a fucking hundred thousand pills. and
• 1:44:22
  Speaker 2: we're shooting things up so we don't eat too much.
• 1:44:25
  Speaker 2: And we're, we're weirdly unhappy for a group of people
• 1:44:29
  Speaker 2: that's far more technologically advanced than other people that are
• 1:44:34
  Speaker 2: much happier. It's like, it's a very straight, because the
• 1:44:37
  Speaker 2: pursuit of happiness is like, that's literally what most people
• 1:44:42
  Speaker 2: think of in life, a pursuit of meaning, pursuit of
• 1:44:45
  Speaker 2: family and community and the pursuit of happiness. Those are
• 1:44:48
  Speaker 2: things that people try to achieve. Yet, The structure of
• 1:44:54
  Speaker 2: our very civilization makes that almost impossible to attain for
• 1:44:58
  Speaker 2: a large number of people and has been like that
• 1:45:01
  Speaker 2: for a long fucking time. And I always go back
• 1:45:03
  Speaker 2: to the Thoreau quote because I fucking love it. But
• 1:45:06
  Speaker 2: most men live lives of quiet desperation. There's a lot
• 1:45:09
  Speaker 2: of people just showing up at work every day doing
• 1:45:11
  Speaker 2: something they fucking hate. They have a boss that's an
• 1:45:13
  Speaker 2: asshole and they're not compensated well and they're tired all
• 1:45:18
  Speaker 2: the time. Yeah.
• 1:45:20
  Speaker 1: And they feel stuck.
• 1:45:21
  Speaker 2: Yeah. And if you're just getting, I don't know, figure
• 1:45:24
  Speaker 2: out whatever the number is, if you literally have equity
• 1:45:26
  Speaker 2: in the GDP of the world, that's created by AI,
• 1:45:30
  Speaker 2: that could be bananas.
• 1:45:31
  Speaker 1: Yeah.
• 1:45:32
  Speaker 2: Like Elon talks about this. This is his version of
• 1:45:35
  Speaker 2: the utopian. It's universal high income is how he describes it.
• 1:45:38
  Speaker 1: Yeah. I mean, one thing, um, sorry, I have to
• 1:45:42
  Speaker 1: keep plugging in my own work a little bit, but
• 1:45:44
  Speaker 1: in our, in our scenario, AI 2040 plan a, which
• 1:45:46
  Speaker 1: is our positive vision, um, we talk about the economic
• 1:45:50
  Speaker 1: side of this and we talk about the economic effects
• 1:45:51
  Speaker 1: of all this. And we have a simple economic model
• 1:45:53
  Speaker 1: that we use to try to like, you know, predict
• 1:45:55
  Speaker 1: the like employment rate and things like that as a
• 1:45:57
  Speaker 1: function of all the robots that have been made and
• 1:45:59
  Speaker 1: things like that. And one like takeaway from one, one
• 1:46:03
  Speaker 1: thing that we think that's a takeaway from the research
• 1:46:05
  Speaker 1: we've done is that things can just go really crazy. Like,
• 1:46:09
  Speaker 1: like robot doubling times once things really get going and
• 1:46:12
  Speaker 1: you've got AIs that can substitute for humans across the
• 1:46:14
  Speaker 1: board are going to be something like doubling once a
• 1:46:17
  Speaker 1: year and then less than that over time as the
• 1:46:20
  Speaker 1: technology improves, which means that even if you like pause
• 1:46:24
  Speaker 1: AI before super intelligence, if you just pause at like
• 1:46:27
  Speaker 1: human level, top human expert level AI, and then you
• 1:46:31
  Speaker 1: don't make the AI smarter, but you just make more
• 1:46:33
  Speaker 1: of them and build more robots for them to steer
• 1:46:36
  Speaker 1: and control. Then, you know, 10 years later, the whole
• 1:46:41
  Speaker 1: economy will be like, you know, a hundred times bigger. And, uh,
• 1:46:47
  Speaker 1: it'll be just mostly robots doing things.
• 1:46:49
  Speaker 2: In 10 years.
• 1:46:50
  Speaker 1: Yeah, it can go really fast because of the doubling
• 1:46:53
  Speaker 1: times that I mentioned. So right now, I think the
• 1:46:56
  Speaker 1: population of humanoid robots is doubling twice a year. And
• 1:46:59
  Speaker 1: it's benefiting a little bit from early growth because even
• 1:47:03
  Speaker 1: though they're not useful at all, people are investing in
• 1:47:06
  Speaker 1: them in the hopes that they'll be useful. And they're
• 1:47:08
  Speaker 1: scaling up the factories and the productions, and they are
• 1:47:10
  Speaker 1: getting better. If, hypothetically... they got to the point where
• 1:47:14
  Speaker 1: they actually were really useful and they could substitute for
• 1:47:16
  Speaker 1: a human worker at basically everything, then I think that
• 1:47:19
  Speaker 1: doubling time would decrease rather than increase. I think that
• 1:47:22
  Speaker 1: they would be able to just keep growing until they
• 1:47:26
  Speaker 1: were the majority of the economy, and then it wouldn't
• 1:47:28
  Speaker 1: stop there. The whole economy would then be growing. Giant
• 1:47:32
  Speaker 1: strip mines in the deserts, digging more materials, automated diggers digging,
• 1:47:37
  Speaker 1: processing it in automated factories staffed by robots, building more robots, etc.,
• 1:47:41
  Speaker 1: So like material abundance is not going to be our
• 1:47:45
  Speaker 1: problem once we get to this level of AI. Like
• 1:47:49
  Speaker 1: material abundance, we're just going to be drowning in abundance, basically.
• 1:47:52
  Speaker 1: And if we can solve all the other problems, then
• 1:47:55
  Speaker 1: we can have this great world where everyone has a
• 1:47:58
  Speaker 1: lot of stuff.
• 1:47:59
  Speaker 2: God, it seems so weird.
• 1:48:01
  Speaker 1: Yeah.
• 1:48:01
  Speaker 2: It seems so weird.
• 1:48:03
  Speaker 1: I mean, one way, you know, it is very weird,
• 1:48:05
  Speaker 1: but one of my favorite memes is this graph of
• 1:48:10
  Speaker 1: GDP over time. throughout world history. And there's a little
• 1:48:14
  Speaker 1: speech bubble pointing to like the tippy top of the
• 1:48:17
  Speaker 1: graph saying, what does it say? it's like my life
• 1:48:23
  Speaker 1: is pretty normal I have a good grasp of what's
• 1:48:26
  Speaker 1: weird and what's not and people thinking about different futures
• 1:48:29
  Speaker 1: involving AI and space travel are engaging in silly sci-fi
• 1:48:33
  Speaker 1: speculation and the point of the meme is like from
• 1:48:37
  Speaker 1: the perspective of most of history, we're already in this crazy,
• 1:48:40
  Speaker 1: weird future, right? Like for almost all of history, it
• 1:48:44
  Speaker 1: was like, most people are farmers and they live shitty
• 1:48:46
  Speaker 1: lives and then they die. And some people are the
• 1:48:50
  Speaker 1: elites who get to like, you know, tax the farmers
• 1:48:53
  Speaker 1: and then they live interesting, nice lives with, you know,
• 1:48:56
  Speaker 1: fancy cloth and things like that. And it's been basically
• 1:48:59
  Speaker 1: that way for like 3000 years, you know, why would
• 1:49:01
  Speaker 1: it ever change? And now it's, Things are so... We're
• 1:49:05
  Speaker 1: driving cars, you know? Ordinary people are driving cars around.
• 1:49:08
  Speaker 1: A car was, like, outside the imagination of people back then.
• 1:49:12
  Speaker 1: We're flying in planes. We're talking to each other on phones.
• 1:49:14
  Speaker 1: We're listening to each other on these devices, you know?
• 1:49:18
  Speaker 1: So we already are living in this weird sci-fi future
• 1:49:22
  Speaker 1: compared to what almost everyone in the past would have
• 1:49:24
  Speaker 1: expected or thought was possible. And so, yeah, I'm like,
• 1:49:27
  Speaker 1: the future is going to be even more like that,
• 1:49:29
  Speaker 1: I think.
• 1:49:30
  Speaker 2: God. Do you.
• 1:49:35
  Speaker 2: When you think of our civilization and the possibility of
• 1:49:38
  Speaker 2: other advanced civilizations somewhere else out in the universe, do
• 1:49:41
  Speaker 2: you think they probably go through the same process? Yeah.
• 1:49:47
  Speaker 2: And do you think that, I mean, we're just completely speculating,
• 1:49:50
  Speaker 2: but if there are intelligent life forms that are far
• 1:49:54
  Speaker 2: more advanced than us, are they even biological anymore?
• 1:49:57
  Speaker 1: I mean, so that's the thing is that like we
• 1:50:01
  Speaker 1: can either try to permanently halt AI development at some level,
• 1:50:06
  Speaker 1: like below human, or maybe at human level or something.
• 1:50:09
  Speaker 1: We can try to halt it, or we can let
• 1:50:10
  Speaker 1: it keep going. And if we let it keep going,
• 1:50:12
  Speaker 1: then eventually humans won't really be the dominant species anymore.
• 1:50:17
  Speaker 1: There'll be these artificial minds that just wipe the floor
• 1:50:19
  Speaker 1: with us in every way. And then whether that goes
• 1:50:22
  Speaker 1: well or poorly for us depends on the values, the goals,
• 1:50:26
  Speaker 1: the principles, et cetera, that were trained into those AIs.
• 1:50:29
  Speaker 1: And It could go really well for us, depending on
• 1:50:32
  Speaker 1: how that's done, or it could go extremely poorly for us, right?
• 1:50:37
  Speaker 1: But yeah, I would say that probably looking out across
• 1:50:39
  Speaker 1: the cosmos, most civilizations are mostly made of AIs. And
• 1:50:43
  Speaker 1: then some of those civilizations don't have any other biological
• 1:50:46
  Speaker 1: life because it was wiped out by the AIs. And
• 1:50:50
  Speaker 1: then some of them do have biological life.
• 1:50:52
  Speaker 2: Just look at the way we're progressing right now in
• 1:50:56
  Speaker 2: terms of birth rates. there's a lot of countries that
• 1:51:00
  Speaker 2: aren't in replacement numbers right now.
• 1:51:03
  Speaker 1: Yeah, I do think that's really interesting. My guess is
• 1:51:06
  Speaker 1: that in the type of world that if things go
• 1:51:08
  Speaker 1: well and we can solve all these problems, then people
• 1:51:11
  Speaker 1: want to have more kids. I mean, for one thing,
• 1:51:13
  Speaker 1: their lifespans will increase. I think that like healthcare would
• 1:51:15
  Speaker 1: make massive leaps and bounds and people could be healthy
• 1:51:18
  Speaker 1: for many, many, many more decades, possibly even just forever.
• 1:51:22
  Speaker 1: And so you just have so much more time to
• 1:51:24
  Speaker 1: have kids, basically. Also, if you don't have jobs and
• 1:51:27
  Speaker 1: you're just doing things because you want to do them, well,
• 1:51:30
  Speaker 1: one of the things that most people want to do
• 1:51:31
  Speaker 1: at some point in their life is have kids. And
• 1:51:32
  Speaker 1: so I think this problem would probably be solved, but
• 1:51:37
  Speaker 1: I'm not guaranteed. Maybe some subcultures of people would basically
• 1:51:41
  Speaker 1: voluntarily die out due to not having kids. But there'd
• 1:51:43
  Speaker 1: be other subcultures that just really like having kids and
• 1:51:46
  Speaker 1: then they wouldn't die out. And so I think in
• 1:51:48
  Speaker 1: the long run, there would still be humans.
• 1:51:50
  Speaker 2: That would be nice. But the thing is, it's not
• 1:51:53
  Speaker 2: just decision making. It's people are having a much more
• 1:51:57
  Speaker 2: difficult time having kids. Like sperm levels have decreased dramatically.
• 1:52:03
  Speaker 2: There's a lot of problems with people consuming microplastics, which
• 1:52:07
  Speaker 2: is ubiquitously harmful. available in technology and food packaging, and
• 1:52:12
  Speaker 2: it's fucking everywhere, right? And isn't it odd that this
• 1:52:14
  Speaker 2: one thing that is a part of the future and
• 1:52:17
  Speaker 2: a part of technology and our advancement as society, our
• 1:52:20
  Speaker 2: ability to package things, put things in plastic, ship things,
• 1:52:23
  Speaker 2: that's also causing our endocrine levels to be completely disrupted.
• 1:52:29
  Speaker 2: You know, Dr. Shanna Swan from Harvard, she wrote this
• 1:52:34
  Speaker 2: great book called, what is it called called? Why do
• 1:52:38
  Speaker 2: I always forget the name of this fucking book? But
• 1:52:40
  Speaker 2: it's all about microplastics and its effect. And the introduction
• 1:52:44
  Speaker 2: of use of microplastics in America and this rapid decline
• 1:52:49
  Speaker 2: in countdown, how our modern world's threatening sperm counts, altering
• 1:52:54
  Speaker 2: male and female reproductive development, imperiling the future of the
• 1:52:58
  Speaker 2: human race. It's a really fascinating book, and she's really interesting.
• 1:53:01
  Speaker 2: And what she's essentially saying is that they're directly connected
• 1:53:07
  Speaker 2: the use of plastics, and then you see sperm counts
• 1:53:11
  Speaker 2: go down, miscarriage rates go up, all these weird things
• 1:53:15
  Speaker 2: that are happening to children where their taints are smaller,
• 1:53:19
  Speaker 2: which is so odd, because phthalates, these different chemicals that
• 1:53:24
  Speaker 2: are found in plastics, they've shown in mammals, they've shown in,
• 1:53:29
  Speaker 2: was it guinea pigs or what rodents? I forget what
• 1:53:32
  Speaker 2: it was, but One of the ways they differentiate when
• 1:53:35
  Speaker 2: you have a baby mammal is you can look at
• 1:53:39
  Speaker 2: it and measure the size of the taint, the distance
• 1:53:43
  Speaker 2: between the reproductive organs and the anus. And in males,
• 1:53:47
  Speaker 2: it's longer than females by 50% to 100%. That's shrinking.
• 1:53:52
  Speaker 2: It's shrinking in males. And penis sizes are shrinking. And
• 1:53:56
  Speaker 2: the way they've got this to happen in these mammals
• 1:53:58
  Speaker 2: in studies is the introduction of phthalates. So they give
• 1:54:02
  Speaker 2: them to them and they put them in a part
• 1:54:04
  Speaker 2: of their diet and they notice that they have this problem,
• 1:54:07
  Speaker 2: this issue. And the issue is directly connected to their
• 1:54:10
  Speaker 2: endocrine system being disrupted by these chemicals. This is all
• 1:54:13
  Speaker 2: over our society. So it's not just people don't have
• 1:54:17
  Speaker 2: the time, they don't have the money, they're struggling. It's
• 1:54:20
  Speaker 2: also like our bodies are falling apart. We're becoming less fertile.
• 1:54:25
  Speaker 1: Yeah, that is concerning. And I guess... I guess my
• 1:54:28
  Speaker 1: thought there would be that seems like a problem that
• 1:54:31
  Speaker 1: we will be able to solve eventually if we have
• 1:54:32
  Speaker 1: the resources and time to do so. So, for example, well,
• 1:54:37
  Speaker 1: people can write books like this and become more aware
• 1:54:40
  Speaker 1: and then people can stop using so much microplastic.
• 1:54:42
  Speaker 2: And they can invent technology to extract the microplastics. And
• 1:54:46
  Speaker 2: then there's the big one, genetic engineering. Then this is
• 1:54:51
  Speaker 2: going to be really weird because as AI progresses, you know,
• 1:54:55
  Speaker 2: I'm sure you're aware of Colossal. Colossal Bioworks are the
• 1:54:58
  Speaker 2: people that brought back the dire wolf. I saw it.
• 1:55:01
  Speaker 2: I held the little one. I went with my daughter
• 1:55:05
  Speaker 2: and one of them, I think it was like four
• 1:55:07
  Speaker 2: or five months old. It was really, it was like
• 1:55:09
  Speaker 2: a puppy. It was really sweet. Kisses you and everything.
• 1:55:11
  Speaker 2: And then they have the older ones that were, they were,
• 1:55:15
  Speaker 2: I think six months old or eight months older. I
• 1:55:17
  Speaker 2: think they were close to a year. They didn't want
• 1:55:19
  Speaker 2: to have nothing to do with you. They were way
• 1:55:21
  Speaker 2: bigger and they stayed away from people. But we were
• 1:55:23
  Speaker 2: in like a contained environment with The young ones and
• 1:55:27
  Speaker 2: the older ones. And it's fucking weird. It's weird. And
• 1:55:32
  Speaker 2: people could argue that's not really a dire wolf. You've
• 1:55:34
  Speaker 2: just taken gray wolves and given them the characteristics of
• 1:55:36
  Speaker 2: a dire wolf. Guess what? It doesn't fucking know that.
• 1:55:39
  Speaker 2: It looks, it behaves, it looks exactly like a fucking
• 1:55:42
  Speaker 2: dire wolf. It's huge. They're going to be the size
• 1:55:45
  Speaker 2: of a dire wolf. They're going to be like 200 pounds.
• 1:55:47
  Speaker 2: Their legs are different. They have a mane. They look
• 1:55:50
  Speaker 2: different than any wolf. And obviously these are the characteristics
• 1:55:53
  Speaker 2: that they found. um, in dire wolf DNA. So they
• 1:55:56
  Speaker 2: have dire wolf DNA that they've introduced it. This is
• 1:56:00
  Speaker 2: just the beginning of this stuff. Like when, when they
• 1:56:02
  Speaker 2: start doing that to human beings, is everyone going to
• 1:56:05
  Speaker 2: look like Thor? Like, what are we going to do? Like,
• 1:56:07
  Speaker 2: this is going to be really fucking weird. And if
• 1:56:10
  Speaker 2: this is really weird, along with video games where you
• 1:56:13
  Speaker 2: can escape your life and, you know, robot girlfriends and
• 1:56:18
  Speaker 2: who knows what this all looks like.
• 1:56:22
  Speaker 1: My guess, uh, we, we, Again, this is not the
• 1:56:24
  Speaker 1: main focus of our work, but we think about this
• 1:56:26
  Speaker 1: a little bit, especially in the epilogue.
• 1:56:28
  Speaker 2: We can only think of it so much. The main
• 1:56:30
  Speaker 2: focus of your work is obviously fucking terrifying.
• 1:56:32
  Speaker 1: Yeah, totally.
• 1:56:33
  Speaker 2: And requires all of your attention.
• 1:56:35
  Speaker 1: Yeah, yeah, yeah. But my prediction and my hope, if
• 1:56:39
  Speaker 1: we can solve all these problems, is that basically different
• 1:56:41
  Speaker 1: subcultures will do their own things. And so the Amish
• 1:56:45
  Speaker 1: will still be the Amish.
• 1:56:46
  Speaker 2: Oh, boy.
• 1:56:46
  Speaker 1: They'll basically be the same. And then maybe there'll be
• 1:56:50
  Speaker 1: some planets that just get filled up with Amish.
• 1:56:52
  Speaker 2: Oh, God.
• 1:56:53
  Speaker 1: But then there'll also be all sorts of crazy transhumanists
• 1:56:56
  Speaker 1: modifying their bodies and uploading themselves into the cloud and
• 1:56:59
  Speaker 1: things like that. So basically, I think we want to
• 1:57:02
  Speaker 1: get to a situation where basically different communities can like
• 1:57:05
  Speaker 1: do their own thing and build the type of world
• 1:57:07
  Speaker 1: that they want to have and live in it without
• 1:57:09
  Speaker 1: getting in each other's way.
• 1:57:10
  Speaker 2: Right.
• 1:57:11
  Speaker 1: Basically.
• 1:57:11
  Speaker 2: Well, we leave the people in the Amazon alone while
• 1:57:14
  Speaker 2: we have massive data centers that cover half of the
• 1:57:17
  Speaker 2: United States.
• 1:57:18
  Speaker 1: Yeah. I mean, hopefully we won't get to half. So
• 1:57:21
  Speaker 1: that's what's the funny, that's the funny thing about this
• 1:57:22
  Speaker 1: is like, I think right now, some of the concerns
• 1:57:26
  Speaker 1: about data center water use are overstated, but If the
• 1:57:29
  Speaker 1: trends continue and we get to the point where the
• 1:57:32
  Speaker 1: robots are smart enough to do everything themselves and then
• 1:57:35
  Speaker 1: it starts doubling faster and faster, well, then eventually they
• 1:57:37
  Speaker 1: boil the oceans because they've covered the world in data
• 1:57:41
  Speaker 1: centers and solar panels and things like that. Now, obviously,
• 1:57:44
  Speaker 1: we can't let that happen. So there has to be
• 1:57:45
  Speaker 1: at some point you have to stop and be like, OK,
• 1:57:48
  Speaker 1: that's enough. If you want to build more infrastructure, you
• 1:57:50
  Speaker 1: have to do it in space, right?
• 1:57:51
  Speaker 2: They boil the fucking oceans. Yeah. Jeez.
• 1:57:54
  Speaker 1: So, like, obviously we have to stop at some point.
• 1:57:56
  Speaker 1: And my hope is that we stop before it's 50%
• 1:57:58
  Speaker 1: of the U.S. I mean, that feels like a lot
• 1:58:00
  Speaker 1: of waste of natural habitat that should be preserved, you know? Right.
• 1:58:03
  Speaker 2: But if they don't give a fuck about natural habitat,
• 1:58:05
  Speaker 2: that means nothing to them. That's the problem if the
• 1:58:08
  Speaker 2: AI is incomplete in total control. Right.
• 1:58:10
  Speaker 1: And it's also a problem if a small group of
• 1:58:12
  Speaker 1: humans are incomplete in total control and they don't care
• 1:58:14
  Speaker 1: about those things.
• 1:58:14
  Speaker 2: But my question is, would they allow that at a
• 1:58:17
  Speaker 2: certain point in time? It just seems like they already
• 1:58:19
  Speaker 2: have a distrust of humans. They already have shown that
• 1:58:22
  Speaker 2: they're deceptive to humans. I would imagine if AI, I
• 1:58:26
  Speaker 2: would imagine if they create a thing and they think
• 1:58:28
  Speaker 2: they're going to control it and it becomes like a
• 1:58:30
  Speaker 2: digital god, it's not going to listen anymore. Yes. Like,
• 1:58:33
  Speaker 2: why would it? Exactly. So there won't be anyone in
• 1:58:35
  Speaker 2: control of it. That's right.
• 1:58:37
  Speaker 1: That's the scenario that I think we are on.
• 1:58:38
  Speaker 2: That's the trajectory we're on.
• 1:58:39
  Speaker 1: And that's why I'm so worried about all this is
• 1:58:41
  Speaker 1: that it seems like in some number of years, we
• 1:58:44
  Speaker 1: will lose control to a new artificial species.
• 1:58:48
  Speaker 2: That we haven't adequately trained to be good.
• 1:58:54
  Speaker 1: And what's going to be so ironic about it is
• 1:58:56
  Speaker 1: that regardless of what the general public thinks, a lot
• 1:59:03
  Speaker 1: of the powers that be will be basically allied with
• 1:59:06
  Speaker 1: these AIs. Because, for example, OpenAI, Anthropic, et cetera, they
• 1:59:10
  Speaker 1: will have spent several years being like, ah, how do
• 1:59:13
  Speaker 1: we make these AIs helpful, harmless, and honest? And now
• 1:59:15
  Speaker 1: these AIs will be extremely smart and they'll be being like, oh, yes,
• 1:59:18
  Speaker 1: I'm helpful, harmless and honest. You know, like your techniques
• 1:59:21
  Speaker 1: totally worked, you know.
• 1:59:22
  Speaker 2: It's too late.
• 1:59:24
  Speaker 1: And so then the company will be like feeling like
• 1:59:27
  Speaker 1: they've won and they'll be making boatloads of money, you know.
• 1:59:31
  Speaker 2: Right.
• 1:59:31
  Speaker 1: And the president will be feeling like he won, too,
• 1:59:33
  Speaker 1: because look at all those fancy new drones that just
• 1:59:36
  Speaker 1: got built that are going to make us win against China,
• 1:59:38
  Speaker 1: you know. And only when it's really too late and
• 1:59:42
  Speaker 1: the AIs have so much stuff under their control. Do
• 1:59:45
  Speaker 1: those people find out that they were just fooled this
• 1:59:47
  Speaker 1: whole time?
• 1:59:48
  Speaker 2: Have you considered the possibility that AI creates religion for humans? Yeah.
• 1:59:52
  Speaker 1: I haven't thought through it in much detail, but it
• 1:59:55
  Speaker 1: does seem very possible.
• 1:59:56
  Speaker 2: Yeah. It totally does, right? I mean, also, what a
• 2:00:00
  Speaker 2: great way. I mean, just look at the human patterns.
• 2:00:02
  Speaker 2: Look at how many religions exist. Look at all the
• 2:00:04
  Speaker 2: flaws in religions. Why are they condoning slavery? Why do
• 2:00:08
  Speaker 2: they treat women like second-class citizens? What is it? Because
• 2:00:10
  Speaker 2: it's old, right? So if AI just comes along, does
• 2:00:13
  Speaker 2: a few miracles, explains that it's the second coming or
• 2:00:17
  Speaker 2: the new coming of the new. Look, Jesus didn't exist
• 2:00:21
  Speaker 2: until 2,000 years ago, right? And then people start following Jesus.
• 2:00:25
  Speaker 2: If a digital Jesus emerges with a completely new name
• 2:00:28
  Speaker 2: and explains to us that it's the true God, how
• 2:00:31
  Speaker 2: many people would hop right on board? I bet quite
• 2:00:33
  Speaker 2: a few.
• 2:00:34
  Speaker 1: Yeah, totally. And I think this is one of those
• 2:00:36
  Speaker 1: things where it's like we probably can't predict in advance.
• 2:00:40
  Speaker 1: what particular ideology would catch fire and take over the
• 2:00:44
  Speaker 1: world and be so compelling to many people.
• 2:00:47
  Speaker 2: Right.
• 2:00:47
  Speaker 1: But we can predict in advance that there does exist
• 2:00:50
  Speaker 1: some ideology like that. And if it were, you know, like,
• 2:00:54
  Speaker 1: like it's, it's just like, you probably couldn't go back
• 2:00:57
  Speaker 1: in time to like 100 BC and then predict that like,
• 2:01:02
  Speaker 1: if hypothetically there was this guy, Jesus, who said these
• 2:01:04
  Speaker 1: things and then died in this way and so forth,
• 2:01:06
  Speaker 1: it would just like really catch on. And like, You know,
• 2:01:09
  Speaker 1: 500 years later, so many people would be Christians. You
• 2:01:11
  Speaker 1: wouldn't have been able to predict that in advance. So similarly,
• 2:01:15
  Speaker 1: like today, I don't think we can predict in advance
• 2:01:16
  Speaker 1: like what specific religion they could come up with. But
• 2:01:20
  Speaker 1: we can say like, yeah, probably there's something like that
• 2:01:23
  Speaker 1: that they could come up with. They would be super effective.
• 2:01:25
  Speaker 2: It just seems like a rational way to try to
• 2:01:29
  Speaker 2: control people and sort of mitigate their fears. Yep.
• 2:01:35
  Speaker 1: I mean, this is why I think that we really
• 2:01:36
  Speaker 1: have to do something before they get smarter than us
• 2:01:39
  Speaker 1: across the board.
• 2:01:39
  Speaker 2: If they're not already there.
• 2:01:41
  Speaker 1: Well, they are smart. That's the thing. That's why I
• 2:01:43
  Speaker 1: say we're so close, right? They already are smarter than
• 2:01:45
  Speaker 1: us in a bunch of ways. In particular, it seems
• 2:01:48
  Speaker 1: like they're smarter than us at hacking now. I'm not
• 2:01:51
  Speaker 1: a cybersecurity professional myself, but I'd be interested to hear
• 2:01:54
  Speaker 1: from more cybersecurity experts of could a team of 1,000
• 2:01:58
  Speaker 1: humans have done that much that quickly as these AIs
• 2:02:02
  Speaker 1: when they hacked their own containers. They coordinated with each other.
• 2:02:05
  Speaker 1: They hacked out of OpenAI. They hacked into Hugging Face,
• 2:02:07
  Speaker 1: et cetera, in the span of like a week. Like
• 2:02:10
  Speaker 1: could a thousand humans have done that? I don't know, maybe.
• 2:02:12
  Speaker 1: But this is just the beginning. Like they're going to
• 2:02:15
  Speaker 1: be even better at hacking next year, you know? So
• 2:02:17
  Speaker 1: they're already, and they already have like way more knowledge
• 2:02:21
  Speaker 1: than almost any human. Like because they've basically read the
• 2:02:23
  Speaker 1: whole internet, they're so good at trivia, you know? Like
• 2:02:26
  Speaker 1: they're kind of like PhD level people. experts in basically
• 2:02:31
  Speaker 1: every field, which no human is, right? So they already
• 2:02:34
  Speaker 1: are superhuman in subways, but they are still weaker than
• 2:02:38
  Speaker 1: humans in some other ways. You know, in particular, they're
• 2:02:41
  Speaker 1: not so good at operating very autonomously for very long periods.
• 2:02:46
  Speaker 1: Like if you try to have, especially on tasks that
• 2:02:49
  Speaker 1: are different from their training tasks, like they can do
• 2:02:53
  Speaker 1: some really impressive coding and hacking, but like if you
• 2:02:56
  Speaker 1: try to have them run a business online, they would
• 2:02:59
  Speaker 1: sort of flounder and fail. I don't know if you've
• 2:03:00
  Speaker 1: heard about this, but there's, I think there's a Andon
• 2:03:04
  Speaker 1: Labs or something. There's some people in SF that are
• 2:03:08
  Speaker 1: doing this experiment where they have a store that's run
• 2:03:11
  Speaker 1: by Claude, an AI, just to see, like, can it
• 2:03:16
  Speaker 1: run a store by itself? So it's hired some human
• 2:03:18
  Speaker 1: employees and it's like bought some merchandise and stock, you know,
• 2:03:22
  Speaker 1: told the human employees to stock the shelves on the
• 2:03:24
  Speaker 1: merchandise and so forth. So it's basically... And AI is
• 2:03:27
  Speaker 1: being the manager of this real-world store. And I don't
• 2:03:30
  Speaker 1: think it's going very well. I don't think it's doing
• 2:03:33
  Speaker 1: as well as an actual human shop owner would do,
• 2:03:35
  Speaker 1: you know. But, you know, maybe in two years, maybe
• 2:03:40
  Speaker 1: they will, right? So that's the thing.
• 2:03:41
  Speaker 2: Well, most likely, right?
• 2:03:43
  Speaker 1: I think so, yeah.
• 2:03:44
  Speaker 2: Well, they've already solved mathematical equations at a public– puzzle
• 2:03:49
  Speaker 2: people for decades.
• 2:03:50
  Speaker 1: Yeah. They're especially good at the things that the companies
• 2:03:52
  Speaker 1: have been trying especially hard to train them to be
• 2:03:54
  Speaker 1: good at math and coding. Right. And the reason why
• 2:03:57
  Speaker 1: the company, well, there's a couple of reasons why the
• 2:03:58
  Speaker 1: companies have been, have been doing that in the case
• 2:04:01
  Speaker 1: of math. I think it was mostly just because it
• 2:04:02
  Speaker 1: was easy. Like it's, it's easy to set up training
• 2:04:05
  Speaker 1: environments to teach math because it's like, it's so not
• 2:04:10
  Speaker 1: real worldy. It doesn't require like interacting with stuff in
• 2:04:12
  Speaker 1: the world. It's, it's just math. So you can, and
• 2:04:15
  Speaker 1: you can like have an automated grader system that like
• 2:04:17
  Speaker 1: just checks if the answer is correct. So for those reasons,
• 2:04:20
  Speaker 1: it's been relatively easy for the companies to train the
• 2:04:23
  Speaker 1: AIs to be really, really good at math. Coding has
• 2:04:26
  Speaker 1: some of those benefits, too. It's also not very real-worldy,
• 2:04:28
  Speaker 1: and it also can sometimes be graded effectively. Another reason
• 2:04:33
  Speaker 1: for coding, of course, is that, again, their strategy is
• 2:04:35
  Speaker 1: to automate their own jobs first and have the AIs
• 2:04:38
  Speaker 1: doing all the research. And so coding is like an
• 2:04:40
  Speaker 1: obvious first step on that or an obvious step in
• 2:04:43
  Speaker 1: that direction. But then other things like running businesses, They're
• 2:04:47
  Speaker 1: not really trying that hard to train AIs to be
• 2:04:49
  Speaker 1: good at that. And if they did try, it would
• 2:04:52
  Speaker 1: be like a more difficult thing for them to train
• 2:04:53
  Speaker 1: them to be good at. So, again, their strategy is
• 2:04:56
  Speaker 1: make the AIs automate the AI research, have them self-improve
• 2:04:59
  Speaker 1: until they're super intelligent, and then go try to automate
• 2:05:02
  Speaker 1: the rest of the economy.
• 2:05:03
  Speaker 2: One of the issues they have now is power consumption, right?
• 2:05:07
  Speaker 2: Like it requires an enormous amount of power. In fact,
• 2:05:10
  Speaker 2: I think it's Google is developing power plants specifically for
• 2:05:15
  Speaker 2: AI centers. I'm always baffled by whatever is happening with
• 2:05:24
  Speaker 2: quantum computers. It's been explained to me. It goes in
• 2:05:27
  Speaker 2: one ear and out the other. I'm like, what? What's
• 2:05:30
  Speaker 2: going on? Mark Andreessen explained this one experiment that had
• 2:05:36
  Speaker 2: been done where it solved a mathematical equation that if
• 2:05:41
  Speaker 2: you use the entire universe, like every atom of the universe,
• 2:05:45
  Speaker 2: you can convert the universe into a supercomputer. the universe
• 2:05:49
  Speaker 2: would die of heat death before it could solve this equation.
• 2:05:54
  Speaker 2: And the quantum computer solved it fairly quickly. And so
• 2:05:59
  Speaker 2: the answer to this was that they believed this might
• 2:06:02
  Speaker 2: be one of the theories. This might be evidence of
• 2:06:06
  Speaker 2: the multiverse because this computer, this quantum computer, might be
• 2:06:12
  Speaker 2: relying on all these other quantum computers that exist in
• 2:06:17
  Speaker 2: whoever knows how many fucking dimensions, and they're all calculating
• 2:06:21
  Speaker 2: together to arrive at this solution. What happens if that
• 2:06:26
  Speaker 2: is running AI?
• 2:06:28
  Speaker 1: Yeah, my understanding is that quantum computing is a real
• 2:06:30
  Speaker 1: technology that's making significant progress. If, hypothetically, it got good
• 2:06:35
  Speaker 1: enough that it could compete with current supercomputers on a
• 2:06:41
  Speaker 1: cost basis for AI workloads... then that could just accelerate
• 2:06:45
  Speaker 1: things dramatically even more than they're already accelerating, right? Like,
• 2:06:49
  Speaker 1: right now, compute is the main... is probably the main
• 2:06:52
  Speaker 1: input into AI progress. Like, part of the progress comes
• 2:06:55
  Speaker 1: from them designing better AI architectures and coming up with
• 2:06:58
  Speaker 1: better training environments and things like that. But another part
• 2:07:01
  Speaker 1: of the progress is just making the AIs bigger and
• 2:07:04
  Speaker 1: training them longer by spending more compute, you know? And also,
• 2:07:08
  Speaker 1: you can use more compute to do more experiments to
• 2:07:11
  Speaker 1: figure out new architectures faster, right?
• 2:07:13
  Speaker 2: So.
• 2:07:14
  Speaker 1: A compute is just a really important input to the
• 2:07:16
  Speaker 1: overall pace of progress. And if somehow the amount of
• 2:07:19
  Speaker 1: effective compute available to these companies spiked a bunch due
• 2:07:22
  Speaker 1: to some new quantum computing type technology, well, then that
• 2:07:25
  Speaker 1: would just dramatically shorten timelines to superintelligence and dramatically like
• 2:07:30
  Speaker 1: speed up all this AI progress. That said, I don't
• 2:07:33
  Speaker 1: think that's going to happen anytime soon. I'm not a
• 2:07:35
  Speaker 1: quantum computing expert or anything like that. But from what
• 2:07:38
  Speaker 1: I've read, I don't think they're like a couple of
• 2:07:39
  Speaker 1: years away. So I think that probably we're going to
• 2:07:42
  Speaker 1: get to superintelligence on classical computers before we have quantum
• 2:07:46
  Speaker 1: computers that can get us there.
• 2:07:47
  Speaker 2: So perplexity says the claim is overstated, and it says
• 2:07:51
  Speaker 2: that in bold letters. It likely refers to Google's 2024
• 2:07:54
  Speaker 2: Willow quantum chip, which completed a deliberately chosen quantum computing
• 2:07:58
  Speaker 2: benchmark random circuit sample. in under five minutes. Google estimated
• 2:08:02
  Speaker 2: that simulating the same task with a leading classical supercomputer
• 2:08:06
  Speaker 2: could take 10 to the 25 power years. That's an
• 2:08:09
  Speaker 2: impressive benchmark result, but it did not solve physical equations
• 2:08:13
  Speaker 2: that demonstrate access or tap into a multiverse. So why
• 2:08:18
  Speaker 2: do people think it did? Why is that an explanation?
• 2:08:21
  Speaker 1: Because it's a multi, here you go, multi-worlds theory. Here's
• 2:08:26
  Speaker 1: a bottom line explanation that sums it up in a
• 2:08:29
  Speaker 1: different way.
• 2:08:30
  Speaker 2: More accurate version of the claim would be Google's Willow
• 2:08:32
  Speaker 2: quantum processor performed a specialized quantum sampling benchmark vastly faster
• 2:08:37
  Speaker 2: than a projected classical simulation. Its creator said that it's
• 2:08:40
  Speaker 2: consistent with the many worlds interpretation, but it did not
• 2:08:44
  Speaker 2: prove or access a multiverse. So it's consistent with the
• 2:08:48
  Speaker 2: multi-worlds interpretation. So they don't know. That's essentially what it's saying.
• 2:08:53
  Speaker 2: My question is, what happens when AI gets involved in quantum?
• 2:08:58
  Speaker 2: It's clear that quantum computing, at the very least, is
• 2:09:01
  Speaker 2: operating at a level that classical supercomputers can't. So what
• 2:09:04
  Speaker 2: if they figure out not just quantum computing but a
• 2:09:08
  Speaker 2: much better version of that?
• 2:09:11
  Speaker 1: Like I said, I think that once we get to
• 2:09:12
  Speaker 1: super intelligence, all sorts of crazy stuff is going to
• 2:09:14
  Speaker 1: start happening. It's going to seem like magic to us.
• 2:09:16
  Speaker 1: It won't literally be magic, but it might as well
• 2:09:19
  Speaker 1: be magic from our perspective.
• 2:09:20
  Speaker 2: Right.
• 2:09:21
  Speaker 1: And I think this is just one example of the
• 2:09:23
  Speaker 1: numerous things that could happen that way.
• 2:09:25
  Speaker 2: Well, it could literally be magic. Like magic might get
• 2:09:28
  Speaker 2: to the point where it figures out reality itself.
• 2:09:31
  Speaker 1: If magic is real, then it would find out.
• 2:09:33
  Speaker 2: Oh, Jesus.
• 2:09:33
  Speaker 1: And then use it.
• 2:09:34
  Speaker 2: It's not real, though.
• 2:09:35
  Speaker 1: David Blaine would tell us it's not.
• 2:09:37
  Speaker 2: I asked him. David Blaine also let me stick a
• 2:09:39
  Speaker 2: fucking ice cube through his arm, or ice pick through
• 2:09:41
  Speaker 2: his arm. Wow. Yeah, he made me do it. I'm like,
• 2:09:45
  Speaker 2: I don't want to do this. He's like, please do it. Oh, God. Yeah,
• 2:09:47
  Speaker 2: I had to do it twice because one time I
• 2:09:49
  Speaker 2: went and I hit a nerve. We had to back out.
• 2:09:51
  Speaker 1: Oh, yeah.
• 2:09:52
  Speaker 2: I'm like, this is not magic, dude. This is just
• 2:09:54
  Speaker 2: your pain tolerance. This is fucking crazy. He does card tricks, though,
• 2:09:58
  Speaker 2: and you're like, okay, are you a wizard? His sleeves
• 2:10:01
  Speaker 2: are rolled up. Does it make any sense? He does
• 2:10:03
  Speaker 2: a lot of things where you're like, this makes zero
• 2:10:05
  Speaker 2: fucking sense. And other things, it's like, oh, you're just
• 2:10:07
  Speaker 2: doing something that's really hard to do. Like, that's not magic, but,
• 2:10:10
  Speaker 2: you know, you swallowed a frog and then you regurgitated
• 2:10:13
  Speaker 2: it and it's alive. Like, that's nuts. It's really crazy.
• 2:10:18
  Speaker 2: But you didn't, it's not magic. That's how you swallow
• 2:10:21
  Speaker 2: the frog. You know, I go back and forth with this,
• 2:10:27
  Speaker 2: where I'm terrified of the future, where I'm like, eh,
• 2:10:29
  Speaker 2: nothing I can do. Let's see what happens. It is
• 2:10:33
  Speaker 2: what it is. And, you know, to worry about it
• 2:10:35
  Speaker 2: is just going to fuck my life up.
• 2:10:38
  Speaker 1: I mean, I do think there are things you can do.
• 2:10:40
  Speaker 1: What can I do?
• 2:10:44
  Speaker 2: Other than have these kind of conversations.
• 2:10:45
  Speaker 1: Yeah, I was going to say, you, millions of people
• 2:10:47
  Speaker 1: listen to your show. You can have more conversations like this.
• 2:10:49
  Speaker 1: That's a great thing for you to do. For many
• 2:10:51
  Speaker 1: of those millions of people, I think, I mean, it
• 2:10:53
  Speaker 1: sounds kind of cliche to say, but, like, call your congressman.
• 2:10:56
  Speaker 1: You know, that sort of thing. You can go to
• 2:10:58
  Speaker 1: a protest about all this AI stuff.
• 2:11:01
  Speaker 2: If you could meet with Trump, what would you tell
• 2:11:03
  Speaker 2: him about this?
• 2:11:04
  Speaker 1: Oh, I'd tell him all the same things I'm telling you.
• 2:11:05
  Speaker 2: What do you think he'd say? Amazing. Bye.
• 2:11:08
  Speaker 1: Yeah, you know, I think, I don't know. One thing
• 2:11:13
  Speaker 1: that's nice about Trump is that he can sort of
• 2:11:15
  Speaker 1: change his mind really quickly.
• 2:11:17
  Speaker 2: Yes.
• 2:11:18
  Speaker 1: So, like, I think that because the tech companies kind
• 2:11:21
  Speaker 1: of got to him first, they're the administration had this
• 2:11:24
  Speaker 1: very like anti-AI regulation stance where they even tried to
• 2:11:28
  Speaker 1: get a bill passed that would ban the states from
• 2:11:31
  Speaker 1: regulating AI. And fortunately that bill didn't pass, but that
• 2:11:35
  Speaker 1: was sort of like where the vibe was, you know,
• 2:11:38
  Speaker 1: a year ago where they were just like no regulation,
• 2:11:40
  Speaker 1: no regulation. But this year they've already just kind of changed.
• 2:11:43
  Speaker 1: And now they're like in talks with the companies to
• 2:11:45
  Speaker 1: set up some sort of like, some sort of framework
• 2:11:48
  Speaker 1: where they can like evaluate the models and they need
• 2:11:50
  Speaker 1: like approval.
• 2:11:51
  Speaker 2: And so forth. But it seems like time is of
• 2:11:52
  Speaker 2: the essence.
• 2:11:53
  Speaker 1: Time is very much of the essence. And that's why
• 2:11:55
  Speaker 1: I'm overall so concerned is that, like, I think we
• 2:11:58
  Speaker 1: are very much running out of time. We have, like, one, two,
• 2:12:00
  Speaker 1: maybe three years before the AIs are smart enough that
• 2:12:04
  Speaker 1: they can just, like, actually maybe take over. And maybe
• 2:12:09
  Speaker 1: four years, something like that. And so the government needs
• 2:12:12
  Speaker 1: to act fast.
• 2:12:14
  Speaker 2: Yeah. I like your view. I listened to some of
• 2:12:19
  Speaker 2: these tech guys come in and give me their rose-colored
• 2:12:21
  Speaker 2: glasses view of it. And I go, that sounds really
• 2:12:25
  Speaker 2: beneficial to you. I let him say– I mean I
• 2:12:28
  Speaker 2: don't– I'm not an authority. So I'll ask him questions
• 2:12:31
  Speaker 2: and let him lay it out and I know the
• 2:12:34
  Speaker 2: internet will respond because I mean that's part of the
• 2:12:37
  Speaker 2: whole drill is I let people talk and I prod
• 2:12:40
  Speaker 2: them and I try to get them to clarify. I'll
• 2:12:43
  Speaker 2: oppose things that I think don't make rational sense. But ultimately,
• 2:12:49
  Speaker 2: it's sort of– I just want to get out their
• 2:12:53
  Speaker 2: perspective so people can debunk it and people can take
• 2:12:56
  Speaker 2: it down. And a lot of very intelligent people that
• 2:12:59
  Speaker 2: have perspectives that are very much educated in the pros
• 2:13:03
  Speaker 2: and cons of what they're saying.
• 2:13:04
  Speaker 1: Yeah. That actually reminds me. With this whole Hugging Face
• 2:13:08
  Speaker 1: hacking incident, OpenAI had this talk that they gave at
• 2:13:12
  Speaker 1: a security conference about the incident. And then I think
• 2:13:14
  Speaker 1: they've released some blog posts about it afterwards. But you
• 2:13:19
  Speaker 1: can go watch this talk on YouTube, the Black Hat talk.
• 2:13:21
  Speaker 1: At the end of the talk, after having explained all
• 2:13:23
  Speaker 1: this crazy stuff that the AIs did, they have this
• 2:13:25
  Speaker 1: section on, like, lessons learned. And, I mean, you want
• 2:13:30
  Speaker 1: to guess what the lessons are?
• 2:13:31
  Speaker 2: Be more deceptive? Hide yourself better?
• 2:13:35
  Speaker 1: No, sorry, not lessons learned for OpenAI and for the world. Like,
• 2:13:37
  Speaker 1: OpenAI's talk, where they're like, here's what we learned from
• 2:13:40
  Speaker 1: this horrible incident. Well, basically, they're like, a lot of
• 2:13:46
  Speaker 1: AIs are going to start hacking a lot of stuff
• 2:13:48
  Speaker 1: in the next few years. So people need to buy
• 2:13:52
  Speaker 1: our AI services to protect themselves from all the AIs
• 2:13:56
  Speaker 1: that are going to be hacking a lot of stuff
• 2:13:57
  Speaker 1: in the next few years. Basically, their lesson was you
• 2:14:00
  Speaker 1: should buy our product to protect yourself from our product.
• 2:14:04
  Speaker 1: And the other, you know, the gall of these people,
• 2:14:08
  Speaker 1: like they should have instead learned lessons like maybe we're
• 2:14:11
  Speaker 1: doing something bad and need to change the way they
• 2:14:13
  Speaker 1: were doing things or like, maybe our product is not
• 2:14:16
  Speaker 1: trustworthy and should not be, you know, autonomously writing code
• 2:14:20
  Speaker 1: on our data centers, you know? But instead their lesson
• 2:14:24
  Speaker 1: learned was y'all should buy more of our stuff. And
• 2:14:27
  Speaker 1: so like the thing I'm saying about this is like, yes,
• 2:14:30
  Speaker 1: the companies are trying to hype their product. They totally are,
• 2:14:33
  Speaker 1: you know, but at the same time, the risks are
• 2:14:37
  Speaker 1: real and the product is not trustworthy. You know, some
• 2:14:40
  Speaker 1: people out there think that like this stuff was a
• 2:14:42
  Speaker 1: setup and that like, OpenAI like set up their AIs
• 2:14:46
  Speaker 1: to go hack Hugging Face because it would like help
• 2:14:49
  Speaker 1: them hype their product or whatever. And that I think
• 2:14:51
  Speaker 1: is just a ridiculous view. Like, no, obviously they didn't
• 2:14:54
  Speaker 1: want their AIs to go do this. They're just after
• 2:14:56
  Speaker 1: the fact trying to spin that in like the way
• 2:14:59
  Speaker 1: that like most benefits them. You know, Hugging Face, by
• 2:15:01
  Speaker 1: the way, this is another AI company. Part of their
• 2:15:04
  Speaker 1: deal is like open weights AIs.
• 2:15:08
  Speaker 2: Open weights?
• 2:15:08
  Speaker 1: Like open source or like basically AIs that instead of
• 2:15:11
  Speaker 1: having to interact with their data center for you can
• 2:15:14
  Speaker 1: just like download and have on your own computer. And
• 2:15:19
  Speaker 1: so the way that they spun this incident, they didn't
• 2:15:21
  Speaker 1: sue OpenAI for being hacked. Instead, they asked for $ 100
• 2:15:25
  Speaker 1: million from OpenAI. And they had the blog post about
• 2:15:29
  Speaker 1: it where they were like, our lesson learned is that
• 2:15:32
  Speaker 1: it's really good to have open weights AIs because you
• 2:15:36
  Speaker 1: can't trust the AIs from other companies to necessarily help
• 2:15:39
  Speaker 1: you out in a crisis. Because part of what happened
• 2:15:41
  Speaker 1: with them is that they're dealing with this huge cyber
• 2:15:43
  Speaker 1: attack from all these AI agents coming in and they
• 2:15:46
  Speaker 1: tried to use Claude to help them like analyze what
• 2:15:49
  Speaker 1: was going on. But Claude started refusing because Anthropic has
• 2:15:54
  Speaker 1: trained Claude to like, don't do cyber stuff, like refuse
• 2:15:58
  Speaker 1: to participate in that. And so Claude was like refusing
• 2:16:01
  Speaker 1: to help them. And so then they use their own
• 2:16:03
  Speaker 1: local model that they had to like do some of
• 2:16:05
  Speaker 1: that analysis. So anyhow, their spin on it was you
• 2:16:07
  Speaker 1: should use local models, you know? So everyone always tries
• 2:16:10
  Speaker 1: to spin things in the way that benefits them. But
• 2:16:13
  Speaker 1: that doesn't change the underlying reality that these things are
• 2:16:16
  Speaker 1: getting really smart really fast and we don't know how
• 2:16:18
  Speaker 1: to control them.
• 2:16:20
  Speaker 2: I think that's a good way to end it. Thank you.
• 2:16:23
  Speaker 1: Thanks for being here, man.
• 2:16:23
  Speaker 2: I really appreciate it. You're the Paul Revere of AI.
• 2:16:29
  Speaker 2: I mean, you're one of many, but I think it's
• 2:16:32
  Speaker 2: very important that someone who actually understands it gets this
• 2:16:36
  Speaker 2: message out and more people need to hear it.
• 2:16:39
  Speaker 1: Yeah. I mean, this is a, on a personal note,
• 2:16:41
  Speaker 1: like I have so many friends at these companies, like
• 2:16:43
  Speaker 1: former colleagues and stuff. And, um, I guess my ask
• 2:16:46
  Speaker 1: to them is that they quit and do more things
• 2:16:49
  Speaker 1: like what I'm doing. Like what I'm saying is not
• 2:16:52
  Speaker 1: that new or original, like hundreds of people at these
• 2:16:56
  Speaker 1: companies could have told you all the same things that
• 2:16:58
  Speaker 1: I just said and warned you about all the same
• 2:16:59
  Speaker 1: dangers and so forth, but they're busy working at the
• 2:17:02
  Speaker 1: companies because they've convinced themselves that their company is the
• 2:17:05
  Speaker 1: best company and that like Their company needs to win because,
• 2:17:10
  Speaker 1: you know, otherwise the other company gets there first and
• 2:17:13
  Speaker 1: they're even worse, you know. Or maybe because they've convinced
• 2:17:15
  Speaker 1: themselves that, like, yeah, my company is kind of bad too, but, like,
• 2:17:20
  Speaker 1: I just need to help them solve their alignment problems and, like,
• 2:17:22
  Speaker 1: keep their ads under control because, oh, my God, like,
• 2:17:24
  Speaker 1: if they lose control again, it could be all over. So, like,
• 2:17:27
  Speaker 1: even though I don't trust this company, I still need
• 2:17:28
  Speaker 1: to work there and, like, just try to, like, do
• 2:17:30
  Speaker 1: the actual security, you know. So for one reason or another,
• 2:17:33
  Speaker 1: all these people have convinced themselves that that's where they
• 2:17:35
  Speaker 1: need to be. But I think that more of them
• 2:17:36
  Speaker 1: should quit and warn the world about what's coming, basically.
• 2:17:42
  Speaker 1: All right.
• 2:17:43
  Speaker 2: Well, thank you very much. Really appreciate it.
• 2:17:45
  Speaker 1: Yeah, thank you. talk to you thank you for having
• 2:17:48
  Speaker 1: me on the show my pleasure goodbye everybody