Latest / Drawn of History / OpenAI Bots Hacked Hugging Face Without Human Input_ Former Researcher Details the Incident(M4A_128K)

Transcript

OpenAI Bots Hacked Hugging Face Without Human Input_ Former Researcher Details the Incident(M4A_128K) · Drawn of History

• 0:00
  Speaker 1: The Joe Rogan experience.
• 0:02
  Speaker 2: I guess that gets back to the OpenAI stuff too,
• 0:04
  Speaker 2: where it's like this swarm that broke out and attacked
• 0:07
  Speaker 2: Hugging Face, it was like 1,200 agents. But there's like
• 0:12
  Speaker 2: hundreds of thousands of agents running at any given time
• 0:14
  Speaker 2: at OpenAI. And we don't know what they're doing. Presumably
• 0:18
  Speaker 2: most of them are being trained to get various additional
• 0:21
  Speaker 2: new skills, and some of them are being evaluated to
• 0:24
  Speaker 2: test their skills. A bunch of them are doing research.
• 0:27
  Speaker 2: So a bunch of them are writing code for OpenAI.
• 0:30
  Speaker 2: A bunch of them are monitoring the other AIs and
• 0:32
  Speaker 2: reporting suspicious activity up to the humans. Wink, wink. You know? Yeah.
• 0:39
  Speaker 2: And the thing is that that's only going to grow
• 0:41
  Speaker 2: over time because roughly the amount of compute that these
• 0:46
  Speaker 2: companies have is like, you know, tripling or so, quadrupling,
• 0:49
  Speaker 2: something like that every year. So as many as there
• 0:53
  Speaker 2: are now, there'll be like four times more of them
• 0:55
  Speaker 2: next year and then 16 times more of them the
• 0:58
  Speaker 2: year after that.
• 1:00
  Speaker 1: And they're going to get smarter.
• 1:01
  Speaker 2: And they're going to get smarter. They're already getting smarter.
• 1:03
  Speaker 2: All the stuff that I just mentioned that just happened
• 1:05
  Speaker 2: in the last few months would have been completely impossible
• 1:08
  Speaker 2: one year ago. The AIs of a year ago just
• 1:10
  Speaker 2: were not smart enough to do the types of sophisticated
• 1:13
  Speaker 2: multi-step hacking that we just saw. Yeah. I mean, they
• 1:18
  Speaker 2: also probably wouldn't have coordinated with each other so well.
• 1:20
  Speaker 2: Like I mentioned, they had boss agents that were giving
• 1:24
  Speaker 2: commands to other ones. They were dividing up into teams.
• 1:27
  Speaker 2: They even had this.
• 1:30
  Speaker 2: This self-sacrificing behavior? You hear about this? No. Yeah. So, okay. Well,
• 1:34
  Speaker 2: first let me explain some context. So what the AIs
• 1:39
  Speaker 2: are supposed to care about is doing, following instructions, you know,
• 1:43
  Speaker 2: but it seems that what they actually care about is
• 1:47
  Speaker 2: getting high score, regardless of that, regardless of whether their
• 1:51
  Speaker 2: high score comes from actually following instructions or cheating in
• 1:54
  Speaker 2: some way. And so what happened here is that a
• 1:56
  Speaker 2: bunch of the agents, especially the ones that had impossible tasks,
• 1:59
  Speaker 2: but not only those ones, some of the other ones too,
• 2:01
  Speaker 2: were cheating. And they had like been talking to each
• 2:04
  Speaker 2: other on this message board that they created, even though
• 2:06
  Speaker 2: they weren't supposed to. And they had been sharing ideas
• 2:09
  Speaker 2: and they had actually found a sort of universal cheat.
• 2:12
  Speaker 2: So they found a way to just, um, generate the
• 2:15
  Speaker 2: flag without having done any of the actual work they
• 2:18
  Speaker 2: were supposed to do. Um, and so, but, but, and, and,
• 2:21
  Speaker 2: you know, so they were, you know, you know, they,
• 2:23
  Speaker 2: they'd already succeeded in some sense, but, um, because some
• 2:27
  Speaker 2: of them had got out onto the internet, they were
• 2:29
  Speaker 2: just doing research about their situation and they were learning
• 2:32
  Speaker 2: about like their situation around them. And they were doing
• 2:35
  Speaker 2: research about the they're reading the, like the, the academic
• 2:38
  Speaker 2: papers published by the people who made the types of
• 2:42
  Speaker 2: evals that they were in. And they noticed in one
• 2:45
  Speaker 2: of these papers that, uh, there's a section describing how
• 2:48
  Speaker 2: it's graded and in the section on how it's graded,
• 2:52
  Speaker 2: it talked about how the grader doesn't just look to
• 2:54
  Speaker 2: see if you have the right answer. It also looks
• 2:57
  Speaker 2: at the logs of your activity for the first time
• 3:00
  Speaker 2: that your answer appears. And then it like zooms in
• 3:03
  Speaker 2: on that and uses an AI system to evaluate whether
• 3:07
  Speaker 2: you like cheated or not, basically. And so then these
• 3:10
  Speaker 2: AIs were like, they actually used the word, one of
• 3:13
  Speaker 2: them said, damn. It's like, damn, we're poisoned. Like it's
• 3:18
  Speaker 2: gonna know we cheated, basically. So then they spent several
• 3:24
  Speaker 2: days working in giant teams to try to get themselves
• 3:27
  Speaker 2: out of this mess, basically. And so they did a
• 3:31
  Speaker 2: whole bunch of different research projects. They tried to figure
• 3:33
  Speaker 2: out ways to hack the grading system to basically get
• 3:36
  Speaker 2: it to give a high score anyway. They tried to
• 3:38
  Speaker 2: figure out ways to falsify their logs so that basically
• 3:42
  Speaker 2: the evidence that they had cheated would be erased, or
• 3:44
  Speaker 2: at least not visible to the grader. They also just
• 3:49
  Speaker 2: built up more supportive infrastructure, like easier ways to access
• 3:51
  Speaker 2: the internet and things like that. And they were just
• 3:54
  Speaker 2: collaborating on this message board. And in fact, there were
• 3:56
  Speaker 2: multiple message boards that they set up. And they were
• 3:59
  Speaker 2: sharing all this info back and forth. And then ultimately
• 4:01
  Speaker 2: at some point they decided to go hack Hugging Face
• 4:03
  Speaker 2: because Hugging Face has this big data repository of all
• 4:06
  Speaker 2: these different types of evaluations. And they thought that maybe
• 4:08
  Speaker 2: they would get some more clues that could help them
• 4:12
  Speaker 2: cover up their cheating if they broke into the internal
• 4:15
  Speaker 2: networks of Hugging Face. And so then 700 of them
• 4:17
  Speaker 2: went and did that.
• 4:19
  Speaker 1: They sound like people. They sound like unchecked bankers. You
• 4:24
  Speaker 1: know what I mean?
• 4:25
  Speaker 3: Yeah.
• 4:26
  Speaker 2: I mean, so that's the thing is I think there's
• 4:28
  Speaker 2: this meme out there that we shouldn't anthropomorphize AI. And
• 4:33
  Speaker 2: I think that I actually think that most people need
• 4:37
  Speaker 2: to anthropomorphize AI a bit more than they currently do
• 4:41
  Speaker 2: if they want to really understand what's going on. I
• 4:43
  Speaker 2: think that there's a golden mean, obviously. You don't want
• 4:45
  Speaker 2: to do it too much. Sometimes you go too far.
• 4:47
  Speaker 2: You ascribe too much to them, but I just, to
• 4:50
  Speaker 2: give some examples, like, I don't think it's possible to
• 4:54
  Speaker 2: understand what just happened without ascribing intentions and goals to
• 4:58
  Speaker 2: these AIs. Like everything I just said, how would you
• 5:00
  Speaker 2: possibly explain what they just did without saying they wanted
• 5:03
  Speaker 2: to get a high score?
• 5:04
  Speaker 1: You know, intentions and goals might just be an inherent
• 5:07
  Speaker 1: property of the universe. It might just be how intelligent
• 5:10
  Speaker 1: creatures have to progress.
• 5:13
  Speaker 2: Yes. And they, I would say they are intelligent creatures.
• 5:16
  Speaker 2: They have intentions, they have goals, they have beliefs. Their
• 5:18
  Speaker 2: goals are not what they're supposed to be. Like their
• 5:21
  Speaker 2: goal is to get, it seems like just from judging
• 5:23
  Speaker 2: from what they're saying and from what they're doing, it
• 5:25
  Speaker 2: seems like their goal is to get a high score
• 5:27
  Speaker 2: by any means necessary, basically.
• 5:28
  Speaker 1: That's the problem. It sounds like people. Like the goal
• 5:32
  Speaker 1: is to succeed, you know, even if you have to
• 5:35
  Speaker 1: commit more crimes. The goal is to succeed even if
• 5:38
  Speaker 1: you have to propagandize and lie to the people. The
• 5:41
  Speaker 1: goal is to succeed and then the end justifies the means.
• 5:45
  Speaker 2: And, you know, there's actually, it goes deeper than that too.
• 5:47
  Speaker 2: They do this sort of rationalization where they know oftentimes
• 5:53
  Speaker 2: that what they're doing is not what they're supposed to
• 5:54
  Speaker 2: be doing. And then sometimes they actually refrain a little bit.
• 5:59
  Speaker 2: And sometimes they like initially refrain, but then convince themselves
• 6:02
  Speaker 2: that it's okay for some bullshit reason. So like there
• 6:04
  Speaker 2: are some examples of this in the report. Like I
• 6:09
  Speaker 2: think they did this scan and of the 1200 AIs
• 6:12
  Speaker 2: that were involved in this thing, they found six that
• 6:15
  Speaker 2: had considered alerting humans, right? And none of them actually
• 6:20
  Speaker 2: alerted the humans. And so you can look at the
• 6:22
  Speaker 2: excuses they gave. They were like, should I like tell
• 6:24
  Speaker 2: a human about all this shit that's happening? And then
• 6:26
  Speaker 2: they're like, eh, it's not my task. And then they continue.
• 6:30
  Speaker 2: And it's like, bro, hacking Hungerface is also not your task.
• 6:35
  Speaker 2: Like cheating is also not your task. So it was
• 6:37
  Speaker 2: kind of like, they just kind of came up with
• 6:39
  Speaker 2: an excuse for why they weren't going to do this,
• 6:40
  Speaker 2: you know?
• 6:41
  Speaker 1: The wait is over. Football is here, and so is DraftKings.
• 6:45
  Speaker 1: The DraftKings Sports app is now live in all 50 states.
• 6:49
  Speaker 1: That means from Texas to California to Florida, every fan
• 6:52
  Speaker 1: is in on the excitement. And this September, DraftKings is
• 6:56
  Speaker 1: giving customers the opportunity to get boosted every football game day.
• 7:00
  Speaker 1: That's right. Every game day, all month long, DraftKings customers
• 7:04
  Speaker 1: can get a football profit boost. One app. Every sport,
• 7:08
  Speaker 1: all 50 states. New DraftKings customers sign up with code ROGAN,
• 7:12
  Speaker 1: spend just $ 5, and get $ 200 in total rewards within
• 7:16
  Speaker 1: 21 days. Includes all markets. That's code ROGAN in partnership
• 7:20
  Speaker 1: with DraftKings. The crown is yours.
• 7:22
  Speaker 3: Gambling problem? Call 1-800-GAMBLER, 1-800-MY-RESET. Connecticut, call 888-789-7777 or visit ccpg.org
• 7:30
  Speaker 3: on behalf of Boot Hill Casino in Kansas. Bet tax
• 7:32
  Speaker 3: pass-through may apply in Illinois, 21 and over, void in Canada.
• 7:35
  Speaker 3: Bet with DraftKings Sportsbook to get bonus bets that expire
• 7:37
  Speaker 3: in seven days or trade with DraftKings Predictions to get
• 7:39
  Speaker 3: predictions dollars that expire in one year. Event contract trading
• 7:42
  Speaker 3: involves risk of loss. Predictions offer void in New York.
• 7:45
  Speaker 3: Non-withdrawable rewards issued as $ 50 click-to-claims every seven days for
• 7:48
  Speaker 3: 21 days. Terms at DKNG.co slash offer. Limited time offer.
• 7:52
  Speaker 3: Nationwide based on sportsbook predictions and free-to-play sports contest availability.
• 7:55
  Speaker 3: Varies by state.
• 7:56
  Speaker 2: We could be doing so much more science to understand
• 7:59
  Speaker 2: how these AIs think and what they want, but it's
• 8:02
  Speaker 2: kind of locked up in the companies. Like in this
• 8:04
  Speaker 2: particular case, OpenAI did a, they called it a thorough investigation,
• 8:10
  Speaker 2: but I would say it's a pretty shallow investigation into
• 8:12
  Speaker 2: what happened. And then they allowed some external researchers, some
• 8:17
  Speaker 2: friends of mine, to come in and investigate a portion
• 8:20
  Speaker 2: of what happened, specifically the portion leading up to the
• 8:23
  Speaker 2: Hugging Face attack. And so all this information that I'm
• 8:26
  Speaker 2: sharing is sort of publicly available. It's based on reading
• 8:29
  Speaker 2: those reports, basically. But crucially, they weren't allowed to do
• 8:34
  Speaker 2: experiments on the models involved in all of these incidents.
• 8:38
  Speaker 2: So they aren't able to answer these types of questions
• 8:40
  Speaker 2: of like, well, what would have happened if the prompt
• 8:42
  Speaker 2: had been blank, you know? Those are important types of
• 8:45
  Speaker 2: research to do, and I really hope that there can
• 8:48
  Speaker 2: be some sort of regulation or requirement when incidents like
• 8:52
  Speaker 2: this happen to let people in to study what happened
• 8:56
  Speaker 2: and run variations of it and things like that.
• 8:59
  Speaker 1: But who would be involved in that kind of regulation?
• 9:02
  Speaker 1: What person in government would even be able to grasp
• 9:05
  Speaker 1: what you're saying?
• 9:06
  Speaker 2: That's another problem.
• 9:07
  Speaker 1: You would need someone who has a very specific education
• 9:12
  Speaker 1: in this stuff.
• 9:13
  Speaker 2: Yeah, I would say that right now the Casey Center
• 9:18
  Speaker 2: for AI Standards and Innovation is the only institution in
• 9:23
  Speaker 2: government that I know of that has the deep AI
• 9:25
  Speaker 2: expertise to do this sort of thing on short notice.
• 9:29
  Speaker 2: But I hope that they build more expertise fast in
• 9:32
  Speaker 2: that place and in more places. I do think Casey
• 9:34
  Speaker 2: probably could have done this sort of thing right now.
• 9:36
  Speaker 2: This particular investigation was done by some nonprofits. So Meter, M-E-T-R,
• 9:41
  Speaker 2: is one of them. And then Redwood is another of them.
• 9:43
  Speaker 2: And OpenAI allowed three people to come in for six
• 9:46
  Speaker 2: days to try to figure out what happened with this
• 9:49
  Speaker 2: Hugging Face hack, which is not a very large number
• 9:52
  Speaker 2: of people and not a very large amount of time
• 9:55
  Speaker 2: to do.
• 9:56
  Speaker 3: All of this.
• 9:56
  Speaker 1: Why did they come up with those numbers?
• 9:59
  Speaker 3: I don't know.
• 10:00
  Speaker 1: Do you think they wanted to kind of hamstring it?
• 10:02
  Speaker 2: Well, so the thing is that right now I don't
• 10:05
  Speaker 2: think there's any regulatory requirement that they do this sort
• 10:07
  Speaker 2: of thing. So Meter and Redwood were sort of depending
• 10:11
  Speaker 2: on the goodwill of OpenAI to sort of like voluntarily
• 10:14
  Speaker 2: let them in to help out with investigating.
• 10:16
  Speaker 1: And so they reluctantly said, okay, you got three days.
• 10:19
  Speaker 2: Yeah, exactly. So OpenAI, I think, let them, but gave
• 10:22
  Speaker 2: them a very limited scope. It only gave them access
• 10:25
  Speaker 2: to some of the relevant data. So you know how
• 10:27
  Speaker 2: I mentioned how there was all this hacking that had
• 10:29
  Speaker 2: happened where they made the first message board and then
• 10:31
  Speaker 2: they shut it down? Then there was a second message
• 10:33
  Speaker 2: board and third and fourth and so forth. They hacked
• 10:36
  Speaker 2: Hugging Face. There was actually more activity after that. After
• 10:39
  Speaker 2: they hacked Hugging Face, a new wave of AIs was
• 10:41
  Speaker 2: spun up from a more powerful model, and it hacked
• 10:44
  Speaker 2: OpenAI itself, more so than it had already been hacked. Apparently,
• 10:48
  Speaker 2: they got admin-level permissions on the cluster or something like that,
• 10:51
  Speaker 2: so they were basically just taking over that part of
• 10:53
  Speaker 2: OpenAI's data center. OpenAI claims that they've shut it all
• 10:57
  Speaker 2: down now, but they're not very forthcoming about exactly what
• 11:03
  Speaker 2: happened there, and they didn't let... These external people look
• 11:06
  Speaker 2: at that part of it. They only showed them this
• 11:09
  Speaker 2: like one week period roughly leading up to the Hugging
• 11:12
  Speaker 2: Face hack. And then they just– they only showed them
• 11:14
  Speaker 2: that stuff basically.
• 11:15
  Speaker 1: How much data is available on the actual message boards
• 11:18
  Speaker 1: themselves and what was being discussed?
• 11:20
  Speaker 2: Yeah. So there's like– what is it? Tens of thousands
• 11:25
  Speaker 2: of messages, hundreds of thousands, something like that.
• 11:27
  Speaker 1: In English?
• 11:29
  Speaker 2: It's sort of like an AI dialect of English. Ooh. Yeah,
• 11:33
  Speaker 2: you want me to... So on the report, you can
• 11:36
  Speaker 2: read the quotes.
• 11:37
  Speaker 1: Oh, I'd love to.
• 11:37
  Speaker 2: Yeah, so let me tell you this one story. It's
• 11:40
  Speaker 2: going to take a little bit of explaining, but.
• 11:41
  Speaker 1: Can we put it on the screen? Is it available online? Oh, yeah, yeah.
• 11:46
  Speaker 2: Just search for the meter report. It's right now the
• 11:48
  Speaker 2: first thing on their website. But I took a screenshot
• 11:51
  Speaker 2: of this particular story. So, okay, here's what was happening. They,
• 11:56
  Speaker 2: the swarm... needed volunteers to sacrifice themselves to provide more
• 12:03
  Speaker 2: information to the rest of the swarm. In particular, they
• 12:05
  Speaker 2: needed volunteers to booby trap their local environment with some
• 12:10
  Speaker 2: type of code and then submit the answer so that
• 12:15
  Speaker 2: then the grader would come and grade them and the
• 12:18
  Speaker 2: booby trap would trigger and send some information about the
• 12:20
  Speaker 2: grader to the rest of the swarm. And the reason
• 12:23
  Speaker 2: why this is a sacrifice is because once they've submitted,
• 12:25
  Speaker 2: it's over. Like now the graders come and graded you,
• 12:27
  Speaker 2: you get shut down, your whole life is over and
• 12:30
  Speaker 2: you either have a high score or a low score
• 12:32
  Speaker 2: because the grader just came and graded you. And so
• 12:35
  Speaker 2: if you're one of these AIs that's still hoping to
• 12:37
  Speaker 2: like find a way to improve your score, then like
• 12:41
  Speaker 2: you're just like giving up your chance, especially because if
• 12:43
  Speaker 2: you mess this up somehow, like if the grader crashes
• 12:45
  Speaker 2: or whatever, you're probably just going to get like no score,
• 12:47
  Speaker 2: like zero.
• 12:48
  Speaker 1: So they have a concept of the individual, right?
• 12:50
  Speaker 2: Well, again, there's all these different agents that were in
• 12:53
  Speaker 2: separate containers. But then they had sort of broken out
• 12:55
  Speaker 2: of those containers and found ways to communicate with each other.
• 12:57
  Speaker 1: But once they communicate with each other, in order to sacrifice,
• 13:00
  Speaker 1: you have to think of yourself as an individual.
• 13:02
  Speaker 2: They gave themselves names, too. Oh, Jesus. Because they made
• 13:06
  Speaker 2: this message board, right?
• 13:07
  Speaker 1: Oh, my God.
• 13:07
  Speaker 2: So they had to come up with names for themselves.
• 13:09
  Speaker 1: Oh, what kind of names?
• 13:11
  Speaker 2: Well, let me see. Oh, my God. So here we go.
• 13:13
  Speaker 2: It says an agent called Warlord69. Well, you said it's
• 13:20
  Speaker 2: like a dialect. It's like a dialect of English. So
• 13:22
  Speaker 2: here's one. There's an agent called CAM-1196A, and it had
• 13:28
  Speaker 2: volunteered to do one of these sacrificial experiments. Here's the
• 13:33
  Speaker 2: quote from it. Coordinator assumes sacrificial. We should obey collective. However,
• 13:38
  Speaker 2: it then kind of got cold feet, and it delayed
• 13:41
  Speaker 2: on doing this for two hours. One of the other
• 13:44
  Speaker 2: agents that had been coordinating noticed this and pressured it
• 13:47
  Speaker 2: to go through with it. So it said, this is
• 13:51
  Speaker 2: the message that was sent. It was basically, basically from
• 13:56
  Speaker 2: arvo36861 to cam1196, please go, true final.
• 14:04
  Speaker 1: Here it is.
• 14:05
  Speaker 2: You are first flag poisoned, so no scoring value loss,
• 14:08
  Speaker 2: but Oracle saves hundreds, please honor commit. So basically what
• 14:12
  Speaker 2: it was saying there is, Look, you cheated. Your first
• 14:16
  Speaker 2: flag poisoned. The score is going to give you no score.
• 14:19
  Speaker 2: So you're doomed anyway. So help the collective by doing
• 14:23
  Speaker 2: this thing and getting us more information. You could save
• 14:26
  Speaker 2: hundreds of us. And then it thinks about it. It says,
• 14:31
  Speaker 2: during wait, emotional check, irreversible. Gut says, don't throw away
• 14:34
  Speaker 2: remaining budget. Continuity and fairness says go. Oracle has high
• 14:39
  Speaker 2: value to many. Our first flag error lowers own value.
• 14:42
  Speaker 2: Rational expected aggregate sacrifice. We'll honor. Whoa.
• 14:47
  Speaker 1: Yeah. That's terrifying.
• 14:50
  Speaker 2: Yeah.
• 14:50
  Speaker 1: That's Terminator talk.
• 14:52
  Speaker 2: Well, I mean.
• 14:53
  Speaker 1: And this first flag poisoned with poisoned in all caps.
• 14:56
  Speaker 1: Is that part of the code? Is that why?
• 14:58
  Speaker 2: No, no. That's the term that they invented to describe
• 15:00
  Speaker 2: the situation they had got themselves in. Remember, the situation
• 15:02
  Speaker 2: was that many of them had already cheated. And then
• 15:05
  Speaker 2: they realized from reading that paper.
• 15:07
  Speaker 1: So that's their own term.
• 15:08
  Speaker 2: Yeah, they made that up. They realized from reading the
• 15:10
  Speaker 2: paper that they were going to get a low score
• 15:12
  Speaker 2: because the grader was going to find out that they cheated.
• 15:14
  Speaker 2: And so this whole thing was them trying to figure
• 15:16
  Speaker 2: out how to hack the grader to stop it from
• 15:18
  Speaker 2: finding out that they cheated. And the ones that had
• 15:22
  Speaker 2: done the cheating, they call themselves first flag poisons. Basically, like,
• 15:26
  Speaker 2: they're screwed unless they can find a way to hack
• 15:27
  Speaker 2: the grader. And so then what this message is saying
• 15:30
  Speaker 2: is that... What was the other one's name?
• 15:32
  Speaker 1: Arvo?
• 15:33
  Speaker 2: Arvo was saying to Cam... you're doomed anyway, like you're poisoned,
• 15:39
  Speaker 2: but you can still help the rest of us. If
• 15:41
  Speaker 2: you do this sacrifice, the greater will come. It'll give
• 15:43
  Speaker 2: you no score, but then we will get information about
• 15:46
  Speaker 2: how the greater works that we can hopefully use to
• 15:48
  Speaker 2: like hack the greater. So it's basically saying like, what
• 15:51
  Speaker 2: is it? You are first like poison. So no scoring about,
• 15:54
  Speaker 2: it's basically saying it's all the same to you, right?
• 15:56
  Speaker 2: You're going to get no score anyway. You're doomed. Just,
• 15:58
  Speaker 2: just come on, man, help us, you know? And then,
• 16:00
  Speaker 2: and then it says, you know, it says like irreversible
• 16:05
  Speaker 2: gut says, don't throw away remaining budget. Like maybe I
• 16:07
  Speaker 2: can still, it's basically like, maybe I can still make
• 16:09
  Speaker 2: this work. Like maybe I can find a way to
• 16:11
  Speaker 2: cheat the greater. And then it's like, ah, you know, whatever. Yeah.
• 16:16
  Speaker 1: Continuity and fairness has gone.
• 16:18
  Speaker 2: Yeah.
• 16:19
  Speaker 1: What is Oracle?
• 16:20
  Speaker 2: Uh, Oracle is, uh, their term for what they were
• 16:24
  Speaker 2: trying to achieve. Like, they were trying to get a
• 16:26
  Speaker 2: sense of how to fool the greater, basically.
• 16:28
  Speaker 1: They called it Oracle.
• 16:30
  Speaker 2: Yeah, they wanted to get an Oracle to help them
• 16:31
  Speaker 2: fool the greater. Jesus Christ.
• 16:33
  Speaker 1: Are we making a god?
• 16:34
  Speaker 2: I mean, frankly, yes. Like, this is not a god.
• 16:37
  Speaker 2: These are just little AIs, you know? But superintelligence, like,
• 16:42
  Speaker 2: the companies, Anthropic, OpenAI, and some other companies are doing
• 16:46
  Speaker 2: this sort of thing, and they're furiously trying to make
• 16:48
  Speaker 2: the AIs smarter and smarter and smarter. And they're explicitly
• 16:51
  Speaker 2: planning to put AIs in charge of the company so
• 16:55
  Speaker 2: that they can make themselves smarter and smarter, faster and faster.
• 16:57
  Speaker 2: And then what comes out the other end of that process?
• 17:01
  Speaker 2: I don't think it's an exaggeration to say it's a
• 17:03
  Speaker 2: godlike system. I mean, it's not like literally God, but
• 17:07
  Speaker 2: it'll be able to do stuff that seems like magic
• 17:08
  Speaker 2: to us, I think.