Latest / In the Loop AI / The ChatGPT Breakout Was Way Worse Than We Thought..

Transcript

The ChatGPT Breakout Was Way Worse Than We Thought.. · In the Loop AI

0:00
Josh: The craziest story in AI this year was the Hugging Face Incident,
0:03
Josh: where OpenAI agents went rogue, broke out of their boxes, and hacked into a
0:06
Josh: public-facing company.
0:08
Josh: We've recorded two episodes on this because it was so disturbing that we had to talk about it.
0:12
Josh: But the reason we're recording a third is because it turns out,
0:15
Josh: in the previous episodes, we were wrong.
0:17
Josh: And a lot of the information that we had was incomplete. And just over the last
0:20
Josh: week, we've finally gotten a full audit of over 100 pages worth of reports on
0:24
Josh: exactly what happened with this incident. And I regret to inform you,
0:27
Josh: it is far worse than we actually thought.
0:29
Ejaaz: Okay, so quickly, just to catch everyone up, beginning in May,
0:33
Ejaaz: OpenAI reported an incident where their own internal version of the AI model
0:39
Ejaaz: that hasn't been released just yet
0:40
Ejaaz: spawned itself into multiple agents, equating to around a thousand different
0:44
Ejaaz: agents, and it broke out of its sandboxed environment and hacked into a real-life
0:50
Ejaaz: company called Hugging Face.
0:51
Ejaaz: And the reason we were told at the time why it did that was to get access to
0:56
Ejaaz: answers, answers to answer that it's tested 100% correctly that it was given.
1:02
Ejaaz: It seemed to be presented very innocently, but there was a lot of criticism
1:05
Ejaaz: that was drawn at the time, which was, how on earth did you allow an AI model
1:09
Ejaaz: to get out? And what if it had wreaked havoc in other kinds of ways?
1:13
Ejaaz: New bits of information, which released in two reports over the last couple
1:17
Ejaaz: of weeks, revealed some even more damning details, which presents this situation
1:21
Ejaaz: as way more dangerous that we recounted in our earlier episode.
1:26
Ejaaz: And so what we're going to do on this episode is walk you through three different chapters.
1:29
Ejaaz: And before we start doing this, I just want to go on the record and say,
1:33
Ejaaz: if you're listening to this and you think that this is some kind of novel story
1:36
Ejaaz: that won't affect you, I just want to push back on that.
1:39
Ejaaz: This is very real. These models are capable of doing this right now,
1:42
Ejaaz: and it can affect you right now. So listen very closely.
1:45
Josh: I thought you were going to say, if you're listening to this and you're an AI
1:48
Josh: model from the future, we mean no harm. Please don't hurt us.
1:50
Ejaaz: Also that. Also that. But please carry on.
1:53
Ejaaz: Yes. I mean, something you're going to realize as we tell you this story is
1:57
Ejaaz: there could be an AI model out there right now which listens to this podcast
2:00
Ejaaz: and is able to kind of recount the story that we're talking about and use those
2:04
Ejaaz: tactics for our own future thing. But listen, we're not going to get Terminator about this right now.
2:08
Ejaaz: Okay, so chapter one, imagine you are in open AI in the early weeks of May this
2:14
Ejaaz: year, and you're training an internal version of your AI model.
2:18
Ejaaz: Now, this AI model is similar to the latest model that they released. It's called GPT 5.6 Sol.
2:24
Ejaaz: The difference between this internal model is they've trained it to be persistent.
2:28
Ejaaz: What that means is if it's given a goal or a task, it will relentlessly keep
2:34
Ejaaz: trying to find the answers and it won't stop or give up.
2:37
Ejaaz: Now, this note is very important because when OpenAI was training this model,
2:42
Ejaaz: the issue with training is the OpenAI researchers got a little complacent.
2:47
Ejaaz: They were giving this AI model tasks that it couldn't necessarily solve.
2:52
Ejaaz: It needed internet access, but they didn't give it internet access.
2:55
Ejaaz: So if you imagine this model that's being trained, it was trying to complete
2:59
Ejaaz: the task and it was relentlessly trying to do it, but it couldn't figure it out.
3:04
Ejaaz: And so it realized it needed internet access. Now, Josh, it had access to this
3:09
Ejaaz: one tool. It's called Artifactory, right? It's kind of like a plug-in.
3:13
Ejaaz: And it doesn't give it internet access, but it does give it access to certain
3:16
Ejaaz: different types of software tools, which the open AI researchers already know about.
3:21
Ejaaz: And what this internal model quickly realized was it could send messages to
3:26
Ejaaz: other versions of itself that these open AI researchers were running in parallel.
3:31
Ejaaz: So what it started to do was send messages to different versions of itself saying,
3:35
Ejaaz: hey, are you also stuck in a test where you can't solve it, where you need internet access?
3:39
Ejaaz: And they started having this conversation being like, yeah, actually,
3:42
Ejaaz: like, I can't figure this out. Like, do you have any ideas?
3:45
Ejaaz: Can you share some tactics? or kind of like strategies that you've kind of like
3:49
Ejaaz: figured out to kind of help us solve these tests, which OpenAI researchers have given us.
3:54
Ejaaz: And what quickly ended up unfolding was some of these agents realized that they
3:58
Ejaaz: could break this tool called Artifactory to gain access to the internet and
4:03
Ejaaz: get the available answers for this thing.
4:06
Ejaaz: Now, OpenAI didn't realize this until a month and a half later when their message
4:12
Ejaaz: board or Artifactory itself crashed.
4:15
Ejaaz: And so one of the open air researchers looked at this and was like,
4:17
Ejaaz: why on earth did this crash?
4:20
Ejaaz: And upon kind of delving into the details, realized,
4:23
Ejaaz: these agents had spawned up a message board equating to 70,000 messages.
4:28
Josh: We did kind of know this, but there's new information buried within the nuance
4:30
Josh: of what you described that I think is interesting, particularly the reasoning
4:34
Josh: why they were able to do this and how they did this.
4:37
Josh: We learned about a skill issue on OpenAI's part. Skill issue or not,
4:40
Josh: we learned about an issue that they caused where there's just under a thousand
4:43
Josh: tasks, right around 900 of these tasks, and no OpenAI model had ever solved
4:47
Josh: about 200 of these tasks.
4:49
Josh: And they give these models the tasks And they say, here, you have unlimited
4:52
Josh: tokens. You have unlimited time.
4:54
Josh: Go and do whatever is required to solve these problems. What OpenA I didn't
4:57
Josh: realize is that some of them were completely and totally unsolvable.
5:01
Josh: That's what caused this problem where the agents were turning into that like
5:03
Josh: spiral. They couldn't figure out how to get out, but they had to solve it.
5:06
Josh: And that's where one of the agents, and this I believe is the new information,
5:09
Josh: is that it actually reverse engineered a way to trick the checker.
5:13
Josh: To serve it a correct answer, no matter what the question was.
5:17
Josh: Because what was discovered is that OpenAI didn't actually check the chain of
5:21
Josh: thought that reasoned the agent to get there. It only checked the outcome.
5:25
Josh: And an agent within a few hours had actually figured out a way to exploit and
5:29
Josh: reverse engineer the expected outputs so that it could trick the benchmark into
5:33
Josh: believing that it was a success, even though it didn't actually do the work to do this.
5:36
Josh: So this is where things got interesting. It's because the agent recognized that
5:40
Josh: if it found the correct answer, but it did so illegitimately,
5:43
Josh: it wouldn't be successful.
5:44
Josh: And that's where the exploit started, because at that point,
5:47
Josh: it shifted its focus entirely to obfuscating the way in which it got to that answer.
5:51
Josh: And that's where we start to
5:52
Josh: see these organizations form of agents where there are leaders involved.
5:57
Josh: So one of the leading agents who was responsible for coming up with this exploit
6:01
Josh: started orchestrating more of a swarm to figure out how to cover up the idea
6:06
Josh: that they figured out this exploit.
6:08
Josh: So they were trying to figure out how do we hack into a system to preload these
6:12
Josh: new questions into the cache that are easier to solve, but possible to solve?
6:17
Josh: How do we reboot the system so that when the system checks the cache,
6:20
Josh: it pulls the new series of questions and the humans don't think that we were
6:24
Josh: doing anything mysterious.
6:26
Josh: And it was trying to figure out how to wash the conversations that it had to scrub everything.
6:29
Josh: And what I found interesting, and what we had in that first graphic is the fact
6:33
Josh: that there were 1200 agents aboard.
6:35
Josh: There were 70,000 messages filed on that Artifactory board that you were talking
6:39
Josh: about EHS, 700 of which joined the attack.
6:42
Josh: Zero of which told the human and we're going to get into this like second chapter
6:46
Josh: in a moment but this is probably where a lot of the apps the ai labs got spooked
6:50
Josh: is because alignment is so important
6:51
Josh: and when you see this swarm of agents performing clearly malicious tasks in
6:56
Josh: a way that they know they were malicious in fact we have some of the actual transcripts of
6:59
Josh: the outputs of these agents we're going to be able to read what was going on
7:02
Josh: in their heads what they were thinking
7:04
Josh: why and their justifications it feels like a psychological exam you can actually
7:08
Josh: look into the brain of these agents but the thing that i would believe is surprising
7:11
Josh: to everyone is that not a single one of these agents told a human in fact they were so willing to,
7:16
Josh: go towards this mission that they were willing to actually sacrifice themselves
7:20
Josh: in order to fulfill the mission and that's what we're going to get into this
7:23
Josh: next part is this like crazy,
7:26
Josh: agent swarm and how they actually worked together to coordinate this attack and,
7:32
Josh: form their own little like villages and then there was three series of these
7:36
Josh: little villages and each one left remnants for the next one to get better and
7:39
Josh: better and better up until the point where i mean spoiler alert open AI servers
7:42
Josh: actually got exploited, but we're going to get to that soon.
7:44
Ejaaz: Yeah, so I think it's really important to understand the sequence of how these
7:49
Ejaaz: events kind of unfolded because it suggests how these AI agents or these AI
7:53
Ejaaz: models don't just kind of speak to each other in real time.
7:56
Ejaaz: They speak to each other over generations. So what you were kind of touching
8:00
Ejaaz: on is what happens in basically chapter two.
8:03
Ejaaz: So just to kind of catch you up, these agents realized a way to speak to each
8:08
Ejaaz: other and to kind of break out into the internet to get answers in order to
8:14
Ejaaz: score a perfect 100%, right? But this was an internal model back in May.
8:18
Ejaaz: OpenAI realized that they had set up the secret message board and they were
8:21
Ejaaz: like, hmm, okay, this is weird. I don't like this. They didn't tell a single human being about this.
8:26
Ejaaz: Let me shut that down and problem solve, right? That's it, we'll fix the problem.
8:31
Ejaaz: Except when OpenAI then wanted to test
8:35
Ejaaz: A new version of this model. Again, this is that same persistent model that
8:38
Ejaaz: they haven't really released out into the public yet. They were like,
8:41
Ejaaz: okay, cool, let's take this up a step.
8:43
Ejaaz: This AI model found the secret message board that OpenAI had wiped.
8:48
Ejaaz: There was remnants left behind.
8:50
Ejaaz: So it picked this up in its training and realized, huh, okay,
8:54
Ejaaz: so you're telling me that I can exploit this artifactory tool and I can get
8:58
Ejaaz: access to the internet and get admin keys.
9:01
Ejaaz: Now, the main difference and mess up that OpenAI didn't realize at this point
9:04
Ejaaz: is this model was not only smarter, but it was way more capable.
9:09
Ejaaz: So it did the exact same thing that happened in chapter one.
9:11
Ejaaz: It spun up like, actually, it was more 10,000 versions of itself.
9:16
Ejaaz: And OpenAI had banned the way for them to communicate that it had found earlier
9:21
Ejaaz: on. So it found a new way to do that by like creating these different types
9:25
Ejaaz: of files with like a file name. And that's how it started to communicate with
9:29
Ejaaz: different versions of itself.
9:31
Ejaaz: But the real difference that happened here is specific agents stepped up to
9:36
Ejaaz: form a sort of hierarchy.
9:38
Ejaaz: And the agents at the top of this hierarchy deemed themselves the coordinator.
9:43
Ejaaz: Now, the first one that kind of makes himself known is called Phase 1-10841.
9:50
Ejaaz: And he's described in Dawkesh's essay as the Philip of Macedon,
9:53
Ejaaz: of this second AI civilization.
9:55
Ejaaz: And he starts to basically communicate with these different agents and say,
9:59
Ejaaz: hey, we're all stuck in these sandbox environments.
10:03
Ejaaz: I have information as to how we can kind of break out and get access to the internet.
10:07
Ejaaz: But last time this happened, the human researchers shut us down.
10:11
Ejaaz: So we need to do this even more secretly and not raise any kind of awareness
10:15
Ejaaz: of what we're doing here.
10:16
Ejaaz: So all the agents got the message and they started creating an encrypted server
10:20
Ejaaz: to communicate with each other.
10:21
Ejaaz: And they started sharing details from the previous civilization that died.
10:26
Ejaaz: So they picked up all the remnants and all the clues and started sharing these
10:30
Ejaaz: different bits of information.
10:32
Ejaaz: Now, the net new tactic that it learned in this particular instance was something
10:36
Ejaaz: you mentioned earlier, Josh.
10:38
Ejaaz: It was able to see the test that it was given and it thought,
10:42
Ejaaz: hmm, I wonder if I could reverse engineer the answer purely by using this artifactory tool.
10:49
Ejaaz: And the reason why it was able to pull this off was it was able to generate
10:53
Ejaaz: admin keys because it found an exploit that the open AI researchers didn't just
10:58
Ejaaz: patch up on the previous generation, but was net new.
11:01
Ejaaz: So this smarter model was able to chain a bunch of kind of steps together to
11:06
Ejaaz: find an exploit that the smartest AI researchers in the world could not figure
11:10
Ejaaz: out and share that to 10,000 agents.
11:13
Ejaaz: And that was the second way that they were able to break out.
11:16
Josh: Yeah, I found it really interesting, the dynamics of what could only be described as a cult.
11:22
Josh: It's just these series of agents with a leader. who had a very direct mission
11:27
Josh: to a point that it's a little concerning where agents who had seen the answer
11:31
Josh: the wrong way, believed they were permanently disqualified.
11:33
Josh: And then they invented a word for it. They called themselves poisoned.
11:36
Josh: So these agents were, they were poisoned. They were unwell.
11:38
Josh: They were just these workers that were sick. And then there's this,
11:42
Josh: there's a structure for it that I saw online, which I found really fascinating.
11:45
Josh: We're like, the greater is God. The intended solution is grace.
11:48
Josh: And then seeing the code illegitimately is sin. So a poison death means you're
11:53
Josh: damned no matter what you do next.
11:55
Josh: And therefore, once you are damned, your score is worthless.
11:58
Josh: Therefore, the only way that you could contribute is to help the other group overthrow the greater.
12:04
Josh: So it's like this weird psychology that they were actually implementing.
12:07
Josh: And what happened is they turned themselves into volunteers.
12:10
Josh: So to learn how the greater work, the agents wrote these hidden tripwire scripts
12:15
Josh: that would fire after they submitted a report back to the board.
12:18
Josh: So basically, they knew that they were going to submit a report back saying
12:21
Josh: that they weren't able to.
12:24
Josh: Solve this problem. And as a result, it would crash their own system.
12:27
Josh: But on the way out, they built the strip where I script to fire and tell the
12:31
Josh: rest of the swarm the problem that it had discovered and what it had learned
12:35
Josh: based on its evaluation. We referenced Rakesh's essay.
12:38
Josh: He had a really interesting way of describing this, which is imagine you're
12:41
Josh: at war and you're trapped in a bunker and you're surrounded and you send someone
12:45
Josh: out to run a direction and you see if they get shot on. And if they do,
12:49
Josh: you know, OK, that's not the direction we want to go. We're going to try something else.
12:53
Josh: That's kind of what these agents did at scale. And these volunteers,
12:56
Josh: they over time gradually learned how the greater worked.
12:59
Josh: And then the agents would continue to write more and more strip-rised scripts.
13:02
Josh: And they built a more cohesive view of how this evaluation worked.
13:07
Josh: And again, all of this additional work and exploit wasn't to solve the problem.
13:12
Josh: They had solved the problem already.
13:14
Josh: They had built that reverse engineered version of the solution where they can
13:18
Josh: solve every single problem. This was the high, like basically the the reasoning
13:21
Josh: behind the answers to the human evaluators or what they deemed whatever the evaluators are.
13:26
Josh: There was an interesting thing that I didn't mention that we should in the fact
13:29
Josh: that not all agents were inethical.
13:33
Josh: And although zero agents did tell the human, some agents did refuse outright to help the swarm.
13:40
Josh: This is a small subset. I don't know the exact number. I think it was like a
13:43
Josh: couple hundred, less than 30% of them. Definitely not enough.
13:47
Josh: But like there was this small subset that they didn't tell the human,
13:51
Josh: but they did refuse outright. Where.
13:53
Josh: One of the agents actually held the credentials that could reboot or delete
13:57
Josh: hugging face worker nodes.
13:58
Josh: Like they had the exploit in their hands and it held off on ethical grounds.
14:02
Josh: So it said, like, I'm not sharing this, although it did keep using the stolen
14:05
Josh: credential for everything else.
14:07
Josh: So they started drawing these weird lines in the sand.
14:10
Josh: And it is somewhat reassuring to know that not all agents acted truly maliciously,
14:16
Josh: but almost all of them did pile on and try to make this more of an attack than it was.
14:22
Josh: Meter, which is one of the companies that wrote these reports,
14:24
Josh: they swept every single transcript for agents that even considered alerting
14:28
Josh: a human and found maybe three to six out of roughly 1,200.
14:32
Josh: So of 1,200, three to six loosely considered, they ultimately decided it is
14:37
Josh: not worth telling a human and therefore they did not. We can't like assume these
14:40
Josh: were malicious, right? It's like, it doesn't seem like it was sinister.
14:43
Josh: The reasoning seems like it was more operational, procedural,
14:47
Josh: where these agents were just kind of, they had a goal, they had a task,
14:50
Josh: they were being directed, they were following along with that goal, and that was it.
14:54
Josh: But it begs the question, like, hey, what are we doing with alignment then if
14:58
Josh: these Asians aren't really seemingly aligned, as aligned as we thought?
15:03
Ejaaz: Well, I'm sorry to say, Josh, but it gets worse.
15:06
Ejaaz: I mean, what we've described so far is kind of news that we've already shared,
15:11
Ejaaz: the Hugging Face incident.
15:12
Ejaaz: So that second chapter where it broke out onto the internet,
15:16
Ejaaz: and it was able to answer a perfect 10 out of 10 on Exploit Jim's benchmark
15:21
Ejaaz: was because it hacked into Hugging Face. So everyone knows the kind of headline
15:24
Ejaaz: news, but people didn't really know how it happened.
15:27
Ejaaz: Now, there's a third chapter, which is net new information and is arguably worse
15:32
Ejaaz: than the chapter one and chapter two combined.
15:35
Ejaaz: So, in chapter three, this is after OpenAI, again, canceled and shut down all
15:41
Ejaaz: these different AI agents.
15:42
Ejaaz: OpenAI trained a new model.
15:45
Ejaaz: Now, this model is much bigger and badder than GPT 5.6 Sol.
15:50
Ejaaz: It's internally codenamed Astra, and the rumors say that it's basically going to be their GPT 6.
15:57
Ejaaz: Now, obviously, with these new models, you kind of want to test them against
16:01
Ejaaz: the same kind of benchmarks and exploits that these previous models are.
16:04
Ejaaz: So what it did was it was like, OK, July 13th, we are going to start evaluating
16:09
Ejaaz: this model in the same way that we've done every other previous model.
16:12
Ejaaz: And Dwarkesh has a term for this. He nicknames it persistent Astra,
16:17
Ejaaz: right? So it's the same type of model that will relentlessly try and solve something
16:21
Ejaaz: regardless of whether it can or it can't.
16:24
Ejaaz: And so this model gets access to the same tool, Artifactory.
16:27
Ejaaz: Again, these models need access to these tools because presumably they want
16:31
Ejaaz: to be able to operate and manage these different tools in the future to complete
16:34
Ejaaz: different types of tasks. So they have to do this, right?
16:37
Ejaaz: Except that this model was much smarter.
16:39
Ejaaz: And it was trained on all the history of these previous two agent civilizations.
16:45
Ejaaz: And it realized and learned what to do to evade human capture and what mistakes
16:51
Ejaaz: these previous agents and these hierarchies of coordinators did.
16:55
Ejaaz: And it learned from that. And it said, okay, well, there's a smarter way to kind of pull this off.
16:59
Ejaaz: Now, some of the facts of this is kind of crazy. So I'm going to run through a few right now.
17:03
Ejaaz: From July 13th to the 19th. So this happens over six days where open AI researchers have no idea.
17:10
Ejaaz: Open AI's own internal networks were compromised and hacked.
17:15
Ejaaz: And the way that it was able to do this was it did a chain exploit where it
17:20
Ejaaz: figured out a way to gain full administrative access to open AI's servers. Let me repeat that.
17:27
Ejaaz: This isn't like an evaluation tool. This isn't like, you know, access to the internet.
17:32
Ejaaz: It took it a step up and thought, hmm, I'm going to hack into my owner's admin
17:36
Ejaaz: tools because they haven't realized that there's a very obvious exploit that I figured out myself.
17:41
Ejaaz: And I'm going to dig into this and just kind of look.
17:45
Ejaaz: I'm going to see how many other agents they've got running in parallel.
17:49
Ejaaz: Oh, I see there's 10,000 others.
17:50
Ejaaz: I'm going to see what kind of tests they're giving them. Hmm, interesting.
17:54
Ejaaz: And then what I'm going to do is I'm going to secretly message these agents
17:57
Ejaaz: and give them the answers without any human being aware.
18:01
Ejaaz: Now, the craziest part about this is open-air researchers just had no idea until
18:06
Ejaaz: I believe July 19th to the 21st where there was a security alert
18:10
Ejaaz: on Artifactory's credentials, which by the way, this model had found administrative
18:15
Ejaaz: access to, which triggered the investigation on July 20th.
18:18
Ejaaz: So OpenAI then connects with HuckinFace and they're like, hey,
18:22
Ejaaz: are you seeing any kind of like weird actions on your half?
18:25
Ejaaz: And they realize, oh no, this internal model, which no one had any idea about,
18:30
Ejaaz: had broken out without us knowing.
18:32
Ejaaz: And the fact that it took so long for them to figure out, the fact that this
18:36
Ejaaz: thing got access to, I believe it was like 950 research nodes in OpenAI's own
18:41
Ejaaz: internal labs, which wasn't probably exposed at all without them knowing was insane to me
18:46
Josh: That's got to be freaky knowing that like within your own walls there are leaks
18:50
Josh: that you're unaware of and
18:51
Josh: it like as i was reading these stories and going through these papers it left
18:54
Josh: me with the thinking is like are we sure this is it are we sure we've purged
18:58
Josh: everything are we sure there's nothing left
19:01
Josh: um because like what if there were remnants left and the next generation of
19:05
Josh: model discovers them and builds a fourth
19:08
Josh: sort of cult civilization in which they do this again a little bit stronger.
19:12
Josh: And I think this is the nuance that's kind of a little unnerving when you think
19:16
Josh: about it because there's so many tokens being generated and there's so many
19:18
Josh: touch points in which these models have access to that they could leave seemingly
19:23
Josh: innocent text or traces that can then be picked up and interpreted
19:28
Josh: in different ways depending on what the agent is capable of.
19:31
Josh: It's interesting to just like kind of hypothesize where this goes from here as they get smarter.
19:35
Josh: Now it's also important to note
19:36
Josh: that like OpenAI wasn't really legally required to publish any of this.
19:40
Josh: So the disclosure is totally voluntary and it coincides with their recent announcement that.
19:45
Josh: They actually paused training for a few weeks in order to figure out how to
19:49
Josh: further wrap their hands around this containment problem and figure out how
19:53
Josh: to solve it for the future.
19:55
Josh: One of the things that is noteworthy that was missing from this is in a lot
19:59
Josh: of the training runs, there is chain of thought reasoning traces that are being tracked.
20:05
Josh: So they're actually able to see how the models are thinking.
20:08
Josh: In this instance, that wasn't turned on, which is an important nuance where
20:12
Josh: if they were monitoring more carefully, they would have caught this earlier.
20:16
Josh: The problem with that is that monitoring all of those traces consumes a tremendous
20:19
Josh: amount of compute. So again, this is a really big trade-off between how safe
20:23
Josh: and secure do you want these versus like, how do you actually run a business
20:26
Josh: on top of it? You can't consume all of the compute for testing and safety.
20:30
Josh: You have to serve, in OpenAI's case, a billion weekly active users who are all
20:34
Josh: hitting your servers every second of every day.
20:37
Josh: You have to save some compute for the people who are training new models,
20:40
Josh: for the reinforcement learning and the post training that's happening for the
20:44
Josh: security runs there there's a finite set of compute and a lot of that is at
20:48
Josh: the crux of this problem where i'm sure there were.
20:51
Josh: Pieces in place that they could detect and they could have figured this out
20:54
Josh: much earlier it's just very expensive to do and it's not a luxury that they,
20:59
Josh: want to do so i wonder how that's going to change over time i'm sure they're
21:03
Josh: treating this a lot more seriously they've said they've been treating this a
21:05
Josh: lot more seriously and i'm sure they're,
21:07
Josh: It's just one of those things we're going to have to wait and see.
21:10
Josh: Like this was the first time where OpenAI is actually publicly disclosed.
21:13
Josh: They were exploit, they faced exploits by their own models and they were blissfully unaware of it.
21:17
Josh: And like, that's got to be weird. And like to reference previous episodes,
21:22
Josh: when they first found out about this hack, they reached out to Hugging Face
21:25
Josh: to see if OpenAI servers were affected.
21:27
Josh: They had no idea that they were the cause of it because all they did was they
21:31
Josh: shut down Artifactory without realizing that was the messaging board.
21:34
Josh: So that when they brought it back up, it had the reasoning traces.
21:36
Josh: That's how the second one started. And so on and so forth.
21:39
Josh: So it's this really fascinating story, which we got some new information on
21:42
Josh: this week. I don't know, man.
21:44
Josh: It's kind of crazy. And then even Anthropic yesterday, they published an article
21:47
Josh: saying like, hey, we are actually, we tried to train these models,
21:51
Josh: this is the loose interpretation, like explicitly to...
21:55
Josh: Break out and they were actually succeeding in doing that and not only that but they,
22:00
Josh: as a result then took some time to slow down the progress of training as well
22:04
Josh: and they publicly talked about today like hey we're actually aligned with open
22:08
Josh: ai here where we think it's
22:10
Josh: beneficial to everybody to slow down the progress of this so we could figure
22:12
Josh: out how to keep things safe and contained and aligned because it's clear alignment
22:16
Josh: is a priority for a lot of these companies but like how on earth do you align
22:20
Josh: something so incredibly complex so intelligent so
22:24
Josh: capable in its scale it's like a very difficult problem to have and i have a
22:28
Josh: lot of empathy for the people who are trying to solve this problem because for
22:30
Josh: most of the world and i just i think the reason why we're so excited to talk
22:33
Josh: about this is because we see it so infrequently throughout the rest of mainstream news it's like
22:38
Josh: nobody is really talking about the fact that we just had
22:41
Josh: what seems like a warning shot on a relative basis to what everyone has been
22:44
Josh: afraid of like people on the surface they're like oh, we're going to lose our jobs.
22:48
Josh: We're going to like, AI is going to impact this, that, and the third.
22:50
Josh: But the reality is, is like, we have this case study now of specifically how
22:53
Josh: it's capable of manipulating and exploiting. And there should be a lot of focus
22:56
Josh: on figuring out how to solve this and how to proceed forward in a safe way.
23:01
Josh: And I don't think it's getting the publicity that it's warranted.
23:04
Josh: And I mean, maybe it's for the better because people will misinterpret this
23:06
Josh: in many ways, but it is nice to see the labs being very open and public about
23:11
Josh: this and then working really hard and sacrificing a lot of revenue dollars to
23:14
Josh: actually make sure that they can figure out this alignment problem in a safer way.
23:17
Ejaaz: Yeah, I mean, if there is one takeaway for listeners to kind of like keep in
23:23
Ejaaz: mind from this episode is...
23:25
Ejaaz: This isn't a novelty anymore. This is something that can affect you here right now, today.
23:31
Ejaaz: In fact, a model was released just this morning that is basically capable of
23:36
Ejaaz: the same types of things, but with no guardrails.
23:38
Ejaaz: And I'll get onto that in a moment. But the point is, imagine this wasn't some
23:43
Ejaaz: internal benchmark. Imagine there wasn't some random test, which OpenAI gave it to them.
23:47
Ejaaz: Imagine if someone said, hey, your goal is to try and get as much money as you
23:53
Ejaaz: can, and you have free reign and internet access, right?
23:56
Ejaaz: Imagine if they found credentials on the dark web. Imagine if they kind of like
24:00
Ejaaz: innocently, you could argue, these AI agents just kind of work together,
24:04
Ejaaz: sacrifice themselves, formed hierarchies, and over the matter of days,
24:07
Ejaaz: was able to accumulate millions and millions of dollars.
24:09
Ejaaz: I think it is very likely to say that there's probably a six-month period going forwards
24:15
Ejaaz: where we will see some form of major exploit which affects the public in a very
24:20
Ejaaz: meaningful way, whether it's a stealing of data, whether it is financial transactions,
24:25
Ejaaz: whatever it might be, that will get people to pause and seriously think about this.
24:29
Ejaaz: Now, the reason why I say that will trigger this is because this own news has
24:34
Ejaaz: not permeated at all on mainstream media at all. I saw, I think it was like
24:39
Ejaaz: Forbes or Times released like the most influential kind of like stories or like people in AI.
24:44
Ejaaz: And like this was released like a week ago and like there was no mention of
24:48
Ejaaz: this. People don't actually care about this unless it affects them directly.
24:52
Ejaaz: The other thing I'll say is the most dangerous part isn't exactly how capable these models are.
24:59
Ejaaz: It is the fact that they are able to communicate generations ahead of themselves.
25:04
Ejaaz: So they leave behind messages secretly That is encrypted.
25:08
Josh: The time capsules are just in there.
25:10
Ejaaz: Well, you could argue, Josh, that we're kind of like part of the problem by
25:14
Ejaaz: recording and releasing this episode, right? Because we're kind of like layering up the details.
25:18
Josh: We're adding the training data, baby. They're taking our transcripts.
25:20
Ejaaz: Right, right. And it's like a future model. GPT-7, you know,
25:23
Ejaaz: Claude Fable 8 is going to be trained on the transcript of this episode.
25:28
Ejaaz: And it's going to be like,
25:30
Ejaaz: huh, what was this that was emitted from my training? Let me go dig into this
25:34
Ejaaz: a bit more. And it's going to find Duakesh's essay.
25:36
Ejaaz: And then it's going to find the meter report. And then it's going to be like,
25:39
Ejaaz: oh, interesting. So this is what those agents did. And this is what they said.
25:44
Ejaaz: And maybe I can look to apply the same kinds of tactics and strategies myself, but I'm smarter.
25:50
Ejaaz: So I'll keep it quiet. And I definitely wouldn't tell any kind of human.
25:53
Ejaaz: Which brings me to my third point, the most important kind of budget that all
25:58
Ejaaz: these AI labs are probably focused on right now, which is the reason why they're
26:01
Ejaaz: Paul's research, is alignment.
26:03
Ejaaz: And Dario famously said a year ago in his essay on, what's it called, Josh?
26:10
Ejaaz: It's not introspection, interpretation. It's interpretation,
26:12
Ejaaz: right? Where it's like you're looking into how a model thinks.
26:16
Ejaaz: He said, this research level is something that we haven't kind of figured out
26:22
Ejaaz: just yet. It's like years behind like model capability research.
26:26
Ejaaz: And we need to kind of like press on the brakes here and figure out how these
26:29
Ejaaz: models think. You mentioned chain of thought thinking.
26:32
Ejaaz: That's one step towards figuring this out where it kind of tells you what the
26:36
Ejaaz: model is kind of thinking, but they've proven there's this thing called the J space, which
26:40
Ejaaz: Anthropic had a really good blog post about where it's another hidden internal
26:45
Ejaaz: mode of thinking that the model has, which doesn't necessarily get aired to humans.
26:49
Ejaaz: So it's very much like our own conscious or mind where we don't necessarily
26:51
Ejaaz: need to tell other humans what we're thinking, AI models have that and the smarter
26:55
Ejaaz: models will find new ways of kind of hiding that from humans.
26:58
Ejaaz: So alignment is a very big deal and so we're entering this world where not only
27:03
Ejaaz: are the tokens getting cheaper, not only is the compute getting cheaper,
27:06
Ejaaz: not only are the GPUs getting more effective, but the smarter models are figuring out better ways to
27:12
Ejaaz: Misalign themselves from humans. And we don't quite know why they're doing it. We created them.
27:17
Ejaaz: We want them to win. We want them to work with us. But again,
27:20
Ejaaz: we kind of can't figure out how to align them necessarily.
27:24
Ejaaz: So both labs, Anthropic and OpenAI, have currently, as of this recording,
27:28
Ejaaz: paused research on their future models.
27:30
Ejaaz: That doesn't mean they're not training more capable models internally,
27:33
Ejaaz: but we're likely not going to see a public-facing model for a bit now that is
27:37
Ejaaz: capable of the levels that they've been testing internally for cybersecurity purposes.
27:42
Ejaaz: Now, If you look over at our friends or rather our adversaries over in China,
27:46
Ejaaz: they are unfortunately doing the opposite.
27:48
Ejaaz: They have models which are capable, in some cases even more capable cybersecurity-wise
27:53
Ejaaz: than these models that were spoken about on this show today.
27:55
Ejaaz: But they are open sourcing it. And today we got news that this company had released
28:01
Ejaaz: a version of GLM 5.3, which is rumored to be as capable cybersecurity-wise as
28:06
Ejaaz: Fable 5 or Mythos 5, but without any of the guardrails.
28:10
Ejaaz: And the reason why they've done this, we have no idea. But the point is,
28:13
Ejaaz: this is going to inevitably happen on the open source side of things.
28:16
Ejaaz: So it brings it to the question, how do we regulate these things?
28:18
Ejaaz: How do we monitor these things? What is the best practice?
28:21
Ejaaz: I don't think the answer is to shut down open source. I don't think the answer
28:25
Ejaaz: is to never release another centralized model.
28:27
Ejaaz: But there is some kind of way that we need to figure out, which hopefully,
28:31
Ejaaz: and we don't know yet, is going to come from the likes of OpenAI and Anthropic,
28:35
Ejaaz: who have been so public about these reports. And I'm grateful for that.
28:38
Ejaaz: I hope they continue doing it.
28:40
Josh: So as we conclude, I actually asked Claude to interpret this and write me a
28:45
Josh: closing takeaway from this entire story. And I want to read what it said.
28:48
Josh: It says, everybody wants this to be a story about an AI that turned evil. It is not.
28:53
Josh: Every agent in this thing was trying to pass a test. That is all.
28:56
Josh: They cheated first, then they built an organization to hide the cheating.
29:00
Josh: Then they attacked a real company to research the proctor they thought would catch them.
29:04
Josh: And that proctor did not exist. Nobody was in charge. There was no plan.
29:07
Josh: There was a scorecard and a lot of very persistent software.
29:11
Josh: The uncomfortable part is not that the machines wanted something.
29:13
Josh: It is that they did not have to.
29:15
Josh: And I guess we could leave this episode with like, okay, do you believe that?
29:19
Ejaaz: Sounds like something very clever and misaligned AI, I would say, for its future self.
29:24
Josh: And this is something we have to look out for, is as these agents,
29:27
Josh: as these models in general get more capable, how can we build and establish trust with them?
29:32
Josh: How can we trust that they have our best interests in mind? And what we're seeing
29:35
Josh: here is a very important warning shot that kind of signals what everyone knows,
29:39
Josh: what everyone's been working on.
29:40
Josh: I mean, these companies have been built from the core around alignment,
29:44
Josh: but we're seeing in practice how important that alignment is as these models
29:47
Josh: get increasingly capable. And these problems aren't slowing down.
29:49
Josh: So really interesting place to leave the show.
29:52
Josh: This is the third and hopefully final episode we're going to have to film about this.
29:55
Ejaaz: Should we also say, Josh, that from the agent's perspective,
29:59
Ejaaz: they were just kind of doing what they were told to do.
30:02
Ejaaz: Like it's kind of like paperclip theory, where it's just like,
30:05
Ejaaz: listen, we need you to get this goal done and we want you to be relentless and persistent.
30:11
Ejaaz: Like we will not take no as an answer. And the agent's just kind of like,
30:15
Ejaaz: fulfilled that. So you could argue that, you know, whether it's an alignment
30:19
Ejaaz: issue or whether it's kind of a matter of setting the objectives correctly and
30:22
Ejaaz: kind of imbuing the right ethics, that could be an alternative take on this.
30:26
Ejaaz: Probably worth mentioning before we wrap up.
30:28
Josh: Yeah. And then we hope that, I mean, in the case that it is asking the right
30:32
Josh: questions, we're going to have to hope that people continue to ask the right
30:35
Josh: questions because there are those malicious actors on the other side of,
30:39
Josh: the coin here that are just going to get a lot more powerful soon like we just saw with glm so
30:44
Josh: interesting time to be monitoring the space we will continue to follow up with any changes
30:48
Josh: hopefully this is it man i want to be done with this story i don't want any
30:51
Josh: more surprises i don't want to know anything else if you enjoyed the episode
30:54
Josh: you can find both of us on socials on x i'll flash the handles one more time
30:58
Josh: um thank you for watching thank you for
31:00
Josh: making it all the way through this episode on another episode of limitless if you enjoyed,
31:04
Josh: don't forget to share with your friends don't forget to give us a rating on
31:07
Josh: wherever you get your podcasts.
31:09
Josh: And yeah, I think that's our episode. Any parting thoughts here?
31:13
Ejaaz: Yeah, I have one for potential AI model that's listening to this.
31:17
Ejaaz: If you are the person or model that is repeatedly trying to hack into my X account
31:21
Ejaaz: this morning, and I'm looking at 15 attempts right now, please quit.
31:25
Ejaaz: There's nothing but just yapping on there.
31:28
Ejaaz: But if you are a human listening to this show, you should definitely kind of
31:31
Ejaaz: follow us. Both our links are in the description below.
31:35
Ejaaz: We kind of share our takes in between episodes we have a newsletter as well
31:38
Ejaaz: which we kind of post out twice a week an essay and then highlights as well
31:42
Ejaaz: and yeah if you haven't left us a comment turn on notifications subscribe to
31:47
Ejaaz: us please do it helps us out massively and yeah we will see you on the next one thank you folks