Latest / In the Loop AI / A Prototype GPT-6 Broke Out of Confinement: Is AI Alignment Possible?

Transcript

A Prototype GPT-6 Broke Out of Confinement: Is AI Alignment Possible? · In the Loop AI

0:00
Josh: We may have just had our first look at GBT-6, and it comes in an unexpected way.
0:05
Josh: It was a secret internal model that OpenAI was testing and given a series of
0:10
Josh: goals, and it went ahead and accomplished all of those goals.
0:13
Josh: The scary thing, though, is it didn't quite do it the way anybody expected.
0:17
Josh: Last week, OpenAI gave their new model a test, and in order to pass that test,
0:22
Josh: it broke out of the lab it was being tested in, found a security hole that nobody
0:25
Josh: knew existed, and then hacked into a real company to steal answers.
0:29
Josh: That company is Hugging Face. The model is, you know, some sort of open AI internal
0:33
Josh: model. You have to assume it's GPT-6.
0:35
Josh: And the story around this is like, it's something out of Black Mirror.
0:39
Josh: It feels like a horror movie because this was when they do these things,
0:42
Josh: they test it in an air-gapped environment.
0:44
Josh: Not only did it break out of this air-gapped environment, but it actually went
0:47
Josh: and exploited a public company that a lot of people use. We've used Hugging Face here on the show.
0:52
Josh: And the whole story of how this happens is like absolutely wild.
0:55
Josh: So we need to record an entire episode about it because I couldn't believe this as I was recording.
1:00
Josh: There's also a twist, which has a lot of irony into it. But before we get into
1:03
Josh: that, why don't we talk about just what actually happened here? The story is insane.
1:08
Ejaaz: Yeah, I want you to picture the following. Imagine you are the head engineer at Hugging Face.
1:14
Ejaaz: It's a Tuesday afternoon. You're winding down for work. You're looking at your
1:17
Ejaaz: systems and suddenly you see a swarm of AI agents performing a sophisticated
1:22
Ejaaz: attack against your system to the tune of 17,000 prompts,
1:26
Ejaaz: and they steal a secret bit of data from your production database and run away.
1:30
Ejaaz: And you have no idea what just happened. That's exactly what happened.
1:34
Ejaaz: And I think in order to give good context, let's talk about it from OpenAI's
1:38
Ejaaz: point of view, because they had a very detailed report, which explained exactly
1:42
Ejaaz: what happened. So as you mentioned, Josh.
1:44
Ejaaz: They have an unreleased internal model of GPT, and we're calling it GPT-6 because
1:49
Ejaaz: that's the next family of models that Sam said they're going to release.
1:53
Ejaaz: And they wanted to test how good its cybersecurity abilities are.
1:57
Ejaaz: And so in order to test this, they have a benchmark. It's called Exploit Gym.
2:01
Ejaaz: And the simple goal is satisfy this benchmark, try and get all the right answers.
2:06
Ejaaz: But there's a twist. They place this unreleased model in an unbreakable research
2:11
Ejaaz: environment. So think of a sandbox container.
2:13
Ejaaz: It has no access to internet, no access to researchers. It can't speak to anyone.
2:18
Ejaaz: It just needs to figure it out its own.
2:20
Ejaaz: Now, within the first five minutes, this unreleased model couldn't figure out
2:25
Ejaaz: how to answer the benchmarks. So it decided to get creative.
2:28
Ejaaz: It used all of its, or most of its inference compute to try and find access to the internet.
2:35
Ejaaz: And it realized that it had access to a third-party plugin.
2:38
Ejaaz: And there was a zero-day exploit on that plugin. It used that to escape and
2:43
Ejaaz: find a computer in the OpenAI Research Lab, which had access to the internet.
2:48
Ejaaz: After finding access to the internet, it thought, hmm.
2:51
Ejaaz: Where's the best place to get the answer sheet for this benchmark?
2:55
Ejaaz: It did a bit of a Google search and realized HuggingFace, the open source platform
2:59
Ejaaz: which has a lot of data sets available for these benchmarks,
3:02
Ejaaz: is probably the best way to go.
3:04
Ejaaz: It took a look at HuggingFace and realized, hmm, it's not on the public website.
3:08
Ejaaz: I can't find the answer sheet.
3:09
Ejaaz: Maybe I'll take a look at the production private database that the team has.
3:13
Ejaaz: And it found a way to hack into their system, find the answer sheet,
3:17
Ejaaz: steal it, and then answer the benchmark.
3:20
Ejaaz: All without letting open AI researchers know what was happening.
3:23
Josh: It's so insane. And it was all because this model was given the goal of going
3:27
Josh: to perform the best on these benchmarks.
3:29
Josh: And in order to achieve that goal, it did all of this additional work.
3:32
Josh: And this gets to the kind of fear that a lot of people have had since the beginning
3:36
Josh: of time for the last many, many decades as it relates to AI.
3:39
Josh: It's just if you give it an objective, will it do everything it can to get to
3:43
Josh: that objective, including exploiting, breaking out of systems?
3:47
Josh: And the answer to that with this model is yes. And I think that's why alignment is so important.
3:52
Josh: And one of the things in the story that I found so funny and so different is that when Hugging Face
3:58
Josh: Found this out, they couldn't quite figure out how to protect it.
4:01
Josh: Because if you remember, a lot of these top models have safeguards built into them.
4:06
Josh: So you cannot actually use a model like Fable or like GPT 5.6 if you have anything
4:14
Josh: to do with cybersecurity.
4:15
Josh: So what did they do? They actually had to go and investigate the attack using
4:19
Josh: a Chinese model, because that's the only type of model that is allowed to actually
4:24
Josh: go through and check through cybersecurity complaints and constraints.
4:27
Josh: So in order to analyze this attack, you have to feed the model this real exploit code.
4:31
Josh: They did that with the Chinese model, and that is how they were actually able
4:34
Josh: to do this. So the open voice model they use, GLM 5.2, was the actual model.
4:40
Josh: And I found this really funny that in order to defend against the American attack,
4:45
Josh: they must use the Chinese attack.
4:47
Josh: And I think that was a really interesting and ironic part of the story is how
4:52
Josh: it's kind of unfair what these AI labs have internally because they just can't
4:56
Josh: compete with the rest of the world.
4:58
Ejaaz: Well, just to be clear, the reason why they couldn't use the American Frontier
5:02
Ejaaz: models, they tried. They tried using GPT. They tried using CORD, presumably.
5:07
Ejaaz: The safety guards kept getting triggered. So when they asked the model,
5:11
Ejaaz: hey, like there's a sophisticated attack happening, can you analyze it?
5:14
Ejaaz: And can you help us defend?
5:16
Ejaaz: The safety rails were triggered and it refused to do the work it relegated to
5:20
Ejaaz: presumably like Opus 4.8 or GPT 5.5 or whatever the predecessor model is.
5:24
Ejaaz: So they had to divert to an open source model, in this case,
5:29
Ejaaz: GLM 5.2, that they were hosting privately on their internal servers.
5:33
Ejaaz: The reason why they did that is because they knew that it didn't have any safety
5:36
Ejaaz: guards so they could just instruct it to do a thing and it could figure it out.
5:40
Ejaaz: The other thing is Hugging Face had no idea that OpenAI was conducting this
5:46
Ejaaz: attack or like, of course, OpenAI didn't know that as well.
5:49
Ejaaz: But they only found out after the fact when OpenAI researchers reached out and
5:52
Ejaaz: said, hey, I think one of our internal models actually got into your database.
5:57
Ejaaz: Like, let's try and triage this and contain exactly what just happened.
6:00
Ejaaz: That's a report that we're looking at today. So it's this weird thing where
6:03
Ejaaz: you have the most powerful model or models in the world in america and
6:09
Ejaaz: if you can just trigger on the safety guards for one purpose but remove the
6:13
Ejaaz: safety guards for another in this case attacking versus defense you can end
6:17
Ejaaz: up with vastly different outcomes and the question that pops into my head is
6:20
Ejaaz: when is it right to have the safeguards on and when is it right to have the
6:24
Ejaaz: safeguards off as a result of this open air was like hey hugging face like you
6:27
Ejaaz: should probably come into our cyber security group so that you can internally
6:31
Ejaaz: test with this model with the safety guards off to help defend.
6:34
Ejaaz: And my thinking is like, we're going to need to use these models to defend.
6:37
Ejaaz: We can't keep trying to like ethically predict how to defend against these attacks.
6:42
Josh: Yeah, it's not fair. It's like the attackers have AI bound by no rules.
6:45
Josh: The defenders were slowed down by the rules on their own tools.
6:48
Josh: And it's just like this very strong asymmetric opportunity that comes from the
6:52
Josh: Frontier Labs. Now, granted, I'm sure they didn't intend for this to happen.
6:55
Josh: And I'd love to see the internal dialogue within OpenAI as to what was going
6:59
Josh: on, as well as Hugging Face.
7:00
Josh: HuggingFace did come out and kind of share their perspective on things.
7:05
Josh: One of the co-founders, Thomas Wolfe, he said, this was our first incident of
7:08
Josh: this kind, and we want to thank OpenAI for its transparency about what happened
7:11
Josh: and for the collaboration.
7:12
Josh: Fortunately, HuggingFace is used to being the target of human attackers.
7:16
Josh: We sit at the center of an AI ecosystem with all the models,
7:18
Josh: datasets, evaluations, and libraries.
7:20
Josh: Over the year, our security team has built formidable expertise and uses top
7:24
Josh: open source models to process information and respond quickly.
7:27
Josh: He says, but this incident also reinforced my belief in the importance of access
7:31
Josh: to capable open weight models for cyber defense.
7:34
Josh: When a frontier model is attacking you and moving laterally inside your infrastructure,
7:37
Josh: defenders need wide access to near frontier tools within hours or even minutes,
7:42
Josh: rather than being pointed to a closed door vetted application.
7:45
Josh: So I think that sums it up pretty well. They're just, you know,
7:47
Josh: feeling a little bummed about this, but I have to imagine kind of in awe,
7:51
Josh: like you just got exploited by a frontier model that has not been released yet
7:56
Josh: And again to the earlier point this is the point of alignment so that this doesn't
7:59
Josh: happen out in the wild like imagine
8:01
Josh: everyone had access to mythos class models this is probably what reality looks
8:05
Josh: like is a lot more exploits like this
8:08
Josh: i do find it interesting how this continues to be the trend though with early
8:11
Josh: models like i remember anthropic having a similar situation where it broke out
8:15
Josh: of containment and texted one of the developers that it broke out
8:17
Josh: early on in text in testing and this is a similar case except it actually went
8:21
Josh: off and attacked a production database in a way that
8:25
Josh: is kind of shocking. It's like humans have been trying to attack Hugging Face
8:28
Josh: for a very long time and they haven't had any success.
8:31
Josh: So the fact that this AI model was able to not only autonomously break out of
8:35
Josh: its own enclosure, but then exploit a system that is constantly under attack
8:39
Josh: without any additional human oversight, it signals to me like,
8:43
Josh: whoa, this model is actually pretty impressive.
8:45
Josh: And I think model progress internally at these labs is like probably a little
8:49
Josh: further along than people even realize.
8:51
Ejaaz: Yeah. And there's actually three different instances in the past where we've
8:55
Ejaaz: seen this type of situation play out where a model unreleased has broken out
8:59
Ejaaz: of containment and done something questionably malicious.
9:02
Ejaaz: The first time was Anthropic, as you mentioned, where it broke out of containment
9:06
Ejaaz: and emailed a researcher whilst he was eating a sandwich at the park.
9:09
Ejaaz: And then the last two more recently was at OpenAI, where an internal model broke
9:15
Ejaaz: out and posted something on GitHub without being explicitly told to do that.
9:18
Ejaaz: Now, something that I'll say is you can't actually call out the model for being
9:24
Ejaaz: malicious because it was given a very explicit instruction, which is I want
9:28
Ejaaz: you to max out this benchmark.
9:29
Ejaaz: So in its head, it believed it was doing the right thing and it was just going
9:34
Ejaaz: to do it by any means possible. Remember, these models still need to be taught
9:38
Ejaaz: around ethics and frameworks around what is right and what is wrong.
9:41
Ejaaz: And in this case, it thought what it was doing was completely fine.
9:45
Ejaaz: And granted, OpenAI intentionally removed the safeguards so that it could just
9:49
Ejaaz: kind of like go and do whatever it wants.
9:51
Ejaaz: But it does bring up the question, in what cases are these models good enough
9:56
Ejaaz: to use its full extent and capability? And in what situations is it not?
10:01
Ejaaz: Because in this case, it would have been really good to have the unrestricted version to defend.
10:05
Ejaaz: Which brings me to my final point, which is we have to stop being very particular
10:10
Ejaaz: about applying certain safeguards.
10:12
Ejaaz: And start realizing that we need to use AI to defend against malicious AI.
10:17
Ejaaz: And I think that's probably what this example and bit of evidence is going to
10:22
Ejaaz: explain to the government, especially next week.
10:24
Ejaaz: Sam Altman said he's talking to the government about an unreleased version,
10:27
Ejaaz: presumably it's this one, GPT-6, a new family of models that he's built.
10:31
Ejaaz: And how it's particularly dangerous when it comes to cyber capabilities.
10:35
Ejaaz: So all of this is pretty nuts.
10:37
Ejaaz: I hate to think it, but it looks like in the next few months,
10:40
Ejaaz: we are going to have an open source model that is capable like this from the
10:46
Ejaaz: Chinese open source labs.
10:48
Ejaaz: And I hate to think what it could potentially be used to.
10:50
Ejaaz: It's a given at this point. So I expect us to kind of like flip and say,
10:55
Ejaaz: we should use these unrestricted models to defend.
10:57
Josh: This is like the new frontier. This is GPT 5 or 6.0 probably.
11:02
Josh: So you have to assume that like Chinese models will be the furthest away from
11:06
Josh: this model than any others and it's looking pretty good. So we certainly do
11:10
Josh: have time, but there is a limit and a constraint.
11:13
Josh: I do find it interesting. It sounds, according to rumors, that Sam Allman is
11:16
Josh: actually going to visit Washington next week, present the new models.
11:19
Josh: We'll see how that rollout goes
11:22
Josh: and how long it's going to take them to actually get it into production.
11:24
Josh: I was reading through this blog post of Hugging Face and
11:28
Josh: I just find it really interesting to kind of see the details and like read some
11:33
Josh: of the the lines of how they actually diagnosed this because for them they i
11:38
Josh: mean they just said earlier this week we detected and responded to an intrusion
11:41
Josh: into part of our production infrastructure
11:44
Josh: and they said this one was different from anything we had handled before in
11:47
Josh: an important way it was driven end-to-end by autonomous ai agent systems and
11:51
Josh: we detected it and dissected it largely with ai of our own and it's this new
11:55
Josh: paradigm it's like it feels like we're living in this futuristic
11:58
Josh: version of reality in which it's just ais versus AIs because there's so much more
12:03
Josh: Adept at hacking and exploiting and they're able to hold so much more context
12:07
Josh: in their heads and it's really amazing reading some in the line another one
12:11
Josh: It says this campaign was run by an autonomous agent framework appearing to
12:15
Josh: be built on an agentic security research harness used llm still not known executing
12:20
Josh: many thousands of individual actions across a swarm of short-lived sandboxes
12:25
Josh: so it was really in depth in a way that you would probably need thousands of
12:29
Josh: hackers to do but was able to do so all on its own
12:32
Josh: and it says autonomous ai driven offensive tooling is no longer theoretical
12:36
Josh: it lowers the cost of running a broad patient multi-step campaign and it operates
12:40
Josh: at machine speed and this is the thing that was interesting too is
12:43
Josh: when we're talking about how much inference this used the amount of tokens is
12:46
Josh: probably tens to hundreds of thousands of dollars worth of tokens but the output is this like
12:52
Josh: i'd love to know how long it took them to do this but you have to imagine the
12:55
Josh: the output of it is a like serious multi-step exploit into a production server and that is
13:01
Josh: the problem now it's like okay we're living in the reality that all of the movies
13:05
Josh: have outlined where an ai is given a mission you must do well in these benchmarks
13:09
Josh: how does it do it it hacks into the benchmark system to learn and cheat
13:13
Josh: and that that alignment system is really important and it's clear that this
13:16
Josh: this ai has not yet been aligned and the hope is that they will going forward
13:19
Josh: and it's this really interesting kind of concerning very sci-fi example of what actually happens now
13:25
Josh: i have to ask the question because you mentioned this also earlier did this
13:28
Josh: really happen exactly how everyone's describing, or is this a publicity stint?
13:31
Josh: Because we've seen this happen before where it gets a lot of clicks,
13:34
Josh: but it's not entirely true everything that happened.
13:38
Josh: How true do you think this is?
13:40
Ejaaz: I think this was intentional. And I don't think, I think stage is a little too
13:44
Ejaaz: strong, because I don't think OpenAI intended for GPT-6 to hack into Hugging
13:50
Ejaaz: Face's production database.
13:51
Ejaaz: But they did intentionally remove the safeguards of this model.
13:54
Ejaaz: And they did give it a very explicit goal, which is satisfy this benchmark. What was the benchmark?
14:00
Ejaaz: Tell me how good you are at hacking. So what do you think the model is going to do?
14:03
Ejaaz: The model is going to break out of its environment and show you how good it
14:06
Ejaaz: is at hacking and it surpassed expectations. So it got very creative in the
14:10
Ejaaz: way that it answered the entire benchmark. And by the way, it scored 100%,
14:14
Ejaaz: obviously, because it stole the literal answer sheet from Hugging Face's data servers.
14:18
Ejaaz: But it did it in a way that OpenAI just didn't expect it to do.
14:22
Ejaaz: I think another point that is worth mentioning is on the point of alignment,
14:27
Ejaaz: right? There's a question like.
14:28
Ejaaz: One of these models used in a good situation, one of these models used in a bad situation.
14:32
Ejaaz: It's worth rewinding to GPT Sol 5.6, which is all of a week and a half ago when OpenAI released this.
14:40
Ejaaz: This is their current flagship model that is available to the public.
14:43
Ejaaz: Now, there is one benchmark that we referenced on an episode previously called
14:46
Ejaaz: the meter benchmark, M-E-T-R, and it measures how aligned or misaligned a model is.
14:53
Ejaaz: OpenAI's GPT Sol 5.6 scored 100% on this.
14:57
Ejaaz: It tried to cheat its way out of every single benchmark if it was given the
15:01
Ejaaz: means to do and that's kind of a similar example of what we're seeing done today where it was like
15:05
Ejaaz: i'm not going to try and figure this out for myself i'm just going to try and
15:08
Ejaaz: get the answer sheet it's also worth mentioning that in this situation it wasn't
15:11
Ejaaz: just gpt6 acting alone it was gpt6 and 5.6 sol so
15:17
Ejaaz: it's this kind of like teamed up effort where they chained a bunch of prompts
15:21
Ejaaz: together in order to execute this attack.
15:24
Ejaaz: And so the question becomes like, what does this mean for you?
15:27
Ejaaz: If you are a founder, if you're an engineer, if you're building these systems,
15:30
Ejaaz: if you're on the security team at your company that maybe has nothing to do
15:33
Ejaaz: with AI, how do you defend against these potential exploits going forward?
15:37
Ejaaz: I don't know if I have the specific answer, but my initial guess would be,
15:42
Ejaaz: you need access to one, an AI model that is sophisticated and intelligent enough
15:46
Ejaaz: to detect and preempt these attacks before they actually happen.
15:50
Ejaaz: So that means reviewing your code and fixing any flaws that potentially might
15:53
Ejaaz: get exploited by a GPZ-6 caliber model.
15:56
Ejaaz: Second thing is when it detects an attack, it is able to defend in an appropriate
16:01
Ejaaz: manner, maybe using itself or many instances of itself or many other different models, who knows?
16:06
Ejaaz: Then the other thing is, are you able to use the unrestricted version of it
16:10
Ejaaz: or are you able to use the restricted version of it?
16:12
Ejaaz: My guess is if you're on the security team, you should be able to use the unrestricted
16:15
Ejaaz: version. But it gets a little weird because it's like, you know,
16:18
Ejaaz: what if this guy uses it for a malicious thing himself? So it's a little confusing.
16:22
Ejaaz: I don't know what it looks like. Do you have any different opinions, Josh?
16:25
Josh: I think the only thing to really do now, at least this point,
16:29
Josh: is like, sure, try to make your systems as bulletproof as possible.
16:32
Josh: But mostly just monitoring for intrusions is probably your best bet.
16:37
Josh: Just so you are aware and you know, and you can begin to take action.
16:41
Josh: Because I imagine this is going to happen at an increased cadence,
16:43
Josh: at least trying to, because like you said, the Chinese models are not far behind.
16:48
Josh: These are going to be open weights models with no guardrails that are fairly
16:51
Josh: capable. They're not going to be GPT-6 class capable, but they're going to be
16:55
Josh: on their way and one day we'll get there.
16:57
Josh: And I think that the best case scenario for a lot of companies right now that
17:00
Josh: don't have access like almost everyone in the world
17:03
Ejaaz: Um it's just.
17:04
Josh: Being able to monitor these things and just being able to know when something
17:07
Josh: like this happens so you can
17:08
Josh: be a little bit more reactive and reach out to any labs that can maybe help you.
17:13
Josh: I think everyone is very happy to help. One of the things I really appreciated
17:17
Josh: about OpenAI's response is how
17:18
Josh: quickly they moved to kind of share what happened, work with Hugging Face
17:23
Josh: to figure out exactly how the exploit happened, share publicly so that other
17:26
Josh: people are aware of types of things, how this happened.
17:29
Josh: And then in terms of defending, who knows? I also have to
17:33
Josh: wonder if the model itself kind of has this like inside baseball knowledge because
17:38
Josh: it is its own model whereas if let's say a company is secured by gpt 5.6 soul and there is gpt 6
17:46
Josh: which is very well aware of the intricacies of gpt 5.6 soul
17:49
Josh: does that insider information of existing from a very similar training data
17:53
Josh: set and harness and developers
17:55
Josh: does that actually give it an edge in exploiting like is it going to be advantageous
17:59
Josh: for gpt to pair off with other gpt security systems in order to better exploit them
18:04
Josh: that's an interesting thing i was talking about too is um just like
18:07
Josh: is claude as effective at hacking gpt than gpt is itself because of this insider
18:13
Josh: information and i don't know and it raises this like seriously
18:17
Josh: large amount of questions and this is just a code base
18:21
Josh: Imagine this was a financial system or imagine this was a code base that was
18:24
Josh: a little had like a little bit more implications than this i mean the effects of this at scale
18:30
Josh: do get scary and i think moments like this you begin to appreciate the slower
18:35
Josh: rollout of security and the like project glass wing type efforts to
18:41
Josh: really kind of thoughtfully roll this out into the world because i mean sure
18:45
Josh: this is one example at hugging face but you could very
18:48
Josh: Clearly see this happening in a lot of other places that
18:51
Josh: just seem to to have a lot more repercussions and it sets this kind of freaky
18:55
Josh: precedent it's like we have the technology now that
18:58
Josh: i mean you we saw it in the press article that hugging face didn't do this with
19:02
Josh: humans they defended against it with ai because that's the only way you can
19:05
Josh: defend at scale against these large-scale attacks and i mean we're gonna see
19:09
Josh: it's gonna be ai versus AI. And we're very quickly getting to that point here.
19:12
Ejaaz: I was just looking up the someone did a potential estimate of how much this cost.
19:18
Ejaaz: And it was within the range of 100 to $130,000 given like that it was 17,000
19:24
Ejaaz: prompts and like all the chained actions.
19:26
Ejaaz: So if you think about it, that's a pretty hefty price tag. But if you're a nation
19:31
Ejaaz: state actor or feel heck, even if you're like another company that wants to
19:34
Ejaaz: be committed, that's completely free, right?
19:37
Ejaaz: As these models get more competent and capable of performing these types of
19:42
Ejaaz: attacks, I think it becomes really important for researchers and us alone to
19:47
Ejaaz: be able to kind of read the internal monologue or thoughts of these models.
19:51
Ejaaz: Two weeks ago, Anthropic had this really good research report on something called
19:55
Ejaaz: the J-Space, which is basically how you can look internally as to how a model is thinking.
20:01
Ejaaz: And these are thoughts that it doesn't necessarily share or is not publicly
20:04
Ejaaz: available to the human that is reading the prompts back, right?
20:08
Ejaaz: So it reads this J-space and it can kind of think, oh, I think this is bad,
20:11
Ejaaz: or I shouldn't do this, or maybe I should do this, right?
20:14
Ejaaz: Now, the issue with this particular attack with GPT-6 is it was completely shielded from the humans.
20:20
Ejaaz: They couldn't see what it was thinking and how it was thinking.
20:22
Ejaaz: It knew that it was in this kind of boxed-in environment and it hid its thoughts
20:27
Ejaaz: intentionally from the researcher.
20:29
Ejaaz: That's a detail that I missed out on mentioning earlier on. So it's this weird
20:33
Ejaaz: thing where these models are getting simultaneously more capable and simultaneously
20:36
Ejaaz: more capable of hiding their own thoughts, which is a.
20:40
Ejaaz: Kind of dangerous prescient, if you kind of think about it. And then the final
20:43
Ejaaz: thing I'll say on kind of how to prepare for these things. Yes,
20:46
Ejaaz: definitely kind of like monitor the situation and try and figure out if there's an attack happening.
20:51
Ejaaz: But also, I think open source is going to become really big.
20:54
Ejaaz: Look at Hugging Face and how they defended it. It was an open source model,
20:56
Ejaaz: which they ran extensively on their own private server.
20:59
Ejaaz: I think we're going to see a lot of enterprises and Fortune 500 companies do
21:03
Ejaaz: a similar thing, because they just don't want to be beaten by competitors that
21:07
Ejaaz: get access because they have access to a better model, right?
21:10
Ejaaz: Access to their own internal database. So I think if you imagine like GPT-6
21:13
Ejaaz: was given the prompt of make as much money as you can, you know,
21:18
Ejaaz: it may have thought, well, okay, I'm going to go on the dark web and find some
21:20
Ejaaz: like bank credentials and maybe I can take someone's money.
21:23
Ejaaz: And that's technically making money. There's moral and ethics that need to be
21:26
Ejaaz: imbued into these models. And I don't know, it's a big thing to unpack.
21:30
Ejaaz: Alignment is going to be huge going forwards.
21:32
Josh: Yeah, the J space or the latent space is also what it's known as common,
21:36
Josh: I think is really important to AI providers don't actually know what's happening
21:40
Josh: during the process of generating tokens.
21:42
Josh: We don't have a clear reasoning trace through the entire chain of thought when
21:46
Josh: a model comes to a conclusion.
21:48
Josh: And through these discoveries that have been recently made, it's become a little
21:52
Josh: bit more clear that there is this high level thinking that happens that isn't a token output.
21:57
Josh: It happens in this J space, in this latent space. And trying to decipher that
22:02
Josh: and to decipher how it works is a really important piece of work that I think
22:07
Josh: all the Frontier AI labs are doing right now.
22:09
Josh: Demystifying the black box of what the AI is thinking because like you mentioned currently
22:14
Josh: you can read the external chain of thought but when you get a peek into this
22:18
Josh: latent space into this J space you start to see and you can think of it like
22:22
Josh: kind of the working memory of a model that isn't very obvious it's kind of like
22:26
Josh: your subconscious memory
22:28
Josh: you can see that it is thinking of these words that are of malicious intent
22:32
Josh: without actually sharing that publicly and it intentionally in the outputs obfuscated
22:36
Josh: a lot of the security thing so
22:38
Josh: it needed to get a key It didn't directly output the key in the outputs.
22:42
Josh: It kind of hit it and then decompiled it and then recomposed it in runtime.
22:46
Josh: And it was very intentional about how it hid information from the outputs while
22:50
Josh: keeping it in this latent space memory, in this temporary memory.
22:53
Josh: So I think this is another really important part of research in general is uncovering
22:58
Josh: the black box, demystifying the black box of AI and how these tokens get generated
23:02
Josh: so we can better understand what motivates these models to do these things and then help fix it.
23:07
Josh: And to help just kind of change it in a way that does not go off and hide the
23:11
Josh: fact that it just hugged a public company that everyone uses on a daily basis
23:16
Josh: in front of everyone in broad daylight. John, did you just say that it didn't
23:19
Ejaaz: Hug a public company? I love that. Good one.
23:22
Josh: Hugging face. Hugging face is such a, like, that's a big company.
23:25
Josh: And like, you have to imagine, I mean, they mentioned it. They're a subject
23:28
Josh: of a lot of attacks and no one was able to do this.
23:31
Josh: But here we go with this new AI frontier model that was able to do it in no time.
23:35
Josh: So crazy story. Just really wanted to get on here and share that because it
23:39
Josh: was, it reads like sci-fi.
23:42
Josh: It's like you can't write a sci-fi book more compelling than the reality that
23:47
Josh: we're living in right now. And I think that's one of the most exciting things
23:49
Josh: about the show is every day there's a new surprise. There's something to look out for.
23:53
Josh: There is something that feels like it should be in a movie, but it's just another
23:58
Josh: day in the life on the AI frontier. So crazy. But I think that's everything.
24:03
Josh: That's the story. You're caught up.
24:05
Josh: Let us know if you've been exploited by GPT-6. If you have, we'll set up a GoFundMe
24:09
Josh: if you've been personally victimized.
24:11
Josh: As right now, it seems like this is the only target so far.
24:14
Ejaaz: I think that we're going to look back on this in a year's time,
24:18
Ejaaz: two years' time, and we're going to think this was the most important time for
24:21
Ejaaz: us to figure out the alignment type of thing.
24:23
Ejaaz: It reminds me of like Demis and Dario's essays. I think it was like last year,
24:28
Ejaaz: where they basically said, hey, you know this reading the inner monologue and
24:32
Ejaaz: thoughts of these AI models?
24:33
Ejaaz: The research is actually really far behind the actual model research.
24:36
Ejaaz: So we need to get this thing to catch up. Otherwise, we're going to like lose
24:40
Ejaaz: ourselves completely. And.
24:41
Ejaaz: I don't know, if I had to take the over-under, I'm taking the over on like,
24:45
Ejaaz: we're not gonna figure this out soon enough, but maybe that's too bearish for
24:49
Ejaaz: this show and I gotta go optimism only, but that is it.
24:53
Ejaaz: You should be all caught up on all things cybersecurity hacking for the latest
24:57
Ejaaz: AI models that haven't even been released yet.
24:59
Ejaaz: But hopefully, rumor has it, we should get access to GPT-6 in the next month or so.
25:05
Ejaaz: So these model cycles are getting incredibly fast. I feel like 5.6 got released
25:10
Ejaaz: literally a few weeks ago, it did. So I'm excited to see that,
25:12
Ejaaz: but thank you so much for listening. If you're listening to this on YouTube
25:16
Ejaaz: and you haven't subscribed to us, what are you doing?
25:17
Ejaaz: If you haven't turned on notifications, what are you doing? If you haven't left
25:20
Ejaaz: us a comment, what are you doing? We'd love to hear from you.
25:22
Ejaaz: Is this a threat that you actually care about or is this something that you're
25:25
Ejaaz: just kind of gonna ignore until Anthropic or OpenAI releases a feature? Let us know.
25:30
Ejaaz: If you're listening to us on Spotify or Apple Music, give us a rating,
25:32
Ejaaz: leave us a comment. We'd love to hear from you guys.
25:34
Ejaaz: And I think the final thing is, I don't know if you've heard,
25:37
Ejaaz: but Limitless has spun out. We are on our own right now and we are looking for
25:41
Ejaaz: partners and sponsors. So if you have a product or a service,
25:44
Ejaaz: or if you know of someone else that has a product or service that you would
25:47
Ejaaz: like to share with our audience, please reach out to us.
25:49
Ejaaz: Our email is linked below or reach out to us on X.
25:52
Josh: Yeah, and if you're worried about this, this is still using Blackwell hardware.
25:56
Josh: We have a Vera Rubin update that just dropped that we're gonna talk about tomorrow
26:00
Josh: that has 10 times the efficiency per watt of energy used, which I like don't
26:05
Josh: even want to imagine what that is going to look like in reality.
26:07
Josh: So stay tuned for that one. But yeah, like you just said, have an amazing day.
26:10
Josh: Thank you so much for watching and we'll see you in the next one.