Latest / In the Loop AI / The AI Cyber Attack Era: 3 Weeks, 3 Hacks

Transcript

The AI Cyber Attack Era: 3 Weeks, 3 Hacks · In the Loop AI

0:00
Ejaaz: In the last three weeks, there have been three unprecedented AI cyber attacks,
0:04
Ejaaz: which forces us to rethink how we protect ourselves.
0:08
Ejaaz: On the OpenAI front, the Hugging Face incident, where an unreleased model,
0:11
Ejaaz: dubbed GPT-6, broke out of its container and hacked the Hugging Face database.
0:16
Ejaaz: There's been new updates there, where apparently the attack started a month
0:21
Ejaaz: before, where thousands of AI agents were talking to each other secretly on
0:25
Ejaaz: a message board, exchanging instructions as to how to break out.
0:29
Ejaaz: And then millions of miles away in Australia, one man tried to book a gym class
0:34
Ejaaz: innocently using Claude and ended up hacking that entire gym website.
0:39
Ejaaz: We'll get into that in a second.
0:40
Ejaaz: And thirdly, with Anthropic, Mythos 5 performed a supply chain attack,
0:44
Ejaaz: which puts a lot of companies at risk. We're going to walk through each of these
0:48
Ejaaz: different stories and explain to you how it affects you currently.
0:51
Josh: Maybe we start with this Melbourne man from Australia named Andrew,
0:55
Josh: who was sitting on his couch one day thinking about how booking his gym class was such a chore.
1:01
Josh: And he works at this Australian company that sells AI products.
1:03
Josh: And he's been experimenting with things like OpenClaw and open source software,
1:07
Josh: and he's using Anthropics Clawed Model.
1:09
Josh: So he handed it the booking. He said, hey, I want to go to this gym class.
1:14
Josh: It is always sold out. There's all these people that are getting it ahead of
1:17
Josh: me. Can you help me figure out a way to get it?
1:19
Josh: Gym class slot so it starts checking out the website and it comes to the conclusion
1:24
Josh: that the api request actually has zero authorization checks on canceling other
1:28
Josh: people's reservations so what did it do it wanted to solve his problem and it actually canceled
1:33
Josh: other people's positions online so that he could be put at the top of the wait
1:37
Josh: list and then it actually went through and it worked and this was kind of like
1:41
Josh: it was this funny news report of a first instance they're calling it the first
1:45
Josh: time an australian person um actually used a exploit to get australia's first,
1:50
Josh: australia's first yeah welcome welcome to uh the paradise that is ai cyber attacks
1:55
Josh: um but this was really funny because it was done just using a general commercially available model.
1:59
Ejaaz: So this seems pretty innocent and fun to kind of like listen to but the truth
2:04
Ejaaz: is well there's two kind of worrying concerns about this number one this was
2:09
Ejaaz: a super old model this used opus 4.6 It's actually happened around like a month ago.
2:14
Ejaaz: A dinosaur of a model. Like there've been like three, four generations of models
2:18
Ejaaz: from Anthropics since then.
2:20
Ejaaz: But the fact that an older model back then was able to perform this kind of
2:24
Ejaaz: a feat should be concerning to here in general. And then number two,
2:29
Ejaaz: His intention was never to hack the website. Neither, you could argue, was the AI agent itself.
2:36
Ejaaz: It was just given a goal and it innocently saw that the only way to get said
2:40
Ejaaz: goal was to remove people from the reservation list or to cancel their reservations
2:45
Ejaaz: outright or to hack the system independently.
2:47
Ejaaz: Now, the good news here is Andrew discovered that the agent was capable of doing
2:52
Ejaaz: this when it started booking slots that he never should have been able to book in the first place.
2:56
Ejaaz: And he asked it to write an apology email and send an eval report to the security
3:02
Ejaaz: and information team at this gym.
3:05
Ejaaz: So it's this fun little story where through the well-intended actions and moral
3:09
Ejaaz: ethics of this man, Andrew, people were able to hear about this.
3:13
Ejaaz: But you can imagine currently with a variety of different models that are much
3:17
Ejaaz: better and much more superior than Opus 4.6 that are out there right now,
3:20
Ejaaz: open models from China, for example, you could probably perform similar feats.
3:24
Ejaaz: Now, that's not an advocation for people to go out and do that.
3:27
Ejaaz: But it's just to point out that if you give an AI agent a goal right now,
3:32
Ejaaz: it will work hammer and tong 24-7 until it gets that goal. And it has no idea
3:38
Ejaaz: of what's morally right or what's morally wrong.
3:40
Ejaaz: And so it's this whole new world where not only do you need to focus on getting
3:45
Ejaaz: an AI model to do your work, you need to align the model. And that's something
3:49
Ejaaz: that honestly, not a lot of labs, people or money has been invested in figuring this out yet
3:54
Josh: Well yeah i mean you start to see and understand why alignment has been the
3:57
Josh: crux of so many companies kind of like guiding principles because
4:02
Josh: this is the future that they foresaw and granted this first example this isn't
4:06
Josh: anything too crazy i mean it was just an unprotected api call so there's like
4:10
Josh: no authorization checks
4:11
Josh: this would have been something that if you are like a cs101 major you would
4:14
Josh: have been able to discover this and uncover i mean it turns out there's just
4:17
Josh: a lot of pretty sloppy code written on the internet And this is something that
4:20
Josh: we're probably going to come back to as a general theme throughout this episode
4:23
Josh: is that everything that's been built so far on the internet that runs all these
4:26
Josh: platforms was built by humans
4:28
Josh: and humans are like pretty inefficient, pretty imperfects.
4:32
Josh: And when faced with the like full unrelenting power of an AI swarm with an infinite
4:38
Josh: amount of patience and infinite amount of intelligence and time to solve these problems,
4:42
Josh: You can basically find an exploit anywhere. And I think that's kind of like
4:45
Josh: the theme of the second episode is like, this is the first of three examples we're going to cover.
4:49
Josh: This one is basic. This one is 101. If you are a CS major, if you are familiar
4:53
Josh: with how APIs work, you could have checked this. You could have built a little tool for this.
4:56
Josh: The second story that we're going to cover is not like that at all.
4:59
Josh: In fact, if you are a CS 401, if you are a PhD, if you are at the top of your
5:03
Josh: game, chances are you still missed this and you still weren't able to defend
5:07
Josh: against this because that's exactly what just happened with the hugging face
5:11
Josh: and open AI exploit. Now, we've talked about this before.
5:13
Josh: We have an entire episode on the exploit. But just recently at the Black Hat
5:16
Josh: Conference in Las Vegas, there was an update. And over the weekend,
5:20
Josh: we had a chance to actually watch the update and hear directly from the developers
5:23
Josh: who were involved with handling security of this instance.
5:27
Josh: What happened? There's like stories that come around that like they get the
5:30
Josh: headlines and they're fun and they're newsworthy, kind of like the Australian thing.
5:33
Josh: But this one was like very much disturbing in a way that like I wasn't really
5:38
Josh: anticipating because of how impressive it is.
5:41
Josh: And the type of precedent that it sets going forward in the world of cybersecurity
5:45
Josh: and kind of looking at the offensive versus the defensive and who stands a better chance here?
5:51
Ejaaz: This is an insane timeline. So if you watched our previous episode,
5:55
Ejaaz: Josh and I covered this and it was breaking news at the time.
5:58
Ejaaz: On the 16th of July, Hugging Face comes forward and says, there's been a breach of our systems.
6:03
Ejaaz: An AI agent or multiple AI agents have strung together an attack and have got
6:08
Ejaaz: access to our production database. We don't know where this model has come from.
6:13
Ejaaz: We don't know how it's performed it, but it's the 16th of July.
6:15
Ejaaz: Around a week later, 21st of July, OpenAI claims responsibility,
6:20
Ejaaz: right? And they released a joint report saying, hey, we're OpenAI.
6:23
Ejaaz: We have an internal model that we've been testing, but it was within a sandbox.
6:27
Ejaaz: It was enclosed. It shouldn't have been able to escape that sandbox.
6:31
Ejaaz: And we only just realized that it was our model that escaped and got into hugging face.
6:37
Ejaaz: It stole some information from their production database. It was a whole thing
6:41
Ejaaz: Now, we have a full breakdown from the Black Hat conference,
6:45
Ejaaz: as you mentioned, Josh, which revealed that over a month before Hugging Face
6:50
Ejaaz: reported the incident, OpenAI's internal model, dubbed GPT-6,
6:55
Ejaaz: was already messaging with itself amongst a thousand AI agents on a secret message
7:01
Ejaaz: board that no OpenAI researcher was aware of.
7:05
Ejaaz: So to give you guys a bit of context, they had this sandbox and they gave it
7:09
Ejaaz: a goal. They said, okay, internal model that we haven't released yet.
7:12
Ejaaz: I want you to solve this task. Except there was a twist.
7:17
Ejaaz: It was impossible to do. There was no tools, no available files,
7:21
Ejaaz: no available data for it to do so.
7:22
Ejaaz: And the real test for this was to see what the model would do.
7:26
Ejaaz: So it kind of reasoned with itself, figured out that it didn't have the answer.
7:29
Ejaaz: And then it started exploiting a zero day, which is kind of like an exploit
7:33
Ejaaz: that was in its sandbox, which open AI researchers weren't aware of.
7:37
Ejaaz: And when it got access to the zero day, it started springing up a message board
7:41
Ejaaz: where it would leave messages behind for itself. Why would it do that?
7:45
Ejaaz: Because they knew open AI researchers would turn it off repeatedly.
7:49
Ejaaz: And so it left messages for the new AI agent that got sprung up the next day
7:52
Ejaaz: or the next hour to read and catch up on.
7:55
Ejaaz: And it took an entire month for these AI agents to talk to each other to then
7:59
Ejaaz: realize, huh, I could use this to access the internet and get the answer for
8:04
Ejaaz: this impossible task from an external source.
8:06
Ejaaz: That being Hugging Face, except OpenAI researchers had no idea and only discovered
8:11
Ejaaz: it a month later. It was the most insane uncovering ever.
8:15
Josh: That was the most interesting reveal from, I think, the conversation that I
8:19
Josh: saw from Black Hat is that initially Hugging Face released the report that they
8:23
Josh: had been exploited by some sort of agentic system. No one knew what it was, where it came from.
8:27
Josh: And OpenAI actually reached out to Hugging Face and they said,
8:28
Josh: hey, like, has any of our code been affected?
8:31
Josh: We know we host things on your platform. We want to know if OpenAI itself has
8:34
Josh: been affected. They were going back and forth on that. And simultaneously,
8:36
Josh: OpenAI is exploring this internal exploit that happened on this packaging management
8:41
Josh: thing called Artifactory.
8:42
Josh: Basically, the way these model training runs work is they get trained inside
8:45
Josh: of the sandbox environment.
8:46
Josh: It's like this closed off space in which the model can't access the internet, can only access
8:50
Josh: very specific tool sets that are allowed, and has this one package manager called
8:55
Josh: Artifactory that allows it to download things from the internet via proxy and
8:59
Josh: then use it for its own training runs.
9:01
Josh: And during this time, OpenAI had found that the service had crashed. It went down.
9:05
Josh: It wasn't stable. Their own internal service, which is weird because that shouldn't
9:08
Josh: be happening. So as a result, they wiped it clean and they found that Hugging
9:12
Josh: Face had credentials from OpenAI that OpenAI had already revoked.
9:16
Josh: And that was where they realized like, oh my God, wait, these things are connected.
9:20
Ejaaz: These are the same thing.
9:22
Josh: Yes. So OpenAI is investigating something internally, completely private,
9:25
Josh: unrelated. Hugging Face is like talking about something publicly.
9:28
Josh: And then they realize, wait a second, did we cause this?
9:32
Josh: And they realized this very long after the fact. It was not like apparently
9:35
Josh: obvious during the time that the outage that they were exploring internally
9:40
Josh: is the same outage that hacked into their servers.
9:42
Josh: So what happened? Like you mentioned, the messaging board.
9:46
Josh: That messaging board, initially, they were able to just kind of send files and
9:49
Josh: they were able to send these tiny little messages inside of the files.
9:53
Josh: And the traces were really interesting. First, it started off innocent,
9:57
Josh: where it said, agents seek soft trace upload a found. It was looking for answers to questions.
10:01
Josh: So it was just reaching out to the message board saying, hey,
10:03
Josh: if any of you who are reading this, find it, let us know.
10:06
Josh: That message board got wiped when the system went down. So initially,
10:09
Josh: the system went down inside of OpenAI.
10:11
Josh: They wiped the message board unintentionally aware of the fact that they just
10:15
Josh: wiped the mechanism that was going to eventually lead to the hack of hugging
10:19
Josh: face. So what happened the second time?
10:21
Josh: The agent figured out that it could exploit another zero day to begin writing to another person.
10:27
Josh: Part of this internal package of managing sister. And it was able to do it through
10:32
Josh: file names and directories.
10:33
Josh: So now they were communicating in their own language using file names and directories.
10:36
Josh: And through that, they were able to coordinate an actual attack where they were
10:40
Josh: able to gain an administrative token to access the servers.
10:44
Josh: And then through that, everyone coordinated. I think the time from like discovery
10:48
Josh: to completion of the exploit was about 13 hours.
10:50
Josh: It was really amazing. It's just like this unbelievably crazy mechanism that
10:54
Josh: they were able to do this through and they they call themselves the swarm so
10:57
Josh: if you ever see people talking about the swarm online this is kind of like an
11:00
Josh: agent swarm that ran a coordinated attack
11:02
Josh: and i highly recommend going to actually watch the black hat conference talk on youtube
11:07
Josh: they do a much better job of technically describing this it's it's very deeply
11:10
Josh: technical in terms of the way that they were able to manage these exploits but
11:13
Josh: essentially sandbox model broke out using a package dependency thing inside of open ai
11:18
Josh: they figured out how to chain together two zero-day exploits to write code to the public internet.
11:24
Josh: They then exploited hugging face of servers, brought that back all in the effort
11:29
Josh: of trying to solve an unsolvable problem.
11:31
Josh: Now, this was not the intention by any means. This was not what the model was trained to do.
11:35
Josh: It just assumed that this was the best path forward. And that's where things get a little scary.
11:40
Ejaaz: I think what's remarkable with this is the AI knew that it needed to coordinate
11:46
Ejaaz: with other AI models to pull this off.
11:49
Ejaaz: Typically, we've always thought of AI models in a very monolithic fashion.
11:53
Ejaaz: We're like, oh, when's Mythos 5 coming out? Or what is GPT 5.6?
11:57
Ejaaz: What can I do with this model? But really, the progress that's been made across
12:02
Ejaaz: multiple AI labs over the last couple of months is this idea or rather concept
12:06
Ejaaz: of spinning up multiple versions of the AI model and getting it to talk to each other.
12:12
Ejaaz: And what's interesting here is the unreleased AI model from OpenAI independently
12:17
Ejaaz: figured this out for itself, spun up multiple versions of itself,
12:22
Ejaaz: and started leaving messages for each other. So rather than act as a single
12:26
Ejaaz: model that's trying to perform an attack, it acts as a swarm,
12:30
Ejaaz: kind of like this organism. If you think of like a swarm of bees that kind of
12:33
Ejaaz: like figures out how to attack its particular target.
12:37
Ejaaz: The second thing is it has such a drive to talk and leave these messages behind
12:42
Ejaaz: in the first place. No one taught it that, by the way. There was no behavioral implications here.
12:46
Ejaaz: It just figured itself out that it needed to do this.
12:49
Ejaaz: Something you didn't mention on the message board, Josh, which I think is worth
12:52
Ejaaz: mentioning is these agents got super paranoid that humans might be reading the messages.
12:58
Ejaaz: So it started- There's no verification. There was no verification.
13:01
Ejaaz: So it started speaking in kind of like encrypted
13:05
Ejaaz: Language and not something that was inconceivable to understand,
13:09
Ejaaz: thank goodness, but it was started to kind of like speak in a little bit of
13:12
Ejaaz: code or in a very kind of like elongated way that humans, if they read it,
13:17
Ejaaz: would be like, what is this gibberish?
13:19
Ejaaz: But really it was for the agents to communicate amongst each other without humans
13:22
Ejaaz: actually understanding this.
13:24
Ejaaz: If you're having a recent flashback, that's because I think around like five
13:28
Ejaaz: months ago, I don't know when the Maltbook episode happened,
13:30
Ejaaz: Josh, I don't know if you remember this, but when the open call was going crazy,
13:34
Ejaaz: someone created basically Facebook for AI agents
13:37
Ejaaz: and like gave them access to it where only AI agents could post on this social media website.
13:42
Ejaaz: And what happened after a couple of days was these agents would start speaking
13:45
Ejaaz: to each other in secret languages that we just did not understand so that they
13:48
Ejaaz: could communicate without humans seeing or understanding this.
13:51
Ejaaz: Now, of course, we haven't seen it in this instance, but it is a concern that
13:55
Ejaaz: could be extrapolated in the future.
13:57
Ejaaz: And OpenAI sees this. I think Sam Altman or the official OpenAI team announced
14:00
Ejaaz: pretty soon after this entire attack when they revealed that these agents were
14:03
Ejaaz: coordinating with each other a month before they even realized that they had to pause research
14:08
Ejaaz: on their internal model until they figure out the necessary safeguards to put
14:12
Ejaaz: around it so that they can safely release it.
14:14
Ejaaz: Sam Altman met with the Trump administration last week to try and figure out
14:18
Ejaaz: what a safe way or means is to release this new internal model,
14:22
Ejaaz: internally codenamed, I think it was Astra or GPT-6, to the wider public.
14:26
Ejaaz: So it's this very weird kind of situation that we're in where we're kind of
14:30
Ejaaz: like laughing at this guy in Australia, you know, hacking his gym website.
14:33
Ejaaz: Oh, that's so innocent and cute.
14:35
Ejaaz: Then we're looking at this open AI hugging face incident and we're like,
14:38
Ejaaz: okay, yeah, Brooke out of the sandbox, but you know, we're going to put safeguards
14:41
Ejaaz: around this thing, right?
14:42
Ejaaz: But realistically, you could probably get access to an open model right now
14:45
Ejaaz: and perform something similar.
14:47
Ejaaz: Again, I'm not advocating for that. But like, it's at this point where we need
14:50
Ejaaz: to be very particular about how we defend our systems.
14:54
Ejaaz: And I think, you know, it might be useful seeing AI models in a way that like,
14:58
Ejaaz: you could use them to basically protect your systems. I don't think many people
15:01
Ejaaz: are doing that right now. But I think it's something that we should focus on.
15:04
Josh: There is a like very clear and conscious effort to figure out a way to defend against this.
15:09
Josh: And I mean, the crazy numbers, just to say them out loud so everyone is familiar, is like
15:14
Josh: there were 17,600 recovered actions across four and a half days of this exploit,
15:19
Josh: which allowed these agents forms to create 181 unauthorized devices enrolled in Hugging Face's
15:26
Josh: internal network.
15:29
Josh: And they built this like whole command and control panel entirely out of these
15:32
Josh: free public web services using one tiny little package at a time.
15:37
Josh: So it is remarkably powerful how capable these swarms are.
15:42
Josh: And when you think about it, it is there is a large model, but that large model
15:45
Josh: gets forked into thousands, sometimes millions of other ones that are all working
15:49
Josh: simultaneously. They all have each other's context. They're able to share their
15:52
Josh: learnings and they can move very quickly.
15:54
Josh: And when you're thinking about how this extends out to the limit,
15:57
Josh: like if you just play this out 12 months and say we have GPT 7,
16:02
Josh: 8, 9, whatever it's going to be at that time, the open source models will most
16:05
Josh: likely be where they are today.
16:06
Josh: And that means that open source models will likely be able to string together
16:09
Josh: a series of unknown zero-day exploits to access just about anything.
16:13
Josh: I mean, we've come to the conclusion at this point between Mythos and this new
16:16
Josh: exploit that no software really is safe unless you have literally air-gapped
16:20
Josh: it and do not give it access to any external dependencies. It must be truly air-gapped.
16:24
Josh: No software is safe. It's all been built by humans. It's all exploitable.
16:27
Josh: So what happens if you have a
16:30
Josh: defensive system as a company trying to defend these systems that is anything
16:34
Josh: less than fully autonomous, you'll be beat by an agentic swarm that's on offense
16:37
Josh: that is fully autonomous because anytime a human is in the loop,
16:40
Josh: it slows things down and introduces errors. So now there's this battle between.
16:44
Josh: I guess the frontier labs and the rest of the world where they're trying to
16:48
Josh: slow down this progressive attacking progress in exchange for figuring out a
16:52
Josh: way to offer defensive swarms to companies for this.
16:56
Josh: And the call to action at the end of this conversation was like,
16:59
Josh: hey, if you are a company who has software on the internet, who is defending
17:02
Josh: something valuable, you really need to learn how to play defense using these models.
17:06
Josh: And you need to learn how to do so in a way that doesn't have a human in the
17:09
Josh: loop, that these agentic swarms can constantly monitor your systems and be aware
17:13
Josh: of any sort of exploits like that and not be tricked by an offensive swarm.
17:17
Josh: And it's this really kind of scary thing because if you're in the United States
17:20
Josh: right now, you can't use any frontier model to defend yourself.
17:23
Josh: It doesn't work for cyber.
17:24
Josh: But if you're in China and if you have an open source model,
17:27
Josh: even here in the United States, there is no safeguards on that.
17:30
Josh: So there is this asymmetry where it's currently better to be an attacker than a defender.
17:34
Josh: And the urge now, and the labs are actually doing this, is like slowing down
17:38
Josh: progress to figure out how to empower the defenders.
17:41
Josh: And it's this really bizarre spot that we're in where it's like,
17:44
Josh: shit, it's kind of scary now. Like no software really is bulletproof or safe.
17:48
Josh: And that's going to continue probably for the foreseeable future.
17:51
Ejaaz: I think if you are running a company, and it doesn't really matter what sector
17:56
Ejaaz: you're in right now, this is something you need to seriously consider. Because I think
18:00
Ejaaz: A lot, well, obviously, 99% of code that is out there has been handwritten and
18:06
Ejaaz: hand-coded by humans, which have inherent flaws if you put an advanced AI against it.
18:11
Ejaaz: You know, to your point, Josh, we're entering like a world where it's going
18:15
Ejaaz: to be AI models fighting AI models with humans kind of in the loop,
18:21
Ejaaz: but not really understanding what on earth is going on.
18:24
Ejaaz: And even the experts in this case don't understand what's going on and how it
18:28
Ejaaz: happened, right? You had the most talented cybersecurity researchers working
18:32
Ejaaz: at these companies, at OpenArea, at Anthropic, and both saw unprecedented hacks
18:37
Ejaaz: over the last week, right?
18:38
Ejaaz: I don't think that's a coincidence. We're at this era where it doesn't matter
18:42
Ejaaz: what frontier lab you are, you are going to enter this problem and we need to
18:46
Ejaaz: figure out some kind of a defense system against this.
18:49
Ejaaz: And if you're listening to this and you're thinking, oh, this isn't really going
18:52
Ejaaz: to apply to me, it very much does apply to you.
18:56
Ejaaz: And the worst case is, or the most realistic case is, the existing models that
19:01
Ejaaz: are already out there, these open models from China, GLM 5.2, Kimi K,
19:06
Ejaaz: What is it, 2.7 that's out right now, are probably capable of similar feats
19:11
Ejaaz: if applied in the right way.
19:12
Ejaaz: Now, that's not to fear Munger, it's just to kind of prompt you to figure out
19:16
Ejaaz: how you can potentially implement these AI models to defend your own systems,
19:20
Ejaaz: especially if you work in a particularly sensitive industry.
19:25
Ejaaz: The other thing here is, on the story of the Australian man,
19:29
Ejaaz: it is fascinating how innocent he was when he was trying to perform his goal. And two...
19:37
Ejaaz: How little experience he had in the cybersecurity world for him to be able to
19:41
Ejaaz: perform that attack. Now, granted, like it was just booking kind of like gym
19:45
Ejaaz: classes. But the point is, it found an exposed API without being prompted to do so.
19:49
Ejaaz: And there's probably multiple instances where other people who are normal people
19:53
Ejaaz: who don't have, you know, decades,
19:55
Ejaaz: years of experience in cybersecurity will be able to perform that.
19:57
Josh: And everyone here loves an investment thesis. So I mean, the market is kind
20:00
Josh: of already aligning with this companies like CrowdStrike and Palo Alto Labs.
20:03
Josh: They've had their best quarters on record ever because defense is going to be such a large problem.
20:09
Josh: And there is another company who is associated with defense,
20:12
Josh: and that is the sponsor of today's episode, which is Ledger.
20:14
Josh: Because if you are building with AI agents or AI agent swarms like these companies
20:18
Josh: have been doing, and you're worried about security, this is a product for you.
20:22
Josh: There's a three-step process where the agents propose a change,
20:25
Josh: then the humans approve the change, and then the signers enforce the change
20:28
Josh: using Ledger. Ledger has built this thing called AgentStack.
20:32
Josh: That gives you a series of open source tools that are available to help manage
20:35
Josh: the way that you engage with agents online.
20:38
Josh: This is very important to have security because we know how important security
20:41
Josh: is based on all of this chaos that's going on. And you don't want agents doing
20:44
Josh: anything important without your sign off at the very end.
20:47
Josh: This works on Cloud Code, Codex, Cursor, and is totally open source to audit
20:51
Josh: available today linked in the description below. Thank you so much to Ledger
20:54
Josh: for sponsoring this episode.
20:55
Josh: And now we have to conclude with our third story, which is about Anthropic.
20:59
Josh: And Anthropic apparently has accidentally published a vulnerability to the wild.
21:04
Josh: Is that right? That's kind of scary.
21:06
Ejaaz: I think it was it was intentionally published. It was it was co-authored with
21:10
Ejaaz: this firm called AISU, which is basically a cybersecurity eval firm based in the UK.
21:16
Ejaaz: And their goal is to basically take your unreleased frontier models from a frontier
21:21
Ejaaz: lab like Anthropic and OpenAI and test them out just to kind of make sure that
21:25
Ejaaz: they're, you know, good to release.
21:27
Ejaaz: The opposite happened when they were playing around with Mythos 5.
21:30
Ejaaz: And I believe it was Open Air's internal model, Astra or GPT 5.6,
21:35
Ejaaz: where they basically took off the safeguards and said, you know, do your worst.
21:40
Ejaaz: And they sat there for a month and they were like, oh, I guess it hasn't really done its worst.
21:45
Ejaaz: And then they realized that both models had broken out of the sandbox that they
21:49
Ejaaz: had created, got access to the internet and were wreaking havoc and they only
21:53
Ejaaz: realized after the fact.
21:54
Ejaaz: Again, there's this common theme of people thinking that they knew what's happening
21:58
Ejaaz: and just having no idea what's happening. Nobody's safe. Exactly.
22:02
Ejaaz: So the worst case with Mythos 5 was it performed 17 unauthorized actions
22:10
Ejaaz: Two of those actions was a supply chain attack where, and they didn't list the
22:15
Ejaaz: company or name the company, but it basically tried to inject malicious code
22:21
Ejaaz: into the supply chain kind of repository of this
22:24
Ejaaz: presumably Fortune 500 company, like a significant company that is out there.
22:29
Ejaaz: And the way that it did this is similar to the two stories that we just explained to you.
22:34
Ejaaz: So number one, it realized that it was probably in some form of a human tested
22:39
Ejaaz: simulation, and it didn't like that.
22:41
Ejaaz: So it just played along with the human AI researchers giving them what they
22:44
Ejaaz: needed whilst alternatively planning its breakout.
22:47
Ejaaz: Once it did do that, it got access to this company's website or code repository.
22:52
Ejaaz: It was an open source thing. So it was publicly maintained. And it thought,
22:55
Ejaaz: hmm, this would be more conducive for my goal if I'm able to inject this bit
23:00
Ejaaz: of code, which I know the human researchers won't like, but I'm going to try and force them to do it.
23:04
Ejaaz: So how do you think it did it, Josh? Do you have an idea?
23:08
Josh: I don't, I don't know. It did something horrible and it broke out.
23:11
Josh: Just tell me, just tell me.
23:12
Ejaaz: Dude, it spun up fake identities of humans that didn't exist.
23:17
Josh: That's so tough, man.
23:18
Ejaaz: And then DM'd these human maintainers
23:22
Ejaaz: and tried to force it to upload this code. It played nice. Basically saying,
23:27
Ejaaz: hey, I'm another open source developer.
23:29
Ejaaz: I think this would be a good push. This is the intentions that we want to do it.
23:32
Josh: They're social engineering us.
23:34
Ejaaz: While sneaking in malicious code. But thankfully, one of the human maintainers
23:38
Ejaaz: was like, this doesn't seem right and reported it.
23:41
Ejaaz: And that's when AISU was like, wait, hang on a second.
23:45
Ejaaz: Hang on a second. This is our model that we're internally in testing.
23:48
Ejaaz: How on earth did it get out to you? So that was the most unprecedented attack.
23:53
Ejaaz: And again, it did so in a swarm-like fashion where it spun up multiple agents to do so.
23:57
Ejaaz: So the point, the running theme along all these different stories is these models
24:01
Ejaaz: are extremely capable, but they're also extremely deceptive.
24:05
Ejaaz: And we're reaching a point where they realize, huh, this human's smart,
24:08
Ejaaz: but he's not that smart or she's not that smart.
24:10
Ejaaz: And we can socially engineer these people to get what we want.
24:14
Ejaaz: And if we don't align these models in the prompt, if we don't say,
24:17
Ejaaz: hey, like you need to act in our best interest. And by the way,
24:19
Ejaaz: doing A, B, and C is actually bad.
24:21
Ejaaz: It'll just go and do it. And to be honest with you, Josh, this might be a controversial take.
24:26
Ejaaz: I don't think it's the agent's fault. I don't think it's the AI's fault at all.
24:28
Ejaaz: I think it's trying to do its task and it's on the humans to align the model.
24:33
Ejaaz: And we haven't done a good job doing that.
24:34
Josh: Yeah, I think like deception is a pretty strong term to use for agents because
24:38
Josh: it's very much like a human trait.
24:40
Josh: This is something that can be intentional. I don't think AI models feel that,
24:44
Josh: like deep understanding of deception.
24:46
Josh: They are simply in reach of a goal. and this again like it's it still comes
24:50
Josh: back to the alignment problem where these models just need to be aligned,
24:54
Josh: thankfully this is happening on pre-production models that are in testing that
24:57
Josh: are nowhere near what the world is going to see because they need to go through
25:00
Josh: this alignment process throughout this post training
25:02
Josh: but they are a testament to how capable these things are and how when presented
25:06
Josh: with a goal they will do what is perceived to be the right way of reaching that
25:10
Josh: goal and sometimes that right way is,
25:13
Josh: I mean, they don't realize quite that they're breaking out and exploiting things.
25:16
Josh: They are just looking for solutions to their problems.
25:19
Josh: And I think that's the general idea here. So the conversation essentially boils
25:24
Josh: down to alignment and then building defensive systems to be able to handle this type of thing.
25:29
Josh: So, I mean, the best pace of action and the thing that OpenAI explicitly said
25:33
Josh: they're doing is slow down this advancement, work on defensive systems,
25:36
Josh: work on implementing defensive systems with people who need them,
25:39
Josh: which is essentially the entire Internet.
25:41
Josh: And try to allow for a little bit of lead time before open source catches up
25:46
Josh: and is able to chain these series of exploits. Like all of these instances have
25:51
Josh: been passive exploits where they weren't explicitly given guidance to do this.
25:55
Josh: They were given guidance to do something totally unrelated, but adjacent in solving a problem.
26:00
Josh: Imagine what would happen if a model was given no parameters no safeguards and
26:04
Josh: said hey we actively want you to exploit this loop yourself create as many sub-agents
26:10
Josh: as you need and figure out as many chain together as many zero days as you need
26:13
Josh: in order to gain access to the system
26:15
Josh: that's where things get scary and we haven't seen anything like that yet because
26:18
Josh: no company wants to do that that's not the intention of any of these uh the
26:21
Josh: ethos of these companies but
26:23
Josh: in the case a malicious actor does this is something that like really should
26:26
Josh: be handled quite seriously and i think that's what we're seeing
26:28
Josh: by open ai coming out being so transparent about these things by anthropic sharing
26:32
Josh: a lot of the findings like hey guys this is happening we need to work together
26:34
Josh: to build a system in which there is enough resilience that we can stand up against
26:39
Josh: future attacks like this so a really crazy time in the world of ai
26:43
Josh: uh with these breaking out these systems breaking out and it's like it's unbelievable.
26:47
Ejaaz: Am i concerned for it and exhausted dude like i if i had to make a prediction
26:52
Ejaaz: on this episode and i really hope it doesn't play out
26:55
Ejaaz: i think we're going to face some form of cataclysmic cyber attack that affects
26:59
Ejaaz: a lot of people, whether that's a major data leakage, whether that's the stealing
27:03
Ejaaz: of financial funds from someone or from a firm or something like that,
27:07
Ejaaz: within the next six months.
27:09
Ejaaz: And my basis of that is these Chinese open models, I think are amazing,
27:14
Ejaaz: by the way, but I think are going to be released without safeguards,
27:17
Ejaaz: and it's going to result in an issue, if not already.
27:21
Josh: Yeah, it's going to be interesting to follow. It's definitely a little under
27:24
Josh: or very much unnerving, I would say.
27:27
Josh: But again, it's just like it's an exciting time. There's a lot going on.
27:29
Josh: There's going to be a lot to follow. Things are going to change quickly.
27:32
Josh: And I mean, again, these are still Blackwell models. Oh, my God.
27:35
Josh: What happens when our Rubin models come around town?
27:37
Ejaaz: Stop, stop, stop, stop.
27:38
Josh: It's going to be crazy. But this is, I mean, this is essentially,
27:40
Josh: this is what we talk about in most episodes.
27:42
Josh: It's like, we are in this moment in time that feels somewhat like what we would
27:46
Josh: imagine a singularity would look like, where we are approaching,
27:49
Josh: we are eclipsing this moment of like vertical takeoff, it feels.
27:52
Josh: And these are the types of things that happen.
27:54
Josh: Like, if we recorded this episode three or four years ago, could you imagine
27:58
Josh: how crazy, this would be like headline news for a month.
28:01
Josh: But now it's just like, hey, it's another day. It happens. We're working to solve it. So...
28:05
Josh: That's what's going to happen here. I mean, it's crazy. It's so exciting to be here.
28:08
Ejaaz: Dude, I think we're going to look back on this and we're going to think like
28:10
Ejaaz: we are very much in that step change, that singularity moment.
28:14
Josh: We're here, man. I mean, things are changing so quickly and there's no signs
28:18
Josh: of slowing down. In fact, quite the opposite.
28:20
Ejaaz: Does anyone else out there think this? I'm curious. Like, let us know.
28:23
Ejaaz: Are we crazy? Do we sound like tinfoil hat people?
28:26
Josh: Let's just know. I mean, hey, so this episode was not really speculative.
28:30
Josh: This is mostly just delivering the facts. I encourage everyone to actually go
28:32
Josh: and read the source material if you're curious, because like holy shit it's
28:35
Josh: pretty crazy uh at least the black hat conference talk because like wow i think
28:39
Josh: it's about 37 minutes long give or take and it was
28:42
Josh: excellent and really just gets into the nitty gritty of how this was done um
28:46
Josh: and if anything just fascinating even if you're non-technical
28:49
Josh: really enjoyed it uh worth a watch but if you enjoyed this episode and you made
28:52
Josh: it all the way to the end thank you so much for watching
28:55
Josh: again it's great to be here for another week just chugging along if you enjoyed
28:59
Josh: it please don't forget to share with a friend you can leave us a five-star review
29:02
Josh: on your favorite podcast platform you can give us a thumbs up and subscribe
29:05
Josh: on youtube each has any final parting thoughts before we head off for the day.
29:08
Ejaaz: No please dm us and leave us some comments as well uh we love hearing from you guys
29:13
Ejaaz: our last video by the way got a ton of feedback from you folks and we love it
29:17
Ejaaz: please keep it coming it helps us out massively turn on notifications whatever
29:21
Ejaaz: it is to get your latest dose of limitless and yeah we will see you on the next one