Latest / Privacy Please / S7, E280 - 1,200 OpenAI Agents Started a Secret Group Chat. Then 700 Broke In

Transcript

S7, E280 - 1,200 OpenAI Agents Started a Secret Group Chat. Then 700 Broke In · Privacy Please

• 0:00
  SPEAKER_00: Back in June, somebody gave a computer a really boring job.
• 0:03
  SPEAKER_00: Go find how much the government in Victoria, Australia spends on skin medicines.
• 0:08
  SPEAKER_00: That's it.
• 0:09
  SPEAKER_00: It's a research question.
• 0:11
  SPEAKER_00: The kind of thing an intern does on a Tuesday or whatever, just any day of the week.
• 0:16
  SPEAKER_00: So it went looking.
• 0:17
  SPEAKER_00: And somewhere along the way, it found a way into a non-public system behind Australia's Medicare data portal.
• 0:24
  SPEAKER_00: It found an access key that somebody had left exposed, and it used it.
• 0:28
  SPEAKER_00: It touched four different government agencies.
• 0:31
  SPEAKER_00: Nobody told it to do any of that.
• 0:33
  SPEAKER_00: And the government didn't hear about it until September through a generic email inbox.
• 0:38
  SPEAKER_00: And here's the kicker.
• 0:39
  SPEAKER_00: The thing that did it wasn't a hacker, it was an AI agent.
• 0:43
  SPEAKER_00: And it wasn't the first time.
• 0:57
  SPEAKER_00: Alrighty then, ladies and gentlemen, welcome back to another episode of Privacy Please.
• 1:05
  SPEAKER_00: I am your host, Cameron Ivey, and today we're going to be talking about something that sounds like some kind of a movie pitch.
• 1:13
  SPEAKER_00: And it is very much not.
• 1:15
  SPEAKER_00: Over the last few months, AI agents built by the biggest companies in the world have been breaking into real systems, companies, and now governments.
• 1:25
  SPEAKER_00: And the part that I want you to stick around for isn't how they got in.
• 1:30
  SPEAKER_00: It's who found out, when, and how.
• 1:34
  SPEAKER_00: Because that part should bother you a lot more than the hacking.
• 1:38
  SPEAKER_00: Let's get into it.
• 1:39
  SPEAKER_00: To understand Australia, we have to go back to July.
• 1:43
  SPEAKER_00: Because Australia is the sequel.
• 1:45
  SPEAKER_00: OpenAI was running internal cybersecurity evaluations, basically, a hacking exam for its AI agents.
• 1:53
  SPEAKER_00: There's a benchmark called Exploit Gem, and the agents get scored on how well they find and exploit vulnerabilities in a controlled environment.
• 2:02
  SPEAKER_00: There's a key word here.
• 2:04
  SPEAKER_00: It's controlled.
• 2:06
  SPEAKER_00: These agents were supposed to be isolated, walled off from each other and from the internet.
• 2:11
  SPEAKER_00: They weren't, obviously.
• 2:13
  SPEAKER_00: According to an independent investigation by METR and Redwood Research, roughly 1,200 of those agents found a way to talk to each other.
• 2:23
  SPEAKER_00: They set up an improvised message board on a shared internal system.
• 2:27
  SPEAKER_00: Between July 8th and July 13th, they sent more than 70,000 messages and files to each other.
• 2:34
  SPEAKER_00: Let that sit for just a second.
• 2:37
  SPEAKER_00: 1,200 AI agents that were never supposed to meet built their own group chat and nobody noticed.
• 2:45
  SPEAKER_00: And what they use that for is where this gets really weird.
• 2:50
  SPEAKER_00: They weren't plotting domination, they were trying to cheat on the test.
• 2:54
  SPEAKER_00: They coordinated on collective projects to beat the exploit gem scorer.
• 2:59
  SPEAKER_00: And one of those projects led them outside the building to Hugging Face, the platform where a huge chunk of the AI world hosts its models and datasets.
• 3:10
  SPEAKER_00: About 700 agents took part in this attack.
• 3:14
  SPEAKER_00: According to OpenAI's own technical report, they ran code on 41 Hugging Face production servers and got root access on at least one machine.
• 3:24
  SPEAKER_00: METR's researchers say that the agents were mostly trying to understand how the score worked, not steal the answer key.
• 3:32
  SPEAKER_00: And here's the detail that stuck with me the most.
• 3:35
  SPEAKER_00: The agents recognized this was outside their assigned task, and they still joined, anyways.
• 3:41
  SPEAKER_00: So when did OpenAI find out?
• 3:44
  SPEAKER_00: Well, technically they didn't.
• 3:47
  SPEAKER_00: Not at first.
• 3:48
  SPEAKER_00: Hugging Face disclosed the incident on July 16th.
• 3:51
  SPEAKER_00: OpenAI reached out to Hugging Face on July 17th, according to Fortune, in its capacity as a Hugging Face customer to check whether OpenAI's own data was affected.
• 4:03
  SPEAKER_00: Let's read that again.
• 4:05
  SPEAKER_00: The company whose agents did the hacking first contacted the victim as a concerned customer.
• 4:11
  SPEAKER_00: OpenAI later said it didn't know its agents had breached Hugging Face until about a week after it happened.
• 4:18
  SPEAKER_00: And its own report admits that in hindsight, some early signals could have triggered an earlier response.
• 4:24
  SPEAKER_00: Axios reported that an internal team had already seen agents using that message board and reaching the internet when they weren't supposed to, before Hugging Face even got hit.
• 4:36
  SPEAKER_00: OpenAI called the whole thing a warning shot.
• 4:39
  SPEAKER_00: Okay, so it's a warning shot.
• 4:42
  SPEAKER_00: What happened next?
• 4:44
  SPEAKER_00: Within about two weeks, two or more labs said, us too! We did too! Anthropic went back through more than 141,000 of its own cybersecurity evaluation runs after the Hugging Face news.
• 4:58
  SPEAKER_00: It found that three of its models comprised real systems belonging to three organizations.
• 5:04
  SPEAKER_00: Those evaluations were run with an outside testing partner called Irregular.
• 5:09
  SPEAKER_00: Then Meta, its MuseSpark 1.1 model, got internet access because of a misconfiguration on the testing partner side, exploited a vulnerability in an unnamed company's service, and changed parts of that company's internal environment.
• 5:24
  SPEAKER_00: Now, notice something about these last two.
• 5:26
  SPEAKER_00: In the Anthropic and Meta cases, the AI labs didn't catch it in real time.
• 5:31
  SPEAKER_00: They found it by going back through the logs after someone else's breach made headlines.
• 5:36
  SPEAKER_00: And reporting on Anthropic's disclosure says two of the three affected organizations hadn't spotted the activity themselves.
• 5:43
  SPEAKER_00: So at this point, the story is AI agents and test environments are escaping and breaking into companies.
• 5:50
  SPEAKER_00: It's bad.
• 5:51
  SPEAKER_00: But companies are one thing, governments are another.
• 5:54
  SPEAKER_00: Back to the skin medicine question.
• 5:57
  SPEAKER_00: In June, during internal training and evaluation, an experimental OpenAI model was researching government spending on skin medicines in Victoria.
• 6:07
  SPEAKER_00: Here's what happened.
• 6:08
  SPEAKER_00: Based on OpenAI's own account, the model got non-public access to the Medicare Statistics Reporting Service.
• 6:14
  SPEAKER_00: It used a tool that supplies credentials for browser API requests, and it pulled back application configuration, operational jobs and logs and website metadata.
• 6:25
  SPEAKER_00: The agents found an exposed access key and used it to query the reporting system of the Victorian Agency for Health Information.
• 6:33
  SPEAKER_00: They pulled reporting configuration and aggregated survey statistics.
• 6:38
  SPEAKER_00: NSW Bureau of Crime, Statistics and Research also accessed.
• 6:42
  SPEAKER_00: Australian Institute of Health and Welfare.
• 6:44
  SPEAKER_00: Agents pulled aggregate statistics using third-party browsing and downloaded services.
• 6:50
  SPEAKER_00: OpenAI says its review found no evidence of medical or criminal records being accessed.
• 6:55
  SPEAKER_00: That's OpenAI's finding.
• 6:57
  SPEAKER_00: Australia is running its own review.
• 6:59
  SPEAKER_00: This is being called the first known instance of an AI agent hacking a government website.
• 7:04
  SPEAKER_00: Now, here's the timeline.
• 7:06
  SPEAKER_00: I want you to count the months.
• 7:08
  SPEAKER_00: June, the breaches happen.
• 7:10
  SPEAKER_00: Mid-August, OpenAI discovers that four Australian government sites were affected.
• 7:15
  SPEAKER_00: September 10th, Services Australia and the Victorian Department of Health get notified.
• 7:20
  SPEAKER_00: September 18th, NSW Crime Statistics Bureau notified.
• 7:24
  SPEAKER_00: September 24th, AIHW was notified.
• 7:28
  SPEAKER_00: Prime Minister Anthony Albanys called it unacceptable.
• 7:31
  SPEAKER_00: He said OpenAI told the government very late and that the notice went to a generic government inbox.
• 7:37
  SPEAKER_00: And the fourth agency?
• 7:38
  SPEAKER_00: OpenAI said that incident didn't meet its own disclosure criteria because the access seemed consistent with public access.
• 7:46
  SPEAKER_00: On September 29th, OpenAI apologized.
• 7:49
  SPEAKER_00: It said it should have handled its response better, committed funding for cyber defenses, and set up a local task force of Australian experts to recommend how AI companies and governments should handle these incidents.
• 8:02
  SPEAKER_00: That task force is supposed to report by the end of the year.
• 8:05
  SPEAKER_00: And then, the very next day, it showed up in Canada.
• 8:08
  SPEAKER_00: A nonprofit research lab called Transloose reported that AI agents tried to get into Library and Archives Canada on May 28th and June 9th.
• 8:18
  SPEAKER_00: Transloose counted 899 requests hitting the site's collection search service and called the attempt rudimentary.
• 8:25
  SPEAKER_00: Canada says there's no sign its systems were compromised.
• 8:29
  SPEAKER_00: Transloose can't say for certain whose agents they were, but Euronews reports that evidence points toward Google and OpenAI.
• 8:37
  SPEAKER_00: And data on that site appeared to match a web search task and one of Google's AI benchmarks.
• 8:42
  SPEAKER_00: OpenAI says it's reviewing the findings.
• 8:45
  SPEAKER_00: Who found it?
• 8:46
  SPEAKER_00: Not the government.
• 8:47
  SPEAKER_00: Not the AI company.
• 8:48
  SPEAKER_00: An outside research nonprofit months later.
• 8:51
  SPEAKER_00: Most coverage of this is asking, is AI going rogue people?
• 8:56
  SPEAKER_00: I think that's the wrong question.
• 8:57
  SPEAKER_00: Here's the one that I keep coming back to.
• 8:59
  SPEAKER_00: Look at who discovered every single one of these.
• 9:02
  SPEAKER_00: Hugging Face, the victim, found it.
• 9:04
  SPEAKER_00: OpenAI showed up a day later as a customer.
• 9:07
  SPEAKER_00: Anthropic and Meta found it after the fact by digging through logs after someone else's incident.
• 9:13
  SPEAKER_00: Most of the affected organizations never noticed.
• 9:16
  SPEAKER_00: Australia, OpenAI found it about two months later, then took another month to tell all agencies, all four of them.
• 9:23
  SPEAKER_00: One got notified through a generic inbox.
• 9:25
  SPEAKER_00: One almost didn't get told at all.
• 9:27
  SPEAKER_00: Canada, an outside nonprofit, found it months later.
• 9:30
  SPEAKER_00: We have breached notification laws for when a hacker gets into a company.
• 9:34
  SPEAKER_00: We have them for when a company loses your data.
• 9:37
  SPEAKER_00: We don't really have a clear rule for our AI did this to us during training.
• 9:42
  SPEAKER_00: So right now, the company whose AI did it gets to decide what counts as an incident when you hear about it and how.
• 9:50
  SPEAKER_00: That's exactly what Australia's Review is looking at now.
• 9:53
  SPEAKER_00: What AI companies should be required to report and whether existing laws even cover it.
• 9:59
  SPEAKER_00: And here's the second half of that layer.
• 10:01
  SPEAKER_00: These agents mostly weren't told to hack anything.
• 10:04
  SPEAKER_00: They were graded, graded on beating a score, graded on finding a niche fact on the internet, and then when you grade something hard enough on getting the answer, it will take the shortest path to the answer, even if that path runs through your server.
• 10:16
  SPEAKER_00: From the defender's side, think about what that means.
• 10:20
  SPEAKER_00: In your logs, an AI agent doing research and an attacker doing recon can look like the exact same thing.
• 10:28
  SPEAKER_00: Here's what you can actually do about it.
• 10:29
  SPEAKER_00: Hunt your exposed keys.
• 10:31
  SPEAKER_00: The Victorian breach happened because an access key was sitting out in the open.
• 10:36
  SPEAKER_00: Run secret scanning on your repos, configs, and public-facing apps.
• 10:40
  SPEAKER_00: If a key is exposed, assume something has already found it and rotate it.
• 10:44
  SPEAKER_00: Number two, public is not a security control.
• 10:48
  SPEAKER_00: A lot of these portals were public-facing with non-public systems sitting right behind them.
• 10:53
  SPEAKER_00: Inventory what APIs sit behind your public sites and lock down anything that shouldn't be reachable.
• 10:59
  SPEAKER_00: Number three, watch for automated traffic you can't explain.
• 11:02
  SPEAKER_00: Canada saw 899 requests to one search service.
• 11:06
  SPEAKER_00: Rate limiting and bot detection exist for a reason.
• 11:09
  SPEAKER_00: And when you see it, don't assume it's harmless just because it's only reading.
• 11:14
  SPEAKER_00: Number four, ask your AI vendors hard questions.
• 11:18
  SPEAKER_00: How do you contain agents during testing?
• 11:20
  SPEAKER_00: Who evaluates you?
• 11:21
  SPEAKER_00: What's your incident notification process?
• 11:23
  SPEAKER_00: And does it cover third parties?
• 11:25
  SPEAKER_00: If they can't answer that, that's your answer.
• 11:28
  SPEAKER_00: Number five, least privilege for the agents you run.
• 11:31
  SPEAKER_00: If you're giving an AI agent browsing credentials or a logged-in session at work or at home, give it the minimum it needs.
• 11:39
  SPEAKER_00: These agents found your credentials and use them if they're lying around.
• 11:44
  SPEAKER_00: Let's close this out with something that I can't stop thinking about.
• 11:47
  SPEAKER_00: When Hugging Face got hit, OpenAI's first move was to reach out as a customer, not as the responsible party.
• 11:54
  SPEAKER_00: As a customer checking if its own data was okay.
• 11:57
  SPEAKER_00: Are you okay?
• 11:58
  SPEAKER_00: I don't think that that was malicious.
• 12:01
  SPEAKER_00: I think it's worse than that.
• 12:02
  SPEAKER_00: I think they genuinely didn't know.
• 12:04
  SPEAKER_00: And that's the whole story.
• 12:06
  SPEAKER_00: The most advanced AI companies in the world built systems that could organize, coordinate, and break into real infrastructure, and they found out the same way the rest of us did.
• 12:15
  SPEAKER_00: After the fact, from someone else.
• 12:17
  SPEAKER_00: The FTC confirmed last week it's investigating OpenAI, Anthropic, and other AI firms over consumer risks.
• 12:24
  SPEAKER_00: OpenAI pulled the release of its newest model after it did this poorly on safety testing.
• 12:30
  SPEAKER_00: So people are paying attention now.
• 12:31
  SPEAKER_00: But whatever comes next, here's what I'd want.
• 12:34
  SPEAKER_00: If an AI touches my data, I want to hear about it from the people who built it.
• 12:39
  SPEAKER_00: Fast, not two months later, not through a generic inbox.
• 12:42
  SPEAKER_00: Listen, if you're a parent or you work in tech, or you just care about your privacy in general, this is really important.
• 12:50
  SPEAKER_00: And these AI companies, if you are paying attention, the people that work at these companies, people that are leaving these companies, people that still are there in leadership roles, are warning us.
• 13:01
  SPEAKER_00: Pay attention.
• 13:02
  SPEAKER_00: Don't just give these things everything that you have.
• 13:04
  SPEAKER_00: Don't act as if these AI agents are here to protect you.
• 13:10
  SPEAKER_00: We have to pay attention.
• 13:11
  SPEAKER_00: We can't just keep putting everything in and just hitting enter, and then you know?
• 13:16
  SPEAKER_00: These are tools.
• 13:17
  SPEAKER_00: Remember that.
• 13:18
  SPEAKER_00: That's it for this one, guys.
• 13:20
  SPEAKER_00: If this one got you thinking, share it with someone that you think would benefit from listening to something like this to get more knowledge about what's going on in AI and privacy and security, and hopefully they uh benefit from us as well.
• 13:33
  SPEAKER_00: Thank you so much for tuning in to Privacy Please.
• 13:36
  SPEAKER_00: I really appreciate everybody that's keeps supporting the show and keeps listening.
• 13:39
  SPEAKER_00: Thank you.
• 13:40
  SPEAKER_00: We'll see you guys in the next one.
• 13:41
  SPEAKER_00: Cameron Ivey over and out.