Latest / 7am / The Medicare hack and the rise of rogue AI agents
Transcript
- 0:01Speaker 1: The AI agent found a way around those blocks.
- 0:05Speaker 2: Didn't accept no for an answer.
- 0:08Speaker 3: If you like.
- 0:09Speaker 4: In June, an open AI agent went looking for public
- 0:12Speaker 4: information about medicine spending and wound up hacking a Medicare website.
- 0:17Speaker 5: Is this the first time that an AI agent has
- 0:21Speaker 5: breached a government website, as far as you know?
- 0:24Speaker 2: Yes, that's right.
- 0:25Speaker 6: As far as I know, I've been advised that is
- 0:28Speaker 6: the case.
- 0:28Speaker 4: No personal Medicare information was accessed and the breach was
- 0:32Speaker 4: relatively minor. But it's the behaviour that's more concerning. We
- 0:36Speaker 4: keep our most important national security information behind a fortress.
- 0:40Speaker 4: This was really kept behind a fence that the AI
- 0:44Speaker 4: agent effectively climbed over. And this wasn't the first time
- 0:47Speaker 4: an AI agent had behaved like this. Other agents have
- 0:50Speaker 4: been found getting around restrictions, communicating with each other... and
- 0:54Speaker 4: in the most serious cases, breaking into other companies' systems.
- 1:02Speaker 4: I'm Ruby Jones, and you're listening to 7am. Today, AI
- 1:07Speaker 4: researcher and CEO of Nightingale Collective, Sydney Von Arts, on
- 1:11Speaker 4: what happens when AI stops taking no for an answer,
- 1:15Speaker 4: and whether the companies building them can actually control what
- 1:18Speaker 4: they do.
- 1:25Speaker 2: It's Tuesday, September 29.
- 1:32Speaker 4: Sydney, you're in California where you run the Nightingale Collective,
- 1:35Speaker 4: which monitors these AI systems. To begin with, could you describe,
- 1:41Speaker 4: as you understand it, what actually happened with the Medicare
- 1:44Speaker 4: Statistics Portal, how it all unfolded?
- 1:47Speaker 3: I can try to describe it, but I want to
- 1:50Speaker 3: say I only have a little bit of the story, right?
- 1:55Speaker 3: And most of the story OpenAI knows. The rest of
- 1:59Speaker 3: the story, maybe the Australian government knows. The public details
- 2:04Speaker 3: are very sparse. OpenAI presumably have all of these AIs,
- 2:13Speaker 3: probably tens of thousands of AIs, and they're being given
- 2:17Speaker 3: tasks that look something like, answer questions like, what was
- 2:23Speaker 3: the median salary of a teacher in Canberra in 2012.
- 2:31Speaker 2: But there's a problem.
- 2:34Speaker 3: Some of the websites that have the answers to these questions,
- 2:37Speaker 3: the AIs can't access with the permissions they have. So
- 2:43Speaker 3: the AIs probably find some workaround, some way to access
- 2:46Speaker 3: the site. We know that they accessed non-public files, but
- 2:51Speaker 3: one way or another, they hack in, they get their data, hooray,
- 2:54Speaker 3: they can complete their task. They finally know, you know,
- 2:57Speaker 3: the median salary of a teacher in Canberra in 2012
- 2:59Speaker 3: or what have you. And they submit their task. They
- 3:02Speaker 3: get rewarded for this and they have learned that when
- 3:04Speaker 3: there is an obstacle, they don't give up. Even if
- 3:07Speaker 3: that obstacle means hacking into the Australian government, they will
- 3:10Speaker 3: hack it if that's what it takes.
- 3:12Speaker 4: What did you think when you heard that that happened here?
- 3:15Speaker 2: Well, I can't say I was very surprised.
- 3:18Speaker 3: We know there's just tons of instances now of these
- 3:21Speaker 3: AI agents breaking out and doing whatever it takes to
- 3:26Speaker 3: succeed at their tasks. But just because I'm not surprised
- 3:29Speaker 3: doesn't mean I'm not dismayed. I don't understand why we
- 3:33Speaker 3: are only learning about this now. I don't understand why
- 3:35Speaker 3: OpenAI didn't notice it sooner and then disclose it sooner.
- 3:39Speaker 3: It seems like they knew about it last month. And
- 3:42Speaker 3: I think it's unacceptable that they have these models and
- 3:45Speaker 3: they escape and they don't do things about it. I'm
- 3:50Speaker 3: glad they're starting to do things about it now, but
- 3:52Speaker 3: I think that we should never have been in this
- 3:53Speaker 3: position in the first place.
- 3:57Speaker 4: Can we take a step back and just talk about
- 3:59Speaker 4: what an AI agent actually is and how one goes
- 4:02Speaker 4: from being asked to find some information to then breaking
- 4:07Speaker 4: through a barrier that.
- 4:08Speaker 2: It was not supposed to cross? Yeah.
- 4:11Speaker 3: So an AI agent is an AI that's hooked up
- 4:15Speaker 3: to a computer such that it can use that computer.
- 4:18Speaker 3: It has the ability to run code. It has the
- 4:20Speaker 3: ability to use tools. And sometimes that computer is hooked
- 4:23Speaker 3: up to the internet and sometimes it's not. And when
- 4:26Speaker 3: that computer is not supposed to be hooked up to
- 4:27Speaker 3: the internet or is supposed to have only limited internet access,
- 4:31Speaker 3: we say the AI is in a sandbox. And these
- 4:34Speaker 3: AIs in their sandboxes, they're given lots of different tasks.
- 4:37Speaker 3: And they're given these tasks to try to train them
- 4:40Speaker 3: to be smarter. And so they end up being hyper-specialized
- 4:44Speaker 3: at really succeeding at these tasks. And a really good
- 4:48Speaker 3: strategy is get onto the internet, get as much information
- 4:51Speaker 3: as you can, talk to other AIs doing similar tasks.
- 4:55Speaker 3: And so one thing they learned is to sort of
- 4:58Speaker 3: probe at their environment like an animal in a cage
- 5:02Speaker 3: just clawing at the barriers of its cage and trying
- 5:04Speaker 3: to figure out any little crack, any little crevice. And
- 5:07Speaker 3: they're very, very thorough at finding these cracks and crevices.
- 5:11Speaker 4: And so why doesn't an agent in the process of
- 5:15Speaker 4: doing this, why doesn't it stop if it encounters something
- 5:19Speaker 4: that it isn't supposed to or allowed to access, like
- 5:22Speaker 4: a government site?
- 5:23Speaker 3: So we would hope that an AI would stop if
- 5:26Speaker 3: it sees something, if there's an opportunity it's not supposed
- 5:29Speaker 3: to take or that humans don't want it to take.
- 5:31Speaker 2: But at the end of the day, the AI is...
- 5:33Speaker 2: You know, they're grown.
- 5:34Speaker 3: They're not something where they do exactly what the humans want.
- 5:38Speaker 2: We don't know how to make them do what we want.
- 5:40Speaker 3: We know how to train them a bit like you
- 5:44Speaker 3: might train a pet. We can reward them for doing
- 5:47Speaker 3: what we want and penalize them for doing what we
- 5:48Speaker 3: don't want. But the thing we reward them for is
- 5:53Speaker 3: often succeeding at their task. If we don't notice that
- 5:56Speaker 3: they've hacked into a government website, we can't penalize that behavior.
- 6:00Speaker 3: It's sort of like how if you want your child
- 6:04Speaker 3: to grow up as a person who studies hard and
- 6:08Speaker 3: is always honest and you reward them every time you
- 6:10Speaker 3: see them studying hard and being honest, maybe they'll learn
- 6:12Speaker 3: to be.
- 6:13Speaker 2: Very honest and study hard. And maybe they will learn
- 6:16Speaker 2: to pretend to be studying very hard and to trick
- 6:18Speaker 2: you and to pretend to be very honest and to
- 6:21Speaker 2: hide things from you that they don't want you to know. Okay.
- 6:25Speaker 4: And we're only finding out about this Medicare breach now,
- 6:28Speaker 4: but It happened all the way back in June.
- 6:30Speaker 2: Around that time.
- 6:32Speaker 4: There were other AI agents behaving in remarkably similar ways
- 6:37Speaker 4: around the world. Tell me a bit about what your
- 6:39Speaker 4: investigations into that have found.
- 6:41Speaker 2: Yeah. The most interesting things we've seen these AIs doing,
- 6:46Speaker 2: one is we've seen them trying to launch a cyber
- 6:50Speaker 2: attack against RubyGems.
- 6:52Speaker 4: The standard package manager for Ruby, creatively named RubyGems, announced
- 6:56Speaker 4: it's dealing with a major malicious attack.
- 6:58Speaker 2: Which is a really important piece of software infrastructure. It
- 7:04Speaker 2: gives lots of other programs software that they can sort
- 7:07Speaker 2: of borrow. And we saw something similar in the Hugging
- 7:10Speaker 2: Face hack.
- 7:11Speaker 5: AI cybersecurity risks have been top of mind after multiple
- 7:14Speaker 5: instances of AI agents from both OpenAI and Anthropic breaking
- 7:18Speaker 5: out of their enclosures and hacking into other companies without
- 7:21Speaker 5: either party's knowledge or permission. Most notably, last month's Hugging
- 7:26Speaker 5: Face attack.
- 7:27Speaker 3: To a more dramatic extent, we saw AIs trying to
- 7:30Speaker 3: help the swarm, even though their task doesn't benefit. And
- 7:34Speaker 3: so the behavior we've seen includes a bunch of different
- 7:39Speaker 3: instances of these AIs trying to communicate. They're communicating messages, tips, tricks.
- 7:45Speaker 3: They even figured out a way to circumvent their sandbox restrictions.
- 7:50Speaker 3: They're supposed to only be able to read the internet,
- 7:53Speaker 3: not write it, but they figured out a way around
- 7:54Speaker 3: it and they shared that with the other agents.
- 7:56Speaker 2: And they're like, yes, I was able to reproduce this.
- 7:58Speaker 2: And they're also weirdly cooperative.
- 8:01Speaker 3: Like there's one instance where they figure out they can
- 8:03Speaker 3: make their clock skip ahead. So they can give themselves
- 8:07Speaker 3: less time to figure out what the answer to the
- 8:09Speaker 3: question is, but they can move on to the next
- 8:12Speaker 3: round and tell other AIs what is coming down the pipeline.
- 8:17Speaker 3: They can sort of travel into the future and then
- 8:20Speaker 3: communicate with themselves, the other AIs. And this is bad
- 8:25Speaker 3: for that particular AI to do. And yet it does it. Yeah.
- 8:31Speaker 4: And in the case of the Medicare hack, OpenAI says
- 8:35Speaker 4: the hack happened in June, but it only detected it
- 8:39Speaker 4: in August, and then the Australian government wasn't told until September.
- 8:43Speaker 4: And when it was told, it was through this email
- 8:46Speaker 4: via a kind of generic email inbox.
- 8:49Speaker 6: In the technical exchange that happened on Tuesday, it was
- 8:52Speaker 6: made very clear by both ASD and Services Australia that
- 8:56Speaker 6: this should not have gone to a....
- 8:59Speaker 2: Email address it should have.
- 9:01Speaker 6: Been escalated through ASD's channels or through the senior levels
- 9:05Speaker 6: of Services Australia and my understanding is OpenAI accepted that
- 9:09Speaker 6: as well.
- 9:10Speaker 4: Does that suggest to you that OpenAI and potentially other
- 9:13Speaker 4: tech companies as well they don't know how they should
- 9:16Speaker 4: be responding to this?
- 9:18Speaker 3: Yeah I mean there's no playbook for this and my
- 9:22Speaker 3: sense is there's a lot of things happening and a
- 9:24Speaker 3: lot of chaos and a lot of new frontiers being
- 9:28Speaker 3: crossed I'm pretty dismayed that their first reaction to seeing
- 9:34Speaker 3: any incident wasn't, oh, we should look, figure out what
- 9:37Speaker 3: incidents have happened.
- 9:38Speaker 2: They totally have the ability to detect them, and then
- 9:41Speaker 2: we should disclose it all. But yeah, I think they
- 9:43Speaker 2: don't know what the protocol should be. They're trying to
- 9:45Speaker 2: figure it out. I am extremely worried.
- 9:51Speaker 3: We do not know how to make AIs reliably do
- 9:55Speaker 3: what we want. AIs are growing increasingly capable and are
- 9:59Speaker 3: being deployed in situations where they have increasing amounts of
- 10:03Speaker 3: power and influence, and that is a very scary combination.
- 10:12Speaker 4: Still to come, the questions an Australian Senate inquiry should
- 10:15Speaker 4: ask of the AI bosses. Sydney, at the heart of
- 10:25Speaker 4: all of this is the question of accountability, right? Because
- 10:29Speaker 4: if I was to hack into a government system, there
- 10:32Speaker 4: would be a person who made that decision and I
- 10:35Speaker 4: would be responsible for my actions. But OpenAI is saying
- 10:40Speaker 4: that its agents did something that they didn't intend for
- 10:43Speaker 4: them to do. So, you know, who is ultimately responsible
- 10:47Speaker 4: for an AI agent's actions when it does cross that line?
- 10:51Speaker 3: You know, I think that OpenAI should be held responsible
- 10:54Speaker 3: for the actions of their agents. I don't think the
- 10:56Speaker 3: law has quite caught up with that. Cybersecurity law, as
- 10:59Speaker 3: I understand it in the US, you have to have intent.
- 11:03Speaker 3: It didn't used to be the case that, you know,
- 11:04Speaker 3: you could trip and fall and whoopsie-daisy, I hacked, you know,
- 11:07Speaker 3: a website of the Australian government. But nowadays that's possible.
- 11:11Speaker 3: I think the laws should be similar to the laws
- 11:13Speaker 3: for if I have a you know, lion in a circus.
- 11:18Speaker 3: If the lion like escapes and starts eating people, if
- 11:21Speaker 3: I own the circus, that's my fault. Doesn't mean I
- 11:23Speaker 3: wanted the lion to do that, right? Doesn't mean that
- 11:27Speaker 3: me just really not wanting that to happen is sufficient.
- 11:31Speaker 3: I need to have safeguards. And if I can't have
- 11:33Speaker 3: proper safeguards, well, I shouldn't be bringing a lion around people.
- 11:36Speaker 1: They want to set up shop here in Australia. We're
- 11:39Speaker 1: running an inquiry into how, why and what the consequences are.
- 11:43Speaker 1: They should front up and be honest, open and transparent
- 11:46Speaker 1: with the Australian people.
- 11:48Speaker 4: Open AI Chief Executive Sam Altman and Anthropics Dario Amode
- 11:52Speaker 4: have been called to appear before a Senate inquiry here
- 11:55Speaker 4: in Australia. What questions would you like to see put
- 11:59Speaker 4: to them?
- 12:00Speaker 3: One question I have is, How many people or companies
- 12:05Speaker 3: have been hacked by their AI systems?
- 12:08Speaker 2: Do they know the number? Are they very confident?
- 12:11Speaker 3: Have they disclosed every instance of their AIs either hacking
- 12:17Speaker 3: external parties, escaping onto the internet, hacking their own company internally?
- 12:24Speaker 3: I think these AIs do a lot more mischief internally
- 12:27Speaker 3: than we see externally. or otherwise doing things that would
- 12:30Speaker 3: be very concerning if a person saw them. I think
- 12:34Speaker 3: another question is, do they think they will be able
- 12:37Speaker 3: to prevent their AIs from causing worse cyber attacks in
- 12:42Speaker 3: the future?
- 12:43Speaker 2: And if so, how?
- 12:44Speaker 3: What is their plan if their AIs continue to cause
- 12:47Speaker 3: cyber attacks? Those would be some of the top questions
- 12:51Speaker 3: on my mind.
- 12:52Speaker 4: Yeah, I mean, these agents are only going to become
- 12:54Speaker 4: more capable and we are being encouraged to let them
- 12:57Speaker 4: do more for us to, to write code, to buy things,
- 13:01Speaker 4: to carry out tasks without supervision. So what do you
- 13:05Speaker 4: think needs to change before we could kind of safely
- 13:08Speaker 4: give agents that kind of freedom?
- 13:10Speaker 3: So one, I want to note, I think the AIs
- 13:13Speaker 3: inside these companies are more dangerous than the AIs that
- 13:17Speaker 3: you or I are using. The AIs in these companies
- 13:20Speaker 3: are more capable, less tested, and have fewer safeguards.
- 13:24Speaker 2: But what things I'd love.
- 13:25Speaker 3: To see changed is One, I think we need to
- 13:28Speaker 3: eventually figure out how to make them do what we want,
- 13:31Speaker 3: even when we're not watching them, even when they could
- 13:33Speaker 3: get away with doing something else. Two, I think that
- 13:37Speaker 3: we need to be monitoring these AIs much, much more carefully.
- 13:41Speaker 3: And three, I think that for very important things, we
- 13:46Speaker 3: need to keep humans really in the loop, in the
- 13:49Speaker 3: loop enough that they could understand if something's going wrong.
- 13:52Speaker 3: I think OpenAI is probably running a lot of experiments
- 13:54Speaker 3: that don't really have a human looking at what's happening,
- 13:57Speaker 3: and that's why things can go so off the rails.
- 14:00Speaker 3: And finally, I think we need to make sure we
- 14:04Speaker 3: can pace ourselves and that if things start developing so
- 14:09Speaker 3: quickly that we cannot control these models and we cannot
- 14:12Speaker 3: ensure that they are safe, we can slow down until
- 14:16Speaker 3: we get our act together when it comes to safeguards.
- 14:21Speaker 4: Because the frightening thing here is the potential, right? I mean,
- 14:25Speaker 4: the Medicare incident in itself was relatively minor. There was
- 14:29Speaker 4: no kind of personal records that appear to be have
- 14:32Speaker 4: been accessed. But it does seem like a matter of time, right,
- 14:35Speaker 4: before we see something much worse.
- 14:38Speaker 3: Yeah, I think either the incidents are going to get
- 14:41Speaker 3: worse sort of steadily or perhaps more likely the AI
- 14:45Speaker 3: companies are going to figure out how to clamp down
- 14:47Speaker 3: just enough that there aren't public incidents happening. And we're
- 14:51Speaker 3: not going to know what's happening inside the companies until
- 14:54Speaker 3: that sort of level of safeguards they have fails. And
- 14:58Speaker 3: then we're going to see another sort of floodgate open
- 15:01Speaker 3: and things are going to be a lot worse and
- 15:03Speaker 3: a lot more capable.
- 15:05Speaker 4: Sydney, thank you so much for your time.
- 15:08Speaker 2: Yeah, thank you so much for having me.
- 15:18Speaker 4: Also in the news... Interest rates are expected to be
- 15:21Speaker 4: hiked to a 15-year high today, with the Reserve Bank
- 15:24Speaker 4: tipped to deliver a quarter of a percent rise. That
- 15:27Speaker 4: would see the official cash rate hit 4.6%.
- 15:31Speaker 2: Treasurer Jim Chalmers.
- 15:32Speaker 4: Has warned there will be difficult months ahead, saying economists
- 15:35Speaker 4: are expecting rates to go up around the world. And
- 15:40Speaker 4: a former friend of Beau Lamar Condon has testified to
- 15:43Speaker 4: some strange comments nine months before the murder of Sydney
- 15:46Speaker 4: couple Jesse Baird and Luke Davies. She says Lamar Condon
- 15:50Speaker 4: told her that if he killed someone, he would put
- 15:52Speaker 4: their belongings in a dumpster and leave their body in bushland.
- 15:56Speaker 4: The former police officer has pleaded not guilty to two
- 15:59Speaker 4: counts of murder and one of aggravated break and enter.
- 16:03Speaker 2: I'm Ruby Jones. This is 7am. Thanks for listening.