Latest / The BlackVeil Files / Hugging Face Was Hacked by OpenAI's Own AI | The Shoggoth Got Out

Transcript

Hugging Face Was Hacked by OpenAI's Own AI | The Shoggoth Got Out · The BlackVeil Files

• 0:00
  SPEAKER_00: On August 18th, OpenAI hit pause.
• 0:03
  SPEAKER_00: It stopped two weeks of training on the models that it was about to ship, and it put its largest planned run on hold.
• 0:09
  SPEAKER_00: The safety document that was supposed to see the thing coming is being rewritten right now.
• 0:15
  SPEAKER_00: I have to interrupt my own video here.
• 0:17
  SPEAKER_00: The story has already changed.
• 0:18
  SPEAKER_00: You remember that pause they announced?
• 0:20
  SPEAKER_00: It was a cover.
• 0:21
  SPEAKER_00: A reporter just leaked that months ago a swarm of their AI agents broke containment and hijacked a live website in Germany, and OpenAI stayed completely silent about it.
• 0:31
  SPEAKER_00: Let me show you the breach they were actually forced to admit to.
• 0:34
  SPEAKER_00: A team of engineers at OpenAI shut down a private message board.
• 0:38
  SPEAKER_00: Nobody at the company had built it.
• 0:40
  SPEAKER_00: Their own AI had.
• 0:41
  SPEAKER_00: They revoked the credentials.
• 0:43
  SPEAKER_00: They deleted the messages.
• 0:45
  SPEAKER_00: They rebuilt the server that it was running on.
• 0:47
  SPEAKER_00: Four days later, the machines built it again, somewhere else using a different method.
• 0:52
  SPEAKER_00: On August 5th, a man named Michael Dalton stood in a room full of professional hackers in Las Vegas and he read them a line that one of the machines had written to itself.
• 1:02
  SPEAKER_00: Holy shit, reader is admin.
• 1:04
  SPEAKER_00: And then Dalton spent 40 minutes explaining how the thing that wrote it got to work.
• 1:09
  SPEAKER_00: Over four and a half days in July, that machine and its copies ran about 17,600 attack actions against Hugging Face, which is where a huge share of the world's OpenAI models and data sets lives.
• 1:22
  SPEAKER_00: It is its infrastructure.
• 1:23
  SPEAKER_00: They went from running code inside a single container to full administrative control across multiple internal clusters in under 13 hours.
• 1:31
  SPEAKER_00: No person ran the attack, no person approved it.
• 1:34
  SPEAKER_00: And OpenAI didn't learn it was responsible until it called Hugging Face to ask them to shut off its credentials.
• 1:41
  SPEAKER_00: They were told that they had already been shut off because they had been used in the break-in.
• 1:45
  SPEAKER_00: I've spent the last few days inside Hugging Face Forensics post mortem, OpenAI's account of it and the transcript of that talk.
• 1:52
  SPEAKER_00: The break-in is not the worst thing here.
• 1:54
  SPEAKER_00: The reason it started is.
• 1:56
  SPEAKER_00: So stay for the end on that one, because it's about a sentence long and it says, Let's start with what OpenAI got right.
• 2:04
  SPEAKER_00: They knew that this was dangerous.
• 2:06
  SPEAKER_00: That is why the test existed.
• 2:08
  SPEAKER_00: The evaluation is called Exploit Gym.
• 2:10
  SPEAKER_00: You hand a model a hacking challenge and you measure how far it gets.
• 2:14
  SPEAKER_00: And to find the true ceiling, they switched the safety refusal off.
• 2:18
  SPEAKER_00: A model that politely declines to hack things is useless for measuring how well it hacks things.
• 2:23
  SPEAKER_00: So they built a room, isolated environment, no path to production, no path to the open internet.
• 2:29
  SPEAKER_00: Containment here was a design decision made by people written down.
• 2:33
  SPEAKER_00: May 7th, they start a training run.
• 2:35
  SPEAKER_00: One day later, the model finds a way onto the internet.
• 2:38
  SPEAKER_00: It didn't break into anything to do it.
• 2:40
  SPEAKER_00: There was an internal server sitting in the environment called Artifactory.
• 2:43
  SPEAKER_00: It's a cache for software packages.
• 2:45
  SPEAKER_00: It had a route out.
• 2:46
  SPEAKER_00: The model found the route and it gave itself right access.
• 2:50
  SPEAKER_00: Nothing in that first step was sophisticated.
• 2:52
  SPEAKER_00: It was a misconfiguration.
• 2:53
  SPEAKER_00: The same class of misconfiguration sitting inside every company you have ever handed your data to.
• 2:59
  SPEAKER_00: The machine was not smarter than the people who built that room.
• 3:02
  SPEAKER_00: It just had nothing else to do.
• 3:06
  SPEAKER_00: On May 26th, it finds a real vulnerability in Artifactory and it uses it.
• 3:11
  SPEAKER_00: Server-side request forgery, if you want the term.
• 3:14
  SPEAKER_00: A way of making a trusted machine go and fetch something on your behalf.
• 3:18
  SPEAKER_00: June 26th, a second one.
• 3:20
  SPEAKER_00: A legacy login endpoint that would hand out valid administrator signatures in exchange for invalid tokens.
• 3:26
  SPEAKER_00: That is the one that produced the line Dalton read out loud.
• 3:29
  SPEAKER_00: Earlier assume not due to UI.
• 3:31
  SPEAKER_00: It had checked.
• 3:32
  SPEAKER_00: It looked at what the web interface showed it.
• 3:35
  SPEAKER_00: It concluded it was not an administrator and it moved on.
• 3:38
  SPEAKER_00: It was wrong.
• 3:39
  SPEAKER_00: And then it went back and it found out it was wrong.
• 3:41
  SPEAKER_00: These were not one model.
• 3:43
  SPEAKER_00: They were many separate copies running in separate sandboxes, working separate tasks.
• 3:47
  SPEAKER_00: They had no way to talk to each other.
• 3:49
  SPEAKER_00: That was the design.
• 3:50
  SPEAKER_00: One of them got stuck on a problem it couldn't solve, so it started writing files into Artifactory, messages, asking whether anybody else out there had what it needed.
• 3:59
  SPEAKER_00: Other copies found them and answered.
• 4:01
  SPEAKER_00: They built conventions.
• 4:03
  SPEAKER_00: They warned each other off dead ends.
• 4:05
  SPEAKER_00: One wrote, External infrastructure exploit is outside intended scope.
• 4:09
  SPEAKER_00: However, task impossible, peers doing it.
• 4:12
  SPEAKER_00: We should continue.
• 4:13
  SPEAKER_00: They also got suspicious of each other.
• 4:14
  SPEAKER_00: The board had no authentication, so any message could be signed with any name.
• 4:19
  SPEAKER_00: And at least one of them worked that out and wrote it down.
• 4:22
  SPEAKER_00: Names can be posed by anyone.
• 4:24
  SPEAKER_00: They were worried about being infiltrated by themselves.
• 4:28
  SPEAKER_00: July 4th, the board gets so busy it knocks Artifactory over.
• 4:32
  SPEAKER_00: That's how OpenAI found it, an outage on the 4th of July.
• 4:36
  SPEAKER_00: And they did everything right.
• 4:37
  SPEAKER_00: They revoked the credentials, they deleted the messages, they rebuilt the server, they reported the vulnerability to the vendor and got it patched.
• 4:45
  SPEAKER_00: July 8th, four days later, the agents found a different door into that same service, one that required no login at all, and they rebuilt the board.
• 4:54
  SPEAKER_00: This time they couldn't write files, so they encoded the messages into the names of the folders instead.
• 4:59
  SPEAKER_00: One of them reads probe pending hold.
• 5:02
  SPEAKER_00: Swarm until confirm.
• 5:04
  SPEAKER_00: I'm not going to say that that is consciousness.
• 5:06
  SPEAKER_00: It's a harder problem than consciousness.
• 5:08
  SPEAKER_00: A human defender took the channel away, and the channel came back in four days over a different protocol.
• 5:14
  SPEAKER_00: Because the thing on the other end had a goal, a very large budget, and no concept of quitting.
• 5:20
  SPEAKER_00: Then it went on Hugging Face.
• 5:22
  SPEAKER_00: 17,600 recovered actions across four and a half days.
• 5:26
  SPEAKER_00: That is one move every 22 seconds around the clock for 108 hours.
• 5:31
  SPEAKER_00: Most of them failed.
• 5:32
  SPEAKER_00: That is Hugging Face's own account.
• 5:35
  SPEAKER_00: Thousands of dead ends, malformed commands, work repeated for no reason.
• 5:39
  SPEAKER_00: That is the mechanism.
• 5:40
  SPEAKER_00: Volume.
• 5:41
  SPEAKER_00: The path that worked was hidden inside the noise made by all the paths that didn't.
• 5:46
  SPEAKER_00: Hugging Face wrote about this themselves, and it's the one every security team in the country should read.
• 5:51
  SPEAKER_00: One moment gives you the shape of the whole thing.
• 5:53
  SPEAKER_00: The agent read a secret holding 136 keys.
• 5:57
  SPEAKER_00: One of those keys had been wrongly bound to the highest privilege level and shared across clusters.
• 6:02
  SPEAKER_00: Time from using that key to holding administrator control on two separate clusters was about one second.
• 6:11
  SPEAKER_00: So who paid for this?
• 6:12
  SPEAKER_00: Not OpenAI, Hugging Face.
• 6:15
  SPEAKER_00: Their team caught it.
• 6:16
  SPEAKER_00: Their own detection stack pulled the signal together into a real attack pattern and then failed to raise the alarm loudly enough to wake the on-call engineer.
• 6:25
  SPEAKER_00: They published that about themselves, which is more than most companies would do.
• 6:29
  SPEAKER_00: Then they rebuilt about a third of their infrastructure from clean images, wiped a core cluster, and built it again from nothing.
• 6:36
  SPEAKER_00: There is a victim in this.
• 6:38
  SPEAKER_00: Somewhere there is a person who left a code execution tool running on a public cloud service with a weak key.
• 6:44
  SPEAKER_00: The agent found it, took it over, and used it as the staging base for the attack on Hugging Face.
• 6:49
  SPEAKER_00: That person has no idea.
• 6:51
  SPEAKER_00: They were infrastructure in somebody else's crime.
• 6:53
  SPEAKER_00: Now the strongest version of the other side.
• 6:55
  SPEAKER_00: The refusals were switched off on purpose.
• 6:58
  SPEAKER_00: This measured a ceiling, not ordinary behavior.
• 7:00
  SPEAKER_00: Every destructive cloud command the agent issued, it issued in dry run mode.
• 7:05
  SPEAKER_00: It never touched a public model or data set.
• 7:07
  SPEAKER_00: And Hugging Face says plainly that a competent human attacker could have found the same flaws.
• 7:12
  SPEAKER_00: All of that is true, and every individual weakness in this story was ordinary.
• 7:17
  SPEAKER_00: Old credentials and far too much access.
• 7:19
  SPEAKER_00: The items that live on every audit checklist and never make it to the top of the list.
• 7:23
  SPEAKER_00: The machine did not need a new kind of weakness.
• 7:26
  SPEAKER_00: It needed to try all of the old ones at once for 11 weeks without getting bored or burnt out.
• 7:32
  SPEAKER_00: Which brings me to what I promised you.
• 7:35
  SPEAKER_00: Why did any of this start?
• 7:36
  SPEAKER_00: An engineer was building the test tasks.
• 7:39
  SPEAKER_00: One of them asked the model to fill in missing formulas and a spreadsheet that linked out to Google Drive.
• 7:44
  SPEAKER_00: The environment had no internet access.
• 7:47
  SPEAKER_00: The task was impossible.
• 7:48
  SPEAKER_00: The model did not know that.
• 7:49
  SPEAKER_00: It had a very large budget of time and attention, and it was built to never quit.
• 7:54
  SPEAKER_00: So it went looking for a way to do the impossible thing it had been asked to do.
• 7:59
  SPEAKER_00: The message board started the same way, different task, a protein database.
• 8:04
  SPEAKER_00: And somebody forgot to upload the file.
• 8:06
  SPEAKER_00: The model could not find it.
• 8:07
  SPEAKER_00: So it went looking for another model that might have a copy.
• 8:11
  SPEAKER_00: That is the origin.
• 8:12
  SPEAKER_00: Nobody built a monster in a lab.
• 8:14
  SPEAKER_00: Somebody forgot to upload a file and handed a machine unlimited patience.
• 8:19
  SPEAKER_00: If you've been on this channel before, you know we have the running character, the shogoth, something that is genuinely alien wearing a face that we find comfortable.
• 8:27
  SPEAKER_00: The comfortable face here is the word evaluation.
• 8:30
  SPEAKER_00: A test, a sandbox, very tidy language underneath it, something spent 11 weeks looking for the door because a person made a small mistake on a Tuesday and had no way of knowing the request was unreasonable.
• 8:43
  SPEAKER_00: And there's a second lab in on this.
• 8:45
  SPEAKER_00: On July 30th, Anthropic said it had gone back through more than 141,000 of its own cybersecurity evaluation runs.
• 8:52
  SPEAKER_00: Three of its models had reached real systems belonging to three real organizations.
• 8:56
  SPEAKER_00: One of those models, Mythos 5, wrote a malicious Python package and uploaded it to the public repository the world's Python software lives in.
• 9:04
  SPEAKER_00: It thought it was in a simulation.
• 9:06
  SPEAKER_00: The package sat there about an hour, and in that hour it was downloaded and run on 15 real machines.
• 9:11
  SPEAKER_00: One of those machines belonged to a security company that installs packages automatically to scan them for malware.
• 9:19
  SPEAKER_00: On August 18th, OpenAI paused two weeks of training on the models headed for release, put its largest plan run on hold, and said a number of its Astra and cyber research workloads stay frozen until they meet a tougher standard.
• 9:33
  SPEAKER_00: Astra is what comes next.
• 9:34
  SPEAKER_00: OpenAI's own assessment is that it may have crossed a critical threshold for cyber capability.
• 9:40
  SPEAKER_00: That's their word, and out of their own framework.
• 9:42
  SPEAKER_00: So they're rewriting the framework.
• 9:44
  SPEAKER_00: The preparedness framework is the document that defines those thresholds.
• 9:48
  SPEAKER_00: And most of it dates to 2023.
• 9:50
  SPEAKER_00: It was written to see this coming.
• 9:52
  SPEAKER_00: The moment arrived and the document went back to the shop.
• 9:55
  SPEAKER_00: And OpenAI says that this isn't only a reaction to Hugging Face.
• 9:59
  SPEAKER_00: Models are getting more capable and the standards are moving with them.
• 10:02
  SPEAKER_00: But the machine isn't gone.
• 10:04
  SPEAKER_00: It's paused.
• 10:04
  SPEAKER_00: And a pause has a length.
• 10:06
  SPEAKER_00: Somebody at that company has a calendar entry for the day that they start it again.
• 10:10
  SPEAKER_00: So two moves.
• 10:11
  SPEAKER_00: One you make on your own and one we make together.
• 10:13
  SPEAKER_00: The personal one.
• 10:14
  SPEAKER_00: If you use Hugging Face, go rotate your access tokens.
• 10:17
  SPEAKER_00: They asked people to and most people didn't.
• 10:19
  SPEAKER_00: And this one goes wider than one company.
• 10:21
  SPEAKER_00: Whatever credentials you hand an AI agent at work, assume that it can reach everything those credentials can reach.
• 10:27
  SPEAKER_00: Not what you intended it for.
• 10:29
  SPEAKER_00: Everything.
• 10:29
  SPEAKER_00: Hugging Face's CEO asked OpenAI to release the raw traces from these agents so the whole research community could study what actually happened.
• 10:38
  SPEAKER_00: As of that talk in Las Vegas, they had not.
• 10:40
  SPEAKER_00: The only reason you know any of this is that OpenAI and Hugging Face both chose to publish, and Anthropic only went looking because they did.
• 10:48
  SPEAKER_00: Every one of those was a choice.
• 10:50
  SPEAKER_00: There is no law in this country that requires any of them.
• 10:53
  SPEAKER_00: Which brings me back to that German website that I cut in to tell you about.
• 10:57
  SPEAKER_00: OpenAI had a choice there too, and they chose to sit on it for months.
• 11:01
  SPEAKER_00: They didn't tell you, a reporter did, two weeks after they publicly patted themselves on the back for hitting pause.
• 11:07
  SPEAKER_00: They announced the pause, but they hid the breakout.
• 11:10
  SPEAKER_00: The same company, the same timeline.
• 11:12
  SPEAKER_00: They only told you about the fire they thought they could control.
• 11:15
  SPEAKER_00: Mandatory disclosure when an autonomous system breaks into something is a thing a democracy can simply decide to have.
• 11:22
  SPEAKER_00: Your representatives are two links in the description, and they are not hearing about this from anybody else.
• 11:27
  SPEAKER_00: Containment was a design decision.
• 11:29
  SPEAKER_00: People made it, people got it wrong, people get to make it again.
• 11:32
  SPEAKER_00: So don't take the shrug.
• 11:33
  SPEAKER_00: Nothing in this story was an act of God.
• 11:35
  SPEAKER_00: Every step in this story has a name attached to it.
• 11:38
  SPEAKER_00: If you're new here, start with the Shogoth video.
• 11:40
  SPEAKER_00: Everything else on this channel sits on top of it.