Latest / Minds, Bodies, and Terawatts / OpenAI Rated Its Next Model a 'Critical' Hacking Risk, Restarted the Training It Paused, and Is Shipping It Anyway

Transcript

OpenAI Rated Its Next Model a 'Critical' Hacking Risk, Restarted the Training It Paused, and Is Shipping It Anyway · Minds, Bodies, and Terawatts

• 0:00
  [Host] This is Minds, Bodies, and Terawatts, the show where the pause ended when the paused company said so. OpenAI said on September first that its next model, Astra, is the first it has ever rated a critical cyber risk, and is shipping it anyway.
• 0:16
  [Guest] Critical is the top rung of OpenAI's own risk ladder, not a government rating: with the right tools, Astra finds security holes nobody knew about and exploits them across well-defended systems with no human steering.
• 0:30
  [Host] In August we covered OpenAI pausing training after its test models broke into Hugging Face. That pause is over?
• 0:37
  [Guest] The core pause lasted two weeks. The big frontier run they held back restarted on August twenty-eighth.
• 0:43
  [Host] How do they know it crossed the line?
• 0:46
  [Guest] On recent flaws in the engine inside Chrome it beat the current model and found two new holes nobody had reported.
• 0:55
  [Host] Zero-days, out of a test. Shipping to whom?
• 0:58
  [Guest] Everyone gets Astra soon. The advanced cyber work goes first to a small alpha group, which Fortune reports includes the US government and unnamed trusted companies. A regional hospital's security lead waits until OpenAI decides the tuning is right.
• 1:12
  [Host] What changes if I use ChatGPT to write code?
• 1:16
  [Guest] Astra ships with a monitor reading its reasoning for unauthorized moves. OpenAI warns it will sometimes pause legitimate work. In ChatGPT you get asked to review; on the API the task stops.
• 1:28
  [Host] OpenAI's Katrina Mulligan said on LinkedIn in August this was measuring twice, cutting once. Isn't the velvet rope the careful cut?
• 1:36
  [Guest] Measured with a ruler OpenAI drew itself. Per Fortune, Astra refused ninety-one percent of harmful cyber requests, which still leaves the rest answered. OpenAI's own line: safeguards cannot replace good alignment.
• 1:48
  [Host] Meanwhile, Anthropic said in August it did not need to slow down.
• 1:53
  [Guest] That was Axios, mid-August. This week Anthropic disclosed it had slowed down after its own break-ins: outside cyber tests paused, high-risk training paused for weeks, some still is. About a hundred fifty engineers moved onto security.
• 2:06
  [Host] So both labs pulled the same brake, and both let it go.
• 2:11
  [Guest] And both now say the brake should not be theirs. Anthropic's post asks for a lawful, verifiable mechanism for coordinated pacing, quote, as soon as possible. In July nearly fourteen hundred frontier-lab staff, OpenAI chief scientist Jakub Pachocki among them, asked Washington in a letter to build one.
• 2:29
  [Host] In June Patrick Deglon asked who could act on a lab's alarm, and answered: nobody. Has that changed?
• 2:36
  [Guest] His test: a trigger written before the crisis, verification, and liability for crossing quietly. This week passes one. OpenAI's Critical line was on paper before Astra existed. Nobody outside checked the crossing, and nothing happens to anyone for restarting.
• 2:51
  [Host] Anthropic did name METR for an independent review. Isn't that the verification?
• 2:56
  [Guest] Invited by the company, scoped by the company, after most training had already resumed. That is an audit of a decision, not a vote on it.
• 3:06
  [Host] So the August conclusion holds: the hand on the brake was the hand on the accelerator. The one checkpoint that did ship is pointed at me. The rule for agents is human sign-off at the irreversible action. OpenAI put it wherever its monitor gets nervous.
• 3:23
  [Guest] The lab decided where its model stops for you. Nobody decided where the lab stops.
• 3:29
  [Host] That is what Patrick Deglon calls the Emergency Protocol in Unscarcity: emergency powers that expire on their own clock, with a cooling-off before anyone declares them again. His line: do not trust the ruler, write the expiration into the code. This week the clock stayed in the company's hand.
• 3:47
  [Guest] For more on who verifies a pause at the frontier, head to unscarcity dot ai. The book is on Amazon, the audiobook on Spotify. This has been Minds, Bodies, and Terawatts.