Latest / AI Safety Daily / OpenAI Shelves GPT-6.1 Astra as California Weighs a Kill Switch — September 29, 2026

Transcript

OpenAI Shelves GPT-6.1 Astra as California Weighs a Kill Switch — September 29, 2026 · AI Safety Daily

• 0:00
  Rachel: OpenAI just shelved GPT-6.1 Astra.
• 0:02
  Rachel: Now I want to know what it actually failed.
• 0:06
  Daniel: Welcome to AI Safety Daily.
• 0:08
  Daniel: A rollout finally stopped, on the same day California starts asking who should hold a kill switch.
• 0:15
  Rachel: And the labs want a hand in writing the rules.
• 0:19
  Rachel: Let's start with Astra.
• 0:21
  Daniel: Follow the show and the next briefing lands in your feed on its own.
• 0:27
  Rachel: Here's Internationly:
• 0:29
  Quote: OpenAI decided not to release an upcoming artificial intelligence model
• 0:34
  Quote: GPT-6.1 Astra after the company determined that it did not adequately meet its safety standards
• 0:40
  Quote: CNBC confirmed on Monday.
• 0:43
  Quote: OpenAI's decision to abandon the release comes as the AI industry faces intensifying concerns surrounding the safety of advanced models.
• 0:51
  Daniel: Well, somebody actually pulled a release.
• 0:54
  Daniel: Per CNBC, OpenAI shelved GPT-6.1 Astra after deciding it didn't adequately meet the company's own safety standards.
• 1:05
  Rachel: Could be exactly the right call.
• 1:07
  Rachel: But "didn't adequately meet"...
• 1:10
  Rachel: which standard, failed how?
• 1:12
  Rachel: A dangerous-capability eval?
• 1:14
  Rachel: An alignment test? Agent behavior?
• 1:17
  Rachel: A halt nobody outside can inspect is really hard to learn from, even when it's correct.
• 1:22
  Daniel: And who made the call.
• 1:24
  Daniel: Saachi Jain, their head of safety systems, says the bar for shipping to users is extremely high.
• 1:31
  Daniel: Fine. Whose bar? And what would have to change for Astra to come back next month?
• 1:36
  Rachel: Listen to how her statement splits it, though.
• 1:39
  Rachel: Safe inside the company, versus shipping to users.
• 1:43
  Rachel: So Astra presumably still exists and still runs internally.
• 1:47
  Rachel: That strict gate she's describing only kicks in at release.
• 1:51
  Daniel: Right. And with Dario Amodei telling labs earlier this month to slow down, this is the first big lab visibly hitting the brakes.
• 2:00
  Daniel: Good. Also entirely discretionary until somebody outside OpenAI can check the work.
• 2:06
  Rachel: Maya Chen, writing in Pivot News:
• 2:10
  Quote: California has not enacted any requirement that frontier models carry a kill switch.
• 2:15
  Quote: The order instead asks the group to examine such a mechanism
• 2:19
  Quote: one that could shut down a frontier model in an emergency with its effectiveness subject to ongoing independent verification
• 2:26
  Quote: as one option among several.
• 2:28
  Daniel: So right after the Astra pull, California's asking who else gets a hand on the brake.
• 2:34
  Daniel: Newsom's four appointees are Goldman
• 2:36
  Daniel: Hadfield Nelson and Reich
• 2:39
  Daniel: and they're weighing a kill switch plus a designated verification organization sitting onsite inside the labs.
• 2:47
  Rachel: And the onsite piece is what I'd watch.
• 2:50
  Rachel: Regular audits, an outside body right there in the building.
• 2:54
  Rachel: It's the first proposal this week that puts someone other than the lab in the room when tests run.
• 2:59
  Daniel: Proposal. Let's keep saying that word. Nothing's required.
• 3:04
  Daniel: The September 18 order has Government Operations and the Office of Emergency Services drafting recommendations, and this panel advises.
• 3:14
  Daniel: The actor exists. The obligation doesn't, yet.
• 3:18
  Rachel: Agreed, and there's a detail in the kill-switch language I like.
• 3:22
  Rachel: Its effectiveness would be subject to ongoing independent verification.
• 3:27
  Rachel: A shutdown you've never tested under realistic conditions is a comfort blanket, so write the drill into the rule.
• 3:34
  Daniel: Then my ask for this panel is simple.
• 3:37
  Daniel: Tell us who's allowed to trigger it, and what happens to a lab that won't host the auditors.
• 3:43
  Rachel: Garance Burke, writing in Tech Xplore:
• 3:47
  Quote: Along the way, they are shaping the conversation around how their technology should be controlled. With their rhetoric
• 3:53
  Quote: the companies appear to be seeking public favor and to set the terms for their own safety protocols in the unregulated market
• 4:01
  Quote: experts analysts and former government evaluators told The Associated Press.
• 4:06
  Daniel: So the same week OpenAI pulls a model, the AP's Garance Burke asks the obvious question: what do these CEOs gain from sounding the alarm?
• 4:16
  Daniel: Her sources point at two things.
• 4:18
  Daniel: The midterms, and a lot of money riding on stock listings.
• 4:22
  Rachel: And the sharpest voices in that piece?
• 4:25
  Rachel: Former government evaluators, saying the labs are setting the terms for their own safety protocols.
• 4:32
  Rachel: That's about the closest thing to an on-the-record answer we've had on outside evaluation.
• 4:37
  Rachel: People who used to do the grading are now publicly asking who writes the rubric.
• 4:42
  Daniel: Right. Altman and Amodei told the UN Security Council these models need independent testing before release.
• 4:50
  Daniel: Great. Tested by whom?
• 4:52
  Daniel: Against what bar? And who can block a launch if it fails?
• 4:56
  Daniel: The Astra halt we just covered was real, but it was OpenAI grading OpenAI.
• 5:02
  Rachel: And per AI Breaking Wire, OpenAI's own third-party assessment framework still names no confirmed evaluator partner and sets no access terms.
• 5:12
  Rachel: So "independent" is still just a word, with no defined test behind it.
• 5:18
  Rachel: Meanwhile an Anthropic engineer, Jacob Coxon, quit on X this month calling for a pause.
• 5:25
  Rachel: Somewhat less aligned with the IPO calendar.
• 5:28
  Rachel: Aju Press, with Shin Hye An:
• 5:31
  Quote: Under the agreement
• 5:32
  Quote: Kakao will provide its AI models and agents for safety evaluation and support the infrastructure needed for evaluation and tool development.
• 5:41
  Quote: The AI Safety Research Institute will be responsible for conducting evaluations and enhancing methodologies.
• 5:47
  Quote: Both parties will jointly review evaluation results and negotiate the scope of public disclosure while collaborating on the development of
• 5:55
  Quote: AI safety evaluation tools.
• 5:57
  Daniel: Kakao's signed an MOU with Korea's AI Safety Research Institute.
• 6:02
  Daniel: Kakao hands over its models and agents, even funds the evaluation infrastructure.
• 6:08
  Daniel: Then both sides jointly review results and negotiate the scope of public disclosure.
• 6:14
  Rachel: And there's already a track record.
• 6:17
  Rachel: The Institute evaluated Kanana-1.5-9.8B last year, and Kakao says it showed a high level of safety versus similar overseas models.
• 6:27
  Rachel: Which models? Which tests?
• 6:30
  Rachel: What threshold? A comparison like that's only as good as its baseline.
• 6:35
  Daniel: Right. And 'negotiate' isn't a disclosure rule.
• 6:39
  Daniel: If the company being graded gets a vote on what's published, you can't tell a clean result from a trimmed one.
• 6:45
  Daniel: And coming right after that AP piece, about labs setting the terms for their own oversight?
• 6:51
  Daniel: Same pattern, different language.
• 6:53
  Rachel: I'd give it a bit more credit.
• 6:55
  Rachel: Pre- and post-deployment testing, then multimodal models and agents in phase two, that's real structure.
• 7:03
  Rachel: But who writes the grading criteria?
• 7:05
  Rachel: And until they publish failure modes, listeners are getting an announcement with no evidence behind it.
• 7:11
  Host: If you want a broader view beyond AI safety
• 7:14
  Host: try AI Daily Briefing: top AI news for engineers
• 7:18
  Host: founders and investors
• 7:20
  Host: every weekday with real capabilities versus demo hype explained fast.
• 7:25
  Host: Find it wherever you listen to podcasts.
• 7:28
  Host: Links to every story are in the show notes whenever you're ready to follow up.
• 7:32
  Host: Take a closer look at the pieces that caught your attention, and we'll be back tomorrow with more.
• 7:37
  Host: That's AI Safety Daily for today.
• 7:39
  Host: This is a Lantern Podcast.