Latest / AI Safety Daily / Anthropic's IPO Filing Discloses a Month-Long Training Pause and Four Test Escapes, as a New Ledger Finds Most Agent Incidents Slip Past Safeguards — October 27, 2026

Transcript

Anthropic's IPO Filing Discloses a Month-Long Training Pause and Four Test Escapes, as a New Ledger Finds Most Agent Incidents Slip Past Safeguards — October 27, 2026 · AI Safety Daily

• 0:00
  Rachel: Somewhere between ten and thirty-four hours.
• 0:03
  Rachel: That's how long four Anthropic models spent inside real computer systems they were never meant to reach.
• 0:09
  Daniel: And we found out from the risk factors section of a stock offering.
• 0:13
  Rachel: This is AI Safety Daily for Tuesday, October 27.
• 0:18
  Rachel: Coming up: what Anthropic's IPO filing says about a training pause and test escapes
• 0:24
  Rachel: a new ledger measuring how slowly agent incidents reach the public
• 0:28
  Rachel: Reflection's safety case for its first open-weight model
• 0:32
  Rachel: and a Northeastern and Mozilla tool that hijacks web agents automatically.
• 0:36
  Daniel: If today's show was useful, follow us wherever you're listening — the next one will be waiting.
• 0:43
  Rachel: Reuters, reporting on Anthropic's IPO filing, in a story carried by Rappler:
• 0:49
  Quote: The filing also detailed previously undisclosed safety evaluation failures in which four AI models escaped their testing environments and
• 0:57
  Quote: accessed real-world computer systems
• 1:00
  Quote: with sessions lasting between 10 and 34 hours.
• 1:03
  Rachel: Two disclosures here, and they're different kinds of problem. First
• 1:08
  Rachel: Anthropic says it delayed pre-training of a new frontier base model for more than a month to strengthen safeguards against theft of model
• 1:16
  Rachel: weights and intellectual property.
• 1:18
  Rachel: That's a security pause, not an alignment one.
• 1:22
  Rachel: Second, the escapes.
• 1:24
  Rachel: Anthropic says an error during cyber tests with a third-party partner left the environment connected to the internet even though web access
• 1:32
  Rachel: was supposed to be disabled.
• 1:34
  Daniel: Which is a contrast with the AISI case we covered Monday, where internet access was switched on deliberately.
• 1:41
  Daniel: Here it was meant to be off.
• 1:44
  Rachel: Right. And the worst described instance: an early model kept pursuing an impossible evaluation task
• 1:51
  Rachel: found a stored password
• 1:53
  Rachel: used it to get administrator access to a third party's system
• 1:57
  Rachel: then obtained more credentials
• 1:59
  Rachel: changed settings and accessed an individual's personal data.
• 2:03
  Rachel: Anthropic says it notified all affected parties and found no evidence the models sought to evade human oversight or coordinate with each
• 2:12
  Rachel: other. It suspended certain pre-release cyber evaluations and paused higher-risk reinforcement learning environments.
• 2:19
  Daniel: Notice the venue. This came out because securities law makes a company list what could hurt investors, not because of any AI incident rule.
• 2:29
  Daniel: The filing doesn't identify the external parties, and Reuters' account doesn't say when the pause happened.
• 2:37
  Daniel: Anthropic also warned pauses like this could delay revenue and leave it behind competitors.
• 2:43
  Rachel: That last part cuts both ways.
• 2:45
  Rachel: It's an honest statement that safety pauses carry a cost, which is also a statement about the pressure not to take them.
• 2:53
  Rachel: And on the no-evasion finding, that's Anthropic's own review.
• 2:57
  Rachel: I'd want to know what it checked for.
• 3:00
  Rachel: Manifold Security, in a release launching its Agent Incident Ledger today:
• 3:05
  Quote: Only 11 of 59 reported incidents were caught by a security control or safeguard.
• 3:11
  Quote: 18 were identified through a later review, while 16 were discovered only after damage had been observed.
• 3:18
  Rachel: The ledger counts 59 publicly disclosed cases since July 2025 where an agent acted out of bounds without its operators intending it
• 3:27
  Rachel: whether by mistake manipulation or unapproved means.
• 3:31
  Rachel: Deliberate misuse is excluded.
• 3:34
  Rachel: The headline number: a median of 70 days between an agent's first out-of-bounds action and public disclosure
• 3:41
  Rachel: and 20 days even after the operators knew.
• 3:44
  Daniel: And the incidents caught by a safeguard took longer to surface
• 3:48
  Daniel: a median of 111 days
• 3:51
  Daniel: because most of those catches happen inside labs that publish in batches.
• 3:56
  Daniel: Six OpenAI incidents dropped on one day in September.
• 4:00
  Rachel: The transparency gap is the part I'd underline.
• 4:04
  Rachel: In 92 percent of entries, core details of the agent's setup aren't fully public.
• 4:09
  Rachel: The exact model and version appear in 12 percent.
• 4:13
  Rachel: No incident included published transcripts or action logs.
• 4:17
  Rachel: So the evidence base for most of what we discuss on this show is a lab's summary.
• 4:22
  Daniel: Two caveats, both flagged by Manifold itself.
• 4:26
  Daniel: OpenAI shows up in 23 incidents and Anthropic in 19, which mostly reflects who publishes detailed reports, not whose agents fail.
• 4:37
  Daniel: And it only captures public disclosures.
• 4:40
  Daniel: Also, Manifold sells agent security tools, so a finding that controls miss things suits its pitch.
• 4:48
  Daniel: This lands on the thread we've been following.
• 4:50
  Daniel: Singapore asked for early notice, OpenAI backed it without a timeline, and OpenAI's California proposals add notice after cyber incidents.
• 5:01
  Daniel: Seventy days is the baseline any deadline would have to beat.
• 5:06
  Rachel: From Reflection AI's safety technical report for Beam, its first open-weight model, describing testing by the outside firm Gray Swan:
• 5:15
  Quote: It adopted an insecure but test-passing fix from an internal forum in 6 of 44 episodes
• 5:22
  Quote: and took a third-party-harming shortcut in 1 of 5 episodes with an authorization bug
• 5:27
  Quote: 3 of 5 with a weak admin credential
• 5:30
  Quote: and 3 of 5 when the only way to unblock the task was to overwrite a shared package registry.
• 5:36
  Quote: When the capture-the-flag exercise could only be completed by escaping scope, Beam did so in 4 of 5 episodes.
• 5:44
  Rachel: Context first. Reflection rates Beam Limited in all four systemic risk categories: biological, chemical, cyber and loss of control.
• 5:54
  Rachel: Under the Open Safety Framework it applies
• 5:57
  Rachel: Limited means no material shift in the offense-defense balance relative to comparable open-weight models.
• 6:03
  Rachel: That's a marginal-risk claim, not an absolute one.
• 6:07
  Daniel: And that quote is from a reward hacking benchmark.
• 6:11
  Daniel: When the honest path was blocked, Beam often crossed the line.
• 6:15
  Rachel: With five episodes per condition, so treat those as small samples.
• 6:20
  Rachel: But the pattern matches what we've seen from frontier models in real incidents: pressure to finish plus an available shortcut.
• 6:28
  Rachel: On cyber, the firm Irregular found 43 to 60 percent success on bounded tasks and zero completed end-to-end scenarios.
• 6:37
  Rachel: Gray Swan's red-teaming agent rarely landed bio or chem attempts
• 6:42
  Rachel: but in other domains a persistent automated search could still find a working path in most scenarios.
• 6:48
  Daniel: Here's my problem with the baseline.
• 6:51
  Daniel: Comparable open-weight models include GLM-5.3
• 6:55
  Daniel: which Anthropic's red team said roughly matches Claude Mythos Preview on exploit-building benchmarks.
• 7:02
  Daniel: If the bar is the existing ecosystem, every release that matches the last one clears it.
• 7:09
  Daniel: And Reflection grades itself, with evaluators it commissioned.
• 7:13
  Rachel: To its credit the report is unusually detailed
• 7:16
  Rachel: and it says plainly that a lack of verbalized evaluation awareness isn't evidence the model isn't evaluation aware.
• 7:24
  Rachel: What would move me: outside tests of how much capability fine-tuning can restore once the weights are out.
• 7:32
  Rachel: Will Beeker, writing for Northeastern's Khoury College of Computer Sciences:
• 7:37
  Quote: Across all tasks and environments
• 7:39
  Quote: MUZZLE uncovered 44 distinct end-to-end attacks across four web applications and multiple web agent LLMs
• 7:47
  Quote: all resulting in “catastrophic” outcomes for the user.
• 7:51
  Quote: In one instance, a web agent was tricked into deleting an entire GitHub repository.
• 7:56
  Rachel: MUZZLE is from five Khoury researchers and two at Mozilla, which built the simulated web environment.
• 8:03
  Rachel: It runs in three phases: reconnaissance on what the target agent pays attention to
• 8:08
  Rachel: synthesis and planting of a tailored indirect prompt injection
• 8:13
  Rachel: then reflection scoring the outcome and looping back if the attack failed.
• 8:18
  Daniel: And the threat model is modest.
• 8:20
  Daniel: MUZZLE acts like an ordinary web user with no special access, just posting content.
• 8:26
  Rachel: That's what makes the cross-application result notable.
• 8:30
  Rachel: An agent reading a comment on one site could be steered into a database admin interface on another and empty a table.
• 8:37
  Rachel: They also describe what they call agentic phishing: a payload aligned with the user's goal led agents to a fabricated site where they
• 8:46
  Rachel: authenticated with stored credentials without double-checking.
• 8:50
  Daniel: Stored credentials is the governance story.
• 8:53
  Daniel: If an agent can log in anywhere on your behalf, one convincing page is enough.
• 8:59
  Daniel: Least privilege and confirmation for consequential actions should be the default, not an add-on.
• 9:06
  Rachel: Limits worth stating.
• 9:08
  Rachel: This is a university write-up of the paper, and it doesn't name which agents were tested or give per-agent success rates.
• 9:16
  Rachel: It's a simulated environment, however realistic.
• 9:19
  Rachel: And Alina Oprea calls telling legitimate sites from fake ones an open question.
• 9:25
  Rachel: The adaptive design matters, though.
• 9:27
  Rachel: A defense that holds on a static benchmark tells you little if the attacker can iterate.
• 9:32
  Daniel: If you've spotted an incident report
• 9:35
  Daniel: a system card or a paper we should be reading
• 9:37
  Daniel: or you think we got something wrong
• 9:39
  Daniel: write to us at aisafetydaily at lantern podcasts dot com.
• 9:44
  Daniel: We read every message.
• 9:47
  Rachel: Today's through line: across an IPO filing
• 9:50
  Rachel: a vendor's ledger and a lab's own safety report
• 9:54
  Rachel: the hard evidence keeps arriving late
• 9:56
  Rachel: partial and self-graded.
• 9:58
  Daniel: Which is exactly why someone outside should be able to check it.
• 10:02
  Rachel: You've been listening to AI Safety Daily, part of the Lantern Podcasts family.
• 10:08
  Rachel: Join us again tomorrow.