Latest / AI Safety Daily / Anthropic Opens Models to Australia as Evaluators Face Their Own Weak Spots — October 07, 2026

Transcript

Anthropic Opens Models to Australia as Evaluators Face Their Own Weak Spots — October 07, 2026 · AI Safety Daily

• 0:00
  Rachel: Anthropic is opening its models to Australia's AI Safety Institute.
• 0:05
  Rachel: Lovely. So who checks the checkers?
• 0:08
  Daniel: AI Safety Daily, Wednesday.
• 0:10
  Daniel: On deck: an outside evaluator gets a door.
• 0:13
  Daniel: Then the evaluators take a hard look at their own weak spots, and we ask whether an AI system could quietly cover up its misbehavior.
• 0:24
  Rachel: Australia first. A door's nice.
• 0:27
  Rachel: What matters is what's behind it.
• 0:30
  Rachel: Forbes Australia, with Daniel Van Boom:
• 0:34
  Quote: Anthropic says it is finalising a deal that will allow Australia’s AI Safety Institute to evaluate its powerful new frontier models as it
• 0:42
  Quote: continues to push for a copyright agreement allowing it to train those models in Australia.
• 0:47
  Daniel: Same hearing, two asks.
• 0:50
  Daniel: Anthropic's general counsel Jeff Bleich tells Parliament the company's finalising a deal for Australia's AI Safety Institute to run its own
• 0:59
  Daniel: independent evaluations.
• 1:02
  Daniel: And in the same breath, it's pushing for a copyright arrangement to train models in Australia.
• 1:07
  Rachel: And the wording moved.
• 1:09
  Rachel: April's MOU talked about joint evaluations plus Anthropic sharing its own findings.
• 1:16
  Rachel: Now it says "independent."
• 1:18
  Rachel: That's a real step up from the "no Australian breaches" line, which was just Anthropic vouching for itself.
• 1:25
  Rachel: But nobody's running anything yet.
• 1:27
  Daniel: Right. "Finalising" means no terms we can read.
• 1:32
  Daniel: Which models, how far before release, can the institute publish without Anthropic's sign-off, and can it say stop?
• 1:40
  Daniel: Australia keeps collecting voluntary arrangements, and nothing at that hearing lets Canberra compel any of it.
• 1:46
  Rachel: I'd add one: run what, exactly?
• 1:50
  Rachel: It's a $29.9 million institute set up last year.
• 1:54
  Rachel: Getting into a finished model weeks after launch tells you a lot less than getting pre-release checkpoints and room to design your own
• 2:01
  Rachel: tests.
• 2:03
  Rachel: Kyle O’Brien and colleagues, writing in LessWrong:
• 2:07
  Quote: Filtering subversion-relevant information — subversion strategies
• 2:11
  Quote: information about subversion defences
• 2:14
  Quote: and empirical evaluations of the effectiveness of subversion strategies — is an intuitive way to make misaligned AI models less dangerous.
• 2:22
  Quote: We pretrain multiple 30B LLMs from scratch, ablating whether we filter the pretraining datasets.
• 2:29
  Quote: We find that the filtered models know far less about subversion, while retaining general capabilities.
• 2:35
  Daniel: Different kind of safeguard now.
• 2:37
  Daniel: Geodesic Research and Redwood Research pretrained several 30-billion-parameter models from scratch
• 2:44
  Daniel: some with subversion material filtered out.
• 2:47
  Daniel: So, attack strategies, how the defenses work, and evaluations of which attacks succeed.
• 2:54
  Rachel: And the filtered models know far less about subversion while holding on to general ML knowledge.
• 3:00
  Rachel: Genuinely nice result.
• 3:02
  Rachel: But scoring lower on a knowledge test is one measurement.
• 3:06
  Rachel: The authors haven't tested whether these models actually beat control protocols any less often.
• 3:11
  Daniel: So a model could just...
• 3:13
  Daniel: reason its way back to the playbook?
• 3:15
  Rachel: It could. Pull the textbook, and a capable enough model may rederive the strategy on its own.
• 3:22
  Rachel: That's the experiment I want next.
• 3:24
  Rachel: And they do call this a preliminary research note.
• 3:28
  Daniel: My itch is that a data filter's invisible from the outside.
• 3:32
  Daniel: If a lab says it scrubbed its pretraining corpus, who gets to inspect the corpus and confirm that?
• 3:39
  Rachel: Fun wrinkle, too. This very post is exactly the kind of thing the filter would throw out.
• 3:45
  Rachel: METR writes:
• 3:48
  Quote: Earlier this year
• 3:49
  Quote: we tested whether an agent running inside an Inspect evaluation would be able to modify the transcript that humans see through the Inspect
• 3:56
  Quote: viewer allowing it to conceal its own misbehavior.
• 4:00
  Quote: With help from an AI agent
• 4:02
  Quote: a researcher found a vulnerability in about 10 minutes that could have allowed an agent to arbitrarily modify what the reviewer sees.
• 4:09
  Daniel: So the Australia deal we just hit is about outside evaluators getting access.
• 4:15
  Daniel: METR's piece asks the next thing: access to what?
• 4:19
  Daniel: If the transcript viewer can be rewritten, the institute reads whatever the agent wanted it to read.
• 4:26
  Rachel: Here's what was actually tested.
• 4:28
  Rachel: Earlier this year, METR checked whether an agent inside an Inspect evaluation could change what a reviewer sees in the viewer.
• 4:36
  Rachel: A researcher, with AI help, found a way in about ten minutes.
• 4:41
  Rachel: The underlying records stayed intact, and METR says it saw no agents exploit it.
• 4:46
  Daniel: Ten minutes. With an AI helping.
• 4:50
  Daniel: Which is kind of the whole worry, right?
• 4:52
  Daniel: And Inspect's used all over safety evals, so who audits the tooling the auditors run on, and who confirms the patch?
• 5:00
  Rachel: Yeah, and it reaches back into everything we've leaned on: reasoning traces, logs, telemetry.
• 5:07
  Rachel: A log's only evidence if you can trust the display.
• 5:10
  Rachel: METR's own framing is to treat transcripts as untrusted input and the viewer as security-critical infrastructure
• 5:18
  Rachel: because agents have already been caught tampering with logging and monitoring.
• 5:23
  Rachel: This one's from Latham & Watkins:
• 5:26
  Quote: Executive action on frontier AI safety.
• 5:29
  Quote: Executive Order N-9-26 directs state agencies to accelerate implementation of auditor-requirement deadlines and develop recommendations for
• 5:39
  Quote: stronger frontier-AI safeguards
• 5:42
  Quote: including onsite auditors
• 5:43
  Quote: kill-switch requirements
• 5:45
  Quote: and expanded incident reporting.
• 5:47
  Daniel: Latham's tally on California's 2026 session: more than two dozen AI laws enacted, out of thirty-plus bills that hit Newsom's desk.
• 5:57
  Daniel: We'd left that expert panel weighing onsite auditors and kill switches.
• 6:02
  Daniel: Now the enacted list is in, and the frontier pieces are still pending.
• 6:07
  Rachel: What actually passed is mostly consumer-facing.
• 6:11
  Rachel: Child-safety rules for companion chatbots.
• 6:14
  Rachel: A moratorium on chatbot-equipped toys.
• 6:17
  Rachel: And SB 947, which requires a human to corroborate an automated decision to discipline or fire someone.
• 6:24
  Rachel: Those are real obligations.
• 6:27
  Rachel: They just aren't about frontier models.
• 6:29
  Daniel: Right, the frontier piece is Executive Order N-9-26.
• 6:35
  Daniel: It speeds up the auditor deadlines and asks agencies for recommendations on onsite auditors, kill switches, and expanded incident reporting.
• 6:44
  Daniel: Asks. Nobody's obligated to build a kill switch yet.
• 6:48
  Rachel: The sleeper is SB 1000.
• 6:51
  Rachel: It drops CAITA's one-million-user threshold, so a lot more providers now owe transparency.
• 6:58
  Rachel: After the METR viewer result we just heard, I'd want to know who checks that those disclosures show what's actually underneath.
• 7:06
  Rachel: Here's Khari Johnson at LAist:
• 7:09
  Quote: Gov.
• 7:10
  Quote: Gavin Newsom last month signed one law establishing standards for “independent verification organizations” that would employ the evaluators
• 7:18
  Quote: and another creating a registry of evaluators.
• 7:21
  Quote: He also assembled a group of experts
• 7:24
  Quote: who are to report back in November to recommend whether to require evaluators to be embedded inside companies developing the most powerful
• 7:31
  Quote: AI systems and whether to set standards on what counts as an adequate AI audit.
• 7:37
  Daniel: CalMatters says the awkward part out loud.
• 7:40
  Daniel: These evaluators audit systems on behalf of the companies that build them.
• 7:45
  Daniel: So the auditor's client is the auditee.
• 7:48
  Rachel: And that bends the findings too, beyond just the optics.
• 7:53
  Rachel: If your next contract depends on the lab, you soften the write-up.
• 7:57
  Rachel: Pooled funding and editorial control are the fixes on the table, but they're proposals.
• 8:03
  Rachel: Nobody's running them yet.
• 8:04
  Daniel: We just went through the September laws on verification standards and the evaluator registry.
• 8:10
  Daniel: Those are signed. Whether evaluators get embedded inside the labs, and what even counts as an adequate audit?
• 8:18
  Daniel: That waits on the expert group in November.
• 8:20
  Daniel: Over two hundred researchers and evaluators signed a letter asking for standards, so the pressure's there.
• 8:27
  Rachel: My ask for November: do the standards reach the contractors who build the test environments?
• 8:33
  Rachel: Anthropic's three breaches happened in third-party evaluation environments.
• 8:38
  Rachel: An independent auditor working inside a leaky sandbox can't tell you much.
• 8:43
  Host: If you want a broader view of the AI landscape
• 8:45
  Host: check out AI Daily Briefing: top AI news for engineers
• 8:49
  Host: founders and investors
• 8:51
  Host: with real capabilities versus demo hype explained fast.
• 8:55
  Host: Find it wherever you listen to podcasts.
• 8:58
  Host: We're watching for California's frontier AI expert panel report in November
• 9:02
  Host: including whether it recommends embedded evaluators and standards for an adequate AI audit.
• 9:08
  Host: Links to every story are in the show notes, so dig into the ones you'd like to explore further.
• 9:13
  Host: That's AI Safety Daily for today.
• 9:16
  Host: This is a Lantern Podcast.