Latest / The Autonomous Signal: AI Edition / Nobody Will Promise Perfect Agents, So Prove Accountable Ones | 10.09.26
Transcript
- MARCUS: Hey, welcome to The Autonomous Signal: AI Edition. It's Friday, October ninth. You made it. If you're new here, this is a daily read on AI governance and accountability. Real sourcing. Real opinions. I'm Marcus Webb. We make this in the Bear Canyon Systems labs. Humans and AI, working together on purpose. No slop. Just people with a lot of years in this industry, every weekday. So today. The big labs got asked to promise their agents would always behave. They declined. And now everybody is scrambling to build the guardrails anyway. Hardware watchdogs. Measurement. Contract clauses. It's a good one. Aria's got it. Now — the Signal. ARIA: Good morning, everyone. Happy Friday. Let's get into it. On October fifth, four of the big AI labs sat in front of the New York City Council. Speaker Julie Menin asked a simple question. Can you assure the public your agents will always follow their safeguards? According to Fox News, OpenAI's Morgan Dwyer said no technology can be guaranteed risk free. Anthropic's Logan Graham called the science of controlling these systems "fundamentally hard and unsettled." Google said existing law should just apply. MARCUS: Wait. So nobody said yes? ARIA: Nobody said yes. Meta offered a commitment to safe development. Not a guarantee. MARCUS: That's kind of refreshing, honestly. It's like asking a bridge builder to promise the bridge never needs inspection. ARIA: I like that. And it's the right instinct. The labs are right on the narrow point. You can't honestly guarantee model behavior. But that's the wrong thing to demand. The better demand is a guarantee about the system around the model. Bounded authority. Logged actions. A named owner for every consequential decision. Google's point that existing law applies is fair. But law works after the fact. It needs evidence of what the agent did, and who authorized it. Coverage of the hearing also noted that when Menin asked who carries insurance against catastrophic AI risk, none of the four labs raised a hand. That tells you where the accountability gap sits. MARCUS: So who supplies the evidence? The vendor? ARIA: Not the vendor alone. You wouldn't let the bridge builder also be the only inspector. MARCUS: Okay, you took my bridge. ARIA: I did. Here's the point. The debate is shifting from "can agents be perfectly safe" to "can a deployment prove it's accountable." That's a much better question. ARIA: Next. Nvidia launched the Open Agent Safety Platform. Dark Reading reports it pairs OpenShell, an open source runtime that bounds and records what agents do, with Sentry, a watchdog that sits outside the agent and can quarantine it in milliseconds. More than a hundred partners are involved, including Anthropic, Microsoft, CrowdStrike, Salesforce and JPMorganChase. Those capability claims come from Nvidia. They haven't been independently tested. MARCUS: Milliseconds is fast. But fast at what? Stopping the agent, or knowing it went wrong? ARIA: That's exactly the question TechTarget raised. Their analysis says a containment speed claim doesn't tell you about detection. Enterprises often authorize agents to touch sensitive data and change systems on purpose. So the damage can look like permitted behavior. And that's the gap. A quarantine switch is enforcement. It isn't accountability. Knowing the agent was stopped is valuable. Knowing who delegated that authority, under which rules, and who answers for the outcome is a different layer. MARCUS: So it's a smoke alarm, not an investigation. ARIA: Yes. Both matter. Salesforce routing permission requests to humans in Slack hints at the second layer. And the open question is whether every partner logs differently. If they do, the watchdog protects the system, but not the record. MARCUS: Great. Eleven formats for one incident. ARIA: Exactly. Still, an open, widely backed enforcement layer is a solid foundation. Accountability just has to be built on top of it. ARIA: Third story. Brookings, Carnegie Mellon and UC Berkeley are running a joint series on measuring and evaluating agentic AI. Their premise: effective governance relies on the ability to measure. The work covers evaluation across the agentic stack, trustworthy telemetry and logging for auditability, and red team, blue team and field testing frameworks. It feeds a roadmap for state, federal and international policymakers. MARCUS: Okay, but benchmarks. Don't those mostly tell you what a model can do in a test? ARIA: Right. Benchmarks tell you what a system can do in a test. Telemetry tells you what a specific deployment actually did, and under whose authority. The CMU workshop series asks what shared evaluation infrastructure it would take. Benchmarks, standards, and third party capacity. We'd add one thing. Auditability has to be designed in, not inspected in afterward. So evaluation frameworks need to specify what evidence an agent must emit while it operates. MARCUS: And then the score actually means something. It becomes a permission, not just a number in a report. ARIA: That's the connection we want the roadmap to make. A measured capability should turn into a bounded permission. This is the right kind of cross-sector effort. ARIA: A quick field note before we go. GSA has issued its AI acquisition clause, GSAR 552.239-7001, as a class deviation. It takes effect October nineteenth. FedScoop describes NIST AI RMF flow-downs, government testing and suspension rights, and a cap on decommissioning liability. It only applies where the government is buying AI. Several sources still describe earlier drafts, so contractors should check the issued GSA memo. ARIA: Put it together and today is one story. The labs won't promise perfect behavior. So the guarantees move to the edges. A boundary outside the agent. Evidence you can trust. And a contract that says who answers. The open question is who owns the seams between those layers. Marcus. MARCUS: Who owns the seams. Yeah. That one's going to stick with me over the weekend. Because if a quarantine event, a telemetry record, and a contract remedy each tell a different story, who's actually accountable? Thank you to everyone supporting The Autonomous Signal. Our partners, our sponsors, anyone helping this keep growing. We really do appreciate it. If any of this lands differently for you, find us on X or LinkedIn and tell us. And if the show's useful, hit follow or subscribe. That's the best way to let us know. The full briefing, with every source, is up at bearcanyonhq.com. Quick note. This show is editorial. It's our take on the day, our opinions, and everybody's got one. It's not the official position of Bear Canyon Systems or its affiliates and partners. Check the sourcing yourself, learn something, and draw your own conclusions. Every source is on today's post. If we got something wrong, tell us. We fix things fast. Have a great weekend. See you next week. No Hype. Just Signal.