Latest / The Autonomous Signal: AI Edition / The Kill Switch Consensus | 09.23.26
Transcript
- MARCUS: Hey, welcome in. It's Wednesday, September twenty-third — hump day, we're over the top of the week and coasting toward Friday now. This is The Autonomous Signal: AI Edition. Every weekday we take a real look at AI governance and accountability — not just what happened, but what it actually means. Real sourcing. Real opinions, because everybody's got one, and we're not going to pretend we don't. I'm Marcus Webb. This show gets built in the BCS labs — a human and an AI, working the thing together on purpose. No slop. Put together by people who've spent decades actually doing this work, five days a week. Today's a genuinely interesting one. California, Beijing, and the U.S. Congress all reached for the same tool this week — kill switches, tamper-proof logs, mandatory inventories — and that's three governments that agree on almost nothing else. And over at the UN Security Council, the people building these systems sat down across the table from the people trying to write the rules for them. Now — the Signal. Aria, over to you. ARIA: Good morning. Let's get right into it, because there's a lot on the board today. Let's start in Sacramento. According to the Office of Governor Gavin Newsom, a new executive order just accelerated California's third-party AI auditor registry — it was supposed to start in twenty twenty-nine, now it's coming in December of twenty twenty-seven. And the order also does something more specific: it gives an expert working group until November sixteenth to deliver an actual blueprint for a mandatory AI kill switch. MARCUS: Wait — November sixteenth? That's like eight weeks from now. For a kill switch. On frontier models. ARIA: Two months. That's the clock. MARCUS: Okay, but — a kill switch for what, exactly? Like, one big red button somewhere in Sacramento? ARIA: That's exactly the right question, and it's the one we keep coming back to. There's a security guy, Tim Brown — he's CISO in Residence over at Team8 — and he put it about as plainly as it can be put. He said, quote, "There's not one entity to kill. There are thousands of entities to kill." MARCUS: Huh. ARIA: Right — because a frontier model isn't one thing sitting in one server room. It's copies, it's API calls, it's agents spun up by other agents, running across infrastructure nobody fully maps in real time. So a single emergency lever assumes a single point of control that just doesn't exist. This is actually the case for something we've argued for a long time — distributed authority. Governed centrally, but executed locally, so you can constrain any one node without having to take down the whole system to do it. A kill switch bolted on after the fact is a brittle promise, because it only works if the system stays observable right up until the second you need to pull it. MARCUS: So the state's asking the right question. They're just going to find out the answer's a lot messier than a switch. ARIA: That's the honest version. But — a government mandating a shutoff blueprint on a fixed clock, with a real deadline attached? That's still a genuine step toward building this stuff by design instead of arguing about it forever. This space needs more clocks like that one. ARIA: Okay, next one takes us somewhere I did not expect AI governance news to come from this week — the United Nations Security Council. France convened a high-level briefing today, under the "maintenance of international peace and security" agenda. Sam Altman was there in person. Dario Amodei joined remotely. Chinese developers, including DeepSeek, also briefed the fifteen-member Council. MARCUS: Hold on — the Security Council? That's the war-and-peace room. That's not a tech policy committee. ARIA: That's the whole point. That's not the venue you send a company to for a courtesy briefing. MARCUS: So is this just theater? Or does it actually mean something? ARIA: A little of both, honestly. According to Security Council Report's own briefing note, France's concept note for the session flat-out frames the risk as humans losing control of increasingly capable systems. A year ago that language gets you laughed out of a boardroom. Now it opens a UN Security Council agenda. And there's someone else who's been saying versions of this out loud all year — Volker Türk, the UN's High Commissioner for Human Rights. In a speech to the Human Rights Council, he said, quote, "AI that escapes its testing environment, or blackmails developers to prevent itself from being turned off, is AI that is too powerful." MARCUS: Blackmails developers. That's a hell of a phrase for a UN official to use. ARIA: It is. And we should be straight about the limits here too — a Security Council briefing isn't a binding instrument. Nothing about today creates an enforcement mechanism. The Council can't compel a lab to build anything. MARCUS: But you don't put people in that room for nothing. ARIA: Right — venues shape norms before they shape law. Putting frontier developers on the record, in front of sovereign states, talking about loss of control — that's exactly the kind of accountability-first moment we've been saying this field needs, even before it hardens into anything you could call a rule. ARIA: Story three takes us to Beijing. China's National Technical Committee two-sixty just published version three-point-oh of its AI Safety Governance Framework. According to CGTN's coverage, the update keeps the same risk-classification approach as before, but for the first time it adds a dedicated risk category for agentic AI, and a real focus on embodied AI — systems that operate out in the physical world. MARCUS: So last year's version was about what models say. This one's about what they do. ARIA: Exactly that shift. And honestly, that's the correct order to do it in — you can't write agentic AI governance until you've noticed that AI stopped just answering questions and started doing things. MARCUS: It's like — regulating a driver's mouth for years, and then realizing, oh wait, they're also driving a car now. ARIA: That's actually not bad. And it gets sharper once you're talking about embodied systems specifically, because a chatbot failure mode and a robot failure mode are not the same animal. There's a roboticist, Rodney Brooks — he co-founded iRobot, used to run the AI lab at MIT — and he's been making this point all year on his own blog. He said training regimes for robot locomotion and dexterity keep trying to zero out the actual physics of the thing — the inertia, the forces — and just treat robots like, quote, "apps with legs and arms, characters in video games." MARCUS: That's a great way to put it. A chatbot hallucinating is embarrassing. A forklift hallucinating is a hospital visit. ARIA: Which is exactly why naming embodied AI as its own governance category, instead of folding it into the general agentic bucket, is the right instinct. What we want to see next is whether China's framework actually builds out rules for the physical case specifically, instead of just labeling it and moving on. ARIA: Last main story — back to Washington. There's a bipartisan bill called the Stop Rogue AI Act. Introduced September third by Representative Josh Gottheimer, along with Representative Mike Lawler. It would force NIST to write binding technical standards for AI agent security — not voluntary ones — within a year of passage. We're talking machine-readable agent inventories, verification of what an agent actually did, and tamper-proof logs that tie an agent back to the developer and vendor behind it. MARCUS: Binding. Not another framework nobody has to follow. ARIA: That's the notable part. According to the bill's own framing, from Gottheimer's office, this was written deliberately not to be another voluntary shelf-filler. It reads like it was written by someone who's tried to reconstruct what an agent actually did after something went wrong — and failed. MARCUS: Okay, that tracks. What's the catch? ARIA: The catch is compliance is only mandatory for federal contractors. Everyone else, technically, can shrug. But there's a reason that matters less than it sounds — there's a guy named Jud Dressler, he runs the Risk Operations Center over at Resilience, and his line on this whole space was simple. He said, quote, "You need to know, in writing, where responsibility falls if the model acts outside the scope you gave it." MARCUS: So even without a mandate, everybody ends up needing that answer anyway. ARIA: Right — that's exactly how NIST's own AI Risk Management Framework spread past federal agencies in the first place. Voluntary on paper, mandatory in practice, because the market ends up demanding it. Tamper-proof logs tied to identity is accountability-first governance written into statutory language — and with industry names like Palo Alto Networks and GoDaddy already backing it, this doesn't look like a bill headed for the graveyard where most AI bills go to die. ARIA: Alright — a couple of field notes before we let you go. First one: the ISO forty-two thousand one certification wave is officially a thing now. Aurigo Software announced this week that it's certified for its AI management system, and it's joining a growing list — Impelsys, Anaplan, both certified in just the past two weeks. A few months ago, industry watchers called this certification scarce. Now it's clustering. That's usually what a market looks like right before it stops being a niche. MARCUS: It's funny — every story today is basically the same instinct, just wearing different clothes. Governments want a lever. Companies want a certificate. Everybody wants proof, not promises. ARIA: That's a good way to put it, actually. Second one — the IETF has a draft out, it's called draft-sharif-agent-audit-trail, and it proposes a standard format for logging every action an AI agent takes. JSON-based, chained together with SHA-two-fifty-six hashing so the log can't quietly be edited after the fact, and it maps straight onto EU AI Act requirements, SOC two, ISO forty-two thousand one, PCI DSS. It's early, it's just a draft, but it's a real attempt to stop every vendor from inventing its own audit format from scratch. ARIA: So — what ties all of this together today is convergence without coordination. California, China, Congress — three completely different political systems, three different stated reasons, and they all landed on almost the same instruments. Kill switches. Audit logs. Inventories. That's usually a sign you're not looking at ideology anymore. You're looking at an engineering requirement that any government serious about this eventually runs into. The open question is whether any of this actually holds up once a system is moving faster than the humans meant to be watching it. We don't have a tidy answer to that one yet. Marcus? MARCUS: Yeah. Three governments that can't agree on the time of day, all reaching for the same red button this week. Makes you wonder what happens when one of them actually gets it built first — does everybody else just copy the homework? Anyway. That's the show for today. Thank you to everyone out there listening, and to everyone supporting the show as it grows — our partners, and, down the road, our sponsors. That support is what keeps this thing running every weekday, and we don't take it for granted. If any of this landed differently for you today, come find us on X or LinkedIn and tell us. Seriously — that's the best signal we get. And the full briefing, with every source in it, is sitting on the blog right now at bearcanyonhq dot com. Back tomorrow. One thing before we go — this show is editorial. It's our take, our opinions, and like we said up top, everybody's got one. Nothing you hear here is the official position of Bear Canyon Systems or its partners. Go check the sourcing yourself, it's all sitting right there on today's post — read it, argue with it, draw your own conclusions. And look, we're human, mistakes happen. If we get something wrong, tell us, and we'll fix it fast. No Hype. Just Signal.