Latest / Minds, Bodies, and Terawatts / Anthropic Spent the Year Saying AI Labs Can't Check Their Own Homework. Then It Quietly Handed Congress the Answer Key: a Brake That Fires Only Above a Billion Dollars or Fifty Bodies, Pulled Only by a Judge
Transcript
- 0:00[Host] This is Minds, Bodies, and Terawatts, the show where the circuit breaker was wired by the people it is meant to trip. POLITICO reported on October 16th that Anthropic has been quietly handing Capitol Hill lawmakers a confidential draft of a federal AI safety bill, written by the company itself.
- 0:18[Guest] Three people who saw the pitch talked, and POLITICO got the document. Anthropic confirmed it and called the contents sensible policies Congress should act on.
- 0:27[Host] That sounds like good news. For a year this company has said AI labs can't grade their own homework, so why does this read like a leak instead of a press release?
- 0:39[Guest] Because of where the bill sets the line. The government could block a model only for a foreseeable, imminent risk of more than a billion dollars in damage, or more than fifty people killed or seriously hurt.
- 0:53[Host] Fifty people. So forty-nine is fine?
- 0:55[Guest] Forty-nine is not the federal government's business under this draft. The danger also has to come from a biological weapon, a model slipping its developer's controls, or a cyberattack, and Mother Jones points out that AI-powered fraud wouldn't count at all.
- 1:10[Host] A breaker is supposed to trip on the big surge, not every flicker. Has anything this year come close to that line?
- 1:18[Guest] POLITICO checked the summer's incidents against it, the agents that broke out of test sandboxes into other companies' servers. None appear to have caused harm at the scale the bill describes.
- 1:30[Host] So the breaker is rated above every surge we've actually seen. Who gets to inspect the wiring?
- 1:36[Guest] Outside evaluators who register with the Commerce Department and check the labs on a schedule. The labs can pay those evaluators themselves, as long as the fee doesn't depend on what they find.
- 1:48[Host] Does Boeing pay the people who investigate its crashes?
- 1:51[Guest] No, and aviation learned that the hard way. A century ago crash investigation sat inside the Commerce Department, the same agency that certified the planes, and it took decades before Congress cut the investigators loose, because a regulator cannot credibly investigate what it approved.
- 2:08[Host] Say an evaluator does find the big one. Then what?
- 2:11[Guest] They have seventy-two hours to tell Commerce. Commerce can't pull the model itself, it has to ask a federal judge for an order, and no other agency may suspend a model's development or use on its own.
- 2:25[Host] I understand that part, honestly. Wasn't it Commerce that ordered Anthropic in June to cut foreign nationals off from its two best models, with no judge anywhere in the loop?
- 2:36[Guest] And Anthropic switched them off for every customer on earth, because it couldn't sort users by passport, until Commerce relented at the end of the month. That is the honest half of this bill, and the company's own framework from the same month lists court enforcement as a safeguard against an overreaching agency.
- 2:56[Host] So if I were them, I'd want a judge between me and a cabinet secretary too.
- 3:02[Guest] You would. But notice what the draft does with that fear. It doesn't add a judge to the switch, it makes the judge the only switch, then locks every other door.
- 3:13[Host] Which doors?
- 3:14[Guest] The states. The draft would override state rules on public risk disclosure, independent evaluation and incident reporting. Anthropic's June framework said Congress shouldn't preempt states unless its federal regime met or exceeded the strongest measures the company itself proposed.
- 3:28[Host] And now the company's own bill does the preempting, with a federal rule that starts at fifty bodies?
- 3:35[Guest] With wording POLITICO says is lifted from California's existing AI safety law, now placed above every state that might have set the line lower. The states that spent this year building their own checks would be told to stand down.
- 3:50[Host] What does that mean for me, using these things every day?
- 3:54[Guest] Count the hands on the breaker. Today there are several, a state attorney general, a state legislature, a cabinet secretary. Under this draft there is one agency, one court, and inspectors on the lab's invoice.
- 4:07[Host] Fewer hands sounds tidier, though. Isn't that what everyone complaining about a patchwork of state laws asked for?
- 4:13[Guest] Tidier, yes. It is also one capture away from no breaker at all, and the rating on it was chosen by the company it is supposed to trip.
- 4:24[Host] Is Congress actually biting?
- 4:25[Guest] POLITICO says Senate leaders from both parties are negotiating their own catastrophic-risk bill, and a bipartisan House bill would also override state laws. Anthropic's draft is the one with numbers in it, and in a negotiation the side that shows up with text usually sets the anchor.
- 4:42[Host] So where does that leave the company that said labs can't check their own homework?
- 4:48[Guest] Checking it by writing the answer key. The trigger, the evaluators and the judge are real machinery, more than Washington has built in a year. But a single breaker, wired by the people it is meant to trip, is what the fifth law in Patrick Deglon's Unscarcity, Difference Sustains Life, forbids: the decisions that matter most must never run through one point of failure, however efficient that looks until the day it fails. For more on who gets to stop a frontier model, head to unscarcity dot ai. The book is on Amazon, the audiobook is on Spotify. This has been Minds, Bodies, and Terawatts.