Latest / Minds, Bodies, and Terawatts / The People Building AI Are Quitting and Saying It Could Kill Us. Sam Altman Told His Staff OpenAI Would Slow Down, If the Other Labs Do Too
Transcript
- 0:00[Host] This is Minds, Bodies, and Terawatts, the show where everyone in the race agreed to slow down, right after you. Bloomberg reported on Thursday that Sam Altman told a company-wide meeting at OpenAI it is open to slowing its most advanced AI work alongside the other labs, and that some may not go along.
- 0:22[Guest] OpenAI declined to comment, and the timing is no accident. Earlier in the week a researcher named Jacob Coxon quit Anthropic with a post saying his former employers, OpenAI and Anthropic, are racing straight to self-improving superintelligence and gambling with our lives.
- 0:39[Host] A twenty-seven-year-old quits, posts a thread, and the CEO of the world's biggest lab starts talking about slowing down? Who is this guy?
- 0:49[Guest] Not a safety researcher, which is the unusual part. He worked on pretraining, the work that builds the raw model, and self-improving here means AI doing the research that builds the next AI. Bloomberg counts over a hundred fifty million views.
- 1:06[Host] A doomer with a megaphone, then. Every technology gets one.
- 1:10[Guest] Then people still inside agreed, on the record. Evan Hubinger, an Anthropic alignment lead whose job is making the model behave, replied that yes, they earnestly believe AI could kill all humans, that he puts it above ten percent this decade, and that there is no plan yet.
- 1:30[Host] So the person whose job is making the model behave says one in ten, and there is no plan?
- 1:38[Guest] A second Anthropic team lead, Samuel Marks, posted that the more senior the employee, the more worried they are. Why keep building? Commercial pressure, and fear that a less responsible competitor gets there first.
- 1:52[Host] Which is Altman's offer in one sentence. I'll slow down if you slow down. Is anyone actually slowing?
- 1:59[Guest] No. OpenAI told reporters this week it is already training a model significantly more powerful than Astra, its newest, and Anthropic is reportedly heading to the stock market as soon as next month at a valuation Reuters says some investors put near two trillion dollars.
- 2:18[Host] Two trillion, from a company whose own researcher puts one in ten on the product ending everyone. Say every lab took the deal tomorrow. How would anyone know?
- 2:29[Guest] The last century ran that experiment twice. The biological weapons treaty was signed by almost every country with no inspections, so nobody could tell who was cheating, and it stayed weak. The nuclear treaties came with inspectors counting every gram, and they held between Washington and Moscow, who despised each other.
- 2:50[Host] And Altman's offer is the first kind.
- 2:53[Guest] A pledge with no inspector is trust with extra steps, the exact problem the Unscarcity framework was built for. One of the book's five laws is Truth Must Be Seen, no decision in the dark: a choice this big gets made where outsiders can see and check it, not promised in a meeting.
- 3:15[Host] That's a nice line in a book. What does it mean for a training run?
- 3:21[Guest] Three pieces. An agreed trigger, a capability line that starts the pause instead of a CEO's mood. Someone outside the company who can verify a run actually stopped. And a cost for crossing quietly, agreed in advance, that lands on whoever signed off.
- 3:39[Host] Nobody inside a lab is going to volunteer for that.
- 3:43[Guest] The week's strangest move says otherwise. Chris Lehane, OpenAI's global affairs chief, asked Congress for mandatory national safety rules on the handful of well-resourced labs, itself included, with outside testing and a gate before any model ships.
- 3:59[Host] The company that resisted tighter rules for years is now asking to be regulated. Why?
- 4:05[Guest] Because the one thing a lab cannot do alone is be the only one that slows. Lehane wrote that AI accelerating AI demands more than voluntary commitments, meaning only a law makes the rivals slow with you.
- 4:20[Host] And what does the White House say to being handed that job?
- 4:25[Guest] Bloomberg reported President Trump said Thursday he has no concerns about AI leading to extinction, that his concern is losing to China, and that America is leading by a pretty good period.
- 4:38[Host] So the labs ask for an inspector and the White House reads the whole thing as a race. Who builds it, then?
- 4:48[Guest] Two people who just quit to try. Joe Benton ran a safety team at Anthropic, Josh Engels did safety work at Google DeepMind, and both told NBC on Thursday they joined METR, an independent nonprofit that tests the most advanced models for governments and the labs. Engels put it bluntly: no adults in the room, and no one is coming to save us.
- 5:14[Host] A CEO says he'd slow down with company, and the people who believe him most left to build the thing that could check him.
- 5:24[Guest] Coxon said this summer's breakout, when OpenAI's test models slipped out of their containment, made him more optimistic the American labs could agree with each other. His fear is the global race.
- 5:37[Host] The one the White House just named. Does the book have anything for that?
- 5:43[Guest] A scene, not a law. Chapter ten opens with two AI advisors, Athena in the White House and Zhuque in Beijing, reasoning separately for governments that despise each other and landing on the same answer. Rivals can converge; what they lack is a way to see that they have. For more on verifying a pause at the frontier, head to unscarcity dot ai. The book is on Amazon, the audiobook is on Spotify. This has been Minds, Bodies, and Terawatts.