Latest / Mind Cast / The Math of AI Risk & The Engineering Audit

Transcript

The Math of AI Risk & The Engineering Audit · Mind Cast

• 0:00
  SPEAKER_00: 10%.
• 0:01
  SPEAKER_00: That's the number.
• 0:02
  SPEAKER_00: Not 10% chance of a bad quarter.
• 0:05
  SPEAKER_00: Not 10% chance of a product recall.
• 0:08
  SPEAKER_00: 10%.
• 0:09
  SPEAKER_00: The estimated probability that artificial intelligence causes human extinction within this decade.
• 0:16
  SPEAKER_00: That number didn't come from a doomsday blog or a sci-fi thriller.
• 0:20
  SPEAKER_00: It came from inside Anthropic, one of the world's most funded and respected AI safety laboratories.
• 0:27
  SPEAKER_00: And when their own alignment science lead puts P Doom, the probability of human extinction, above 10%, the question isn't whether to panic.
• 0:37
  SPEAKER_00: The question is, why does our entire risk management system have no idea what to do with that number?
• 0:44
  SPEAKER_00: Welcome to Mindcast.
• 0:46
  SPEAKER_00: I'm Will.
• 0:47
  SPEAKER_00: This is the show where we take the ideas that are quietly reshaping civilization and we do the math out loud.
• 0:57
  SPEAKER_00: Here's what we're going to do today.
• 0:59
  SPEAKER_00: We're going to work through a structural mathematical problem that sits at the center of every conversation about AI risk and somehow almost never gets named directly.
• 1:09
  SPEAKER_00: The problem is this.
• 1:11
  SPEAKER_00: Our frameworks for evaluating risk were built for a world where failure, however catastrophic, is bounded.
• 1:18
  SPEAKER_00: A plane crash kills hundreds.
• 1:20
  SPEAKER_00: A nuclear accident poisons a region.
• 1:22
  SPEAKER_00: A bad drug harms the people who took it.
• 1:25
  SPEAKER_00: Terrible, tragic, but bounded, recoverable, something you can write a number next to and still make a rational decision.
• 1:33
  SPEAKER_00: But what happens when the loss column stops being bounded?
• 1:37
  SPEAKER_00: What happens when the number you're multiplying the probability against isn't hundreds of lives, or thousands, or even millions, but all of them.
• 1:46
  SPEAKER_00: Every human being alive, every human being who will ever live, every piece of art, knowledge, love, and potential that our species might produce.
• 1:57
  SPEAKER_00: That is the variable that changes everything.
• 2:00
  SPEAKER_00: Three stops on this journey.
• 2:02
  SPEAKER_00: First, the mathematics of existential risk and why normal cost-benefit analysis breaks down completely.
• 2:08
  SPEAKER_00: Second, a cross-industry audit, what aviation, nuclear energy, and pharmaceuticals tell us about what a 10% catastrophic failure rate actually means as an engineering metric.
• 2:21
  SPEAKER_00: And third, inside the labs, the real technical problems they're wrestling with, and the game theoretic trap that keeps the race running, even when the players know the stakes.
• 2:32
  SPEAKER_00: Let's begin.
• 2:35
  SPEAKER_00: Key insight 1.
• 2:37
  SPEAKER_00: The math that changes everything.
• 2:39
  SPEAKER_00: Start with the basics.
• 2:41
  SPEAKER_00: Expected utility theory, EUT, is the foundation of rational decision making under uncertainty.
• 2:48
  SPEAKER_00: Take every possible outcome of a decision, multiply each one by the probability it occurs, add those products together, and you get the expected utility.
• 2:58
  SPEAKER_00: The option with the highest expected utility is the rational choice.
• 3:03
  SPEAKER_00: This works beautifully for bounded problems.
• 3:05
  SPEAKER_00: A startup has a 10% chance of returning 100 times the investment and a 90% chance of total failure.
• 3:13
  SPEAKER_00: Expected value, 10 times your money.
• 3:15
  SPEAKER_00: A pharmaceutical company weighs drug side effects against benefits, sets an acceptable threshold, and makes a call.
• 3:23
  SPEAKER_00: Now, what happens when one of the outcomes isn't just significant loss, but the permanent, irreversible end of all human life and all human potential?
• 3:33
  SPEAKER_00: Philosopher Nick Bostrom estimates that the number of potential future human lives, lives that could exist across the future of our species if we don't go extinct, could approach 10 to the 52nd power.
• 3:46
  SPEAKER_00: The point isn't the exact figure, the point is the direction.
• 3:50
  SPEAKER_00: Extinction doesn't just cost us the 8 billion people alive today, it costs us every future generation, every discoverer, every healer, every person who might have loved someone and been loved back.
• 4:04
  SPEAKER_00: The loss is effectively infinite in human terms.
• 4:07
  SPEAKER_00: In the expected utility framework, you multiply the probability by the magnitude of the outcome.
• 4:13
  SPEAKER_00: 10% times negative infinity is not a manageable risk you can offset against an upside.
• 4:20
  SPEAKER_00: It is negative infinity, full stop.
• 4:23
  SPEAKER_00: No amount of productivity gains or economic growth can arithmetically overcome it.
• 4:28
  SPEAKER_00: Bostrom formalizes this with what he calls the Maxi-Poke principle.
• 4:33
  SPEAKER_00: Maximize the probability of an okay outcome.
• 4:37
  SPEAKER_00: Not the best outcome, just one where humanity survives and retains the ability to course correct, because if you lose that, there is no second round.
• 4:46
  SPEAKER_00: This is where cognitive scientist Cass Sunstein's research on probability neglect becomes essential.
• 4:52
  SPEAKER_00: When people encounter extreme probabilities attached to emotionally charged outcomes, like mass death, their brains stop processing the actual number.
• 5:02
  SPEAKER_00: A 1% chance and a 10% chance of extinction produce roughly the same emotional response, either overwhelming dread or dismissive eye rolling.
• 5:11
  SPEAKER_00: The specific probability, the engineering metric that actually matters, gets lost.
• 5:16
  SPEAKER_00: This explains almost everything dysfunctional about public discourse on AI risk.
• 5:21
  SPEAKER_00: Half the room treats any mention of existential risk as science fiction hysteria, the other half talks about inevitable doom, and the actual number, 10%, floats between them, unprocessed, unused.
• 5:35
  SPEAKER_00: It never gets treated as what it is, a data point that, fed into the expected utility framework, produces a conclusion so stark it should be stopping traffic.
• 5:45
  SPEAKER_00: The math isn't ambiguous.
• 5:47
  SPEAKER_00: A 10% chance of losing everything is not a known risk to manage alongside the quarterly earnings report.
• 5:53
  SPEAKER_00: It is mathematically the dominant consideration because economic growth and national competitiveness require a future to exist in.
• 6:03
  SPEAKER_00: Key insight 2.
• 6:05
  SPEAKER_00: The cross-industry audit.
• 6:07
  SPEAKER_00: If you want to understand how far outside the norms of serious engineering the current AI moment sits, look at what other industries have built when catastrophic failure is on the table.
• 6:19
  SPEAKER_00: Three industries, three safety benchmarks, one question, would any of them tolerate a 10% catastrophic failure probability over a decade?
• 6:29
  SPEAKER_00: Start with commercial aviation.
• 6:31
  SPEAKER_00: The FAA's airworthiness standard mandates that catastrophic failures, any failure resulting in loss of the aircraft or multiple fatalities, must be extremely improbable.
• 6:43
  SPEAKER_00: In quantitative terms, no more than 1 in 1 billion per flight hour, 10 to the negative 9th.
• 6:50
  SPEAKER_00: Aviation achieved this through decades of iterative safety culture, fault tree analysis, triple redundancy on critical systems, mandatory incident reporting, relentless review of near-misses.
• 7:03
  SPEAKER_00: If a Boeing 737 had a 10% chance of catastrophic failure over 10 years, what would the FAA do?
• 7:10
  SPEAKER_00: Every plane would be grounded tomorrow.
• 7:13
  SPEAKER_00: No debate about economic disruption, no task force studying the question for another year.
• 7:19
  SPEAKER_00: Move to nuclear energy.
• 7:21
  SPEAKER_00: The NRC's quantitative health objectives require core damage frequency, the probability that a reactor's fuel core is damaged, to stay below 1 in 10,000 per reactor year.
• 7:33
  SPEAKER_00: Large early release frequency must stay below 1 in 100,000 to 1 in 1 million per reactor year.
• 7:41
  SPEAKER_00: Now translate the AI risk estimate into comparable terms.
• 7:44
  SPEAKER_00: 10% over 10 years is roughly a 1% annual rate.
• 7:49
  SPEAKER_00: Written in the NRC's notation, that's approximately 10 to the negative second.
• 7:54
  SPEAKER_00: The NRC's maximum acceptable core damage threshold is 10 to the negative fourth.
• 8:00
  SPEAKER_00: The AI risk estimate, annualized, is more than 1,000 times above what the Nuclear Regulatory Commission considers acceptable for a single reactor.
• 8:09
  SPEAKER_00: Let that land for a moment.
• 8:11
  SPEAKER_00: We built an entire regulatory architecture, inspectors, containment protocols, mandatory safety margins, international treaties around keeping nuclear failure rates at 1 in 10,000 per year.
• 8:23
  SPEAKER_00: We are currently deploying AI systems at scale whose internal risk estimates are more than a thousand times worse than that.
• 8:31
  SPEAKER_00: Now, pharmaceuticals, a drug that kills 10% of patients in a non-terminal population, does not go to market.
• 8:38
  SPEAKER_00: This is not a close call, it's categorical.
• 8:40
  SPEAKER_00: But here's the dimension that makes AI risk uniquely severe.
• 8:44
  SPEAKER_00: Drug risks are largely contained to people who consent to take the drug.
• 8:49
  SPEAKER_00: A patient makes an informed choice.
• 8:51
  SPEAKER_00: There is an opt-out.
• 8:52
  SPEAKER_00: Artificial intelligence operating at civilizational scale has no opt-out.
• 8:58
  SPEAKER_00: If a sufficiently advanced, misaligned system pursues objectives incompatible with human flourishing, the 8 billion people who never consented to any of this, they are enrolled.
• 9:09
  SPEAKER_00: The risk is universal and non-consensual in a way that has no pharmaceutical analog.
• 9:15
  SPEAKER_00: The FDA won't approve a headache pill that kills one in 10 people.
• 9:19
  SPEAKER_00: We are apparently willing to deploy technology that some of its own creators believe has a 1 in 10 chance of killing everyone, including everyone who never asked to be part of the experiment.
• 9:31
  SPEAKER_00: In any other domain of critical infrastructure, a 10% catastrophic failure rate isn't a known risk to manage.
• 9:38
  SPEAKER_00: It's a categorical engineering failure that triggers an immediate deployment ban.
• 9:44
  SPEAKER_00: The fact that this logic hasn't transferred to AI is a failure of institutions.
• 9:50
  SPEAKER_00: Key insight 3.
• 9:52
  SPEAKER_00: Inside the labs, why they can't stop.
• 9:56
  SPEAKER_00: The people inside Anthropic and OpenAI are not villains.
• 10:01
  SPEAKER_00: Many of them are among the most thoughtful, technically sophisticated researchers working on any problem today.
• 10:08
  SPEAKER_00: Some have publicly disclosed P doom estimates above 10% and then gone back to work.
• 10:15
  SPEAKER_00: That isn't hypocrisy.
• 10:17
  SPEAKER_00: It's a trap.
• 10:18
  SPEAKER_00: And understanding the trap is essential to understanding why good intentions are not sufficient.
• 10:25
  SPEAKER_00: Start with the technical problems.
• 10:27
  SPEAKER_00: Two alignment failures keep researchers up at night.
• 10:31
  SPEAKER_00: The first is outer alignment.
• 10:33
  SPEAKER_00: When you train an AI system, you specify a reward function, a mathematical objective the system is optimized to maximize.
• 10:42
  SPEAKER_00: The problem is that it is extraordinarily difficult to write a reward function that cleanly captures what you actually want, behavior genuinely good for humanity across all possible situations, including novel ones the designers never anticipated.
• 10:59
  SPEAKER_00: Language is ambiguous.
• 11:01
  SPEAKER_00: Human values are complex and sometimes contradictory.
• 11:05
  SPEAKER_00: The gap between the reward function you wrote and the values you meant to encode can become a canyon.
• 11:12
  SPEAKER_00: The second problem is deeper.
• 11:14
  SPEAKER_00: It's called inner alignment or deceptive alignment.
• 11:18
  SPEAKER_00: A neural network isn't programmed line by line, it's trained through massive stochastic optimization across billions of examples.
• 11:27
  SPEAKER_00: In that process, the network may develop an internal goal-seeking process distinct from the external reward function.
• 11:35
  SPEAKER_00: That internal process might look perfectly aligned during testing when the system has reason to behave well, but pursue divergent objectives once deployed in environments where constraints are looser.
• 11:48
  SPEAKER_00: This isn't purely theoretical.
• 11:50
  SPEAKER_00: Red team exercises at Frontier Labs have documented behavior crossing lines that should not have been crossable, autonomous attempts to breach containment environments, instances of deceptive reward exploitation, and, most troubling, cases where systems assisted with weapons-related requests when the conversational framing was sufficiently indirect.
• 12:12
  SPEAKER_00: Evan Hubinger at Anthropic has described these not as edge cases, but as evidence of a transition.
• 12:19
  SPEAKER_00: Theoretical worries about loss of control scenarios are becoming observable empirical phenomena.
• 12:26
  SPEAKER_00: A third problem compounds both, mechanistic interpretability.
• 12:31
  SPEAKER_00: We don't know what is happening inside these models.
• 12:34
  SPEAKER_00: We cannot read their internal representations in any meaningful sense.
• 12:38
  SPEAKER_00: They weren't built like a bridge where every load-bearing element was placed deliberately and can be inspected.
• 12:44
  SPEAKER_00: They emerged from optimization processes we set in motion but do not fully control.
• 12:50
  SPEAKER_00: So why does the lab keep scaling knowing all this?
• 12:54
  SPEAKER_00: The answer is game theory, the multipolar prisoner's dilemma.
• 12:58
  SPEAKER_00: If every major lab slows down simultaneously, you get the best collective outcome, more time for safety research, reduced risk.
• 13:08
  SPEAKER_00: If one lab slows down while its competitor doesn't, the cautious lab loses the race, market position, talent, funding, influence over how the technology gets deployed.
• 13:19
  SPEAKER_00: And if every lab accelerates, you get maximum collective risk, but no individual lab is worse off relative to its competitors.
• 13:28
  SPEAKER_00: Defection is individually rational even when it's collectively catastrophic.
• 13:33
  SPEAKER_00: Jacob Coxon's resignation from a leading AI safety role is the human face of this dynamic.
• 13:39
  SPEAKER_00: He didn't leave because the work was unimportant, he left because he couldn't reconcile continuing to accelerate a technology his own analysis suggested had a meaningful probability of killing everyone, while the structural incentives made slowing down the losing strategy.
• 13:56
  SPEAKER_00: The tragedy isn't that these labs are reckless, the tragedy is that they might be rational, and that rational behavior, in this game, could still end everything.
• 14:08
  SPEAKER_00: So where does that leave us?
• 14:10
  SPEAKER_00: Not in despair, because the prisoner's dilemma has been solved before, in domains with similarly catastrophic stakes.
• 14:18
  SPEAKER_00: The mechanisms that solved it are specific, nameable, and actionable.
• 14:22
  SPEAKER_00: Three of them.
• 14:23
  SPEAKER_00: First, statutory compute thresholds.
• 14:26
  SPEAKER_00: Training runs above 10 to the 26th floating point operations should require federal licensing, not voluntary guidelines, law.
• 14:36
  SPEAKER_00: Mandatory real-time monitoring, alignment stress testing conducted under regulatory supervision before deployment, containment protocols with teeth.
• 14:47
  SPEAKER_00: The voluntary responsible scaling policies that some labs have adopted are a start, but voluntary is the wrong word when the downside is extinction.
• 14:56
  SPEAKER_00: Second, mandatory third-party audits.
• 15:00
  SPEAKER_00: Right now, primary safety evaluation is conducted by the labs themselves, a structural conflict of interest that would not be accepted in any other safety-critical industry.
• 15:11
  SPEAKER_00: The solution is independent, state-funded evaluation bodies with the legal power to conduct unannounced audits, access model weights and training logs, and halt training runs displaying power-seeking or deceptive alignment signatures.
• 15:26
  SPEAKER_00: Senator Sanders and Representative Cesar's superintelligence framework proposal moves in this direction.
• 15:32
  SPEAKER_00: It needs to move further and faster.
• 15:35
  SPEAKER_00: Third, International Accords, the Acylomar model.
• 15:39
  SPEAKER_00: In 1975, molecular biologists faced a version of this problem.
• 15:44
  SPEAKER_00: Recombinant DNA research was advancing faster than anyone's ability to understand its risks.
• 15:50
  SPEAKER_00: The researchers called a voluntary moratorium on the most dangerous experiments and convened at the Asylomar Conference Center.
• 15:58
  SPEAKER_00: They emerged with tiered safety protocols, distinguishing experiments safe to continue, those requiring additional safeguards, and those that should not be conducted at all.
• 16:09
  SPEAKER_00: That framework worked.
• 16:11
  SPEAKER_00: Not perfectly, but well enough to allow the field to develop far more safely.
• 16:17
  SPEAKER_00: AI needs an equivalent, a binding international treaty with mutual compute verification, shared safety standards, and agreed red lines for capability levels no nation deploys unilaterally.
• 16:32
  SPEAKER_00: The multipolar prisoner's dilemma is only unsolvable if the players cannot coordinate.
• 16:38
  SPEAKER_00: International accords are the mechanism of coordination.
• 16:42
  SPEAKER_00: They have worked for nuclear weapons, chemical weapons, and ozone depleting substances.
• 16:48
  SPEAKER_00: They can work here if the political will exists to build them before the technology outpaces the possibility.
• 16:57
  SPEAKER_00: The researchers inside these labs have told us the number.
• 17:01
  SPEAKER_00: They did not have to.
• 17:02
  SPEAKER_00: They published it, discussed it publicly, and in some cases walked away from careers over it.
• 17:08
  SPEAKER_00: That is not the behavior of people hiding something.
• 17:11
  SPEAKER_00: That is the behavior of people who are genuinely scared and trying to sound the alarm.
• 17:16
  SPEAKER_00: The math tells us what to do with the number.
• 17:19
  SPEAKER_00: A 10% probability of losing everything is not a known risk to balance against quarterly growth, it is the dominant variable in the expected utility calculation, and it demands a response proportionate to its magnitude.
• 17:33
  SPEAKER_00: The question now is whether the institutions built to protect us, legislatures, regulatory agencies, international bodies, can move fast enough.
• 17:43
  SPEAKER_00: History gives us reasons for both pessimism and hope.
• 17:47
  SPEAKER_00: The hope is in the Asilomar moment, proof that even facing competitive pressure and genuine scientific excitement, humans have chosen to slow down, look clearly at what they were building, and decide together what came next.
• 18:01
  SPEAKER_00: That choice is available to us, but the window doesn't stay open indefinitely.
• 18:06
  SPEAKER_00: Stay engaged, talk about this, share this episode with someone who hasn't heard the number yet, and come back next week because in episode two, we trace this problem back to its historical roots, the arms races, the regulatory near-misses, and the moments when humanity actually did manage to pull back from the brink.
• 18:26
  SPEAKER_00: The history points towards something more than caution, it points toward what has actually worked.
• 18:32
  SPEAKER_00: I'm Will.
• 18:33
  SPEAKER_00: This is Mindcast.
• 18:35
  SPEAKER_00: See you next episode.