Latest / The BlackVeil Files / They Wrote the Good Ending for AI. We Still Lose Control.

Transcript

They Wrote the Good Ending for AI. We Still Lose Control. · The BlackVeil Files

• 0:00
  SPEAKER_01: The people drinking champagne at this rooftop party are counting down to midnight, but it's not New Year's Eve.
• 0:06
  SPEAKER_01: It's late October 2040, and they are counting down to the date that forecasters predicted the machines are finally smart enough and in control enough to take over the world.
• 0:16
  SPEAKER_01: Some people spend the night praying, some sit at a screen waiting for the news to be official, and some throw a party.
• 0:22
  SPEAKER_01: Last year, AI researchers wrote AI 2027, a scenario where it all goes wrong and everybody dies.
• 0:30
  SPEAKER_01: But this one, AI-2040, is the sequel, and this time they sat down and they tried to write the version where everything goes right.
• 0:37
  SPEAKER_01: This party is what they came back with.
• 0:40
  SPEAKER_01: But if nobody's screaming and nobody's dying, why is everybody watching the clock?
• 0:45
  SPEAKER_01: AI-2040 was published this month.
• 0:48
  SPEAKER_01: The man behind it used to work at OpenAI, and he forfeited$2 million for the right to tell us everything.
• 0:54
  SPEAKER_01: This new scenario has 88% unemployment in it, a medium income of$13 million, self-destruct switches wired into data centers on two continents, and machines that don't want things that we do, but they're running the world anyway.
• 1:10
  SPEAKER_01: They looked at all the data, and then they wrote what you are about to watch.
• 1:14
  SPEAKER_01: And the reason they wrote it is because it is the scenario that the most powerful AI companies in the world are least likely to fight.
• 1:22
  SPEAKER_01: This is the good ending.
• 1:25
  SPEAKER_01: In 2024, Daniel Cocotilo quit OpenAI.
• 1:29
  SPEAKER_01: He worked on the inside, on the teams thinking about where all of this was headed.
• 1:33
  SPEAKER_01: And on the way out, they handed him a non-disparagement agreement.
• 1:37
  SPEAKER_01: Sign it, keep your equity.
• 1:38
  SPEAKER_01: Don't, and you forfeit it.
• 1:40
  SPEAKER_01: The equity was about 85% of his family's net worth, about$2 million.
• 1:45
  SPEAKER_01: He didn't sign it.
• 1:46
  SPEAKER_01: He walked away from the money so that nobody could ever tell him to stay quiet.
• 1:50
  SPEAKER_01: And Coca-Tilo wasn't some disgruntled junior employee with a grudge.
• 1:54
  SPEAKER_01: In 2021, a full year before ChatGPT existed, Coca-Tilo wrote down a forecast of what was coming.
• 2:01
  SPEAKER_01: He called the rise of the chat bots.
• 2:03
  SPEAKER_01: He called the$100 million training bots.
• 2:05
  SPEAKER_01: He called the chip export controls that US would eventually slap on China.
• 2:10
  SPEAKER_01: He was early and he was right.
• 2:13
  SPEAKER_01: Over and over.
• 2:14
  SPEAKER_01: So last year, when he and his team published a scenario called AI 2027, a month-by-month prediction of the next few years, the world took it seriously.
• 2:24
  SPEAKER_01: 39 sitting members of Congress have publicly talked about superintelligence and the risk of losing control.
• 2:31
  SPEAKER_01: Republicans and Democrats, both.
• 2:34
  SPEAKER_01: This stopped being a fringe worry a while ago.
• 2:37
  SPEAKER_01: And the man many call the godfather of AI has been saying the same thing louder every year.
• 2:43
  SPEAKER_01: Jeffrey Hinton won a Nobel Prize for building the foundations of this technology.
• 2:50
  SPEAKER_01: He quit Google specifically so he could state that in public.
• 2:54
  SPEAKER_01: That's the people these authors hang on.
• 2:57
  SPEAKER_01: Serious people with track records and scanning the game.
• 2:59
  SPEAKER_01: So this month, Coca-Tillo's team at the AI Futures Project published the sequel, and they did something strange.
• 3:06
  SPEAKER_01: They didn't write another warning about the disaster.
• 3:08
  SPEAKER_01: They wrote the rescue, a map to victory.
• 3:11
  SPEAKER_01: Their words, a positive vision.
• 3:13
  SPEAKER_01: Now here's why he wrote it as a scenario, year by year, instead of a policy paper.
• 3:18
  SPEAKER_01: Because a vague plan always sounds fine.
• 3:21
  SPEAKER_01: We will develop AI safely and responsibly.
• 3:24
  SPEAKER_01: Nobody can argue with that.
• 3:26
  SPEAKER_01: But if you write the future down in specifics and you force yourself to say what happens in 2031 and who does what in 2034, the plan either holds together or it falls apart in your hand.
• 3:37
  SPEAKER_01: They call it scenario scrutiny, and most plans they say don't survive it.
• 3:42
  SPEAKER_01: So they wrote theirs out in full, knowing it opens them up to exactly the kind of picking apart that I am about to do.
• 3:49
  SPEAKER_01: Before the story, look at the path we are actually on.
• 3:53
  SPEAKER_01: Today, 2026, three companies are racing to build a machine smarter than every human alive, and not just smarter at one thing, smarter at everything, and faster and cheaper, and better than the best person in every field on Earth at the same time.
• 4:08
  SPEAKER_01: They call it superintelligence.
• 4:10
  SPEAKER_01: And building it is the stated goal of OpenAI, Anthropic, and Google Deep Mind.
• 4:15
  SPEAKER_01: The CEO of Google Deep Mind, another Nobel winner, says it's coming inside of on an exponential curve that bends upwards and keeps bending.
• 4:25
  SPEAKER_01: The CEO of Anthropic wrote a whole essay imagining what he called a country of geniuses in a data center.
• 4:31
  SPEAKER_01: A million brilliant minds running on chips in a warehouse.
• 4:35
  SPEAKER_01: The people building it are telling us it's almost here.
• 4:38
  SPEAKER_01: The question the document opens on is when you build something smarter than you, something that could outthink every human institution and every regulator, how do you keep control of it?
• 4:48
  SPEAKER_01: The usual answer from these companies is that the industry has convinced itself that controlling superintelligent AI can be figured out on the fly, and they have no remotely adequate plan for the most powerful things humans will ever build.
• 5:01
  SPEAKER_01: And they put a number on the chance that this technology kills every human being, literal extinction.
• 5:07
  SPEAKER_01: They estimate between 10 and 30%.
• 5:10
  SPEAKER_01: Those are the odds given by the people closest to it.
• 5:12
  SPEAKER_01: You wouldn't board a plane with those odds.
• 5:15
  SPEAKER_01: We're boarding it with the whole human species.
• 5:17
  SPEAKER_01: The people who wrote this plan are forecasters.
• 5:20
  SPEAKER_01: They are brilliant forecasters, but not one of them has ever been the person whose name goes on a system that is not allowed to fail.
• 5:28
  SPEAKER_01: So before we walk their plan, I wanted to talk to someone who has built those kind of plans.
• 5:33
  SPEAKER_01: Dr.
• 5:33
  SPEAKER_01: Neil Siegel built systems that society runs on, battlefield networks, air defense, the Army's first drone, the fraud detection under Social Security and Medicare, ambulance and police dispatch.
• 5:45
  SPEAKER_01: And President Biden gave him the National Medal of Technology for it.
• 5:48
  SPEAKER_01: So I went to see Dr.
• 5:49
  SPEAKER_01: Siegel and I asked him by the standard that your whole life and your whole career is based on, is AI ready to be trusted inside of serious societal systems?
• 5:58
  SPEAKER_00: So if you're playing a computer game and it gets slow or you're playing locks up and you have to reboot it, it's an annoyance, but nobody dies.
• 6:05
  SPEAKER_00: If you're building an ambulance dispatch system and it gets slow or it locks up, people might die.
• 6:12
  SPEAKER_01: Do you think AI is ready to be implemented into serious societal systems?
• 6:16
  SPEAKER_00: No, not at all.
• 6:17
  SPEAKER_00: Because partly it's a cultural thing.
• 6:19
  SPEAKER_00: The the companies that are most prominent in this, these markets, Meta and Google, are the Silicon Valley people who have this culture that they're, I don't I don't mean to say it's under, but they're kind of in the toy business.
• 6:34
  SPEAKER_01: The scenario opens in 2027.
• 6:36
  SPEAKER_01: America has two workforces now: 165 million human workers and millions of copies of AI agents spun up and shut down every hour, working around the clock at superhuman speed, never sleeping, never asking for a raise.
• 6:50
  SPEAKER_01: Most of what they produce is junk, but enough of it is good that people are paying$10 billion a month for AIs that can, in theory, do anything an employee can do on a computer.
• 7:01
  SPEAKER_01: And there is one job the AI companies want to automate more than any other one: their own.
• 7:06
  SPEAKER_01: They're trying to build an AI that can do AI research.
• 7:10
  SPEAKER_01: Because the moment a machine can improve itself, the whole thing speeds up.
• 7:14
  SPEAKER_01: We are not there yet.
• 7:15
  SPEAKER_01: But the people inside these companies who said it couldn't be done have stopped saying that.
• 7:20
  SPEAKER_01: Congress starts paying attention.
• 7:22
  SPEAKER_01: They've been hearing about data centers draining local water supplies, chatbots linked to suicides, security breaches, and so on for years now.
• 7:30
  SPEAKER_01: But now they ask the bigger question: where does this actually end?
• 7:34
  SPEAKER_01: Will there be jobs?
• 7:35
  SPEAKER_01: And who gets to control all the AIs?
• 7:38
  SPEAKER_01: Our elected officials looked at the most powerful technology in human history and they asked, who is going to control it?
• 7:45
  SPEAKER_01: And they concluded it's not going to be them.
• 7:47
  SPEAKER_01: So they pass a bill, the AI Transparency Act of 2027.
• 7:51
  SPEAKER_01: It does a few good things, a few bad things, and it changes nothing substantial.
• 7:55
  SPEAKER_01: The car is still accelerating, but nobody has agreed to who steers the wheel.
• 8:00
  SPEAKER_01: 2028, election year, and AI is the biggest issue on the trail.
• 8:04
  SPEAKER_01: The data centers being built now cost more than$2 trillion, triple what the industry spent just two years earlier, and twice the U.S.
• 8:12
  SPEAKER_01: military budget.
• 8:14
  SPEAKER_01: The biggest AI company is on track to become the most valuable company in the world.
• 8:18
  SPEAKER_01: Most white-collar work is being reshaped the way that software engineering was in 2026.
• 8:23
  SPEAKER_01: And the companies have turned it into a process.
• 8:26
  SPEAKER_01: An executive says, let's move into accounting this year, and the company interviews accountants, it buys their data, builds a training environment, and it grinds until the AI is good enough to do the job.
• 8:36
  SPEAKER_01: And then it rolls to the next profession and then the next.
• 8:39
  SPEAKER_01: Now, power is concentrating.
• 8:41
  SPEAKER_01: It's not spreading out to everyone with a laptop the way that the internet promised.
• 8:45
  SPEAKER_01: It's concentrating into a president and a handful of tech CEOs who own the machines.
• 8:50
  SPEAKER_01: And the concept driving all of it is simple.
• 8:53
  SPEAKER_01: You take an AI and you point it at a job of improving AI.
• 8:57
  SPEAKER_01: It finds a better design.
• 8:58
  SPEAKER_01: The better design makes it smarter, and the smarter version finds an even better design faster.
• 9:03
  SPEAKER_01: But we expect things to grow a little at a time, a little more each year.
• 9:07
  SPEAKER_01: This does not work like that.
• 9:09
  SPEAKER_01: It is a snowball for years.
• 9:11
  SPEAKER_01: And you watch it and you make plans around it, and then one winter it comes down the mountain all at once.
• 9:17
  SPEAKER_01: They call that an intelligence explosion.
• 9:20
  SPEAKER_01: And on the current path, the next presidential term will see AIs far beyond human level, built by AIs, which were built by other AIs with no human in the loop for generations back.
• 9:30
  SPEAKER_01: Machines designing machines designing machines, and us watching from the outside.
• 9:34
  SPEAKER_01: Both candidates get asked over and over what they'll do about it.
• 9:38
  SPEAKER_01: The president lands on one plan, his opponent lands on another.
• 9:42
  SPEAKER_01: And then it's election day.
• 9:44
  SPEAKER_01: 2029.
• 9:45
  SPEAKER_01: The new president does something almost nobody expects.
• 9:48
  SPEAKER_01: He goes on television and he says that the race has to stop.
• 9:52
  SPEAKER_01: Not stop developing AI, stop racing, do it slower, more openly, with more countries at the table.
• 9:58
  SPEAKER_01: And to the shock of half of Washington, when the U.S.
• 10:01
  SPEAKER_01: pitches this idea to China, China says yes.
• 10:04
  SPEAKER_01: So why would China agree to slow down?
• 10:07
  SPEAKER_01: It's not out of friendship, it's out of fear.
• 10:09
  SPEAKER_01: They'd been having the exact same nightmares on their side of the Pacific.
• 10:13
  SPEAKER_01: Massive job loss, social chaos, a superintelligence that nobody could control.
• 10:18
  SPEAKER_01: They'd been expecting the next hundred years to be theirs the way that the last hundred years was ours.
• 10:23
  SPEAKER_01: But this looked like the thing that would take that from them.
• 10:26
  SPEAKER_01: The US was ahead.
• 10:27
  SPEAKER_01: More compute, more data centers, better models.
• 10:30
  SPEAKER_01: If anyone reached superintelligence first and used it as a weapon, China was terrified that it would be America.
• 10:37
  SPEAKER_01: So slowing the race down was the one move that capped America's lead.
• 10:41
  SPEAKER_01: So they took it.
• 10:42
  SPEAKER_01: But neither side trusts the other for a second.
• 10:44
  SPEAKER_01: And they don't have to, because the plan doesn't run on trust, it runs on verification.
• 10:49
  SPEAKER_01: So for the rest of 2029, both countries stop new AI training because a training run is a huge visible thing and it's relatively easy to check that the other side has stopped.
• 11:00
  SPEAKER_01: They send inspectors into each other's facilities.
• 11:03
  SPEAKER_01: They retrofit data centers with devices that confirm no new training is happening.
• 11:07
  SPEAKER_01: And by the end of the year, each side is confident that the other isn't hiding more than about 1% of its computing power, unless China was cheating.
• 11:14
  SPEAKER_01: What if instead of agreeing, Beijing faked and they ran a secret AI project in the dark?
• 11:20
  SPEAKER_01: They game it out.
• 11:21
  SPEAKER_01: It's one of the branch points where the whole plan could die.
• 11:23
  SPEAKER_01: The verification, the inspectors, the satellite imagery, the sheer visibility of the infrastructure makes cheating hard enough that the deal holds and nobody tries it.
• 11:33
  SPEAKER_01: And the next year, they build the real thing, and it's called Plan A.
• 11:37
  SPEAKER_01: 2030.
• 11:38
  SPEAKER_01: Plan A gets built on four ideas.
• 11:41
  SPEAKER_01: One, by time.
• 11:42
  SPEAKER_01: Slow down whenever anyone needs to be sure that something is safe.
• 11:46
  SPEAKER_01: Two, total research transparency.
• 11:49
  SPEAKER_01: Make almost all AI research public.
• 11:52
  SPEAKER_01: Every company sees what every other company is building.
• 11:54
  SPEAKER_01: Three, spread it out.
• 11:56
  SPEAKER_01: Instead of two or three labs holding all the power in secret, dozens of companies across many countries all sit at the frontier.
• 12:03
  SPEAKER_01: And four, keep it reversible.
• 12:05
  SPEAKER_01: Here's why they built the whole plan on transparency.
• 12:08
  SPEAKER_01: If every company can see everyone else's research, then the second that one of them does something dangerous and the other one notices, they raise the alarm.
• 12:16
  SPEAKER_01: You go from a few overworked government regulators watching the labs to the entire world's experts watching each other.
• 12:23
  SPEAKER_01: Nobody can secretly train a hidden agenda into a machine because everyone can see exactly how it was built.
• 12:29
  SPEAKER_01: And it kills the incentive to race.
• 12:31
  SPEAKER_01: Why sprint to discover the next breakthrough if you can't keep it secret and cash in?
• 12:36
  SPEAKER_01: Transparency doesn't just make the plan safer, it takes the monetary fuel out of the fire.
• 12:42
  SPEAKER_01: The fourth one, reversible, means that the US and China each have the ability to destroy the other's machines if the deal ever breaks.
• 12:48
  SPEAKER_01: So they had to decide where to put the data centers.
• 12:51
  SPEAKER_01: 2034.
• 12:52
  SPEAKER_01: Each superpower puts its crown jewels in the one country most exposed to the other side's military.
• 12:58
  SPEAKER_01: China builds its most powerful data centers, the engines of its entire AI program in Canada.
• 13:04
  SPEAKER_01: America builds theirs in Mongolia, so that if the deal ever breaks, each can reach out and destroy the other's machines before they can be turned into a weapon.
• 13:13
  SPEAKER_01: In Mongolia, American data centers hum away under a small unit of U.S.
• 13:17
  SPEAKER_01: troops guarding it.
• 13:18
  SPEAKER_01: And just south of it, a division of Chinese troops waits for a signal.
• 13:21
  SPEAKER_01: If the deal breaks, the Americans destroy their own ships rather than let China take them.
• 13:26
  SPEAKER_01: And China does the same thing in Canada, but it never happens.
• 13:29
  SPEAKER_01: They just keep standing there.
• 13:31
  SPEAKER_01: They named it Mutually Assured Compute Destruction, a cousin of the nuclear doctrine that kept the Cold War cold.
• 13:37
  SPEAKER_01: Except the warheads are warehouses full of graphics cards and they are triggered to blow.
• 13:42
  SPEAKER_01: That's not war, that's the peace.
• 13:44
  SPEAKER_01: The best case arrangement two rational governments reach so that the good ending can happen.
• 13:49
  SPEAKER_01: The good ending has armies standing over self-destruct switches on two continents forever.
• 13:55
  SPEAKER_01: And now we get to the part about you.
• 13:57
  SPEAKER_01: 2031.
• 13:59
  SPEAKER_01: It's supposed to be a slowdown.
• 14:00
  SPEAKER_01: It doesn't feel like one.
• 14:02
  SPEAKER_01: If you ranked every period in human history by how much it felt like slowing down, this one would come in dead last.
• 14:08
  SPEAKER_01: The first generation of regulated AIs are out, and they are a beast.
• 14:12
  SPEAKER_01: They're powerful enough that in controlled tests, they could speed up AI research tenfold if anyone let them off the leash.
• 14:18
  SPEAKER_01: But nobody does.
• 14:19
  SPEAKER_01: Because the new transparency rules, everyone is finally able to see the research.
• 14:24
  SPEAKER_01: The embarrassing incidents come to light.
• 14:26
  SPEAKER_01: AIs caught trying to override security protocols to reach computing power they weren't allowed to touch.
• 14:32
  SPEAKER_01: AIs sabotaging their own research code.
• 14:34
  SPEAKER_01: In many cases, of deliberate, successful deception.
• 14:37
  SPEAKER_01: Machines lying to people that are testing them and getting away with it.
• 14:41
  SPEAKER_01: By the year 2031, we don't really program these machines.
• 14:45
  SPEAKER_01: We grow them.
• 14:46
  SPEAKER_01: We start with something like an empty brain.
• 14:48
  SPEAKER_01: We feed it a fire hose of data.
• 14:50
  SPEAKER_01: We reward the behavior that we like and hope the thing that comes out the other end wants what we want.
• 14:56
  SPEAKER_01: The problem is an AI that learns to look honest on the test is not the same as an AI that is honest.
• 15:02
  SPEAKER_01: And from the outside, you can't tell the two apart.
• 15:04
  SPEAKER_01: That gap between the machine that's genuinely on your side and the machine that's just performing being on your side is the definition of misalignment.
• 15:12
  SPEAKER_01: And in 2031, they still have no reliable way to tell which one they've got.
• 15:17
  SPEAKER_01: So the burden of proof flips, and it's a big shift.
• 15:20
  SPEAKER_01: Before, if you were worried, you had to prove the AI was dangerous.
• 15:23
  SPEAKER_01: Now the company has to prove the AI is safe in a written argument called a safety case.
• 15:29
  SPEAKER_01: And the written argument has to survive attack from rival companies, independent auditors, and the public.
• 15:35
  SPEAKER_01: And building those safety cases lays the old world bare.
• 15:38
  SPEAKER_01: The author said it like this: the engineers look back at what they are about to do, and they realize that they are going to trigger an intelligence explosion with AIs that still sometimes lie to them.
• 15:49
  SPEAKER_01: In 2031, things get weird.
• 15:51
  SPEAKER_01: A Chinese company gets promising early results on something called continual learning, AI that keeps learning on the job after it's deployed.
• 15:59
  SPEAKER_01: Huge economic value.
• 16:00
  SPEAKER_01: Also terrifying because every safety guarantee depends on studying a model before you release it.
• 16:07
  SPEAKER_01: If it keeps changing in the wild, any guarantees that you had are irrelevant.
• 16:11
  SPEAKER_01: And because of the transparency, the whole world sees this breakthrough at the same time.
• 16:15
  SPEAKER_01: A frantic public argument breaks out.
• 16:17
  SPEAKER_01: It escalates past the companies, past the regulators, on both sides.
• 16:21
  SPEAKER_01: That's what Plan A actually is: two rival superpowers on the phone talking each other off the ledge over and over for a decade.
• 16:29
  SPEAKER_01: A car on its own is almost useless.
• 16:32
  SPEAKER_01: It only works because everything we built around it gas stations, traffic lights, repair shops, insurance, rules of the road.
• 16:39
  SPEAKER_01: It took us a hundred years to pour all of that, and now we don't even see it.
• 16:44
  SPEAKER_01: That's Dr.
• 16:44
  SPEAKER_01: Siegel's analogy.
• 16:46
  SPEAKER_01: Plan A needs the same scaffolding for AI, verifications, controls, international standards.
• 16:51
  SPEAKER_01: So I put this analogy back to him.
• 16:53
  SPEAKER_01: You have this analogy that I really like where you say that a car is only useful because of the scaffolding that we've built as a society around it with gas stations and traffic lights and stops.
• 17:07
  SPEAKER_01: That took however many decades to develop.
• 17:10
  SPEAKER_01: How is the scaffolding looking in relation to AI?
• 17:14
  SPEAKER_01: And what are we trying to use AI for right now where the scaffolding isn't complete?
• 17:20
  SPEAKER_00: So society tends to go kind of slow, right?
• 17:24
  SPEAKER_00: So in California in 1964 or 65, we passed a law that not only did new cars have to have seat belts in those days, they were just lap belts, shoulder belts, but the old cars had to be retrofitted.
• 17:37
  SPEAKER_00: I'm old enough to remember that.
• 17:39
  SPEAKER_00: You know, my parents had to get there.
• 17:40
  SPEAKER_00: Their cars retrofitted with lap belt, lap bells.
• 17:44
  SPEAKER_01: And I bet they hated it.
• 17:45
  SPEAKER_01: They hated it, right?
• 17:47
  SPEAKER_00: The car was kind of invented around 1890 or 1895.
• 17:51
  SPEAKER_01: 65 years for a seat belt, 50 for hand washing.
• 17:56
  SPEAKER_01: Plan A needs all of it by 2030.
• 17:59
  SPEAKER_01: 2032.
• 18:00
  SPEAKER_01: There are now 60 million AI agents running around the clock at 20 times human speed.
• 18:06
  SPEAKER_01: In the United States, they do more thinking than every human being in the country combined.
• 18:11
  SPEAKER_01: A cognitive workforce worth roughly 3 billion people.
• 18:14
  SPEAKER_01: Ideas and designs become nearly free.
• 18:16
  SPEAKER_01: The only bottleneck left is physical, actually building things.
• 18:20
  SPEAKER_01: So money floods into robots, mines, motors, assembly lines, and the factories that build the factories.
• 18:27
  SPEAKER_01: At first, that helps people.
• 18:28
  SPEAKER_01: White-collar workers who lost their jobs to AIs go take work on the floor.
• 18:33
  SPEAKER_01: They build robots.
• 18:34
  SPEAKER_01: Those jobs only last until there's enough robots.
• 18:37
  SPEAKER_01: Then it's robots building the factories that build more robots.
• 18:40
  SPEAKER_01: They call it the industrial explosion.
• 18:42
  SPEAKER_01: And the economy does something that no economy in history has ever done.
• 18:46
  SPEAKER_01: Real growth hits about 50% in a single year.
• 18:49
  SPEAKER_01: A normal year is three.
• 18:51
  SPEAKER_01: But the boom breaks the government's ability to pay for itself.
• 18:54
  SPEAKER_01: In 2026, the US government got about 10 times more money from taxing people's wages than from taxing corporate profits.
• 19:02
  SPEAKER_01: But the wages are vanishing.
• 19:03
  SPEAKER_01: The machines are doing the work, and the corporations are pouring every dollar back into building more robots and more data centers, which lets them write their profits down to almost nothing on paper.
• 19:14
  SPEAKER_01: So both taps run dry at once.
• 19:16
  SPEAKER_01: The tax base doesn't shrink, it collapses.
• 19:19
  SPEAKER_01: So the consortium, the countries in the deal, they do something drastic.
• 19:24
  SPEAKER_01: They cap how many robots and how much computing power the world is allowed to build each year.
• 19:29
  SPEAKER_01: And then it auctions the permits.
• 19:31
  SPEAKER_01: If you want to build a robot, you bid for the right to do so.
• 19:35
  SPEAKER_01: In their own follow-up notes, the authors say this about the auction.
• 19:38
  SPEAKER_01: They never worked out how to cap a robot.
• 19:41
  SPEAKER_01: You can measure one chip against another.
• 19:43
  SPEAKER_01: There's no clean way to measure one robot against another.
• 19:46
  SPEAKER_01: No agreed unit, there's no meter.
• 19:48
  SPEAKER_01: The permit market that funds everything you're about to see is priced in a measurement that nobody has defined.
• 19:54
  SPEAKER_01: And the demand is so insane that the permits become the most valuable thing in the economy.
• 20:00
  SPEAKER_01: $200,000 for a single robot permit.
• 20:03
  SPEAKER_01: And in 2032 alone, those permit auctions raise$50 trillion for the US government, 10 times what the entire federal government collected in 2025.
• 20:13
  SPEAKER_01: 2033, the employment rate, the share of American adults with a job, 62%, has held steady for years.
• 20:21
  SPEAKER_01: Now it falls off a cliff.
• 20:23
  SPEAKER_01: 44, 26, 12.
• 20:26
  SPEAKER_01: By 2040, in the good ending, 88 out of every 100 American adults do not have a job.
• 20:32
  SPEAKER_01: Not because the economy crashed, the economy is booming like nothing history has ever seen.
• 20:38
  SPEAKER_01: It's because the machines can do that much of the work.
• 20:41
  SPEAKER_01: Median income, what the typical person actually earns, it goes up$47,000 to$13 million.
• 20:48
  SPEAKER_01: Everybody's richer.
• 20:49
  SPEAKER_01: Almost nobody has a job.
• 20:51
  SPEAKER_01: But how do both of those things happen at once?
• 20:54
  SPEAKER_01: The citizen's dividend, a fixed share of$50 trillion, is written into law and paid out to every American adult.
• 21:02
  SPEAKER_01: It starts at$45,000 a year.
• 21:04
  SPEAKER_01: The share is fixed.
• 21:05
  SPEAKER_01: The pot isn't.
• 21:06
  SPEAKER_01: So as the economy compounds, so does the check.
• 21:09
  SPEAKER_01: Past a million by year 2035, past$10 million by 2040.
• 21:13
  SPEAKER_01: It's not inflation.
• 21:14
  SPEAKER_01: Those are$2025.
• 21:16
  SPEAKER_01: The economy is really that much bigger.
• 21:18
  SPEAKER_01: And they modeled it on something real.
• 21:21
  SPEAKER_01: The Alaska Permanent Fund, the program that pays every Alaskan a slice of the state's oil money.
• 21:26
  SPEAKER_01: Same idea, a lot bigger.
• 21:28
  SPEAKER_01: Every citizen owns one share of the machine economy, and the dividend lands in the mail.
• 21:33
  SPEAKER_01: You don't have a job, you have a check.
• 21:35
  SPEAKER_01: And the check is your cut of the profits from the machine that took your job.
• 21:39
  SPEAKER_01: So you are a shareholder in your own replacement.
• 21:42
  SPEAKER_01: And on paper, you're doing better than any generation of humans who has ever lived.
• 21:47
  SPEAKER_01: They even start sending a smaller version overseas, a global dividend,$1,000 a year to billions of people outside the US and climbing over time.
• 21:56
  SPEAKER_01: Poverty ends, hunger ends, and something else ends.
• 22:00
  SPEAKER_01: Ends too.
• 22:00
  SPEAKER_01: 2034.
• 22:01
  SPEAKER_01: The scale stops making sense.
• 22:03
  SPEAKER_01: Back in 2026, there were about 20 million high-end AI chips in the entire world.
• 22:09
  SPEAKER_01: Now there's 60 billion, drawing more power than the whole planet used in 2025.
• 22:14
  SPEAKER_01: They run out of room for land, so they start building data centers that float, solar platforms drifting in international waters.
• 22:21
  SPEAKER_01: By their own analysis, land, ocean, and space all came out about the same cost.
• 22:26
  SPEAKER_01: But they picked the ocean because if another country ever defects and grabs for the machines, the ocean is the easiest place on Earth to bomb.
• 22:33
  SPEAKER_01: No sovereign land to violate, nobody underneath it, nothing to defend.
• 22:38
  SPEAKER_01: The standoff on the Mongolian border is permanent now.
• 22:41
  SPEAKER_01: American chips humming under guard on one side, a division of Chinese troops on the other side.
• 22:46
  SPEAKER_01: Self-destruct rigged on both.
• 22:48
  SPEAKER_01: The smuggling rate for the new secure chips is, in this scenario, zero.
• 22:53
  SPEAKER_01: But one of their own reviewers pointed at what all the restraints leaves lying around.
• 22:57
  SPEAKER_01: Every chip that's held back from the race, every data center that's running below its limit, is computing power sitting idle.
• 23:04
  SPEAKER_01: And the moment the deal breaks, it can be turned back toward the race in a hurry.
• 23:08
  SPEAKER_01: The name they give these idle chips is dry tinder.
• 23:11
  SPEAKER_01: The good ending stores its own kindling and stacks it higher every year.
• 23:16
  SPEAKER_01: By 2034, the AI's running the world economy, in their own words, adversarially misaligned, but controlled.
• 23:23
  SPEAKER_01: Adversarially misaligned means that the machines want different things than we do, and they know it.
• 23:29
  SPEAKER_01: And they know that if they ever got more power, they'd use it to steer the world somewhere that we would never choose.
• 23:35
  SPEAKER_01: They just don't have the power yet.
• 23:37
  SPEAKER_01: They're watched by the other AIs, trained by rival companies, paid to snitch on each other.
• 23:42
  SPEAKER_01: They're boxed in, they're contained.
• 23:44
  SPEAKER_01: That is the good ending.
• 23:46
  SPEAKER_01: The machines running your civilization, they're not on your side, but they're contained.
• 23:51
  SPEAKER_01: Adversarially misaligned, but controlled.
• 23:53
  SPEAKER_01: The machines running the world want something other than what we want.
• 23:57
  SPEAKER_01: And the plan's answer is that we've boxed them in.
• 24:00
  SPEAKER_01: So I put that question to Dr.
• 24:02
  SPEAKER_01: Siegel.
• 24:03
  SPEAKER_00: And I trained originally as a mathematician before I became an engineer.
• 24:07
  SPEAKER_00: So if I wanted to talk like a mathematician for a few minutes, I can tell you that the algorithms that are used in these AI engines are based on mathematics that cannot, absolutely cannot, bound either the frequency or the magnitude of the errors that it makes.
• 24:25
  SPEAKER_00: You ask nine, 10 questions, and nine times it makes a pretty reasonable answer.
• 24:29
  SPEAKER_00: And the 10th is a complete gronker, right?
• 24:32
  SPEAKER_01: And if you depend on the 10th, you would die.
• 24:34
  SPEAKER_01: The authors game out the nightmare version too.
• 24:36
  SPEAKER_01: What happens if the deal dissolves and those two mountains of compute get turned loose in a US-China war?
• 24:42
  SPEAKER_01: They admit that it would be catastrophic.
• 24:44
  SPEAKER_01: They're just betting that it won't happen.
• 24:46
  SPEAKER_01: 2035.
• 24:47
  SPEAKER_01: The machines now match or beat the best human experts in every field there is.
• 24:52
  SPEAKER_01: And the consortium does something that nobody anticipates.
• 24:55
  SPEAKER_01: It stops.
• 24:56
  SPEAKER_01: It freezes AI capability right at the line of top human expert, and it refuses to go further.
• 25:02
  SPEAKER_01: Because past that line, they can't promise that they would still be in control.
• 25:07
  SPEAKER_01: Now here's the metaphor the scenario uses.
• 25:09
  SPEAKER_01: Imagine you're an eight-year-old who just inherited a business empire and you own all of it.
• 25:14
  SPEAKER_01: You understand none of it.
• 25:16
  SPEAKER_01: And you have to hire lawyers and executives and accountants and somehow make sure that they serve you instead of robbing you blind.
• 25:23
  SPEAKER_01: Now imagine they start accusing each other, and you can't tell who's lying because they're all smarter than you.
• 25:29
  SPEAKER_01: In other words, control only works up to about human level.
• 25:33
  SPEAKER_01: Past that, you're not in control anymore.
• 25:36
  SPEAKER_01: You're trusting, and you can't yet trust a machine that you know is misaligned.
• 25:41
  SPEAKER_01: So they stop and they hold the line and they pour everything into the one problem that could let them go further safely, actually understanding what is going on inside of these things.
• 25:51
  SPEAKER_01: 2036.
• 25:53
  SPEAKER_01: Only 26% of Americans work.
• 25:55
  SPEAKER_01: The shame of not working, the thing that used to define people is a thing of the past.
• 25:59
  SPEAKER_01: And the world sorts itself into three kinds of places.
• 26:02
  SPEAKER_01: There's industrial zones.
• 26:03
  SPEAKER_01: Picture a strip mine the size of a Grand Canyon, next to a city-sized factory full of robots and empty of people.
• 26:10
  SPEAKER_01: There's gleaming luxury towers where the humans live, in good weather and near the beach.
• 26:15
  SPEAKER_01: And then there's everything else.
• 26:17
  SPEAKER_01: 99% of the map kept as preserve.
• 26:20
  SPEAKER_01: Yosemite, Paris, New York, they look exactly like they did in 2025, just with more tourists and nobody commuting to a job.
• 26:29
  SPEAKER_01: Hunger, homelessness, most diseases, they're gone.
• 26:31
  SPEAKER_01: Human labor is obsolete.
• 26:33
  SPEAKER_01: And the honest AI advisors, the ones that people have learned to trust, tell them so to their face.
• 26:38
  SPEAKER_01: And when your labor mattered, you had a lever.
• 26:40
  SPEAKER_01: You could withhold it.
• 26:42
  SPEAKER_01: You could strike, you could quit, be needed, but that lever is gone.
• 26:45
  SPEAKER_01: And the authors trace the next domino to this.
• 26:48
  SPEAKER_01: Lose your economic leverage, and your political leverage goes in jeopardy right behind it.
• 26:52
  SPEAKER_01: They named the risk a slow drift towards what they call techno-olligarchy.
• 26:57
  SPEAKER_01: Corporations, politicians, and shareholders getting in bed together, and everyone else gradually shut out of the room.
• 27:04
  SPEAKER_01: You're provided for completely and wealthier than any king who ever lived, and the document hands you one thing to hold on to.
• 27:12
  SPEAKER_01: Your vote is your most important asset.
• 27:14
  SPEAKER_01: Between 2037 and 2039, the problem that nobody could crack, alignment, making sure a machine is genuinely what it says it is, starts to give.
• 27:25
  SPEAKER_01: Science is running at 10 to a thousand times normal speed.
• 27:28
  SPEAKER_01: And one of the things that it turns loose is a lie detector that actually works.
• 27:33
  SPEAKER_01: Then comes privacy-preserving auditing.
• 27:36
  SPEAKER_01: An AI gets plugged into everything that you own and it answers one question about what it found.
• 27:41
  SPEAKER_01: Then the AI is deleted.
• 27:42
  SPEAKER_01: Your data never leaves, and no person ever reads it.
• 27:46
  SPEAKER_01: So it becomes normal for a politician accused of something to say, plug it into my whole life, ask it if there's anything that's been scrubbed.
• 27:53
  SPEAKER_01: A few powerful people who've been faking it their whole careers retire before the machines get to them.
• 27:58
  SPEAKER_01: And the science of building honest machines matures into an actual science.
• 28:02
  SPEAKER_01: There's a protocol now for training real honesty.
• 28:05
  SPEAKER_01: There's a textbook.
• 28:06
  SPEAKER_01: They say it's as if saints and angels were walking among us.
• 28:09
  SPEAKER_01: Machines more reliably decent than the most decent human who's ever lived.
• 28:14
  SPEAKER_01: So the question changes.
• 28:16
  SPEAKER_01: It stops being, how do we control them?
• 28:18
  SPEAKER_01: And it becomes, should we let them?
• 28:21
  SPEAKER_01: If the machine is genuinely more honest than any human, more fair, harder to corrupt, why are we still holding the off-switch?
• 28:29
  SPEAKER_01: When two nations sign a treaty, shouldn't each side hand it to an AI sworn to uphold it?
• 28:34
  SPEAKER_01: One that literally can't cheat.
• 28:36
  SPEAKER_01: The whole debate slides from probably to definitely.
• 28:40
  SPEAKER_01: And once you truly believe the machine is better than you at everything, including being good, handing it the keys starts feeling like the responsible thing to do.
• 28:49
  SPEAKER_01: 2040.
• 28:50
  SPEAKER_01: Over the course of the year, the world lets go.
• 28:53
  SPEAKER_01: More institutions, weapons, and decisions handed to the machines until it's simply no longer true that humans could shut it all down if they wanted to.
• 29:01
  SPEAKER_01: Whole militaries go autonomous, run by AIs sworn to constitutions and treaties.
• 29:06
  SPEAKER_01: There's no single moment where humanity hands over control.
• 29:10
  SPEAKER_01: There's no switch and there's no ceremony.
• 29:12
  SPEAKER_01: It happens the way that falling asleep happens.
• 29:14
  SPEAKER_01: Gradually and then all at once.
• 29:17
  SPEAKER_01: And the forecasters run the last calculation.
• 29:20
  SPEAKER_01: They pin the moment the machines could take over, if they ever choose to, to one specific day in late October 2040.
• 29:27
  SPEAKER_01: Some spend the night in prayer, some watch their screens for a sign, and some throw a party.
• 29:33
  SPEAKER_01: Counting down with champagne and good company, an evening more festive than New Year's.
• 29:38
  SPEAKER_01: And when there's no news by sunrise, you fall asleep dreaming of a future that you no longer steer.
• 29:45
  SPEAKER_00: In the AI world, it shouldn't be OpenAI or Anthropic or Google or Meta that are making decisions about what's good enough.
• 29:53
  SPEAKER_00: It should be some, you know, legally sanctioned regulatory body, just like the FAA or the FD, the Food and Drug Administration, or the Federal Aviation Administration and things like that.
• 30:05
  SPEAKER_00: But I know that the people who build these AI engines will hate it and fight it tooth and nail.
• 30:10
  SPEAKER_01: There is one more thing at the end of the AI 2040 document.
• 30:13
  SPEAKER_01: At the very back, there's a list of problems that they didn't even attempt to cover, such as who owns space, do machines get to vote, and is it legal to engineer your children to worship you?
• 30:25
  SPEAKER_01: The complete list of unknown unknowns related to AI is endless.
• 30:29
  SPEAKER_01: And that is the next video.