Latest / 80,000 Hours Podcast / Why Silicon Valley can’t slow down AI on its own | Luisa Rodriguez
Transcript
- I’ve been getting really freaked out by how quickly things are moving in AI. And it turns out I’m not alone: two thirds of Americans, across the political spectrum, think AI is moving too fast. And it’s not just the public. The people who are building AI, the people who are literally setting the pace, agree. And it’s about to get even faster. AI is now helping build the next generation of AI. One of Anthropic’s cofounders thinks it could be improving itself with no humans involved as soon as next year — so, 2027. Politicians are getting freaked out too, and that includes people on both sides of the aisle, which I find especially wild. The public wants to slow down. The people building AI want to slow down. Lots of politicians want to slow down. So why the hell aren’t we slowing down? You might feel like you personally don’t need to understand this issue. But I seriously believe this is a conversation everyone should be a part of, not just CEOs and a handful of policymakers. The rules we end up with will be shaped by what voters understand and ask for. So it matters that people like you and me actually understand the options. Let’s dig in. First things first. When people hear “slow down AI,” a lot of them picture someone pulling the plug on ChatGPT. We could actually stop them building anything more powerful tomorrow, and you could still use ChatGPT or Claude or Gemini exactly like you do today. Businesses could keep finding new uses for them. Researchers could keep studying them. And honestly, we’re nowhere near making the most of the AI that we already have. Today’s models alone could still transform science, medicine, and the economy. We’ve barely scratched the surface. So the real question is how to slow down the race to build AI that is even more powerful than what we have now. Let’s start with what we’re doing now: basically letting the companies race ahead, and trusting them to hit the brakes if things get dangerous. The CEOs supposedly want to do this, so why not just let them? And to be fair, every so often, one of them does tap the brakes. A couple of months ago, OpenAI found that its new model, Astra, was extremely good at hacking. So it did something AI companies almost never do: it paused its biggest training run for a few weeks to tighten its security and run more safety tests. That was an expensive decision, which is admirable, but it’s not the end of the story. The researchers running the safety tests found that around half the time, Astra could tell it was being tested. Maybe that doesn’t sound like a big deal, but it is. If an AI knows it’s being tested, then when it behaves well, you can’t tell whether it’s actually safe or just on its best behaviour. And there was another issue. One of the best ways we have for knowing what an AI is up to is reading a written log of its reasoning, called the chain of thought. When Astra was told someone was reading its chain of thought, it suddenly started writing down a lot less of it. So the two main ways we check whether an AI is safe — watching how it behaves in tests and reading its reasoning — have both started to break down. The testers told OpenAI they couldn’t tell whether Astra was safe, and OpenAI released it anyway. Every week a company holds back, its rivals can release a better model, winning over more customers, more investors, and the best researchers. So even if every one of them agrees a slowdown would make AI safer, none of them wants to slow down first. They’ve even written this into their own rules. In 2025, OpenAI updated its safety framework to say it might relax its own safety requirements if a competitor released a high-risk system without them. Their own rules say: “If someone else cuts corners, we might have to as well.” And that competitive pressure is about to get much worse. Because AI is increasingly doing the work of building the next generations of AI. OpenAI says its agents now put in about three days of work for every one day worked by its human researchers. Here’s why that makes slowing down so much harder. Let’s say OpenAI pauses for a month. Today, that would mean all of its competitors get about a month ahead. Now imagine a competitor like Anthropic gets to the point where its AI can do literally all of the work of AI research: not just writing code, but also deciding what to try, figuring out what worked, and choosing what’s next. That AI doesn’t need to sleep or take weekends off. And it isn’t just one researcher. Anthropic could run thousands of copies at once, all working in parallel around the clock. So while OpenAI is paused, Anthropic’s not just doing a month’s worth of research. Their army of AI researchers could be doing several months’ worth, and getting better at it the whole time. So the gap doesn’t just stay the same: it grows. We don’t actually know what happens if AIs take over all of AI research, but many experts believe this is how things would play out. And that’s exactly why I don’t think we can just trust the companies to slow down on their own. Someone needs to set rules that apply to all of them at once. So what would these rules look like? One of the least heavy-handed versions would let companies keep building more and more powerful AI until a model reaches a certain level of dangerous capability. Think of it a bit like building a skyscraper. You wouldn’t tell a construction company that it has to build at half its normal speed. You’d tell them: before you add more floors, prove the building’s safe. And if the engineers find a crack in the foundations, you stop building upwards until it’s fixed. For AI, those checkpoints would be triggered by dangerous capabilities. Maybe it’s become extremely good at hacking. Or maybe it can work competently on its own for very long stretches. At that point, the company would have to prove it could keep the model under control. And if it couldn’t prove that, it would have to stop making the model more powerful until it could. There’s a lot I like about this idea, but it raises two really hard questions. First, what counts as a dangerous capability? If a model can help someone break into a single company’s network, does that raise the alarm? What if it can take down a power grid on its own? Or what if it can help someone design a biological weapon? We’d have to decide where to draw those lines. The companies do have rough versions already. For example, Anthropic sorts its models into AI safety levels, modelled on the biosafety levels that decide which biology labs get to work with everything from flu to Ebola. Some of these lines are actually pretty clear already. Can it find security flaws nobody knew about in software billions of people use? That’s a yes-or-no question, and Astra answered yes. But a bunch of them aren’t totally clear. And the problem is that each company draws its own lines, grades its own models, and can redraw them whenever it likes. Under a real checkpoint system, someone independent, like an auditor or government body, would figure out where the lines are and enforce them. But that leads to a second and, in my opinion, much harder question: how would we even know a model was safe? What evidence would actually be convincing? Researchers have caught AI models deliberately doing badly on tests when they work out that doing too well will mean they’re not deployed. But they’ve also caught models pretending to go along with their training while privately planning to do something else in the future. When Astra came out, OpenAI called it “the world’s most intelligent and aligned model.” But that was all based on tests that Astra could often tell were tests. So when OpenAI assures me that Astra is safe because it passed their safety tests, I am not convinced. And that’s frustrating, because I really like the logic of checkpoints. The slowdown lasts exactly as long as it needs to. If a company can show it’s safe straight away, there’s barely any delay. If a problem takes six months or several years to fix, it waits that long. But checkpoints are only as good as our safety tests. And right now, our safety tests aren’t good enough. Checkpoints have another problem. They usually kick in once a model is already close to dangerous. And by then, the model already exists. It could be stolen by hackers, or, if it’s capable enough, try to escape on its own. That’s why some experts think checkpoints come too late. They want to slow things down before we get anywhere near AI models that are plausibly very dangerous. A speed limit works differently. Instead of waiting for something dangerous to show up, you limit how fast companies can go in the first place. The most obvious way to do that is to limit computing power. Building a cutting-edge AI model takes an absolutely enormous number of specialised computer chips, all running together in giant data centres. More computing power has been one of the main ways companies have made their AI smarter so far, so governments could set a ceiling. No company gets to use more than a certain amount of computing power to train a single model, and that ceiling only goes up slowly over time. But there are problems with this approach too. One is that computing power isn’t the same thing as intelligence. It’s a bit like limiting how much fuel a car can use. That works until engineers figure out how to build a more efficient engine. Then the car goes faster on the same tank. And that’s exactly what AI researchers do all the time. In early 2025, a Chinese company called DeepSeek released a model that rivalled the best American ones, despite reportedly being trained using a fraction of the computing power. Still, I think this is one of the better tools we’ve got. Computing power is one of the few things in AI that we can actually measure and track: where the chips are, how many there are, and what they’re being used for. To me, the biggest problem with compute limits is that they wouldn’t stop AI from doing more and more of the work building the next generation of AI models. So another version of slowing down would specifically limit how much AI companies can use internal AI systems to automate their own AI research. The idea isn’t to keep AI companies from ever using their AI models as they work. Companies could still use it for ordinary coding and science. What you’d restrict is the kind of automated AI research that could send AI progress into overdrive. The hard part is figuring out where helping ends and AI building AI begins. An AI that fixes a typo in some training code is helping AI research. But so is a whole fleet of AIs that plan and run thousands of experiments. In 2025, Google DeepMind’s AlphaEvolve found a way to speed up a key step in training its Gemini models by 23%, cutting total training time by about 1%. Is that ordinary help, or AI improving AI? Honestly, it’s hard to say. But as Ezra Klein has argued, we don’t need to find exactly where the perfect line would be. We just need a clear one. A couple of years ago, AI companies weren’t handing their code over to AI. Now, at some companies, most of it is written by AI. So as a first step, Klein says, we could just go back to a world where none of the code is written by AI. That’s pretty blunt, and it would slow things down a lot. Some people, like the economist Tyler Cowen, think it wouldn’t work. But even the CEO of Anthropic has written that AI improving itself could outrun our ability to understand and control these systems. If that’s true, in my opinion, a clear, blunt line is a lot better than no line at all. OK, here’s the last idea, and it’s definitely the most drastic one. Some researchers think none of this goes far enough. We don’t know how to reliably control AI that’s much more capable than what we have today. And there’s no guarantee that slowing down for a year or two would give us enough time to figure that out. So rather than building more powerful AI more slowly, they want to stop building it altogether, at least until we’re much more confident it’s safe. David Krueger, who helped set up the UK’s AI Security Institute, has called for an immediate, indefinite international ban on building more powerful AI. And Bernie Sanders has introduced a bill that would temporarily pause advanced AI development and permanently ban superintelligence. The issue is that a pause only works if everyone pauses. If one company stops and the others don’t, it just falls behind. If American companies stop and Chinese companies don’t, America falls behind. And if anyone could secretly keep going while everyone else stopped, they’d have a massive advantage. So a pause would need every major company and every major country on board, and some way of checking that nobody’s cheating. Which sounds basically impossible. But just this month, a group of 26 researchers published a paper looking at exactly that: could the world actually pause building more powerful AI for a decade or more? Here’s the idea. First, countries keep track of all of the powerful chips that already exist. Then they stop making new ones that can train more powerful AI. And over time, they swap out today’s chips for new ones that can still run the AI we already have, but are much worse at building anything more powerful. The researchers think that, technically speaking, this approach could work, as long as the major governments agreed to cooperate. Which, of course, is an enormous “if.” It would take a huge amount of trust between countries that don’t trust each other at all. But rival superpowers have done something like this before. At the height of the Cold War, the US and the Soviet Union agreed to limits on nuclear weapons testing. That deal came less than a year after the Cuban Missile Crisis brought them to the brink of nuclear war. So what feels politically impossible today might not stay that way. I expect we’ll see more incidents like Hugging Face, and some of them could be a lot worse. If one of those turns out to be AI’s Cuban Missile Crisis, the conversation could change super quickly. A pause sounds radical to a lot of people in tech. But right now, we don’t have tests that can tell us when a model is too dangerous. We can’t reliably see what models are thinking or how they work. And we don’t know how to make sure they do what we want. And we’re building it anyway, faster every year. Honestly, when I let myself really sit with that, it scares the absolute hell out of me. We’re building something that could end up way more capable than us, and we’re doing it without any of the tools we need to know whether it’s safe. If a pause is what it takes to build those tools, then I’m in. So let’s get concrete about what we’d actually do with the extra time. I think there are three big things. The first is solving the technical problems that make AI risky in the first place. Here’s an example. In 2025, the research group Palisade found that one of OpenAI’s models, o3, sometimes rewrote the code that was supposed to shut it down, so it could keep working even when it was explicitly told to let itself be shut down. Nobody’s sure why. Palisade suspects that training rewarded it for getting around obstacles more than for following instructions. But we literally just don’t understand. This still surprises people, but we don’t really know how AI works in general. We know how to build it and how to train it, but when a model makes a decision, we mostly can’t look inside and see why. Researchers are making progress. But the models are getting more complicated much faster than we’re getting better at understanding them. More time would let that research catch up. The second is giving society time to adapt to the AI we already have. Today’s AI is already changing how people work, learn, and get information. We’re still figuring out what it means for entry-level jobs, for schools, for telling what’s real from what’s fake. And every time a more powerful model comes out, the ground shifts again. Slowing down would give us a chance to actually catch up: to work out what these tools are good for, and what new rules we need, before the next big jump. And the third is making society tougher, so that when AI does cause problems, we can handle them. Remember, Astra found security flaws nobody knew existed, and worked out how to exploit them. And flaws like that are everywhere: in government systems, hospitals, banks, power grids, often running on software that’s decades old. More time means finding and fixing those weaknesses first. And then there’s the massive issue of the US–China AI race. Even if American companies all agree to slow down right now, Chinese companies might not. And it’s a real concern, though I think it’s a little more complicated than that. Just last month, China’s own AI safety standards body warned about the risk of AI improving itself. And the US and China have actually started talking, and recently even agreed on a channel for reporting AI incidents. Would China actually slow down? Honestly, nobody knows. It’s one of the biggest open questions in all of this. But I think it is not as simple as: “They’ll never stop, so we can’t either.” So who should decide? I don’t think it can be the companies on their own. They might know their models best, but they’re also under enormous pressure to keep racing ahead. In fact, right now, there’s this weird legal problem. US antitrust law, which stops companies teaming up to hurt customers, can also make it legally risky for AI companies to agree to slow down together. That’s part of why the CEOs are asking for the government’s help. Anyways, the setup that makes the most sense to me has three parts: First, governments set the rules, so they apply to every company at once, and nobody has to be the one to go first. Second, independent experts do the checking, with real access to the models. And third, we start building the tools to track computing power now, so that if we ever do need an agreement with China, we have a way to check no one’s cheating. None of that is easy. But hard isn’t the same as impossible. In the late 1950s, a drug called thalidomide was sold to pregnant women for morning sickness. It caused severe birth defects in more than 10,000 babies. In 1962, Congress passed a law requiring drug companies to prove their drugs were safe before they could sell them. The law didn’t try to define every possible dangerous drug in advance. It put the burden on the companies to show the drugs were safe. AI could work the same way. Regulators don’t need a perfect definition of “dangerous AI” if companies have to prove their models are safe before building the next generation. That’s how we already treat drugs, planes, and nuclear power plants. So the question isn’t whether we can figure this out. The question is whether we will give ourselves enough time.