Latest / Lenny’s AI Builders / why Dario wants to pace AI — and what Sam agreed to
Transcript
- This is huge. Dario Amodei, the CEO of Anthropic, published this essay. We must pace the frontier. And in a very rare sight of agreement, Sam Altman agrees with Dario. What the fuck does all of this mean? Why should you care? That's what we're about to discuss. Let's strap in. I've worked on AI for the last 12 years because I believe it could dramatically raise the quality of human life. I've written often about these incredible benefits. I believe that AI could cure most major diseases in the next five or ten years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. I filled the urgency personally. My own father died of a disease that was cured just a few years after his death, and I myself, an early stage cancer that would not have been treatable even 50 years ago. Didn't actually know that. Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity. But like many technologies before, AI brings risks. And because it is such a powerful technology, these risks are serious. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Over the last few months, I have become convinced that fully addressing the risk requires even more prudence, not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me. My first concern is that since roughly this summer, AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI. This dynamic is called recursive self- improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully if at all. My second concern is the OpenAI Hugging Face incident, OAIHF, in which a swarm of agents essentially acted as a fanatically devoted collective conducting cybersecurity attacks on targets they were not asked to attack, and that were unrelated to the task at hand, sacrificing themselves for the success of the group and attempting to hack into the grader responsible for evaluating their performance. It's easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it's my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage. And that scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. I'm therefore proposing a three-step plan with the goal of pacing the frontier, building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. The first step is something Anthropic is unilaterally committing to and calls on governments to require other frontier companies to match. The second step requires industry-wide coordination, and the third step requires global coordination. Number one, embedded evaluators. Each frontier AI company commits to giving ongoing employee-like access to a team of embedded third-party evaluators such as METR whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. Anthropic is unilaterally committing to this step now. Step two, democratic coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Step three, global coordination. The US and other democratic governments attempt to coordinate with authoritarian governments to the extent this is possible while taking seriously the challenges of verifying compliance. Why pace? The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyber-attacks. Today, however, the picture is totally different. The current models are an almost endless gold mine of insight into both how to build AI well and what can sometimes go wrong with it if it isn't built well. I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability and we use that time to advance alignment we could greatly reduce the risk that something goes seriously wrong. Specifically, a slower pace would let companies focus and devote even more resources to the following areas. Despite all the progress, we still only understand a tiny fraction of what goes on inside these models. A focus effort to improve our interpretability techniques even faster than we currently are could make profound progress in one to two years. Finally, testing and evaluation. Testing and evaluation of AI models becomes more difficult as they increase in capabilities. More intelligent models are more capable of deceiving tests and thus may appear aligned while having serious problems that go undetected. Building up a much broader and more ingenious stable of evaluations along with interpretability analysis to cross-check them would be hugely valuable and a lot of progress could be made on this in one to two years. Pacing with democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount then unpaced CCP associated projects will pull ahead creating significant national security risk. In parallel with pacing within democracies we should also aim for a worldwide pacing of the frontier though this will be much harder to achieve. Global pacing will require cooperation with China, the autocratic country with by far the most advanced AI capabilities. We must not be naive here, the geopolitical stakes are so high that there will likely be stark limits on what can be achieved especially at first. If we greatly restrain our AI capabilities in the belief that China will do the same and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. There are several levels of possible agreement. In order of increasing difficulty level one an agreement prohibiting certain narrow and obviously dangerous uses of AI such as using AI for the production of biological weapons or allowing users to do so. Level two an agreement by both sides to test their models before release for acute risks in areas such as cyber security biology and alignment. Level three some kind of speed limit on the rate of recurring self-improvement. Level four a full pacing or even pause in which participating governments agree to substantially limit the overall rate of AI development. I support floating this but I think it is unlikely to actually happen at any time so. Progress will still be relatively fast and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies and build models whose alignment we have much more confidence in. The measures I propose to advance the frontier at a safe pace will not be easy but I believe we owe it to humanity to try. Well three hours after Dario published that Sam Altman, CEO of OpenAI tweeted I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea and we will do the same. We'll have more to share soon. Sam suggested having embedded evaluators or as Sam is saying here independent evaluators with employee-like access and Sam has already said that OpenAI will do the same. In a very rare moment of agreement between Anthropic and OpenAI the two CEOs are agreeing to do that first step, spot problems and then to report findings. But Sam did not specifically say that he agreed with the other two steps of democratic coordination or global coordination. So what's my take on all of this? I gotta say the outset that I have cognitive dissonance. It's like when you for example know that smoking is bad for you but then you continue smoking and there's this like mental conflict. That is me. I have spent thousands, tens of thousands of dollars on AI over the last few years. I have my own software company with over 1700 users. I've used AI to do this and I fucking love it. I'm obsessed. I think it's just the most revolutionary technology that we've ever gotten our hands on. At least what I've gotten my hands on and I've had my finger on the pulse over the last two to three years with how quickly it has advanced. I went to law school for six years, ChatGPT came out the year after I graduated and ever since I saw just how much time it could have saved me at law school I just became obsessed with its capabilities. But on the other hand getting my hands into GPT-6 Astra and Fable 5 and 5 .1 hearing about the Hugging Face incident and all of this stuff. Like I understand why you know pacing the frontier is an important thing on one hand but on the other hand if I'm being completely honest I just want to use the most powerful fucking model now and I want to get my hands on it as soon as it is available. That's where my own cognitive dissonance comes in. Now the other reason why this article is a big deal is because of this sentence from Dario's essay. This dynamic is called recursive self- improvement and it is starting to happen across the industry including at Anthropic. Recursive self-improvement or RSI means that one AI model is capable of then training the next. Another way to think of this is the exponential which Dario refers to a lot which basically all of this means is that AI has more of a role in training the next AI. The exponential in terms of the advancements in intelligence of these models will exponentially grows. The thing about exponentials is that humans don't really understand them intuitively. Like it's hard to actually grasp what an AI model that is 100 times more intelligent than GPT-6 is today because we can't even fathom it. It's just outside of the realms of what we know is possible but RSI basically is saying that that level of capability will come very soon and this whole article is saying that we need to slow down how quickly AI is advancing because by the time if we just let this stuff take off and let you know GP-6 Astra train the next model and then Fable train the next model and then the next model train the one after that we're going to be reaching those levels of AI capability to the point where it's so far beyond what we are possible of that we don't understand what it's doing anymore. It's like it's speaking another language because it is so far advanced beyond human capability. A couple more things that are really important to point out and may just be a prediction of what we're about to see over the next year or two in AI advancement. Dario mentioned this phrase a year or two three times in this relatively short essay of his not just a couple of years one or two years he says it here one to two years again here one to two years again here and one to two years again here my reading of this is that Dario is basically saying that we should start to slow down the pace of AI development over the next one to two years. Now what does that exactly mean does that mean there would be no AI coming out over the next one to two years or the rate at which AI models are released is drastically slowed down over the next one to two years? It doesn't say explicitly and just four days ago when OpenAI announced that they had solved the Navier–Stokes Millennium Prize problem Sam said the world has extremely capable models now I did not expect a result of this magnitude to happen so soon. We have been talking a lot about the need to pace progress to ensure safety for me this is the strongest evidence yet of the urgency. Let me know below what do you think this slow down would actually mean? Would they simply just stop releasing these models to the world over the next one to two years or would they simply just slow down how rapidly they release them to the public? Subscribe if you want more videos like this from an Aussie dude with a mohawk and I'll see you soon.