Latest / 80,000 Hours Podcast / The case for and against AGI by 2030 (article by Benjamin Todd)

Transcript

The case for and against AGI by 2030 (article by Benjamin Todd) · 80,000 Hours Podcast

• Hey, this is Benjamin Todd. More and more people have been saying that we might have AGI, artificial general intelligence, before 2030. Is that really plausible? I wrote this article to look into the case for and against and try and summarise all the key things you need to know about that argument. I definitely don't think it's guaranteed to happen, but I think you can make a surprisingly good argument for it. That's what we're going to dive into here. You can see all the images and many footnotes in the original article. The Case for AGI by 2030. In recent months, the CEOs of leading AI companies have grown increasingly confident about rapid progress. In November, OpenAI's Sam Altman went from saying he expects the rate of progress continues to by January, we are now confident we know how to build AGI. Also in January, Anthropic CEO Dario Amodei said, I'm more confident than I've ever been that we're close to powerful capabilities in the next two to three years. Google DeepMind's more cautious CEO, Demis Hassabis, has switched from saying as soon as 10 years in autumn to by January we're probably three to five years away. What explains this shift? Is it just hype or could we really have AGI by 2030? In this article I look at the four drivers of recent progress, estimate how far those drivers can continue, and explain why they're likely to continue for at least four more years. And that means we should expect major additional AI progress in that time. In particular, while in 2024 progress in LLM chatbots seemed to slow, a new approach started to work, teaching the models to reason using what's called reinforcement learning, which I'll explain later in the article. In just a year, this technique let them surpass human PhDs at answering difficult scientific reasoning questions and achieve expert level performance on one hour coding tasks. We don't know how capable AI will become, but just extrapolating the recent rate of progress suggests that by 2028 we could reach AI models with beyond human reasoning abilities, expert level knowledge of every domain, and that can autonomously complete multi week projects, and progress would likely continue from there. No longer just chatbots, these agent models could satisfy many people's definitions of AGI, roughly AI systems that match human performance at almost all knowledge work. I give a much more detailed definition of AGI in the footnotes. This would mean that while the company CEOs are probably a bit overoptimistic, there's enough evidence to take their position very seriously, and it's also important not to get caught up in definitions. Ultimately, what matters is that these models could start to accelerate AI research itself, unlocking vastly greater numbers of more capable AI workers. And then in turn, sufficient automation could trigger explosive economic growth and 100 years of scientific progress in 10 -- a transition society isn't prepared for. While all this might sound outlandish, it's within the range of possibilities that many experts think is possible. This article aims to give you a primer on what you need to know to understand why they think that, and also the best arguments against that position. I've been writing about AGI since 2014. Back then, AGI arriving within five years seemed very unlikely, but today the situation seems dramatically different. We can see the outlines of how AGI might work and exactly who could build it, and in fact, the next five years seems unusually crucial. The basic drivers of AI progress, investments in computational power and algorithmic research, cannot continue increasing at current rates much beyond 2030. That means that we either reach AI systems capable of triggering an acceleration soon, or progress will most likely slow significantly. Either way, the next five years is when we'll find out. Let's see why. The article In a nutshell. Four key factors are driving AI progress: larger base models, teaching models to reason, increasing how long models think about each question, and building agent scaffolding for multi step tasks. These in turn are underpinned by increasing computational power to run and train AI systems, as well as increasing human capital going into algorithmic research. All of these drivers are set to continue until 2028 and perhaps until 2032. This means in that time we should expect major further advances in AI performance. We don't know how large these advances will be, but extrapolating recent trends on benchmarks suggests that we'll reach systems with beyond human performance in coding and scientific reasoning and that can autonomously complete multi week projects. Whether we call these systems AGI or not, they could be sufficient to enable AI research itself, robotics, the technology industry, and scientific research all to accelerate, leading to transformative impacts on society. Alternatively, AI might fail to overcome issues with ill defined high context work over long time horizons and remain a tool even if much improved compared to today. Increasing AI performance requires exponential growth and investment in the research workforce. At current rates, we will likely start to reach bottlenecks around 2030. Simplifying a bit, that means we'll either likely reach AGI by around 2030 or see progress slow significantly. Hybrid scenarios are also possible, but the next five years seems especially crucial. Section 1: What's driven recent AI progress and will it continue? Entering the deep learning era. In 2022, Yann LeCun, the chief AI scientist at Meta and Turing Award winner, said, and I'm sorry I can't do a French accent. “I take an object, I put it on the table, and I push the table. It’s completely obvious to you that the object will be pushed with the table…There’s no text in the world I believe that explains this. If you train a machine as powerful as could be…your GPT-5000, it’s never gonna learn about this.” But within just two months of LeCun's statement, GPT-3.5 could answer this easily. And that's not the only example of experts being wrong footed. Before 2011, AI was famously dead. But that totally changed when conceptual insights from the ‘70s and ‘80s combined with massive amounts of data and computing power to produce the deep learning paradigm. Since then, we've repeatedly seen AI systems going from total incompetence to greater than human performance in many tasks within just a couple of years. For example, in 2022, Midjourney could not draw an otter on a plane using Wifi. But just two years later, Veo 2 can make a hyperrealistic movie. In 2019, GPT-2 could just about stay on topic for a couple of paragraphs, and that was considered remarkable progress at the time. Critics like LeCun were quick to point out that GPT-2 couldn't reason, show common sense, exhibit understanding of the physical world, and so on. But many of these limitations were overcome within just a couple of years. Over and over again, it's been dangerous to bet against deep learning. Today, even LeCun says he expects AGI in several years. And the limitations of current systems aren't what to focus on anyway. The more interesting question is where might this be heading? What explains the leap from GPT-2 to 4, and could we see another leap like that? So what's coming up? At the broadest level, AI progress has been driven by more computational power, called compute, and better algorithms. Both are improving rapidly. More specifically, we can break down recent progress into four key drivers, which I'll explain through the rest of the article. The first is called scaling pre-training. That lets you create a base model with basic intelligence, a basic understanding of the world. Secondly is using reinforcement learning to teach that base model to reason about complicated problems like in maths and coding. Third is having the model think longer about each question or each problem it's posed. This is called increasing test time compute. And then fourth is building an agent scaffolding around that model which lets it complete complex tasks and take actions in the world. In the rest of this section, I'll explain how each of these works and try to project them forward. As GPT would say, delve in and you'll understand the basics of how AI is being improved. Then in section 2, I'll use this to forecast future AI progress, and then finally explain why it makes the next five years especially crucial. So the first driver: scaling pre-training to create base models with basic intelligence. People often imagine that AI progress requires huge intellectual breakthroughs, but a lot of it is more like engineering: just do a lot more of the same and the models get better. In the leap from GPT-2 to 4, the biggest driver of progress was just applying dramatically more computational power to the same techniques, especially to what's called pre-training. Modern AI works by using artificial neural nets involving billions of interconnected parameters organised into layers. During pre-training -- a misleading name which simply means it's the first type of training -- here's what happens: Data is fed into the network, such as the image of a cat. The values of the parameters in that neural net then convert that data into a predicted output, such as a description, This is a cat. The accuracy of those outputs is graded versus the reference data. Then the model's parameters are adjusted in a way that's expected to increase the accuracy of those predictions. This is repeated over and over with trillions of pieces of data until the model becomes better and better at predicting accurately. This method has been used to train all kinds of AI, but it's been most useful when used to predict language. The data is the text on the internet, and LLMs are trained to predict gaps in that text. More computational power for training, so-called training compute, means that you can use more parameters, which means the model can learn more sophisticated and more abstract patterns in the data. It also means you can just use more data for training. Since we entered the deep learning era around 2011, the number of calculations used to train AI models has been growing at a staggering rate, more than four times per year. That's the amount of additional computing power used for training the largest AI model each year. This in turn has been enabled by spending far more money, as well as using much more efficient chips. Historically, each time training compute has increased 10 times, there's been a steady gain in performance across many tasks and benchmarks. For example, as training compute has grown a thousandfold, AI models have steadily improved at answering diverse questions, from common sense reasoning to understanding social situations and physics. This is demonstrated on the BIG-Bench Hard benchmark. This is a benchmark of diverse questions specifically chosen to challenge LLMs. In the article you can see a graph showing a linear increase in performance as training compute is scaled up. Likewise, OpenAI created a coding model that could solve simple coding problems. Then they used 100,000 times more compute to train an improved version. They showed that as training compute increased, the model correctly answered progressively more difficult questions. These test problems weren't in the original training data, so this wasn't merely better searched through memorised problems. This relationship between training computes and performance is called a scaling law. Papers about these laws have been published by 2020. To those following this research, GPT-4 wasn't a surprise. It was just a continuation of trend. The second contribution to pretraining is algorithmic efficiency. Training compute has not only increased, but researchers have found far more efficient ways to use it. Every two years, the compute needed to get the same performance across a wide range of models has decreased tenfold. In the article I show the example of image recognition algorithms. The amount of compute to get the same accuracy at recognising images has decreased roughly 10 times every two years. But a very similar pattern applies across a wide range of algorithms. These gains usually make the models much cheaper to run. DeepSeek-V3 was reported in the media as a revolutionary efficiency breakthrough, but in fact it was roughly on this preexisting trend. It was released roughly two years after GPT-4 and it's about 10 times more efficient than GPT-4. Now, algorithmic efficiency means that not only is four times as much compute used on training each year, but also that compute goes about three times further each year. The two effects multiply together to mean that effective compute has increased around 12 times each year. This is an insane rate of increase. For instance, the famous Moore's law about semiconductor efficiency is only 35% growth per year. This AI growth is over 10 times as large. That means that the computer chips that were used to train GPT-4 over three months could have been used to train the model with the performance of GPT-2 about 300,000 times over just four years later. This increase in effective compute took us from a model that was just about able to string some sentences together to GPT-4 being able to do things like: Beat most high schoolers at college entrance exams Converse in natural language -- which in the long forgotten past was considered a mark of true intelligence called the Turing Test Solve the Winograd schemas, a test of common sense reasoning that in the 2010s was regarded as requiring true understanding And create art that most people can't distinguish from the human produced stuff. So how much further can this driver of progress, pretraining, continue to scale? If current trends continue, then by around 2028 someone will have trained a model with 300,000 times more effective compute than GPT-4. That's the same as the increase that we saw from GPT-2 to 4. So if that was spent on pretraining, we could call that hypothetical model GPT-6. And so far we seem to be on that trend. GPT-4.5 was released in early 2025 and forecasters expect a GPT-5 size model to be released in the second half of the year. Can the trend continue all the way to GPT-6? The CEO of Anthropic, Dario Amodei, projects that a GPT-6 size model will cost about $10 billion to train. That's expensive, but still affordable for companies like Google, Microsoft, and Meta, which earn $50 to $100 billion in profits each year. In fact, these companies are already building data centres big enough for such training runs. And that was before the hundred billion dollar plus Stargate project was announced. In addition, frontier models are already generating over $10 billion of revenue, and that has been more than tripling each year. So soon AI revenue alone will be able to pay for a $10 billion training run. I'll discuss what could bottleneck this process more later. But the most plausible bottleneck is training data. GPT-4 already used the most easily accessible data on the internet for training, and we only have one internet. However, the best analysis I've found, by Epoch AI, suggests that there will be enough data to carry out a GPT-6 training run by 2028. And even if that isn't the case, it's no longer crucial -- because the AI companies have discovered a way to circumvent the data bottleneck, as I'll explain next. So the second driver of progress is training the models to reason with reinforcement learning. People often say ChatGPT is just predicting the next word, but that's never been quite true. Raw prediction of words from the internet produces outputs that are regularly crazy, as you might expect given that it's the Internet. GPT only became truly useful with the addition of reinforcement learning from human feedback, RLHF. In this process, outputs from the base model are shown to human raters. Then secondly, the raters are asked to judge which are most useful. Then third, the model is adjusted in a way that's expected to produce more outputs like the helpful ones, which is called reinforcement. A model that's undergone RLHF isn't just predicting the next token, it's predicting what human raters will find most helpful. You can think of the initial LLM as providing a foundation of conceptual structure, but RLHF is essential for directing that structure towards a particular useful end. RLHF is just one form of post training. Post training is named because it happens after pretraining, though in fact both are simply types of training. There are many other kinds of post training enhancements, including things as simple as letting the model access a calculator or the internet. But there's one that's especially crucial right now: reinforcement learning to train the models to reason. The idea is that instead of training the model to do what humans find helpful, it's trained to correctly answer problems. Here's the process: Show the model a problem with a verifiable answer, like a math puzzle. Ask it to produce a chain of reasoning to solve that problem, which is called chain of thought. If the answer is correct, adjust the model in a way that's expected to produce more outputs like that: reinforcement. Repeat that process over and over. This process teaches the LLM to construct long chains of hopefully correct reasoning about logical problems. Before 2023, this didn't really seem to work. That's because if each step of reasoning is too unreliable, the chains quickly go wrong. And if you can't even get close to a right answer, then you can't give the model any reinforcement. But in 2024, just as many were saying that AI progress had stalled, this new paradigm was in fact starting to take off. Consider the GPQA Diamond benchmark, a set of scientific questions designed so that people with PhDs in the field can mostly answer them but non experts can't, even with 30 minutes of access to Google. It contains questions like advanced quantum physics that I can't make any sense of, even though I studied physics at university. In 2023, GPT-4 performed only slightly better than random guessing on this benchmark. That means it could handle the reasoning required for high school level science problems, but it couldn't manage PhD level reasoning. However, in October 2024, OpenAI took the GPT-4o base model and used reinforcement learning to create o1. o1 achieved 70% accuracy at this benchmark, making it about equal to PhDs in the relevant field at answering these questions. It's no longer tenable to claim these models are just regurgitating their training data. Neither the answers nor the chains of reasoning required to produce them exist on the internet. Most people aren't answering PhD level science questions in their daily life, so they simply haven't noticed this progress. They still think of LLMs as basic chatbots. And it turns out o1 was just the start. At the beginning of a new paradigm, it's possible to get gains especially quickly. Just three months after o1, OpenAI released results from o3. o3 is the second version. It's named that because O2 is a telecom company. But please don't ask me to explain any other part of OpenAI's model-naming practices. o3 is probably just o1, but with even more reinforcement learning and another change I'll explain shortly. o3 surpassed human-level expert performance on GPQA. Now, reinforcement learning should be most useful for problems that have verifiable answers, such as in science, maths and coding. And in fact o3 performs much better in all of these areas than its base model GPT-4o. Most benchmarks of maths questions have now been saturated, which means that leading models can get basically every question right. In response, the research group Epoch AI created Frontier Math, a benchmark of insanely hard mathematical problems. The easiest 20% are similar to Olympiad level problems. The most difficult are, according to Fields Medalist Terence Tao, extremely challenging. They would typically need an expert in that branch of mathematics to solve them. Previous models, including o1, could hardly solve any of these questions. But in December 2024, OpenAI tested a version of o3 with better scaffolding than the now publicly released version, which they claimed could solve 25%. More recent testing of Google's Gemini 2.5 after this article was released showed that it could solve about 20% of problems on the Maths Olympiad, so it would be about in line with these results. At the time, these results went entirely unreported in the media. In fact, on the very day of the o3 results, the Wall Street Journal was running a story about how GPT-5 was behind and expensive. But this misses the crucial point that GPT-5 is no longer necessary. A new paradigm has started which can make even faster gains than before, even without GPT-5. In January, DeepSeek replicated many of o1's results. Their paper revealed that even basically the simplest possible version of the process works. That suggests there's a huge amount more to try. DeepSeek R1 also reveals its entire chain of reasoning to the user, and from that we can see its sophistication and surprisingly human quality. It'll reflect on its answers, backtrack when wrong, consider multiple hypotheses, have insights, and more. All of this behaviour emerges out of simple reinforcement learning. OpenAI researcher Sabastian Bubeck observed, “No tactic was given to the model. Everything is emergent. Everything is learned through reinforcement learning. This is insane.” The compute for the reinforcement learning stage of training DeepSeek R1 likely only cost about $1 million. If that keeps working, then OpenAI, Anthropic, and Google could now spend billions of dollars on the same process, approximately a 1000 times scaleup. One reason this is possible is that the models generate their own data. This might sound circular, and the idea that synthetic data causes model collapse has been widely discussed, but there's nothing circular in this case. You can ask o1 to solve 100,000 math problems and then only take the cases where it got the right answer, and then use those to train the next model. Because the solutions can be quickly verified, you've generated more examples of genuinely good reasoning. And in fact this data is much higher quality than the data you'll find on the internet because it contains that whole chain of reasoning and it's known to be correct. Not something the internet is famous for. This potentially creates a flywheel. Have your model solve a bunch of problems, use those solutions to train the next model, the next model can solve even harder problems, that generates even more solutions, and so on. If the models can already perform PhD level reasoning, the next stage would be researcher level reasoning and then generating novel insights. This likely explains the unusually optimistic statements from the AI company leaders that I mentioned at the start. Sam Altman's shift in opinion coincides exactly with the o3 release in December 2024. Although most powerful in verifiable domains, the reasoning skills developed will probably generalise at least a bit. It's common to see an AI model get reasoning training in one domain, like coding problems, and then also improve in other domains that weren't part of that training process. In more fuzzy domains, like business strategy or writing, it's harder to quickly judge success. So the process of reinforcement learning will take longer. But we should expect it to work to some degree, and it's a major focus of the companies right now. Exactly how well it will work is a crucial question going forward. The third driver of progress: increasing how long models think. If you could only think about a problem for a minute, you probably wouldn't get very far. If you could think for a month, you'd make a lot more progress, even though your raw intelligence isn't higher. LLMs used to be unable to think about a problem for more than about a minute before mistakes would compound or they would drift off topic, which really limited what they could do. But as models have become more reliable at reasoning, they've become better at thinking for longer. OpenAI showed that you can have o1 think 100 times longer than normal and get linear increases in accuracy on coding problems. This is called using test time compute: compute spent when the model is being run rather than trained. If GPT-4o could think usefully for about a minute, o1 and DeepSeek seem like they can think for the equivalent of about an hour. As reasoning models get more reliable, they will be able to think for longer and longer. At current rates, we'll soon have models that can think for a month and then a year. It's particularly intriguing to consider what happens if they could think indefinitely. That would mean that given sufficient compute, and assuming progress is possible in principle, they could continuously improve their answers to any question. Using more test time compute can be used to solve problems via brute force. One technique is to try to solve a problem 10, 100 or 1000 times and then to pick the solution with the most votes. This is probably another way that o3 was able to beat o1. The immediate practical upshot of all this is that you can pay more to get more advanced capabilities earlier. Quantitatively, in 2026, I'll expect you'll be able to pay 100,000 times more to get performance that would have previously only been accessible in 2028. Most users, of course, won't be willing to do this, but if you have a crucial engineering, scientific, or business problem, even $1 million is a bargain. In particular, AI researchers may be able to use this technique to create another flywheel for AI research. This is a process called iterated distillation and amplification, which you can read about in an article I link to. But here's roughly how it would work: Have your model think for longer to get better answers. This is called amplification. Use those answers to train a new model. That new model can now produce almost the same answers immediately without needing to think for longer, which is called distillation. Now take that new distilled model and have it think for longer. It'll be able to generate even more and better answers than the original model. Then you can repeat that process over and over. This process is essentially how DeepMind made AlphaZero superhuman at go within a couple of days even without using any human data. The fourth driver of progress: building better agents. GPT-4 resembles a co-worker on their first day of work who is smart and knowledgeable, but only answers a question or two before leaving the company. Unsurprisingly, this is only a little bit useful, but the AI companies are now turning chatbots into agents. An AI agent is capable of doing a long chain of tasks in pursuit of a goal. For example, if you want to build an app, rather than asking the model for help with each step, question by question, you'd simply say, Build an app that does X. It'll then ask you clarifying questions, build a prototype, test, fix bugs, and deliver a finished product -- much more like a great human software engineer would. Agents work by taking a reasoning model and then giving it a memory and access tools, which is called a scaffolding. Here's how it works: You tell the reasoning module a goal and it makes a plan to achieve that goal. And based on that plan, it uses the tools it's been given access to take some actions. The results of those actions are fed back into the memory module. The reasoning module then updates the plan based on that. And then the loop continues until the goal is achieved or it's determined not to be possible. AI agents already work a bit. SWE-bench Verified is a benchmark of real world software engineering problems taken from GitHub that typically take about an hour to complete. GPT-4 basically can't do these problems because they involve using multiple applications on your computer. However, when put into a simple agent scaffolding, GPT-4 can solve about 20% of these problems. Claude Sonnet 3.5 could solve about 50%, and O3 reportedly could solve over 70%. This means O3 is basically as good as professional software engineers at completing these discrete tasks. In fact, on competition coding problems, o3 would have ranked about top 200 in the world. Now consider perhaps the world's most important benchmark: METR’s set of difficult AI research engineering problems called RE-bench. These include problems like fine tuning models or predicting experimental results that engineers tackle to improve cutting-edge AI systems. These problems were chosen to be genuinely difficult problems that closely approximate actual AI research engineering. A simple agent built on o1 and Claude Sonnet 3.5 turns out to be better than human experts when given two hours. This performance exceeded the expectations of many forecasters and we haven't even seen the results of o3 yet. However, AI performance increases more slowly than human performance when given more time. So it turns out that human experts still surpass the AIs around the four-hour mark. So the AIs are better over two hours, but then the humans are better over four hours or more. But the AI models are catching up fast. GPT-4o was only able to do tasks that took humans about 30 minutes. To measure this rate of increase more precisely, METR made a broader benchmark of computer use tasks categorised by how long they normally took humans to do, which they called time horizon. GPT-2 was only able to do tasks that took humans a few seconds, GPT-4 a few minutes, and the latest reasoning models like o1 could do tasks that took humans just under an hour. This time is doubling roughly every seven months. If that trend continues to the end of 2028, AI will be able to do AI research engineering and software engineering tasks that take several weeks as well as many human experts. And interestingly, it looks like the trend since 2024 has been even faster, doubling every four months. And since this article was published, o3 was tested and it appears to be on the new even faster trend. This trend could be due to the new reasoning models paradigm that started in 2024, unlocking a faster rate of progress. If the faster trend continues, then we'll have models that can do multiweek software engineering tasks in under two years, almost twice as fast progress as before. AI models are also increasingly understanding their context. They can correctly answer questions about their own architecture, past outputs, whether they're being trained or deployed -- another precondition for agency. On a lighter note, while Claude 3.5 is still terrible at playing Pokemon, just a year ago Claude 3 couldn't really play at all. So we could say that AIs still don't make great agents, but they are improving fast. These results and graphs explain why although AI models can be very intelligent at answering questions, they haven't yet automated many jobs. Most jobs are not just a list of discrete one hour tasks. They involve figuring out what to do, coordinating with a team, long novel projects with lots of context, and so on. If even in one of AI's strongest areas, software engineering, it can only do tasks that take under an hour, then it's a long way from being able to fully replace software engineers. However, the trend suggests that there's a good chance this soon changes. As we said, if we project the 2020 onwards rate of progress forward, then we'll be reaching models that can do one day and one week tasks within a couple of years. An AI that could do one day or one week tasks would be able to automate dramatically more work than current models. Companies could start to hire hundreds of digital workers overseen by a small number of humans. So how far can this trend of improving agents continue? OpenAI dubbed 2025 the year of agents. While AI agent scaffolding is still primitive, it's the top priority for the leading labs, and that means we should expect more progress. More concretely, gains will come from hooking up the agent scaffolding to ever more powerful reasoning models, giving the agent a better, more reliable planning brain. Those in turn will be based on base models that have been trained on a lot more video data, which might make the agents much better at perception, a major bottleneck currently. The models are often unable to do things like recognise a button on a website, but that could get solved. Once agents start working a bit, that unlocks more progress. You can set an agent a task like making a purchase or writing a popular tweet. Then if it succeeds, use reinforcement learning to make it more likely to succeed next time. In addition, each successfully completed task can be used as training data for the next generation of agents. The world is ultimately an unending source of data, which lets the agents naturally develop a causal model of the world. Any of these measures listed above could significantly increase the reliability of agents. And as we've seen several times in this article, reliability improvements can suddenly unlock new capabilities. Even a simple task, like finding and booking a hotel that meets your preferences, requires tens of steps. With a 90% chance of completing each step correctly, there's only a 10% chance of completing 20 steps. However, with a 99% reliability per step, the overall chance of success of all 20 steps leaps from 10% to 80%: the difference between an agent that's not very useful to a very useful one. So progress could feel quite explosive. All this said, agency is the most uncertain of the four drivers. We don't yet have great benchmarks to measure it. And so while there might be a lot of progress at navigating certain types of tasks, progress could remain slow on other dimensions. A few significant areas of weakness could hamstring AIs’ applications. More fundamental breakthroughs might be required to make it really work. Nonetheless, the recent trends and the improvements already in the pipeline mean I expect to see significant progress. How good will AI become by 2030? The four drivers projected forwards. Let's recap everything we've covered so far. Looking ahead the next two years, all four drivers of AI progress seem set to continue and to build on each other. A base model trained with 500 times more effective compute than GPT-4 will be released, which we could call GPT-5. That model could be trained to reason with up to 100 times more compute than o1. So we could call that o5. It'll be able to think for the equivalent of a month per task when needed. It'll be hooked up to an improved agent scaffolding and further reinforced to be more agentic. And that won't be the end. The leading companies are on track to carry out $10 billion training runs by 2028. That would be enough to pre-train a GPT-6 sized base model and to do another 100 times more reinforcement learning or some other combination of the two. In addition, new drivers like reasoning models seem to appear roughly every one to two years. So we should project at least one more discovery like this in the next four years, and there's some chance we might see an even more fundamental advance, more akin to deep learning itself. In the article you can see a table summarising the four drivers of progress over the last four years and how they might evolve over the next four years. Putting all this together, people who picture the future as slightly better chatbots are making a mistake. Absent a major disruption, perhaps like an invasion of Taiwan or a major economic crisis, progress is not going to plateau here. The multitrillion dollar question is how advanced AI will get. Ultimately, no one knows, but one way to get a more precise answer is to extrapolate progress on benchmarks measuring AI capabilities, such as those I've mentioned earlier. Since all the drivers of progress are continuing at similar rates to the past, we can roughly extrapolate the recent rate of progress. In the article I have a summary of all the benchmarks we've discussed, plus a couple of others, and where we might expect them to be in 2026. Most of them seem set to be saturated, including BIG-Bench Hard, SWE-bench Verified, GPQA Diamond, most math benchmarks. More interesting is perhaps Humanity's Last Exam, a compilation of 3,000 questions at the frontier of human knowledge. Previously models could only answer under 3% of these in 2022, but by the end of 2024 that had risen to 9%, and by February 2025 that had already hit 25%. Projecting to 2026, I’d guess somewhere between 40% unsaturated. With frontier math, as we've said, that's risen from 0% in 2022 to maybe about 25% today, and I would guess perhaps 50% to saturated by the end of 2026. Finally, on METR’s time horizon benchmark, in 2022, models could do tasks that humans could do in about one minute. By the end of 2024 that had risen to 30 minutes. And if we project forward the slower rate of progress, by the end of 2026 they'll be able to do tasks that humans can do in six hours. At the faster rate of progress that we seem to have been on since 2024, that would be almost twice as long. So roughly one day long tasks. Putting all this together implies that in two years we should expect AI systems that have expert level knowledge of every field, can answer math and science questions as well as many professional researchers, are better than humans at coding, have general reasoning skills better than almost all humans, can autonomously complete many day long tasks on a computer, and are still rapidly improving. The next leap might take us to beyond human level problem solving, the ability to answer as yet unsolved scientific questions independently. So what jobs would these systems be able to do? Many bottlenecks hinder real world AI agent deployment, even for those tasks that can be done on computer. These include regulation, reluctance to let AIs make decisions, insufficient reliability, institutional inertia, and lack of physical presence. Initially, powerful systems will also be expensive, and their deployment will be limited by available compute, so they will be directed at only the most valuable tasks. That means that most of the economy will probably still continue much as normal for a while. You'll still consult human doctors even if they have AI tools advising them. You'll get coffee from human baristas and hire human plumbers. However, there are a few crucial areas where despite these bottlenecks, these systems could be more rapidly deployed with major consequences. The first of these is software engineering. This is where AI is being most aggressively applied today. Google has said about 25% of their new code is written by AI. And actually since I wrote this article, that's risen to maybe 50%. Y Combinator startups say that for some of their companies it's 95%, and those companies are growing several times faster than before. If coding becomes 10 times cheaper, we'll use far more of it. Maybe fairly soon we'll see billion dollar software startups with a small number of human employees managing the equivalent of hundreds of AI agents. When OpenAI launched, it became the fastest growing startup of all time in terms of revenue. Since then, several other AI companies have taken the record. Most recently Cursor, a coding agent. It reached $100 million of annual recurring revenue several times faster than previously very successful software startups in the past. So even this very narrow application of AI could still produce hundreds of billions of dollars of economic value pretty quickly, sufficient to fund continued AI scaling. And AI's application to the economy could expand significantly from there. Epoch AI, for instance, estimate that perhaps a third of work tasks could be performed remotely through a computer, and that automation only of those tasks could more than double the economy. The second area is scientific research. The creators of AlphaFold already won the Nobel Prize for designing an AI that solves protein folding. A recent study found that an AI tool made top material science researchers 80% faster at finding novel materials. And I expect many more results like this once scientists have adapted AI to solve specific problems, for instance by training on genetic or cosmological data. Future models might even have genuinely novel insights simply by asking them. But even if not, a lot of science is amenable to brute force. In particular, any domain that's mainly virtual but has verifiable answers, such as mathematics, economic modelling, theoretical physics, computer science, research could be accelerated by just generating thousands of ideas and then verifying which ones work. In fact, even in an experimental field like biology research, it's also bottlenecked by things like programming and data analysis, constraints that could be substantially alleviated by AI. A single invention like nuclear weapons can change the course of history, so the impact of any acceleration here could be dramatic. A field that's especially amenable to acceleration is AI research itself. Besides being fully virtual, it's the field that AI researchers understand best, have huge incentives to automate, and face no barriers to deploying AI in. Initially, this will look like researchers using intern level AI agents to unblock them on specific tasks like software engineering capacity, which is a major bottleneck, or even to help them brainstorm ideas. Later, it could look like having the models read all the literature, generate thousands of ideas to improve the algorithms, and automatically testing those algorithms in small scale experiments. An AI model has already produced an AI research paper that was accepted to a conference workshop. In the article, I linked to a long list of other ways that AI is already being applied to speed up AI research. Given all this, it's quite plausible that we'll have AI agents doing AI research before people have figured out all the kinks that enable AI to do most remote workshops. Broad economic application of AI is therefore not necessarily a good way to gauge AI progress. It might follow explosively after AI capabilities have already advanced substantially. So what's the best case against impressive AI progress by 2030? Here's the strongest case against it in my mind: first, concede that AI will likely become superhuman at clearly defined discrete tasks, which means that we'll see continued rapid progress on benchmarks, but it'll remain poor at ill defined high context and long time horizon tasks. That's because these kinds of tasks don't have clearly and quickly verifiable answers, so they can't be trained easily with reinforcement learning. They're also not normally contained in the training data either. That could mean the rate of progress on these kinds of tasks will be slow and might even hit a plateau. If you also argue that AI is very bad at these types of tasks today, then even after four to six more years of progress, it might still be bad. Secondly, argue that most knowledge work jobs consist significantly of these long horizon, messy high context tasks. For example, software engineers spend a lot of their time figuring out what to build, coordinating with others and understanding massive codebases, rather than just knocking off a list of well defined tasks. So even if their productivity at coding increases 10 times, if coding is only 50% of their work, their productivity will only roughly double. A prime example of a messy ill defined task is whatever's involved with having novel research taste. So you could argue that this task, which is especially important for unlocking an acceleration, is likely to be the hardest to automate. In this kind of scenario, we'll have extremely smart and knowledgeable AI assistants and perhaps an acceleration in some limited virtual domains, perhaps like mathematics research, but AIs will remain tools and humans will remain the main economic and scientific bottleneck. Human AI researchers will see their productivity increase, but not enough to start a positive feedback loop. AI progress will remain bottlenecked by novel insights, human coordination, and compute. These limits , maybe combined with problems finding a business model and other barriers to deploying AI, could mean that the models won't create enough revenue to justify training runs over $10 billion. That'll mean progress slows massively after about 2028. Then once progress slows, the profit margins on frontier models could collapse, because after one or two years, competitors will release free versions that are basically just as good. And once the profit margins are down, that will make it even harder to fund continued scaling. So I think that's the strongest case against I can make. The primary counterargument is the earlier graph from METR: models are improving at acting over longer and longer time horizons, which requires deeper contextual understanding and handling of more abstract complex tasks. Projecting this trend forward suggests much more autonomous models within four years. And as I've shown, this could be achieved via many incremental advances of the type I've sketched, but it might also happen via a more fundamental innovation that arises in the coming years. The human brain itself proves that such capabilities are possible. Moreover, long horizon tasks can most likely be broken down into shorter tasks, like making a plan, executing the first step, and so on. If AI gets good enough at shorter tasks, then long horizon tasks might rapidly start to work too. This is then perhaps the central question of AI forecasting right now: Will the horizon over which AIs can act plateau or continue to improve or perhaps even accelerate like it seemed like it maybe is recently? Here are some other ways AI progress could be slower or unimpressive: Disembodied cognitive labour could turn out not to be very useful even in science, because innovation, you could argue, arises mainly out of learning by doing across the whole economy. Broader automation, which will take much longer, could be required for innovation. Second, pre training could have big diminishing returns, so maybe GPT-5 and 6 will be disappointing. That could be due to diminishing data quality. AI could continue to be bad at visual perception, limiting its ability to use a computer -- see Moravec’s paradox. More generally, AI capabilities could remain very spiky, perhaps weak on dimensions that aren't yet even well understood, and these weak spots could really limit their application. Benchmarks could seriously overstate progress due to issues with data contamination and the difficulty of capturing messy tasks. An economic crisis, Taiwan conflict, other disaster, or massive regulatory crackdown could delay investment by several years. There could be other unforeseen bottlenecks. The planning fallacy is the observation that everything takes longer than it expects. And the reason for the planning fallacy is we don't anticipate all of the ways that something can go wrong. For deeper exploration of the skeptical view, see “Are we on the brink of AGI?” by Steve Newman, “The promise of reasoning models” by Matthew Barnnett, “A bear case: My predictions regarding AI progress,” by Thane Ruthenis, and the Dwarkesh podcast with Epoch AI. Ultimately, the evidence will never be decisive one way or another, and estimates will rely on judgement calls over which people can reasonably differ. However, I find it hard to look at the evidence and not put significant probability on AGI by 2030. When do the experts expect AGI to arrive? I've made some big claims, and as a non-expert, it would be great if there were some experts who could just tell us what to think. Unfortunately there aren't. There are only different groups, each with different drawbacks. I reviewed the views of these different groups of experts in a separate article, but one striking point is that every group has shortened their estimates dramatically. Today even many AI sceptics think AGI will be achieved in 20 years -- mid-career for today's college students. Since 2020 the mean estimate on Metaculus for when AGI will be developed has plummeted from 50 years to 5 years. There's problems with the definition used on Metaculus, but this graph reflects a broader trend of declining estimates. My overall read is that AGI by 2030 is within scope of expert opinion, so dismissing it as sci-fi is not justified. Indeed, the people who know the most about the technology seem to have the shortest timelines. Of course, many experts also think it'll take much longer, but if 30% of experts think a plane will explode and the other 70% say it'll be fine, as non-experts, we shouldn't conclude that it definitely won't. If something is uncertain, that doesn't mean it won't happen. Section 3: Why the next five years are crucial. It's natural to assume that since we don't know when AGI will arrive, it might arrive soon, or maybe in the 2030s or the 2040s and so on. Although that's a common perspective, I'm not sure it's right. As we've seen, the core drivers of AI progress are more compute and better algorithms. That means more powerful AI is most likely to be discovered when compute and the labour used to improve AIs is growing most dramatically. Right now, the total compute available for training and running AI is growing about three times per year, and the workforce is growing rapidly too. That means that each year the number of AI models that can be run increases three times. In addition, three times more compute can be used for training, and that training can use better algorithms, which means they get more capable as well as more numerous. Earlier I argued these trends can continue till 2028, but now we'll show that it most likely runs into bottlenecks shortly thereafter. The first bottleneck is money. Google, Microsoft and Meta are spending tens of billions of dollars to build AI chip clusters that could train a GPT-6 size model in 2028. But another 10 times scaleup would require hundreds of billions of investments. That's still doable, but it's more than their current annual profits and would be similar to another Apollo programme or Manhattan Project in scale. Then GPT-8 would require trillions of dollars. AI would need to become a top military priority or already be generating trillions of dollars of revenues to generate that type of investment, and that would probably already be AGI if we had that. Second, even if the money is available, there will be other bottlenecks such as the following: Electricity: Current levels of AI chip sales, if sustained, will mean that AI chips will use about 4% of US electricity by 2028, but another 10 times scaleup would require 40% of US electricity. That's possible, but it would require building a lot of power plants pretty fast. Chip production: Taiwan Semiconductor Manufacturing Company, TSMC, manufactures all of the world's leading AI chips, but its most advanced capacity is still mostly used for mobile phones. That means that TSMC can still produce five times more AI chips than it does now. However, reaching 50 times more chips would be a huge challenge, requiring massive construction of chip fabs. Third, latency limitations could also prevent training runs as large as GPT-7. So most likely the rate of growth in compute used for training slows around 2028 to 2032. Algorithmic progress is also very rapid right now, but as each discovery gets made, the next one becomes harder and harder because the easier ones get taken first. That means maintaining a constant rate of progress requires exponentially growing research workforce. In 2021, OpenAI had about 300 employees. Today it has about 3,000. Anthropic and DeepMind have also grown more than three times, and new companies have entered the space. The number of ML papers produced per year has roughly doubled every two years. It's hard to know exactly how to define the workforce of people who are truly advancing AI capabilities versus just selling the product or doing other broader ML research. But if the workforce needs to double every one to three years to maintain recent progress, that can only last so long before the talent pool runs out. My read is that growth can easily continue until the end of the decade, but will probably start to slow in the early 2030s, unless by then AI has already become good enough to substitute for AI researchers. Algorithmic progress also depends on increasing compute, because more compute enables more experiments. In fact, with enough compute, researchers can even conduct brute force searches for optimal algorithms. That means that slowing compute growth will correspondingly slow algorithmic progress too. If compute and algorithmic efficiency increased by only 50% annually, rather than the threefold per year they've been increasing recently, a leap equivalent to the leap from GPT-3 to 4 would take over 14 years instead of the two and a half that it actually did. Slower growth of compute and the workforce also reduces the probability of discovering a new AI paradigm. So putting all this together, there's a race: Can AI models improve enough to generate enough revenue to pay for their next round of training before it's no longer affordable? Can the models start to contribute to algorithmic research before we run out of human researchers to throw at the problem? The moment of truth will be around 2028 to 2032: either progress slows or AI itself overcomes these bottlenecks, allowing progress to continue or even accelerate. Two potential futures for AI. If AI capable of contributing to AI research isn't achieved before around 2030, then the annual probability of its discovery decreases substantially. Progress, of course, won't suddenly halt, it'll slow more gradually. In the article, you can see a graph where I look at the probability of AGI being discovered each year. I think it's increasing from now until around 2027, and then it starts to gradually decline, reaching much lower levels, perhaps 10 times lower than today, by the mid-2030s. So roughly, we can plan for two scenarios. Either we hit AI that can cause transformative effects by 2030, AI progress continues or even accelerates, and we probably enter a period of explosive change; or AI progress will slow. The models will get much better at clearly defined tasks, but they won't be able to do the ill defined long horizon work required to unlock a new growth regime. We'll see a lot of AI automation, but otherwise the world will look much more like normal. We'll know a lot more about which scenario we're in within the next few years. I roughly think of these two scenarios as a 50/50, though my estimates can vary between 30% and 80% depending on the day. And of course, hybrid scenarios are also possible. Scaling could slow more gradually or be delayed several years by a Taiwan conflict pushing AGI into the early ‘30s. But I find it useful to just start with a simple model. And of course the numbers you put on each scenario also depend on your definition of AGI, and also what kind of AGI you think will be transformative. I'm most interested in forecasting AI that can meaningfully contribute to AI research. AGI in the sense of a model that can do almost all remote work tasks cheaper and better than a human may well take longer due to a long tail of deployment bottlenecks. On the other hand, AGI in the sense of better than almost all humans at reasoning when given an hour, seems to be basically here already. Conclusion. So, will we have AGI by 2030? Whatever the exact definition, significant evidence supports this possibility. We may only need to sustain current trends for a few more years to get there. We're never going to have decisive evidence either way, but it seems clearly overconfident to me to think that the probability before 2030 is below 10%. Given the massive implications and serious risks, that's enough evidence to take this possibility extremely seriously. Today's situation feels like February 2020, just before the COVID lockdowns. A clear trend suggests imminent massive change, yet most people continue their lives as normal. In an upcoming article, I'll argue that AGI automating much of remote work and doubling the economy could be a conservative outcome. If AI can do AI research, the gap between AGI and superintelligence -- AI that's more capable than humans at almost all tasks -- could be short. This could enable the equivalent of a massive expansion in the research workforce, potentially delivering a century's worth of scientific progress in under a decade. Robotics, bioengineering, and space settlements could all arrive far sooner than is commonly anticipated. The next five years would be the start of one of the most pivotal periods in history. So thank you for listening. This was the first chapter in a new guide to how to help AI go well that I'm writing with 80,000 Hours. You can see a summary of the guide on 80000hours.org/agi/guide/summary. It includes a summary of our current thoughts about what to do about this issue, and also some tactical advice on how to switch into it. If you already know you'd like to switch, apply to speak to the team one on one. They can help you with planning, job opportunities, and introductions to people in the field. Otherwise, stay tuned for more chapters. Thanks for listening. Bye.