Latest / Embodied AI 101 / OpenAI o1: Teaching LLMs to Think Slow and Deep
Transcript
- OpenAI o1: Teaching LLMs to Think Slow and Deep. In September 2024, OpenAI introduced o1, a new large language model that radically rethinks how AI solves hard problems. Unlike previous GPT-4 variants, which generate answers quickly, o1 was trained to take its time – to literally “think” through a question with a lengthy chain of reasoning before answering. In OpenAI’s words, o1 “thinks before it answers” by producing an internal chain-of-thought then delivers its final response. The results are striking: on difficult math and science exams o1 dramatically outperforms older models. For example, on the USA Mathematical Olympiad qualifier (AIME), the previous model (GPT-4o) solved only 12% of problems on average, whereas o1 solved 74% with a single answer and up to 93% when given extra inference-time pathways. On a benchmark of challenging physics, biology, and chemistry questions (GPQA Diamond), o1 even surpassed the accuracy of human PhD experts. In competitive programming (Codeforces) o1 ranks in the 89th percentile of human contestants, whereas GPT-4 was around the 11th percentile. These gains come from re-engineering the model’s training and inference, embedding a native chain-of-thought and explicit reasoning rewards into the learning process. OpenAI reports that o1’s performance continues to improve if you give it more compute at inference time – the model is designed to keep getting smarter the longer you let it think. In short, o1 marks a shift from “fast AI” to a “slow, careful thinker”. The key idea behind o1 is to blend the successes of “chain-of-thought” prompting with reinforcement learning and search. In earlier work, researchers found that prompting an LLM to work out a problem step by step (“Think step by step”) often improves its reasoning. But GPT-4 and its predecessors were not explicitly trained to think that way; they tended to produce quick answers that please human raters. In fact, standard supervised fine-tuning and RLHF trains models to give fluent, quick responses, without rewarding deep, slow deliberation. OpenAI realized this needed to change. They devised a new training paradigm: during learning, the model generates a full chain of reasoning (potentially thousands of tokens long) and receives feedback on each step, not just on the final answer. A separate “process reward model” scores the quality of each reasoning step, and a verifier checks the final answer for correctness. The policy model (o1) is then updated via reinforcement learning to maximize a combination of the step-by-step rewards and answer correctness. In effect, the model learns how to reason, not just what output to give. As OpenAI explains, “to train a model to reason well, you need a reward signal that evaluates reasoning quality, not just final answer correctness”. The result is an LLM that, by design, thinks before answering. When o1 faces a complex question, it generates an internal chain of thought – a sequence of intermediate reasoning steps – completely hidden from the user. OpenAI describes these as “thinking tokens” that serve as the model’s scratchpad. These hidden tokens may run into the thousands for hard problems, as o1 explores multiple approaches, catches its own errors, backtracks, and refines its reasoning. Only when the chain-of-thought is complete does o1 emit a polished final answer. In practice this means o1 can do a lot of silent pondering; for example, in the demonstration OpenAI showed, the model actually displayed a “Thought for 5 seconds” message and then output a detailed breakdown of a cipher solution. This introspective reasoning is fundamentally different from earlier models. Traditional chain-of-thought models simply printed each step for the user, which can be noisy or misleading. In o1, that internal reasoning is kept private (and indeed OpenAI decided not to show the raw chain-of-thought to users). Instead, o1 is trained to distill its hidden analysis into a succinct final answer, or at most a short summary of its reasoning. Under the hood, the o1 training loop is roughly as follows (in broad strokes): for each problem in the training batch, the policy model generates a chain of thought and an answer. Those reasoning steps are parsed into logical chunks. A process reward model scores each step (evaluating whether that intermediate reasoning seems productive), and an outcome verifier gives a reward for getting the final answer right. The system then computes a combined reward – a mixture of per-step scores and the final correctness – and uses RL (e.g. PPO) to update the policy so that high-quality reasoning traces become more likely. OpenAI alludes to distributed PPO in their description of the method. The critical innovation is dense rewards on the reasoning process, not just on accuracy: the model learns why a chain-of-thought was good or bad. In effect, o1 is learning to “think better”: to break hard problems into simpler steps, recognize and correct its own mistakes, and switch strategies if needed. Crucially, OpenAI found that o1’s gains depend both on training cost and on inference cost. That is, the more RL training (compute) they poured in, the better the model got – and they discovered that even at inference time, giving the model more “thinking time” improved results. In the words of the paper, “the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)”. Conceptually this means we have a new scaling knob: instead of only scaling model size or pretraining data, we can scale test-time computation. In practice, OpenAI exploited this by running o1 into the weeds at inference. For example, to answer each AIME math question the model sampled many chains of thought – say, 64 separate attempts – and then took a majority vote (consensus) as the answer. This “64-sample consensus” significantly boosted accuracy. They went even further: generating 1000 candidate answers and using a learned scoring function to pick the best yielded a 93% AIME score. In short, by throwing compute at inference (running o1 many times with varied reasoning) the accuracy kept climbing. This tradeoff – more inference compute yielding better answers – is a hallmark of the o1 design. OpenAI’s evaluation suite shows the payoff. In most benchmarks that stress reasoning, o1 is far ahead of GPT-4 (and even ahead of other state-of-the-art models). For instance, on MMLU (a broad academic question set), o1 beats GPT-4o on 54 out of 57 subdomains. On Codesforces competitive programming tasks, its Elo rating corresponds to the 89th percentile of human coders (vs ~11th for GPT-4o). Most striking: on the highly-novel ARC-AGI visual reasoning benchmark they report, GPT-4o scored ~5%, but o1 scored ~32%, and a follow-up model o3 (which uses the same ideas plus even more compute) scored over 75%. These leaps suggest o1 has qualitative reasoning power we didn’t see before. The improvements carry over to coding tasks as well. OpenAI fine-tuned a version of o1 on programming problems and even entered it in simulated programming contests. Under strict contest rules (10 hours, limited submissions) it scored 213 points in the 2024 International Olympiad in Informatics – about the 49th percentile among human competitors. If relaxed (allowing thousands of tries), it achieved 362 points – well above the gold-medal threshold. In Codeforces competitions, GPT-4o had Elo ~808 (only 11th percentile), whereas this o1-based programming model hit Elo ~1807 (93rd percentile). Again, a lot of this success hinged on smart test-time search: the model sampled many solution candidates and used public test cases plus a learned ranking function to pick the best 50 to submit. Without that selection strategy, it would have scored far lower in the contest. These results illustrate how sweeping the new paradigm is – allowing the model time and trials at inference produces coding performance far beyond previous models. Despite this focus on raw reasoning power, o1 has been vetted through human preferences and safety evaluations. In blind comparisons on open-ended prompts, human raters generally preferred o1’s answers over GPT-4o’s in reasoning-heavy categories (data analysis, math, coding). However, o1 was less preferred on some creative or open-domain tasks, reflecting that its style is tuned for logic over chit-chat. OpenAI also reports that explicitly baking safety and value rules into the chain-of-thought improves robustness. By teaching o1 to reason about policies internally, they saw significant gains in “safe completion” rates on adversarial queries (for example, o1 passed 93.4% of tough harmful-prompt tests vs 71.4% for GPT-4o). In other words, a reasoning model that thinks through rules ends up complying more reliably under scrutiny. One notable choice is that users do not see o1’s raw reasoning. OpenAI argues that the internal chain-of-thought may contain exploratory or conflicting steps that could confuse users. Instead, o1 is trained to surface only the polished result. (In practice, the ChatGPT interface actually shows a short “thinking” delay and then a summary of the reasoning, but not every thought the model considered.) The developers emphasize that hiding the chain still gives them the benefit of “reading the model’s mind” for monitoring, without confusing or unleashing raw, possibly unaligned reasoning to the user. For example, they considered situations where a model’s private thought process might reveal undesirable strategies; their solution is to keep the raw scratchpad invisible, and if needed produce only a clean summary. In summary, OpenAI’s o1 demonstrates a new paradigm: training LLMs to reason like mathematicians or programmers, with built-in chain-of-thought and reward signals on reasoning. This is a departure from the “fast-answer” approach of past models. Under this paradigm, you can dial up the inference-time compute to keep improving accuracy – essentially the model is allowed to think as long as needed. Early results are impressive: math contest feats, human-level science problem solving, advanced coding contest performance. At the same time, the approach adds complexity: inference is no longer one-shot generation but a multi-step search, and the model may be slower. And despite its skill, o1 isn’t a perfect all-purpose chatbot – it excels at logic problems but can be overkill (or even preferred less) on casual dialogue. The o1 model is a preview of where language models are headed. Its successors (e.g. the rumored “o3” and beyond) are expected to push these ideas even further, possibly generating thousands of candidate reasoning paths and selecting among them. In effect, we’re seeing the LLM paradigm evolve toward something like internal multimodal “search” or even Monte Carlo tree search in reasoning space. For practitioners and researchers, the lesson is clear: if you want complex reasoning, throw more compute at inference and train models to use it wisely. The constraints on scale are now changing. Instead of just bigger models or more data, we have “test-time compute” as a resource. As the engineers of o1 showed, letting the AI ruminate longer can yield leaps in capability – in other words, we can teach LLMs not just what to say, but how to deliberate before saying it. Key takeaways: o1 is a new LLM trained with reinforcement learning to produce internal chains-of-thought. Its training gives rewards on the reasoning process, not just final answers. At inference it uses those long chains to “think” before replying, and performance rises if you let it think longer. The outcome is much better reasoning on hard tasks – from math competition problems to coding contests – albeit at the cost of extra compute and some trade-offs in style. This “slow thinking” approach opens up new uses: for instance, safety and alignment can be monitored by inspecting the chain-of-thought (even if users don’t see it). The work represents a shift towards LLMs that plan and self-critique their solutions internally. It’s an exciting advance and a sign of more capable, but more deliberative, AI systems to come.