Latest / Embodied AI 101 / High-Level Automated Reasoning with Qwen2.5-7B

Transcript

High-Level Automated Reasoning with Qwen2.5-7B · Embodied AI 101

• High-Level Automated Reasoning with Qwen2.5-7B. Recent work shows that smaller open models can “out-think” giants like GPT-4 by adopting strategic, search-based reasoning rather than just churning through examples. A new paper – billed as “HiAR-ICL” (High-level Automated Reasoning in In-Context Learning) – demonstrates this on hard math problems. By breaking reasoning into a small set of atomic actions and running a Monte Carlo Tree Search (MCTS) over these actions, the authors reported that a 7-billion-parameter Qwen2.5 model achieved 79.6% accuracy on the MATH benchmark, beating even GPT-4o’s 76.6%. In other words, a 7B open-source model, augmented with clever reasoning machinery, outperformed a much larger closed-source model. This result suggests that the structure of strategic reasoning can matter far more than sheer scale: with the right search-based “think tank” on top, a small model can rival or exceed the reasoning prowess of a system an order of magnitude larger. In this article we unpack how HiAR-ICL works, why MCTS and atomic reasoning actions are so potent for complex problems, and how it relates to parallel efforts like Microsoft’s rStar-Math. We’ll delve into the key ideas (chain-of-thought as a special case of broader actions, “thought cards” as reusable reasoning templates, cognitive complexity metrics, etc.), and explain why this line of work could change how we think about leveraging LLMs – for math and beyond – especially in robotics and embodied settings where logical planning often matters. The Challenge of Mathematical Reasoning for LLMs. Benchmarks like the MATH dataset (math contest problems) have long served as a litmus test for reasoning ability in language models. Despite startling performances on many language tasks, even GPT-4 initially struggled on MATH questions requiring multi-step algebra and problem solving. Traditional chain-of-thought prompting – having the model generate step-by-step reasoning in one long answer – did improve things but still fell well short of 100% correctness. (For context, GPT-4’s earlier published MATH score was only on the order of ~40–50% without special prompting. Even the latest closed-source “GPT-4o” preview only hits ~76.6%.). Why is math reasoning hard? These problems often require (1) exploring intermediate steps or subproblems (for instance, breaking a geometry problem into simpler parts), (2) tracking consistency across multiple steps, and (3) using background knowledge or formulas at the right time. Simply appending a few demonstration examples in the prompt (standard in-context learning) usually isn’t enough to handle such complexity. Large models do better, but we may have hit a plateau where bigger alone won’t easily reach near-human-level accuracy. This has led researchers to look beyond the vanilla prompt-and-complete paradigm. Ideas like self-reflection, verification, and even search-based planning (“Tree-of-Thoughts” and similar schemes) have emerged to give models a more systematic way to explore alternative reasoning paths. One parallel insight is that humans often solve hard problems by planning. We don’t just read a question and spit out an answer; we consider different strategies (sketch a diagram, rephrase the problem, break it down) and iterate. Can we endow LLMs with a similar higher-level “controllers” that decide how to think about a problem? The HiAR-ICL work shows one way to do exactly that: define a small menu of reasoning “moves” or actions, then use MCTS – the same search algorithm that powers AlphaZero’s chess play – to choose and sequence these actions. The result is a model that effectively pauses to think about the next step, rather than just keep auto-regressing tokens. Crucially, HiAR-ICL achieves these gains without additional training or external large teacher models. All the heavy lifting is done at inference time with the existing 7B Qwen model (an open model from Alibaba). The model itself is not fine-tuned on new data or endowed with new knowledge; instead, a search algorithm and a set of reasoning primitives let it use its existing knowledge more efficiently. As one summary puts it, this approach “enables small models to master math reasoning. without relying on a distilled teacher model, through multi-round self-evolution of deep thinking”. In other words, the system bootstraps its own reasoning process, much like playing out different lines in a game tree. HiAR-ICL: Atomic Reasoning Actions and MCTS. At the core of HiAR-ICL is a shift in how we handle in-context learning. Traditional ICL just appends example input–output pairs (or chain-of-thought examples) to the prompt. HiAR-ICL says: context can include something higher-level, namely abstract reasoning patterns. The method introduces five atomic reasoning actions that serve as building blocks for problem-solving. A WeChat summary of the work (translated) lists these five as:. System analysis – Carefully examine and rephrase the problem, identifying key givens and goals. One-step thinking – Perform a single logical or arithmetic inference. Chain-of-thought reasoning – Carry out a short sequence of steps in a row (the usual multi-step answer generation). Divide-and-conquer – Explicitly split the problem into subproblems. Self-reflection and correction – Check partial results or the final answer for mistakes and fix them if needed. These actions are meant to simulate different high-level cognitive moves a human might use. For instance, “system analysis” might involve listing relevant formulas or simplifying an equation before solving it. “Divide-and-conquer” explicitly tells the model: break the problem into smaller pieces first. “Self-reflection” means the model should pause and question, “Does this answer make sense? Did I make an arithmetic error?”. Each atomic action essentially changes how the model is prompted. Under the hood, the model can be told (via the prompt) to perform one of these actions. In effect, the model is equipped with a small toolbox of strategy primitives. During solution generation, it doesn’t just output tokens one by one blindly — instead, the reasoning process is seen as a sequence of these abstract actions, and the model is asked to execute them. This is where Monte Carlo Tree Search enters. Think of the problem-solving process as a game: the “game state” is the current problem plus any intermediate facts derived so far, and a “move” is choosing one of the atomic reasoning actions. Each action leads to a new state (for example, applying “divide-and-conquer” might generate two smaller subproblem descriptions). The model can then continue from there. MCTS explores this space of action sequences by simulating many possible paths. Concretely, the algorithm repeatedly:. Select an action to explore (using a balance of exploitation and exploration, e.g. UCB1). Expand by “applying” that reasoning action: in practice, the LLM is prompted accordingly to produce the next piece of the solution (a short chain of thought, or a division of the problem, etc.). Rollout/Evaluate by letting the model continue to the end (perhaps by default or by additional heuristic) and check if it reaches a correct answer. Backpropagate the success/failure score up the tree to update the values of earlier actions. Over many iterations, MCTS scores each partial plan by how often it leads to a valid solution. The search builds a tree whose branches are sequences of atomic actions. For example, one branch might be “system analysis → chain-of-thought → self-reflect,” while another might be “divide-and-conquer → one-step thinking → chain-of-thought,” and so on. The tree search uncovers which sequence yields the highest probability of solving the problem correctly. As the authors describe it, this process “explore[s] reasoning paths and construct[s] ‘thought cards’ to guide subsequent inference”. The “thought cards” are essentially short templates or strategies that summarize successful reasoning trajectories found by the search. Once MCTS has identified promising action sequences, those sequences can be reused like reusable hints: if a future problem is similar, the system can try the same sequence of actions (the same “card”). Importantly, the approach also includes a cognitive complexity framework. This is a lightweight heuristic that characterizes each problem’s difficulty or structure (e.g. number of subgoals, complexity of given conditions, or semantic similarity to known problem types). The framework matches each new problem to the most appropriate thought cards uncovered during search. In practice this means the model first estimates what kind of problem it’s dealing with (say, easy arithmetic vs. multi-part geometry) and then picks the reasoning strategies best suited for it. This dynamic matching helps the system apply prior successful patterns to analogous problems, rather than always searching from scratch. Putting it all together, HiAR-ICL extends the ordinary “context” of in-context learning. Instead of only passing examples and instructions, the context now includes abstract strategies (the thought cards) and the option to invoke complex search. It is, as the authors put it, a paradigm “that shifts focus from specific examples to abstract thinking patterns, extending the conventional concept of context”. In simpler terms, the model isn’t just mimicking a few sample answers — it’s actively planning out its reasoning in a structured way. Qwen2.5-7B in Practice: Results and Implications. The HiAR-ICL system was evaluated on the official MATH benchmark. Using the 7B version of the open-source Qwen2.5 model (with an instruct tuning), the method achieved 79.6% accuracy on MATH. This is quite striking. By comparison, the open reference says GPT-4o scored 76.6% on the same metric. In effect, the little 7B model can now solve more contest math problems correctly than GPT-4o. (For further perspective, note that high-powered models like GPT-4 usually score much lower on MATH with normal prompting, so 79.6% is pushing into territory previously thought accessible only to supermodels or human experts.). This gain is not just marginal; it reflects a completely new regime of solving problems. In many individual cases, HiAR-ICL found solutions that a raw chain-of-thought attempt would never have discovered. The Chinese writeup summarizes the impact well: “experimental results show that HiAR-ICL significantly improves LLM performance on complex reasoning tasks, especially for small models. In some cases, its performance even exceeds top closed-source models”. It further notes that compared to other tree-based methods, HiAR-ICL had better overall performance, wider generalization, and even better computational efficiency (more on efficiency below). How can we interpret “outperforming larger models”? Essentially, HiAR-ICL equips the 7B model with a kind of extended “thinking time”. Instead of spitting out an answer in one forward pass, it invests compute into exploring multiple reasoning branches. The underlying Qwen model, unchanged, doesn’t suddenly “know” new math — but it uses its existing knowledge more creatively. When it hits a sub-problem it can no longer solve in one shot, it can pause (through an atomic action) and treat that sub-problem as a new mini-goal. The tree search orchestrates this process. The result is that a relatively small network, guided by a strong search strategy, covers for its lack of raw capacity by effectively thinking deeper. This mirrors findings in other domains. For instance, Microsoft’s rStar-Math (an independent system) applies verified chain-of-thought and MCTS in a loop of self-improvement. That pipeline boosted a math-specialized 7B Qwen model from 58.8% to 90.0% on the same MATH benchmark. In concrete terms, Qwen2.5-Math-7B started at 58.8% and with rStar-Math reached 90.0%, easily surpassing GPT-4o’s previous 85.5%. Likewise, on the AIME math contest (similarly hard problems), rStar-Math solved an average of 53.3% of problems versus 44.6% for OpenAI’s model. These numbers underscore a key point: clever reasoning architectures (MCTS + modular actions + self-improvement) let 7B models leap ahead of their natural fixed-capability baseline. Back to HiAR-ICL: the advantage isn’t free – it comes with extra computation at inference time. Running MCTS over many branches means giving the model time to simulate various moves. However, the authors report that the approach is still reasonably efficient compared to other search-based schemes. The Chinese summary explicitly notes that HiAR-ICL “shows superior performance and generalization [to other tree-based methods], and also improvements in computation efficiency”. In practice, fewer branches or a smaller search budget might suffice thanks to the clever selection of atomic actions, so you don't need an astronomical number of simulations. And importantly, this system doesn’t require retraining the LLM or collecting more data. There is no separate large “wizard teacher” model giving solutions. In fact, part of the appeal is exactly that no huge teacher or extensive fine-tuning is needed. As one source remarks about the related rStar approach, it achieves this kind of reasoning uplift “without relying on a distilled teacher model”. HiAR-ICL follows the same spirit: the structure of the problem-solving process itself provides the necessary guidance. The model is essentially learning on the fly via search, rather than learning new weights via gradient descent. How Atomic Actions and Search Make the Difference. To appreciate why this works, let’s unpack the ingredients. At a high level, HiAR-ICL frames reasoning as planning in a symbolic action space. Each atomic action corresponds to a shift in how the language model is being prompted or steered. Contrast this with vanilla chain-of-thought generation: normally an LLM just continues generating text based on the previous tokens. There’s no explicit choice of how to reason at each step – it’s all buried in the weights. HiAR-ICL makes that choice explicit and controllable. Imagine a student solving a math problem. They might start by circling or summarizing what’s given (“system analysis”), or by plugging some obvious values into a known formula (“one-step thinking”), or by writing a brief proof outline (“chain-of-thought”), or by sketching how to break the problem into parts (“divide-and-conquer”), and then later double-checking their calculation (“self-reflect”). A normal LLM prompt might encourage one of these implicitly, but it won’t tell the model at each moment to explicitly perform one of these modes. HiAR-ICL does: it essentially tells the model, “Now use the divide-and-conquer action,” and lets the model produce the corresponding text. These clearly defined actions reduce ambiguity and guide the model’s internal attention. In RL terms, they’re like macro-actions that jump several steps at once. Because there are only five such actions, the search space is much smaller than if we tried to search over all possible token sequences. MCTS can quickly evaluate the utility of each action at each step. For example, if “system analysis” followed by “chain-of-thought” consistently yields solutions in similar problems, the search will learn to favor that branch. If “one-step thinking” seems less successful on a complex problem, the algorithm will explore that branch less. Over many simulated games, a value is estimated for each action, and the best sequence is chosen. The “thought cards” then serve as a memory of what worked. When search discovers a high-scoring reasoning sequence, that sequence is encapsulated as a card (complete with its pattern of breakdowns and partial answers). Later, when a new problem is addressed, the system looks up which card matches its difficulty level (as judged by the cognitive complexity heuristics) and can initialize the search from that card’s plan. In this way, HiAR-ICL doesn’t reinvent the wheel on every problem – it reuses successful blueprints. This architecture has several benefits. First, it expands the breadth of the search. LLMs often get “stuck” in one line of reasoning, but a tree search can revisit alternative strategies. Second, it improves depth selectively. Not all problems need very deep thought; the model might detect a simple problem and just do “one-step thinking” directly. The cognitive complexity framework helps here by assigning the problem to an appropriately shallow or deep plan. Third, it enables verification and correction. By including “self-reflection” as an action, the system allows itself to incorporate a feedback loop: if a partial solution looks wrong, it can backtrack or adjust, much like humans checking their work. This can be done for free in the search because unsuccessful branches (those with wrong answers) get pruned or given low value. Finally, the whole system is essentially a single-agent reasoning process – it doesn’t need a separate feedback agent or critic. All evaluation comes from checking if the answer matches known solutions or satisfies the problem’s conditions. In many setups (like rStar-Math) the model even generates code and runs it to verify correctness. HiAR-ICL uses its existing language model ability to judge correctness (for example, by asking if its answer is consistent with the problem). Because the atomic actions incorporate code and equations naturally, the LLM can literally “compute” to check some answers. A Comparison to Related Approaches. HiAR-ICL is part of a wave of recent work using search and planning to improve LLM reasoning. For context, one of the earliest ideas in this vein was Tree-of-Thoughts (Yao et al., 2023), which also built a search tree of reasoning paths. Others have proposed having the model re-think or self-critiquing its answers after the fact. What makes HiAR-ICL distinctive is the explicit use of a small, hand-crafted action vocabulary and dynamic selection of patterns via “thought cards.”. Microsoft’s rStar-Math is the most direct analog. It too defines a few key operations (including writing and running Python code) and uses MCTS + a learned step-level reward model to iteratively refine solutions. In fact, a writeup of rStar-Math reports very similar philosophy: no teacher model is needed, and multiple rounds of self-improvement (“deep thinking”) yield dramatic gains. The main difference is that rStar-Math involves a training loop that evolves both a policy and a reward model over time with millions of synthetic examples. HiAR-ICL, by contrast, seems to be purely an inference-time search – it does not explicitly train new models or rewards, but accomplishes much by clever prompt engineering and tree search. The result is comparable: rStar-Math reached 90% on MATH with Qwen2.5, and HiAR-ICL hits 79.6% without additional training. Another related trend is the idea of agentic LLMs or planning models. OpenAI’s recent models (sometimes code-named “o1” or “Gemini”) already attempt multi-step planning internally. Claude’s Sonnet or other “reasoning models” from big labs also integrate multi-step reasoning hints. What HiAR-ICL shows is that similar capabilities can be built on smaller models, if we architect their reasoning process carefully. In robotics terms, this is akin to giving a robot a hierarchical planner: the LLM is the low-level worker, and the set of atomic actions plus MCTS is the high-level controller. Of course, deploying MCTS does add complexity and latency. It must evaluate many possible continuations of each action. However, in many applications (especially reasoning or coding tasks done offline), this trade-off is worthwhile. The reports suggest that because the search space is structured (five actions) and the model is relatively fast (7B parameters can be queried thousands of times quickly), the total overhead is manageable. In practice, we might only run the search for a limit (say a few hundred simulations) and still see big accuracy gains. The documented improvements in efficiency suggest that HiAR-ICL may actually need fewer total model queries than some naïve tree-of-thought approaches, thanks to its smarter action design. Why This Matters. The big picture takeaway is that strategy can beat brute force. Until now, the dominant narrative was “bigger models get better at reasoning.” However, results like HiAR-ICL and rStar-Math demonstrate a different axis of progress: more structured thinking. This has several implications for embodied AI and robotics:. Resource efficiency: If a 7B model can outperform a 70B model by planning carefully, we can get away with smaller hardware footprints for on-board intelligence. Robots and drones, for example, often have limited compute – a HiAR-style architecture might let them use a compact neural model yet still solve puzzles or tasks requiring logical chaining. Interpretable reasoning: The atomic actions give us “reason codes” for what the AI is doing at each step (divide the problem, reflect after a step, etc.). This is closer to symbolic planning and might be easier to debug or verify than a black-box chain-of-thought. For safety-critical tasks, having discrete reasoning steps inspected is valuable. Transferable skill modules: The “thought cards” are analogs of reusable skills or subroutines. In robotics, this matches ideas like motion primitives or task templates. The ability to match a new problem to an existing template and attempt it is very much a high-level planning approach. Adaptivity and learning: Although HiAR-ICL itself is an inference-time trick, the framework could be extended to actually learn or refine those thought cards over time. A robot could gather more “solved problems” and generate new reasoning templates, gradually improving its internal planner. This bridges symbolic search and neural learning in a promising way. Of course, there are challenges. Running MCTS online can be slow for real-time tasks. Defining the right atomic actions requires insight (the five chosen here fit math problems) and might need redesign for, say, visual tasks or other domains. The cognitive complexity measures used to match problems to templates were hand-coded; making them robust is nontrivial. Still, these are engineering hurdles, not show-stoppers. The core idea – equip the model with a repertoire of thinking moves and let it search over them – is powerful and general. Conclusion. In summary, “High-Level Automated Reasoning” as exemplified by HiAR-ICL is a leap forward in how we use LLMs for complex tasks. By codifying five atomic reasoning actions, employing Monte Carlo Tree Search to explore action sequences, and storing promising chains as reusable thought cards, a 7B Qwen model can far outthink a much larger counterpart on math problems. This undermines the notion that only gigantic models can handle multi-step reasoning. Instead, it highlights that the architecture of reasoning – how the model decides what to think next – can be even more important than raw parameter count. For roboticists and ML researchers, this suggests exciting new directions. We can imagine integrating HiAR-like planners into robot task planning: the robot’s language model might consult a search module to plan assembly steps, navigation strategies, or dialogue responses. It also indicates that future benchmarks and methods should consider not just how good a model is as-is, but how well it can search and deliberate. In an era where we constantly build bigger neural nets, “smaller but smarter” is a tempting alternative. HiAR-ICL shows that with a bit of strategic scaffolding, a lean model can become an expert reasoner. As we continue to develop embodied AI systems, combining the best of neural pattern recognition with such structured reasoning will likely be a key to truly intelligent behavior. Sources: The above is based on the HiAR-ICL paper (Wu et al. 2024) and related reports, as well as performance summaries from Qwen and rStar-Math evaluations. These confirm that the atomic-action+MCTS approach yields ~79.6% on MATH with Qwen2.5-7B, surpassing GPT-4o’s ~76.6%, and exemplify how structured reasoning enables small models to outperform much larger ones.