Latest / Embodied AI 101 / From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation
Transcript
- From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation. Modern robots can perform impressively on short, well-defined tasks, but long-horizon manipulation — think of tidying an entire room or assembling furniture — remains a formidable challenge. A key obstacle is that success in long workflows depends on subtle, sequential progress, not just end results. So far, many approaches use video-based language models (video MLLMs) or foundation models to “watch” the robot and describe what it sees. These models are trained in a supervised fashion, essentially learning to be passive observers: they can label or caption each action (e.g. “the robot picks up block A”) but do not actively reason about how these steps add up toward the goal. In practice this means they recognize events but aren’t good at judging overall progress. As Liu et al. point out, such models “function as passive ‘Observers’ that recognize ongoing events rather than evaluating the current state relative to the final task goal”. In other words, they see what is happening now but not how well we’re doing. What if instead our model could act like a critic or coach – constantly checking whether the robot is still on the right path? That’s exactly the vision behind From Passive Observer to Active Critic, where the authors introduce PRIMO R1 (Process Reasoning Induced Monitoring), a 7-billion-parameter framework that transforms a video-language model into an active critic using reinforcement learning (RL). The idea is to push the model to generate an explicit chain-of-thought about the robot’s actions, effectively narrating each step and using that reasoning to estimate task progress. By training this critic via RL – rewarding it for accurately judging progress – the robot’s internal model learns to “think” through the process rather than just watch it. In the paper’s words, they “leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation”. In practice, PRIMO R1 sees a short video clip of the robot’s execution, but crucially it’s given as a structured input: the very first frame of the task (initial state) and the current frame are provided as anchors. The model processes everything between those anchors and is asked to articulate what happened and how far along the task is. For example, if a robot is stacking blocks and the video shows it raising a second block, PRIMO would generate a step-by-step reasoning (“First the robot placed a red block; now it is placing blue block number two. That means we’ve built two blocks out of a target of four blocks.”) and then output something like “50% complete”. This chain-of-thought output is explicit – the model is learned to put words to its reasoning. Under this setup, the model is no longer silently inferring progress; it is verbally critiquing its own actions. Behind this transformation is a clever training scheme. PRIMO R1 is first given lots of examples of long tasks with annotated progress. Instead of merely training on those annotations (which would still make it a passive predictor), the authors apply reinforcement learning to fine-tune the model. In this stage, the model proposes its reasoning and a progress estimate, and it receives a reward based on accuracy of that estimate. Because RL only needs a final reward signal, it does not require that the model’s intermediate steps be “exactly right” according to some ground-truth script. As one summary explains, RL “does not require correct ‘answers’ for every step, only a signal indicating how good the final outcome was”. This flexibility is key: it means PRIMO can learn its own reasoning strategy by trial and error, focusing on getting the final answer (progress) correct, rather than memorizing annotated step-by-step instructions. The authors in fact implement this via a variant of policy optimization (which they call Group Relative Policy Optimization, or GRPO) that optimizes the chain-of-thought policy for higher final rewards. In short, the model is rewarded not for mimicking some gold reasoning tape, but for producing reasoning that accurately predicts progress. Beyond training, the PRIMO architecture itself is carefully designed. To give the model clear context, the input is not just a raw video stream; it’s anchored between an initial frame and the current frame. Concretely, the model sees an image of the scene at time 0 and the image right now, plus possibly the frames in between. This explicitly tells PRIMO: “here is where we started, and here is where we are now.” By structuring the temporal input in this way, the model can more easily gauge “we’ve moved from this original state to that state” and reason about the transitions. This is fundamentally different from earlier video-LLM approaches that just take a clip without fixed reference points. With PRIMO’s anchors, the model has a built-in timeline, which greatly aids process reasoning. The heart of PRIMO R1’s power is its ability to articulate why it thinks the task is at a certain progress. During inference or fine-tuning, PRIMO is prompted (implicitly or explicitly) to “show its work” – to output an interleaved narrative of its thoughts (“I see the red block was picked, then the blue block”). This explicit chain-of-thought is something the RL objective encourages. In effect, PRIMO becomes an interpretable critic: instead of invisibly computing a metric, it speaks its reasoning. The authors emphasize that PRIMO “generates an explicit Chain-of-Thought to output the final progress estimate”. Thus, in scenarios where a robot might otherwise blindly push forward, PRIMO checks each step, helping catch errors early. It’s akin to a human instructor saying, “hold on – that object was supposed to go in the bin before opening the drawer; we’re off schedule,” except the model learned to produce that commentary autonomously. To train PRIMO R1, the authors also released a large new dataset. This PRIMO Dataset and Benchmark contains on the order of 300,000 video clips (roughly 116K for initial supervised fine-tuning and 182K for the RL stage). It pulls data from multiple sources: real-robot trials in the “AgiBot” environment, collectives of simulation tasks from BEHAVIOR-1k (household object manipulation), and the RoboTwin simulation suite. Each example in PRIMO has a precise progress annotation (e.g. “this task is 60% complete”) and, importantly, accompanies chain-of-thought annotations – essentially the ideal reasoning path that justifies that progress level. In practice this might look like a list of intermediate actions or a descriptive “dialog” of the task. Armed with this data, the model can first be supervised to imitate human-like reasoning, then fine-tuned with RL to refine it. In short, the PRIMO dataset gives the model not only input-output pairs, but the intermediate reasoning framework needed to learn how to build a narrative of the task. Results from PRIMO R1 are striking. Across a variety of long-horizon manipulation tasks, PRIMO significantly outperforms all baselines. Quantitatively, the model achieves about a 50% reduction in mean absolute error compared to previous specialized reasoning systems. Recall that this is measured on the task of progress estimation: given a video, how accurately (as a percentage) does the model predict how much of the task is finished. Cutting error in half is a huge improvement and shows that the chain-of-thought strategy really helps the model understand the timeline of actions. Even more impressively, this 7B model beats out much larger general-purpose video-language models of around 72B parameters. In the authors’ words, PRIMO “demonstrat[es] significant relative accuracy improvements over 72B-scale general MLLMs”. In other words, by specializing and training smartly, PRIMO R1 outmaneuvered giant multimodal foundations – a reminder that for tricky embodied tasks, targeted reasoning can trump raw scale. The paper also highlights performance on failure detection and generalization tasks. One key benchmark is RoboFail – a recently proposed suite for predicting failure states in robot policies. Essentially, RoboFail tests a model on its ability to foresee when a robot’s action sequence is going awry. Remarkably, PRIMO R1 achieves 67.0% accuracy on RoboFail, beating the state of the art and doing about 6 percentage points better than a strong baseline (OpenAI’s O1 model). This is after zero-shot exposure, meaning PRIMO was not specifically trained on those scenarios but still generalized its reasoning ability. The authors note that PRIMO “exhibits strong zero-shot generalization on difficult failure detection tasks”. In practice, this means that even in novel or unforeseen situations, the active critic catches blunders that a passive observer would miss. Taken together, these results suggest the following: by turning a video-language model into an active reasoning agent, PRIMO R1 provides far better supervision of long tasks. The combination of explicit chain-of-thought and RL fine-tuning allows a relatively small model to achieve state-of-the-art progress estimation and failure detection. The structured input and huge annotated dataset support the learning of process-level understanding. As one reviewer quipped, PRIMO lets the robot “explain its reasoning about how far it has come,” rather than just pointing at what it sees. The implications for robotics are significant. An active critic like PRIMO can serve as a checkpoint mechanism: after each few actions, it could predict whether the robot is still on course. If the predicted progress stalls or a failure is flagged, the robot can re-plan or call for human oversight. In this way, PRIMO-style models could make robot deployments much safer and more reliable, especially for home or industry tasks where failure is costly. Moreover, the use of RL to teach reasoning is a powerful recipe: it suggests that future embodied agents might use similar self-supervised internal feedback loops to hone their thinking. Of course, some caveats remain. Training such a model requires a large, specialized dataset (the PRIMO set with its hundreds of thousands of clips and reasoning annotations) and substantial compute. The current work was done offline (analyzing recorded videos), so integrating this into a live control loop will require further engineering. And while PRIMO outperforms passive vision models on these benchmarks, it’s still limited to the scenarios it was trained on (kitchen tasks, assembly tasks, etc.). Real-world robustness (lighting changes, unexpected objects) will need additional work. But the leap made here – adding “thinking” via RL to robotic perception – is both novel and promising. In summary, From Passive Observer to Active Critic charts a new direction for robot learning. By leveraging reinforcement learning to elicit explicit process reasoning (chain-of-thought) in a multimodal model, Liu et al. show that robots can move beyond mere observation to critical analysis of their own actions. The result is PRIMO R1, a model that can say to itself, “I’ve done X out of Y steps so far, and here is how I figured that out,” and thereby reduce mistakes before they happen. This work highlights the power of hybridizing large-language-model techniques with RL in physical tasks, and it opens the door to more introspective and adaptable embodied AI. References: The facts and quotes above are drawn from the authors’ paper and related summaries. In particular, note their claim that current video foundation models are “passive observers”, their introduction of PRIMO R1 with RL-driven chain-of-thought, the specialized structure of inputs and dataset, and the reported empirical gains (50% error reduction and 67% accuracy on RoboFail). The notion that RL only needs an end reward and not step-by-step labels is also invoked to justify the approach. All numeric results and descriptions above follow directly from the cited material.