Latest / Embodied AI 101 / SOLE-R1: Can Video-Language Rewards Train Manipulation Policies?
Transcript
- SOLE-R1: Can Video-Language Rewards Train Manipulation Policies?. SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning • Philip Schroeder et al. • MIT and RAI Institute • arXiv • 2026. Can a robot learn a manipulation skill from random actions when its only task reward comes from a model watching video?. SOLE-R1 provides a credible, carefully scoped yes. Across the authors’ reported 40-task evaluation, its rewards enable policies to reach at least 50 percent success on 24 tasks. GPT-5 reaches that threshold on seven, and the Gemini baselines on five. That is a substantial difference in this experimental setting—not a small improvement in an offline video-scoring benchmark. But two distinctions matter immediately. “From scratch” describes the robot policy, not the extensively trained reward model. And “zero-shot” describes transferring that reward model to the downstream task, not executing the task successfully without policy training. My assessment is that SOLE-R1 advances reusable reward modeling, particularly the question of whether a learned evaluator remains useful while an optimizer actively searches for its mistakes. It does not establish unattended, general-purpose robot learning. This episode anchors its quantitative assessment to the May 26, 2026 paper revision and checks the released artifacts separately. What the Paper Did. The system separates deciding what to do from judging whether it helped. SOLE-R1 is a Qwen3-VL-based, eight-billion-parameter video-language reward model. Its documented interface accepts a task description, observations from the first, previous, and current timesteps, and the previous progress estimate. Multiple camera views can be included. It returns a textual explanation followed by a numerical progress prediction. Imagine asking the robot to put a container inside a drawer. The evaluator’s job is not to produce a motor command. It should distinguish approaching the container, establishing a grasp, moving it toward the drawer, and actually releasing it inside. A useful evaluator must also recognize reversals: dropping the container cannot remain “almost finished” merely because the gripper is now near the destination. The previous scalar prediction supplies a compact history. The initial observation supplies a reference. Comparing neighboring observations gives the model an opportunity to distinguish an object moving with the gripper from an object merely disappearing behind it. That is my interpretation of why this interface is sensible; the interface itself does not guarantee those distinctions will be made correctly. There is an important architectural consequence. This is not an unlimited-memory video observer. A deployment team should ask whether the supplied observations contain the evidence needed to judge the task. If the relevant event happened earlier and left no visible trace, a fluent explanation cannot recover missing information. The downstream policies use DrQv2-style and SAC implementations. DrQv2 belongs to the off-policy, actor–critic family of visual-control algorithms: it learns from pixel observations using data augmentation and stored experience. “Online learning” therefore should not be confused with a strictly on-policy optimization algorithm. The distinction matters operationally. Newly collected experience can be relabeled by the reward service and reused during learning. There is no requirement to differentiate through SOLE-R1 or make the language model produce actions. The methodological question is whether replacing the task reward supplies enough useful information for an otherwise conventional learner. The main evaluation spans RoboSuite, ManiSkill, Meta-World, LIBERO, and physical manipulation. Policies receive two RGB streams and proprioception, while the task-reward supervision comes from the frozen evaluator rather than downstream demonstrations or ground-truth success labels. The difficult part is preparing that evaluator. In their author discussion, Schroeder and Weng describe starting with RoboCasa demonstrations, inserting random actions to generate deviations, and sometimes interpolating back toward a later expert state. Simulator geometry supplies progress supervision and grounded descriptions. Real-world Open X-Embodiment videos contribute visual diversity, with temporal perturbations providing additional regression examples. These are meaningfully different ways to manufacture negative evidence. Reversing a successful video illustrates undoing a successful action. Injecting actions can expose states that never occur along the expert route. Consider an object knocked beside a receptacle: that is not necessarily represented by running a correct placement backward. However, I would reserve judgment about how completely such synthesized trajectories represent deployment failures. Recovery through state interpolation is a data-generation convenience, not automatically a dynamically realizable recovery policy. For reward learning, the question is whether the resulting visual transitions teach distinctions that remain valid during actual interaction. The project reports more than a million generated reasoning examples from over 40,000 videos, combined with broader spatial and temporal reasoning data. Training then has two stages. Supervised fine-tuning teaches the explanation-and-answer behavior. Reinforcement learning with verifiable rewards emphasizes numerical progress accuracy; the authors motivate this by noting that a scalar answer occupies relatively few tokens compared with its explanation. The GRPO objective rewards parseable formatting and agreement with progress labels. “Verifiable” needs careful interpretation here. Agreement with a supplied progress target is verifiable. The physical meaning of that target is a separate question. An optimization procedure can improve agreement with an imperfect annotation scheme without making the annotations more faithful to task completion. Indeed, simulator labels include task-weighted geometric distances. So my reading is not that reward engineering disappears. It is that engineering is moved upstream and potentially amortized: construct informative supervision once, train a transferable evaluator, then avoid rebuilding the reward for every downstream task. That is still valuable. Reusability, rather than the absence of all human assumptions, is the defensible promise. How Strong Is the Experimental Evidence?. The headline result should be interpreted as task coverage. Twenty-four tasks passing a success threshold is not a 60 percent average rollout success rate. Nor does it mean 24 distinct physical skills were learned on hardware. It counts qualifying task conditions across the combined evaluation. The project reports three training seeds for simulation and one for physical experiments. The gap is large enough to take seriously. But a thresholded count conceals how far each result lies from the cutoff. A method that improves a task from 49 to 51 percent adds one “solved” task; improving another from 60 to 95 percent adds none. I would use this aggregate to establish breadth, then inspect the per-task outcomes before choosing a reward model. The physical-task chart is also not a uniform victory. It shows, for example, 80 percent for picking the can and 70 percent for opening the drawer with SOLE-R1. GPT-5 reaches 100 percent on that drawer-opening condition. Drawer closing reaches 100 percent for multiple methods. These are useful counterexamples to the stronger claim that specialized reasoning always wins. Evaluation typically uses twenty episodes in simulation or ten on hardware, and transient success counts. That last choice changes what the result establishes. If your application requires an object to remain securely held, a benchmark that credits briefly achieving the target state is answering a less demanding question. The benchmark can still demonstrate learning, but it is not interchangeable with a production acceptance test. With ten evaluation episodes, one different outcome moves the reported rate by ten percentage points. More importantly, repeated evaluations of one trained policy do not measure variability across training runs. A second initialization might discover the task quickly, remain stuck, or find another exploit. My conclusion is therefore stronger about the aggregate simulation pattern than about the precise reliability of an individual physical result. The baseline selection is relevant rather than obviously antiquated. However, understanding the comparators requires reading their own papers. RoboReward trains reward models using robot videos, counterfactual instruction relabeling, and temporally clipped examples. Its physical RL experiment fine-tunes a pretrained diffusion policy on a WidowX arm using diffusion steering. That is different from requiring a reward model to support exploration from a random controller. ReWiND likewise pretrains a policy with offline RL before adapting it online to unseen task variations. Its reward-learning pipeline uses demonstration data and video rewinding. These are not excuses for poor performance in SOLE-R1’s protocol. The protocol deliberately asks a demanding question. But they limit the conclusion: failure as the sole reward for random-start learning does not establish that a model is ineffective for improving an already competent policy. Robometer deserves an additional qualification. Its own formulation explicitly incorporates failed and suboptimal trajectories, combining frame-level progress supervision with trajectory-comparison preferences. Its dataset includes substantial mixed-expertise experience. It would therefore be inaccurate to summarize the competing literature as learning only from successful demonstrations. The stronger distinction is about the combination of supervision and evaluation: dense reasoning about local changes, structured non-expert data, and an online optimizer that tests whether the resulting reward can actually support learning. Baseline prompting was also compared across camera and temporal-context configurations. That makes “they probably used one bad prompt” an insufficient rebuttal. Conversely, comparing several prompts does not exhaust every possible adaptation of a general-purpose VLM. For practitioners, I would describe the evidence as a strong result under a demanding, specific operating regime—not a universal ranking of robotic reward models. One additional control would greatly help interpret the remaining failures: train the same controller with a reliable task reward under the same interaction budget. If that learner also fails, exploration or control may be the bottleneck. If it succeeds while the learned-reward version does not, reward quality becomes the more convincing explanation. What the Ablations Say About Reasoning. The authors report different failure changes when components are removed. Removing reasoning makes learning more signal-limited. Removing non-expert videos or broad spatial reasoning data increases reward-hacking failures. They also distinguish policies receiving inflated scores from policies that fail while the evaluator remains pessimistic. That distinction is more informative than simply saying every component improves success. A reward can fail by being too generous: the optimizer finds something easy that looks successful. It can also fail by being uninformative: every reachable behavior looks equally bad, leaving no useful preference among exploratory actions. My interpretation is that a deployment evaluation should test both properties separately. Consider two hypothetical evaluators. One correctly rejects every failed grasp but cannot distinguish approaching the object from moving away. Another gives useful approach rewards but confidently declares success whenever the object is occluded. The first may be difficult to exploit yet useless for bootstrapping. The second may accelerate early learning before producing a persistent exploit. A single offline accuracy number could obscure this difference. The reported reasoning ablation supports retaining the explanation channel in this particular training recipe. It does not establish that natural-language reasoning is inherently necessary for robust rewards. Removing explanations changes the supervision problem as well as the inference procedure. To isolate the mechanism, I would want comparisons against a numerical predictor with carefully matched training effort, a model with additional visual auxiliary supervision, and a supervised objective that gives the progress answer more weight. A further experiment could retain the reasoning-trained representation while removing generated explanations at deployment. Those tests would distinguish several possibilities: explanations might improve representation learning, add useful computation at inference, regularize the dataset, or combine all three effects. The paper also reports an improvement from RLVR over SFT alone. Again, the useful engineering conclusion is narrower than a philosophical claim about reasoning. Explicitly optimizing the numerical output appears worthwhile in this system. Whether GRPO is the most efficient way to achieve that remains a comparison to make, not a settled consequence. I would also avoid treating a convincing explanation as an audit certificate. An evaluator can state the right subgoals and still assign the wrong number. For a reward model, the number’s effect on the learned behavior is ultimately more consequential than the plausibility of its accompanying prose. That is why SOLE-R1’s closed-loop experiments are important. They ask whether the explanation-and-score system helps a policy, not merely whether its answers sound like competent robotics commentary. A Subtle Issue: Progress Is Not Automatically the Right Objective. SOLE-R1 clips and scales absolute predicted progress into reward, rather than relying only on progress differences. This choice is easy to overlook because a progress curve intuitively looks like a reward curve. But I would test the resulting incentives explicitly. Imagine a robot receives substantial credit for holding an object above a bin. Actually placing the object requires entering an uncertain visual configuration and releasing it. If hovering generates a reliable stream of intermediate reward, the controller may prefer hovering to taking the final risky step. This example is a hypothetical diagnostic, not a reported SOLE-R1 failure. It illustrates why even a reasonably accurate progress estimator can induce an undesirable objective when accumulated over time. The classic reward-shaping result from Andrew Ng, Daishi Harada, and Stuart Russell provides a useful reference. Potential-based shaping preserves optimal policies under specified conditions when the shaping term is constructed from discounted differences of a potential and added to the original task reward. Simply replacing the task objective with an arbitrary progress score does not inherit that guarantee. Nor is subtracting successive model outputs an automatic repair. In a system without an independently correct task reward, differencing alone does not supply missing semantics about completion. It can also make sensitivity to prediction noise more important. My practical question would be: what behaviors maximize the actual accumulated return, rather than the score at one attractive frame?. Test stalling near completion. Test repeated approach-and-retreat cycles. Test entering and leaving an ambiguous camera configuration. Test whether the policy prefers a visually clear partial state over an uncertain but genuinely successful one. The interface’s dependence on the previous score suggests another diagnostic. Present the same current visual evidence with different preceding estimates. If the answer changes substantially, determine whether that dependence reflects legitimate history or numerical anchoring. The released client supports a mode that scores timesteps relative to the initial frame rather than propagating the previous prediction. That offers a concrete starting point for this comparison. I would not necessarily deploy that alternative. I would use it to discover how much reward stability depends on the evaluator’s own earlier answers. What Generalization Has—and Has Not—Been Demonstrated. The right object of transfer is the reward model. A robot policy trained through new interaction is not the same thing as a policy transferred directly from simulation to hardware. Consequently, I would not describe this as eliminating the conventional policy sim-to-real problem. It changes where transfer is required: the evaluator must recognize useful progress across visual and task changes, while the controller learns within the downstream setting. This can be a favorable division of labor. A reward model need not reproduce every detail of an arm’s dynamics to recognize that a drawer moved. But it still needs sufficient evidence to distinguish movement caused by successful interaction from a misleading change in viewpoint. We should also distinguish unseen task instances from unseen manipulation concepts. A new drawer, camera, or instruction may test meaningful transfer without requiring a genuinely new understanding of articulation. There is a concrete reason to avoid the strongest interpretation of “unseen”: the training-mixture table lists LIBERO-derived Embodied CoT data. That does not establish leakage into a particular evaluation. It does mean that “no downstream policy demonstrations” should not be restated as “the entire learned system has never encountered related benchmark imagery or task semantics.”. For a stronger generalization claim, I would request a trajectory-level split audit across all reward-training sources, not just the progress-supervision subset. I would also separate new objects, new scenes, new embodiments, and new task primitives in the results. These are different distribution shifts and need not share a solution. The diversity experiment is encouraging. The published plot rises from three qualifying tasks with one training task type to 24 with seven types; the experimental protocol holds total training steps fixed. My reading is that broader training coverage improves this reward-learning recipe. Calling the result a predictive scaling law is a stronger step. Seven curriculum variants, evaluated through thresholded success counts, do not by themselves tell us how performance will grow under a much larger data budget. I would want repeated selections of task categories, uncertainty estimates, and tests where the added categories are deliberately unrelated to the evaluation. Otherwise, additional diversity may partly help because it introduces precisely the progress concepts the tests require. That is not a criticism of useful data. It is a distinction between learning transferable structure and establishing a law that predicts future scaling. Can a Competent Team Reproduce and Deploy It?. The release is more useful than a project page promising future artifacts. As checked for this episode on October 11, 2026, the linked resources expose a model checkpoint, an approximately two-terabyte training dataset, and inference tooling. The dataset includes images, prompts, explanations, and progress labels. The model repository provides sample videos, outputs, and reward client/server components. However, I could not verify a complete end-to-end training and benchmark reproduction pipeline from the visible repository. Its issue tracker still lists an open July 16, 2026 issue titled “SFT and RLVR code.” Availability of inference code should not be conflated with verified reproducibility of every experiment. The paper does provide optimizer, sampling, prompting, and policy hyperparameters. The authors describe roughly a week for supervised training and another couple of days for RL refinement. My assessment is therefore split. Reusing the released evaluator looks substantially easier than reproducing its full training history. Reproducing the physical experiments additionally requires matching the control stack, sensing arrangement, reset distribution, and evaluation procedure. Those physical conditions are consequential: manual resets, fixed gripper orientation, bounded workspaces, one-hertz actions, and twenty-step episodes. These are sensible choices for an experiment. They also show why “no task-specific reward engineering” is not “no task-specific engineering.” Restricting orientation removes a large part of the exploration problem. Constraining the workspace changes which bad states are reachable. Human resets supply recovery that the learning system itself has not demonstrated. Reward inference is another system-level constraint. The serving documentation targets an H100-class GPU, while the paper measures forty seconds to annotate forty frames. That can be compatible with asynchronous learning, but I would measure the complete experience-to-update delay rather than treating model latency in isolation. A fast actor is not useful if reward labeling falls behind, and an apparently sample-efficient learner may still be operationally slow. The physical comparisons also use different sampling and timing arrangements: the API baselines receive ten sampled timesteps versus SOLE-R1’s twenty, with runs lasting up to five hours versus two. My interpretation is that these experiments compare workable systems, not perfectly equalized observation and inference budgets. That is relevant for deployment, but it complicates attributing every difference exclusively to reward-model intelligence. There is one small integration detail worth checking before any robot run. The server’s returned field indicates whether the request succeeded; task-progress predictions are returned separately. An infrastructure acknowledgment must never accidentally become a task-success signal. I inspected the documentation and released artifacts for this analysis. I did not independently run the model or reproduce the learning curves. What This Means for Practitioners. I would consider SOLE-R1 a serious candidate for a research reward service, especially when the desired task outcome is visually observable and writing a reliable task reward would otherwise require considerable effort. I would not initially use it as an unquestioned authority. The first deployment milestone should be demonstrating that its mistakes are understood well enough to train against—not demonstrating that it can narrate a successful video. Start with a deliberately difficult local evaluation set. Include correct completion, partial completion, wrong-object interactions, contact without a stable grasp, occlusion, and actions that look promising but do not change the relevant object state. Where possible, pair visually similar successes and failures. For example, compare putting an object inside a bowl with putting it directly behind the bowl. Compare touching a button with actually actuating it. Compare a held object with one hidden by the fingers. These are proposed tests, not additional SOLE-R1 benchmark results. I would prioritize false high rewards over an aggregate correlation score. A model can order most ordinary trajectories correctly while retaining one reachable, highly rewarded mistake. An optimizer has an incentive to concentrate experience around that mistake. Then perform a limited closed-loop trial. The objective of this trial should be to discover whether optimization changes the error distribution. Keep an independent completion measure for evaluation, even if it is never provided as a training reward. That separation preserves the scientific claim. Not training on a ground-truth signal does not require refusing to measure the ground truth. I would log videos, action timing, raw model responses, parsed scores, model version, prompt version, and labeling delays. When learning fails, ask which of three things happened: the controller never explored a useful state; the evaluator failed to recognize useful progress; or the controller discovered an easy way to receive undeserved credit. Those failure types suggest different interventions. More exploration does not repair a systematic false positive. Better calibration does not automatically create a useful learning signal near the initial state. A more capable starting policy may solve a bootstrapping problem without changing the evaluator. For that reason, I would also test SOLE-R1 with the policy prior I actually intend to deploy. Random initialization is a strong stress test, but it is not necessarily the most economical operational choice. The paper’s separate policy-steering experiment improves success from 2 percent to a 12 percent median. That supports investigating the direction, while remaining far from evidence that this reward service reliably upgrades any generalist policy. Finally, keep task progress separate from the robot’s operational constraints. In my proposed deployment, collision limits, force limits, workspace boundaries, watchdogs, and reset procedures would remain independently specified. I would not ask an uncertain visual progress estimator to become the safety controller simply to preserve the rhetorical purity of “sole reward.”. Once learning stops, there is also no architectural reason to keep the evaluator in the action loop if only the trained actor is being deployed. That makes the economics of training-time reward inference different from the economics of continuous autonomous adaptation. The clearest contribution of SOLE-R1 is therefore not that language can describe a reward, or that a VLM can recognize a successful demonstration. Those are insufficient tests. The useful advance is showing that a specialized video-language evaluator can support substantial learning while exposed to the unusual states produced by optimization. Its training recipe takes failure coverage seriously, and its evaluation asks a harder question than static video scoring. My takeaway is to treat learned rewards as components that must be tested under optimization, not just validated as predictors. SOLE-R1 gives a promising implementation of that idea. The remaining work is to establish how reliably it survives broader tasks, stronger policies, longer runs, and less carefully controlled physical environments.