Latest / Embodied AI 101 / SARM2 + SPIRAL: Multi-Task Reward Models and RL Refinement for Long-Horizon Dexterous Manipulation

Transcript

SARM2 + SPIRAL: Multi-Task Reward Models and RL Refinement for Long-Horizon Dexterous Manipulation · Embodied AI 101

• SARM2 + SPIRAL: Multi-Task Reward Models and RL Refinement for Long-Horizon Dexterous Manipulation. In recent years we’ve seen vision-language-action (VLA) policies emerge as powerful “foundation” controllers for robots. Big models trained on massive data (images, language instructions, and actions) can already perform impressive manipulation behaviors out-of-the-box. However, those pretrained policies still struggle on long-horizon, contact‐rich tasks (think cloth folding, whiteboard cleaning, etc.) without further fine-tuning. Traditionally, robot teams have fine-tuned such policies via behavior cloning or imitation learning from human demos – but that approach is costly and tends to keep the robot “stuck” near the demonstration behaviors. The new work SARM2 + SPIRAL (by Huang et al.) tackles this bottleneck by learning autonomous reward models for manipulation and using them to drive an on-robot reinforcement learning (RL) loop for self-improvement. The result is a system that can automatically generate dense rewards for long tasks and then use RL to refine a dexterous policy, all with minimal human supervision – yielding dramatic gains over pure imitation baselines. In a nutshell, SARM2 is a Multi-Task, Stage-Aware Reward Model that produces a dense reward signal at every step of a manipulation task. Intuitively, it watches the robot’s video + proprioceptive stream and continually estimates how close the robot is to completing the (language-specified) goal. By contrast, most vision-language reward models are either very general (e.g. “GPT-like” models that only say “yes/no” to success) or require expensive per-task annotations. SARM2 sits in between: it uses a learned “stage estimator” to break a task into generic action primitives, and a specialized Mixture-of-Experts value head to predict progress. In their tests, this learned dense reward was far more accurate than prior methods (cutting the value-prediction error by ~80% on a 10-task benchmark) and enabled excellent policy refinement. The paper then introduces SPIRAL (Self-Policy Improvement via Reward-Aligned Learning), an RL fine-tuning pipeline that uses SARM2’s dense rewards in a TD3-based residual RL setup. Rather than relying solely on sparse terminal rewards, SPIRAL deploys the learned dense signal at every timestep and combines it with imitation-like regularization. The overall training loop has several phases: behavior cloning, a one-time reward-model fine-tuning step on a small set of demos, an initial offline RL update, and then an on-policy loop of autonomous rollouts. Notably, after the reward model is primed, the robot can largely self-improve without additional human labeling. On challenging tasks like “Folding Shorts on a table” and “Cleaning a Whiteboard,” applying SPIRAL resulted in success rates jumping from ~60% to nearly 100% over a few rounds of self-training. These gains were nearly monotonically upward, showing that a good dense reward really can bootstrap a robot’s performance. Below we unpack the key ideas in depth: the motivation, the reward-model architecture, the RL algorithm, and the experimental outcomes. We also situate this work in the context of prior stage-aware reward learning (SARM) and the current frenzy around fine-tuning vision-language-action policies (e.g. Pi-0, GR00T, Helix, OpenVLA). Finally we discuss limitations and future directions. The take-home is that learning a dense, multi-task reward model and coupling it with an on-robot RL loop is a promising path toward truly self-improving dexterous agents – one that can reduce dependence on costly human demonstrations and drive continuous “data flywheel” improvements. The Challenge: Fine-Tuning Long-Horizon Manipulation. Modern VLA policies already offer a lot – multimodal transformers can execute complicated instructions like “fold the shirt in half, then put it in the drawer” in simulation or on a real robot arm. But when trial runs start, they fall short on tasks that require many sequential steps, especially if the scenarios differ from the demos they were trained on. Even if you have a handful of good demonstrations, a dangerously tempting but limited approach is to just behavior-clone more data. In practice that means hiring experts to record dozens of demonstrations per task – an expensive and slow process – and the resulting policy can only reliably imitate those exact sequences. It doesn’t generalize well, and you can’t just push it via standard RL easily because of sparse, long-horizon rewards. A more scalable vision is to learn a reward model for each task (or set of tasks) so that the robot can practice on its own. If the robot can evaluate “how good am I doing right now?” at every timestep, it can use reinforcement learning to go beyond the training demos. In theory, reward models promise to transform imitation into learning: they can reweight the demonstration data (giving high scores to good segments and low scores to bad ones), or provide a continuous training signal (“you’re 20% done”) in place of just +1/0 success signals. However, crafting such reward models is tricky. On one end, heavy hand-engineered signals (like specifying exact intermediate goals or using very dense human annotations) do work, but at the cost of generality. On the other end, people have tried large pretrained vision-language models to judge a robot’s video. For example, methods like TOPReward or Robometer feed video frames or language into a big model to estimate success. While these can apply broadly without manual labels, they tend to be very coarse: they might only reliably tell you if the final outcome looks correct, but not the fine-grained progress or stage of a long task. In practice, those VLM-based rewards often turn out slightly “over-optimistic”, giving inflated credit early on and then crashing near the end. You end up with a poor learning signal. The SARM2 authors found this problem firsthand: baselines like TOPReward or Robometer were indeed overly optimistic in their experiments (predicting success too early) and giving unstable learning because they missed how the task unfolds step by step. A contrasting approach (seen in the earlier SARM paper) is to make highly stage-aware reward models: explicitly divide a task into a few named stages or subtasks, and train the model to predict which stage the robot is in and how much progress. This can yield very accurate shaping. For instance, SARM (Stage Aware Reward Modeling) thus predicted both the current “high-level stage” of the task and a fine-grained progress fraction. It even used natural-language annotations of subtasks (from the visual demonstrations) to create training labels. The results were promising, but SARM had a drawback: it was task-specific. For every new task, you needed to annotate what the stages are (like “reach for cloth”, “grasp cloth”, “fold cloth”) and often collect data per stage. This doesn’t scale easily to dozens of tasks. SARM2 (+SPIRAL) attempts to get the best of both worlds. It still uses the idea of “long-horizon tasks decompose into stages or primitives,” but it does so generically across many tasks. Instead of per-task labels, they built a common vocabulary of action primitives – generic segments of manipulation that recur across tasks, like “acquire object,” “release,” “rotate,” “translate (move)”, “insert into container,” etc. The core reward model is then multi-task: it can handle 10 or more manipulation tasks without retraining from scratch, by predicting which primitive is happening at each step and then tailoring a progress value to that stage. Concretely, SARM2 learns an Action-Primitive Stage Estimator and a Multi-Gate Mixture-of-Experts Value Head. Together these produce one unified dense reward for any task in the suite. This design avoids per-task stage annotation but still gives tight, stage-conditioned rewards (instead of a one-size-fits-all generic score). In experiments the result was a remarkably accurate predictor of progress. On the 10-task benchmark they used, SARM2’s value estimates were far closer to ground truth than all prior reward models (value estimation MSE fell by ~80% compared to the best alternative). With these accurate dense rewards in hand, SPIRAL then uses them to finetune the policy with RL, yielding large jumps in success rate over mere imitation. In short: dense, learned rewards + on-robot reinforcement. That is the combination that allowed the robot to largely self-improve on long tasks, trimming away many cycles of human supervision. As one commentary puts it, a reward model that’s “sufficiently dense and precise allows an autonomous data-flywheel, reducing human supervision and adapting policies to new tasks without recollecting data”. This paper shows exactly how to build such a model for dexterous, multi-stage tasks – and how to hook it into a learning loop that drives continual robot improvement. The next sections describe each piece in detail. The SARM2 Reward Model. At the heart of SARM2 is a Multi-Task Stage-Aware Reward Model, abbreviated “RM” in the paper abstract. The goal is simple to state but tricky to implement: given a video and proprioceptive trace of a robot performing any of several known manipulation tasks, predict a score at each timestep that reflects remaining progress to the goal. (Equivalently, predict the normalized “steps remaining” or distance to success.) This score can then be used as a per-step reward or supervision signal. SARM2 achieves this by breaking the problem into two parts:. Action-Primitive Stage Estimator. First, at each time step the model tries to identify which action primitive is currently happening. The designers decided on a fixed vocabulary of about 21 manipulation primitives (plus a “null” class), common across all tasks. Examples of primitives include things like acquire (grasping an object), release (letting go), rotate, translate (moving something in space), insert, shape (like adjusting object shape), force (pressing or pushing), and a catch-all “other.” These primitives capture the recurring segments you see in tabletop manipulation: each of the 10 tasks in their benchmark can be decomposed into a sequence of such primitives. (In fact, they curated a large dataset of behavior – roughly 200 hours from 100 tasks – to define and illustrate these primitives.). The stage estimator is essentially a video-and-state classifier over these primitives. Concretely, it takes as input a short recent history: six frames of video (sampled at a modest stride) from each of three cameras (e.g. one observing each arm plus a wide view) plus the robot’s proprioceptive state (joint angles, gripper status, etc.) at the current time. Each frame is fed through a frozen SigLIP-2 visual encoder (a powerful vision-language transformer merely used for image features), and the state goes through a small MLP. The resulting embeddings are concatenated in time and fed into a 4-layer causal Transformer. The transformer's hidden output at the current time is then passed through a linear head to predict one of the $K+1=22$ classes (21 primitives plus null). Training this stage classifier uses standard cross-entropy on labeled data (they hand-labeled segments from the demonstrations with their primitive classes). The intuition is that by knowing which primitive is happening, the model can contextualize the notion of progress. In practice, this stage estimator achieved near-perfect performance on its validation set: during learning it was able to correctly identify the current primitive in most frames of all tasks. (The 21-primitives assumption and labeled dataset are specific to their setup, but they picked fairly generic primitives, so many new tasks would share them.). Multi-Gate MoE Value Decoder. Second, SARM2 uses the predicted primitive to help compute a dense value (or reward) estimate. Rather than using a single monolithic network head (which would have to learn value regression for all tasks and all primitives at once), they use a multi-gate mixture-of-experts design. The idea is to have a bank of expert sub-networks, each specialized to certain kinds of stages, and to route only the relevant experts for the current primitive. Concretely, the value-prediction network is another Transformer (6 layers) that takes a similar input sequence (several recent frames, plus the first frame of the episode to give context, plus the rewound frames from an earlier time) along with embeddings for the task name (a text label of the task) and the predicted primitive class. The transformer processes this and its output at each time step goes into the mixture-of-experts head. They grouped the 21 primitives into 8 high-level semantic groups (e.g. “Acquire,” “Release,” “Translate,” “Insert,” “Rotate,” etc., plus “Other”). For each of these 8 groups there is a gate network $G_m$ that looks at the transformer’s hidden state and produces a weighting over $E=10$ shared expert MLPs. In practice each gate does a softmax over experts and picks the top-2 experts with nonzero weight. In summary, for time $t$, if the predicted primitive falls into group $m$, they activate gate $G_m$ and linearly combine the outputs of up to two experts chosen by that gate. These 10 experts are 3-layer MLPs that share parameters across all groups (except gating differs). This way, each expert learns some aspect of the value function that could be reused by multiple tasks or stages, but the gates ensure only relevant experts are combined. Finally, the combined expert output $o_t$ is passed through another linear layer to give a scalar value $\hat{r}_t$ predicting the normalized remaining time to completion. Concretely they define the target label for each step as $$ r^_t ;=; -\frac{T - t}{T} \quad\in[-1,0], $$ where $T$ is the time of episode end (finish). So at the end $\hat{r}=0$, and earlier steps are negative, capturing how far to go. The model is trained by mean-squared error on $(\hat{r}_t - r^_t)^2$. They also add a few small auxiliary losses to keep the gate usage balanced (so no single expert hogs everything). In essence, the model learns to regress how “close” the robot is to success, in a way that depends on the type of action it’s currently doing. The result of these two components is that SARM2 can consume a video/state stream of the robot and at every step output a dense reward (e.g. the change in predicted remaining value) beyond just an end-of-episode check. Because it is stage-aware, it gives meaningful intermediate feedback: for example, picking up an object (“Acquire” primitive) may yield a small reward when completed, while rotating an object (“Rotate” primitive) gives rewards on smooth orientation progress. By contrast, a naive model might either give zero reward until the very end, or use a too-general vision-language score that jumps to +1 only once it sees the whiteboard fully clean in the corner of the frame. In this multi-task setup, a single SARM2 reward model can serve ten different tasks, ranging from cloth folding to block stacking, all with no per-task annotations needed at runtime. (Much of the above architecture summary is drawn from the authors’ description and a summary by others.). Self-Improvement via SPIRAL RL. Having a dense reward model in hand is only half the story. The other half is: how do we use it to actually improve a policy? This is where SPIRAL comes in. SPIRAL (Self-Policy Improvement via Reward-Aligned Learning) is the proposed on-policy RL loop that takes the pretrained VLA policy and the SARM2 reward model and iteratively refines the policy. In practice they model this as a kind of residual RL: starting from the original policy $\pi_\text{pre}$ (learned by behavior cloning), we want to learn a small corrective policy $s_\theta(s,z)$ to add on top of it, as in $$ \pi(s,z) = \pi_\text{pre}(s,z) + s_\theta(s,z), $$ where $z$ is any latent noise or conditioning. The end result is the full policy $\pi(s,z)$ used at test time, but the training objective encourages $s_\theta$ to be small so as not to overthrow the good behavior already learned. Specifically, SPIRAL is based on TD3 + BC (twin-delayed DDPG with Behavior Cloning regularization) plus a bit of distributional RL theory from a method called DICE-RL. Here are the key points:. Dense Reward Critic. The critic is trained with the usual TD3 style objective (bellman backup) using SARM2’s dense per-step reward $r^d_t$ instead of only a sparse terminal reward. That is, the TD3 critic minimizes $\mathbb{E}[(y_t - Q_\phi(s_t,a_t))^2]$ with $y_t = r^d_t + \gamma Q_{\phi'}(s_{t+1},a_{t+1})$. Because $r^d_t$ is fine-grained, the value estimates are much more stable and informative. Hybrid Critic Objective. In addition to the TD target, SPIRAL adds a Monte Carlo (MC) return term as a regularizer. Concretely, they mix a term $J_{\text{MC}}(\phi)$ (which tries to match the long-horizon sum of actual returns) into the critic loss. The full critic loss is$$ J_\text{critic}(\phi) = J_{TD}(\phi) + \alpha, J_{MC}(\phi). $$ Empirically this hybrid helps stabilize learning by anchoring the TD estimates to the actual episodic returns (which still measure success in the usual sparse way). They found using both targets was more robust than either alone. Actor + BC Regularization. For the actor, they use a TD3-like deterministic policy with an extra $L_2$ penalty on the residual action (to keep it near zero). In practice the actor loss is$$ \min_\theta \mathbb{E}{s\sim D}[ -Q\phi(s, \pi_\text{pre}(s) + s_\theta(s)) + \beta ,|!s_\theta(s)!|^2_2 ], $$with a coefficient $\beta$ chosen large (e.g. 30) so that the policy doesn’t drift too far from the behavior cloned policy. This is essentially the same idea as TD3+BC or AWAC: the policy is nudged by the Q-value but softly penalized for deviating. Putting it together, SPIRAL’s training pipeline has four main stages (illustrated in a list for clarity):. Behavior Cloning (BC): First, train a base VLA policy $\pi_\text{pre}$ with standard imitation learning on the available demonstrations. This is what we would do normally – yields an initial policy with some success rate (e.g. ~50-60%). Reward-Model Adaptation: Gather a small number (~100) of additional rollouts from the initial policy $\pi_\text{pre}$ and have humans label the true rewards/progress on these. The reason is to reduce distribution shift: SARM2 was originally trained on some dataset, but its predictions might drift on the robot’s actual behavior. So we fine-tune (or adapt) the reward model with this small labeled set, producing a better model $RM_2$ that understands $\pi_\text{pre}$’s trajectories. Importantly, this is only a one-time tuning step and requires manual annotation of very few episodes. Offline SPIRAL Update: Using the (dense) reward model $RM_1$ before fine-tuning, run a round of offline RL on the logged demo data to produce a new policy $\pi_2$. Because offline RL can exploit imperfect data but is still cautious, it gives an intermediate improvement. (In practice they warm-start the policy by supervised learning, then do one offline RL gradient update using $RM_1$.). On-Policy Improvement Loop: Now switch to autonomous self-improvement. At each iteration, deploy the current policy $\pi$ on the robot to collect new experience, and label that experience with the adapted reward model $RM_2$. Then do one step of on-policy SPIRAL (TD3-like) training to get a better policy. Iterate this loop: as the policy improves, you continuously label fresh data with $RM_2$ and refine the policy. The key is that after step 2, no further human labels are needed. The robot can keep improving on its own, with the reward model providing dense feedback. In short, SPIRAL is essentially an on-policy residual-RL procedure guided by SARM2 rewards, with each policy roll generating new training data. The combination of dense rewards and BC-regularized RL ensures safety and stability: the robot improves steadily while never straying wildly from demonstrated behavior. In their experiments, SPIRAL consistently drove the success rate up each round; it was essentially monotonic. By the end, tasks that were barely doable by BC alone became nearly perfect successes under SPIRAL. (This description is condensed from the paper abstract and review notes.). Experimental Results. The authors tested SARM2+SPIRAL on ten long-horizon robotic manipulation tasks. Five of these (“S1” tasks) were fairly standard in robot labs, and five (“S2”) were more unusual or challenging variants. Two representative tasks were Folding Shorts (neatly folding a pair of shorts on a tabletop) and Cleaning Whiteboard (erasing marks on a whiteboard with an eraser). Each task is composed of many steps (cloth manipulation, wiping motions, etc.) and has a binary success at the end (correctly folded, clean whiteboard). They compared against several baselines:. Behavior Cloning (BC): The initial imitation policy trained on demonstrations (this typically solved only ~50-60% of trials for the hardest tasks as-is). Offline RL: A baseline where the dense rewards (as computed by SARM2) were used to do a single offline RL pass on the demonstrations. This gave some improvement over BC but often struggled without on-policy data. Robust Reward Models: They tested the learned SARM2 model (denoted RM or “SARM2 reward model”) against other reward predictors like ReWiND, TOPReward, Robometer, and a LoRA-fine-tuned Robometer. On purely reward modeling metrics, SARM2 dominated. For example, on all 10 tasks, the value-estimation mean squared error (MSE) of SARM2 was about 80% lower than the next best model. It also had the highest correlation with actual progress (“rollout classification score”) on the hardest tasks. In contrast, vision-language-model-based rewards (TOPReward, Robometer) consistently overshot their predictions (they thought the task was nearly done too early). Ablation studies showed that both pieces of SARM2 – the action-primitive estimator and the multi-gate MoE head – contributed significantly; removing either one degraded performance. Policy Learning (SPIRAL) vs BC and other RL: The real test was how the final policies performed. On tasks like Folding Shorts and Cleaning Whiteboard, SPIRAL with the SARM2 reward was dramatically better. For instance, on Folding Shorts (for a “flat folding” variant), the success rate went from 58% with BC to 100% with SPIRAL. On Cleaning Whiteboard, it went from 50% to 90%. In other words, tasks that half-worked under imitation were solved almost perfectly with RL fine-tuning. Moreover, SPIRAL’s improvements were steady over iterations: each self-play round raised performance. Crucially, the ablation results emphasized that reward quality was the bottleneck. When they tried SPIRAL with only the sparse binary reward (instead of the dense SARM2 reward), performance actually degraded. Even using a fine-tuned Robometer (another learned reward) in SPIRAL underperformed the SARM2 reward. This shows that having accurate, stepwise reward feedback is what unlocked the improvements. The paper phrases it well: high-quality dense rewards are key to a stable robot “data flywheel”. With SARM2’s dense guidance, the robot’s own practice data became informative; with a poor reward, the flywheel stalls or even spins backward. We should stress that these results are quite striking. Achieving 100% success on a dexterous folding task from a VLA baseline is non-trivial. The jump in performance came from repeated training: each SPIRAL round lets the robot try, fail, and correct under its new policy. Contrast that with BC, which can only mimic what it saw. By giving the agent a good sense of how much closer it got to the goal at each step, SARM2 encourages novel successful behaviors beyond the demos. This is evidenced by the near-perfect eventual success rates in the experiments. For perspective, typical imitation-only or offline-RL methods often plateau well below these numbers on similar tasks. The improvement “from 58% to 100%” and “50% to 90%” wasn’t just a small tweak – it essentially turned a borderline policy into one that nearly always succeeds. One reviewer notes this increase as “near-perfect task success. showing monotonic gains across self-improvement rounds”. It suggests that SARM2+SPIRAL succeeded where many prior efforts (especially on long-horizon tasks) did not: namely, robust policy refinement with minimal new supervision. Why It Matters: A Path to Self-Improving Robots. SARM2 + SPIRAL is significant not just for the pretty numbers, but for what it represents in the broader robotics landscape. Right now, many teams are racing to build powerful VLA controllers – think of projects like Pi-0 (Physical Intelligence), NVIDIA’s GR00T, Figure AI’s Helix, UC Berkeley’s OpenVLA, and others. These groups train very large models (billions of parameters) to “understand” language and vision and propose actions. But a common theme is that these policies, once pretrained, still need some on-robot adaptation and fine-tuning to hit peak performance in specific environments. Sleeping in a similar distribution of this or that demonstration doesn’t guarantee solving the next drawn-from-hat task exactly. The problem is that this fine-tuning step is often neglected or hand-wavy. If you rely purely on more demonstrations, you end up with a slow, expensive feedback loop. If you try standard RL with sparse feedback, convergence is unreliable. SARM2+SPIRAL suggests a concrete remedy: invest in a self-supervised reward model that can then power an autonomous learning loop. In industry speak, it’s building a “robot data flywheel”: the system practices on its own and improves in situ. The result (at least in this proof-of-concept) is a significant reduction in the reliance on human demos. This idea dovetails with what many companies say they want. For example, robotics firms like Agility Robotics or 1X Technologies constantly emphasize reducing per-task finetuning costs. A robust dense reward model is exactly the kind of component that could integrate into their stack to enable turnkey adaptation. The authors themselves point out that though this is still academic work (no code release yet, tasks still lab-scale), the conceptual takeaway is important: “combining a dense reward model and self‐improvement loop without human demos is precisely what industry players are looking for to lower adaptation costs”. In other words, this is not just a curiosity – it’s a building block for making advanced robot policies practical in the real world. Furthermore, by being multi-task, SARM2 anticipates a world where a single robot platform tackles many related tasks (like different kinds of cloth handling or tool use). The multi-gate design means one trained model can cover variants, rather than requiring a fresh reward function per task. This generality is valuable: an integrator could, in principle, deploy the same MoE reward module on a new variant of cloth-folding without starting from scratch. Of course, the current work is still limited to fairly similar tasks (all bimanual tabletop scenarios with the same kind of hands). But it’s a step toward “universal” rewards for a class of tasks, not just a narrow one. Let’s also revisit the notion of stage-awareness. By leveraging generic action primitives, SARM2 reduces the annotation burden: you don’t need humans to label every new episode with what stage it’s in, and you don’t need to design hand-crafted stage detectors by hand. Instead, the system learns these primitives from data and then uses them to condition a general model. This is appealing because it decouples the semantic meaning of the task (the language instruction) from the low-level state (the video). The robot can learn that “folding” and “stacking” share some primitives (“grasp, lift, release”) and reuse that knowledge. It’s an instance of multitask learning in robotic rewards, which often yields better sample efficiency and transfer. Indeed, they observe positive transfer: e.g. learning the MoE on 10 tasks did better than training separate models, since the shared experts got more training signals. Related Work: SARM and Beyond. It’s worth situating SARM2 among prior attempts. The most direct predecessor is the original SARM (Stage-Aware Reward Modeling) by Chen et al. (ICRA 2024). SARM also used video-based transformers to predict task stage and progress. In fact, SARM’s paper beautifully put it: “Reward modeling offers a natural solution: by providing grounded progress signals, it transforms noisy demonstrations into stable supervision that generalizes across diverse trajectories”. In SARM, those progress signals were derived from natural language annotations of subtasks (for example, knowing that a cloth folding demo is annotated with stages “reach, grasp, fold, place”). This meant they needed structured annotations per task. SARM2 effectively automates that phase: rather than using linguistic stage labels, it trains a classifier on visual data to identify primitives. Qualitatively, SARM2’s stage estimator just replaced the need for language-labeled stages. The MoE head further extends SARM by handling multiple tasks via shared experts. So one can view SARM2 as a multi-task, learned-stage generalization of the SARM idea. (Where SARM needed a separate annotated model for each task, SARM2 pools them.) The empirical gains of SARM2 underline this: by avoiding both oversimplified VLM rewards and per-task semantics, it hits a sweet spot of being “dense, accurate, and general” simultaneously. Other recent work tries different angles on reward modeling. ReWiND (C. Zhang et al., CoRL 2025) learns language-guided rewards by re-ranking demonstration trajectories, but it still uses discrete preferences. TOPReward (Shah et al., 2026) attempts a zero-shot approach by treating token probabilities as rewards. These are creative, but they lack per-step resolution or require careful text prompts. SARM2’s novelty is in combining vision-language priors with structured priors (primitives and MoE) in an end-to-end learned fashion. As for policy improvement, using residual RL with BC regularization is becoming popular (TD3+BC is one example). SPIRAL’s hybrid critic objective (mixing TD and MC) is similar in spirit to other works that combine bootstrapped and Monte Carlo critics to stabilize learning. The innovation here is packaging it into a loop that is explicitly designed for autonomous robot maintenance — they borrowed ideas (like Distributionally Consistent Actor-Critic) but pitched it in the context of on-robot rollouts. Limitations and Future Directions. Of course, no paper is a silver bullet. The authors are candid about their limitations. First, their action-primitive vocabulary was hand-defined from tabletop bimanual data. That means SARM2 as written only covers certain robot embodiments and scenarios. A humanoid ballerina or a legged robot might have entirely different primitives. Extending this to varied robots or to completely different settings (say mobile manipulation, navigation plus manipulation) would require rethinking the primitives or making them learned. Second, as they note, residual RL has its own blind spots. SPIRAL improves the low-level control and mechanics of executing tasks, but it won’t fix high-level planning or strategy errors. If the policy’s failure mode is “I folded the shirt the wrong way, I’m in the wrong subtask”, the reward model can’t magically re-plan. (It might still eventually stumble into the right approach given enough trial, but it's not guaranteed.) True generalization still requires good latent task representations or new language instructions. So SARM2+SPIRAL assumes you at least gave it the correct overall goal in language; it doesn’t learn new tasks from scratch. Third, there’s still some human in the loop. They needed a one-time batch of labeled rollouts to calibrate the reward model to the actual robot policy distribution. Also, even though the runs are “autonomous”, the physical system still needs resets, which in practice often involve a human. So the loop isn’t fully free of supervision – it’s just orders of magnitude less than raw imitation. Sample efficiency is also not solved: running on-policy rollouts to get to 100% success still took many robot attempts (the paper doesn’t specify exactly how many, but “ relatively sample-inefficient” is noted). For real deployment, one would need to integrate resets or use simulations to fill gaps. Finally, it’s worth mentioning that this is a preprint: code isn’t released and industrial-scale benchmarks haven’t been done yet. The tasks are still somewhat controlled. However, this paper is clearly a step in an ongoing effort. The idea of a learned dense reward plus self-improvement has charisma: even if the initial numbers might change with more replication, the high-level concept should inspire further work. Conclusion. The SARM2+SPIRAL approach represents a compelling vision for how robots can steadily improve without being hand-fed every detail. By investing in a multi-task, stage-aware reward model, the authors gave the robot a kind of common sense about progress in manipulation. By coupling that with RL-based policy refinement, they let the robot convert that knowledge into better action strategies. The reported results (e.g. 0.58→1.00 success on folding) are exciting and suggest that a dense “scoreboard” is indeed a powerful motivator for dexterous learning. For roboticists, the takeaways are clear: if you have a suite of related tasks and a base VLA policy, consider investing effort into learning a structured reward model! The architecture shown here – combining vision encoders, transformers, primitives, and Mixture-of-Experts – is one worked example of how to do that scaleably. And even if your tasks or robots differ, the principle of “multitask reward model + on-robot self-improvement loop” could be adapted. In an era where machine learning is king, this is one case where learning to grade yourself (via a reward model) might let the robot become a better student than plain imitation ever could. References: The above summary draws on Huang et al., “SARM2 + SPIRAL: Multi-Task Reward Models and RL Refinement for Long-Horizon Dexterous Manipulation”, arXiv 2025 (author-provided); and related sources, including media summaries and earlier work on SARM.