Latest / Embodied AI 101 / Robometer and the Future of Robotic Reward Modeling
Transcript
- Robometer and the Future of Robotic Reward Modeling. Roboticists have long grappled with a fundamental challenge: how to define what success looks like for a robot performing a task. In reinforcement learning and imitation learning, reward functions or progress signals guide policies toward task goals. Traditionally, building a good reward model has meant carefully annotating or engineering reward signals for each task. Recent work in general-purpose reward modeling aims to learn vision-language reward functions that can generalize to new tasks and embodiments, often by training on large datasets of robot trajectories. Yet most existing models rely exclusively on absolute progress labels interpolated along successful demonstrations, a supervision that is easy to define for clean, expert demonstrations but breaks down on the messier data of real robotics where failures and suboptimal attempts abound. The new 2026 Robometer framework tackles this long-standing limitation by introducing trajectory comparison supervision. In addition to anchoring rewards via frame-by-frame progress on expert demonstrations, Robometer learns to compare entire trajectories and predict which one better satisfies an instruction. This simple twist lets it leverage vast quantities of partially failed or noisy data, yielding more calibrated and general reward models across many robots and tasks. Built on a vision-language model backbone, Robometer is trained on RBM-1M, a one‐million-trajectory dataset that the authors assembled specifically for reward learning. This dataset spans over 21 different robot platforms (bimanual manipulators, single-arms, mobile robots, even human-held cameras) and includes both simulation and real data. Crucially, RBM-1M includes a heavy dose of suboptimal and failed trajectories – data that would normally be discarded in purely success-driven training. By introducing preferences among trajectories rather than only scoring them absolutely, the model can learn useful signals even when ground truth progress isn’t obvious. In this article we’ll unpack Robometer’s design, training, and results in depth. We’ll see how its dual objectives (absolute progress and pairwise comparisons) are implemented, how the data is curated and augmented, and what gains it delivers. Along the way we’ll compare to prior approaches like ReWiND, RoboReward, VLAC, and others. In the end, Robometer achieves significantly better reward alignment and downstream learning across unseen tasks and robots, demonstrating that preference-based training can effectively scale general-purpose reward modeling in robotics. The Problem of Reward Learning in Robotics. At its core, robotic reward learning tries to let a robot figure out “how well am I doing?” at each point in a task. Given a video of a robot performing a task (often with an accompanying language instruction describing the goal), a reward model outputs a dense reward or progress score per frame. This dense feedback can then be used to train policies via reinforcement learning or to help filter demonstrations for imitation learning. Early classic methods like Inverse Reinforcement Learning (IRL) inferred rewards from expert trajectories , but those tend to be task-specific: every new task or environment needed fresh demonstrations and retraining. More recent work has pushed toward general-purpose reward models, especially using big vision-language models (VLMs) and large datasets, so one model can handle many tasks. For example, RoboReward (Lee et al 2026) fine-tuned a Qwen-3 VLM on ~50k expert demos to predict a coarse 1–5 progress score. VLAC (Zhai et al 2025) trained a video-language-action model with 300k trajectories to predict actions and rewards. And others have used large LLMs and VLMs (often closed-source) to query progress (e.g. Generative VLM (GVL) prompts GPT-5 Mini). However, almost all of these methods issue progress scores by assuming you only train on successful trajectories. They assign frame-wise labels (say, 0% to 100% progress linearly) along curated, completed demo videos. This is straightforward for clean benchmarks: if the robot linearly moves from start to finish on a task, you can give 0 at the start and 1 at the end. But in real robotics practice, plenty of data is non-expert: robots fail, get stuck, or take suboptimal routes. In those cases, saying “this trajectory was 30% done” is ambiguous – what does 30% even mean on a failure? As Liang et al. point out, interpolated progress labels “become ill-defined and costly to annotate for failed attempts”. This means vast swaths of collected data (self-play atempting a task, aborted trials, etc.) are essentially unusable under traditional reward-supervision regimes. Robometer’s insight is that we can exploit those suboptimal videos by comparing them to better ones, instead of trying to assign them exact scores. In other words, if video A succeeds and video B fails at the same task, we may not know B’s exact progress, but we certainly know “A is better than B.” This is the idea behind preference learning or relative supervision: the model sees pairs of trajectories and learns to predict which one is preferable with respect to the task. Importantly, this preference signal is agnostic to exact percentages – it only cares which video demonstrates more progress or success. By adding this kind of pairwise signal to the usual progress supervision on expert data, Robometer gets both the local frame-by-frame grounding and the global ordering constraints that tie different trajectories together. This combination helps the model generalize and makes the reward outputs better calibrated for messy real data. The authors formalize this with a dual objective. On one hand, Robometer still tries to predict absolute progress and a final success flag for well-labeled, expert demos (so its outputs are anchored to 0–1 on those known cases). On the other, it predicts pairwise preference on arbitrary trajectory pairs, marking which video is better. During training, the network ingests either a single video (for progress inference) or two videos (for a preference query), and outputs both scalar reward trajectories and a binary preference choice. In practice this was implemented by adding special tokens to a VLM backbone (more on that below). To make it concrete: imagine two videos, A and B, both purportedly executing “pick up the red block.” Video A picks it up successfully, video B knocks it over and fails. Robometer might be trained with A as the positive and B as the negative in a preference loss. Meanwhile, A alone would also have framewise progress labels (0→1 from start to success). B might have no progress labels but still provide a preference example. Over many such comparisons and expert demos, the model learns an internal notion of “on a scale from zero to success, how well is the robot doing right now?” that works even when explicit labels aren’t given. As we’ll see, this hybrid approach yields reward models that are more robust and informative than those trained only on expert data. Robometer learns to sharply penalize regressions (like a robot dropping an object) in the middle of a task, whereas a pure progress-trained model would often just keep giving higher scores even on a failed run. By the authors’ account, Robometer distinguishes suboptimal from successful trajectories significantly better (32% improvement in distinguishing suboptimal from successful) and correlates with ground-truth progress more strongly (14% higher rank correlation) compared to prior methods. Next let’s dig into the key components: the dataset they used, the model architecture, and the training regimen that makes all this work. The RBM-1M Dataset: A Million Trajectories for Reward Learning. A major part of Robometer’s success is having a very diverse and large dataset of robot motion videos. The authors curated RBM-1M, a reward-learning dataset of over one million trajectories pulled from many sources. Table VI (Appendix) lists typical sizes: RoboReward had 45k, RoboDopamine 100k, VLAC 300k, whereas RBM-1M is orders of magnitude bigger at >1M. (Even if the scale includes lots of “unlabeled” data, quantity matters for learning large models.). RBM-1M’s contents are strikingly heterogeneous. First, it heavily mixes sources:. Existing robot demo datasets: e.g. a mix of robots from Open-X Embodiment (diverse arms and grippers) and task suites like AGIBotWorld or Galaxea humanoid data. Human videos: Egocentric videos of humans doing tasks (like Epic-Kitchens) and human-robot paired data (like RH-20T) to inject diverse scenes and perspectives. Simulation benchmarks: Tasks from datasets such as LIBERO and MetaWorld to cover common manipulation skills. Failure rollouts: Trajectories from automated exploration or “fail-safe” datasets where robots attempted tasks imperfectly (e.g. simulated failures in LIBERO, or real-world experiments where policies are trained until some tasks fail). Because of this, RBM-1M claims 21 distinct robot “embodiments” – everything from bimanual humanoid arms to mobile arms, including videos with just human hands. It purposefully includes a large fraction of suboptimal or failed executions that naturally come from autonomous rollouts (for example, Zhou et al. 2024’s autonomous data or Lin et al. 2025’s failure dataset). Failures in this context might be dropping an object, getting stuck, etc. Traditional methods would struggle to annotate a progress score on those, so typically they’d be tossed out. Robometer keeps them as unlabeled examples for preference training. For training, each trajectory $\tau$ in RBM-1M contains: a sequence of images $o_{1:T}$ (subsampled video frames), a language instruction $l$ describing the task, and possibly a progress label $p$ at the end of the trajectory (often 1 for success, or other known scores if provided). In practice, all expert demos get $p=1.0$ (we assume 100% complete at end). Some datasets give partial scores for incomplete demos. Critically, failed trajectories get $p=\texttt{None}$ to indicate “no absolute label”. Progress loss is only applied to those with labels. In addition to training data, the authors built evaluation splits. They created RBM-EVAL-ID (an in-distribution test set from trajectories in the training sources but withheld) and RBM-EVAL-OOD (out-of-distribution). RBM-EVAL-OOD is particularly interesting: it has 976 trajectories from 3 different academic sites that purposely aren’t included in any training data. These cover 6 distinct robot embodiments, including 3 completely new ones (e.g. maybe a robot or a human arm not seen during training). Diverse camera angles and novel scenes test how well the model generalizes. In summary, this dataset architecture is one of the broadest attempts yet to cover “the real world” for reward modeling. Such scale and diversity pay off: the model can learn consistent cross-task preferences. For example, in experiments the authors sample videos of different tasks and check whether Robometer assigns higher rewards when the video does match the instruction versus mismatches. On unseen tasks from RBM-EVAL-OOD, they found Robometer’s reward predictions align the instruction with the correct video far better than alternatives, producing a sharp diagonal in confusion matrices. This indicates the model strongly recognizes “this video is about sweeping, not wiping,” etc., even for new robots and scenes. To compare, other recent reward papers trained on at most hundreds of thousands of frames. Table I in the appendix (line 989-997) shows RBM-1M’s 1M+ against 100k–300k in RoboDopamine, VLAC, RoboReward. In their experiments, the authors compare Robometer (trained on RBM-1M) versus baselines trained on their original data (RoboReward’s 45k, etc.) as well as retraining those baselines on RBM-1M. Robometer tends to pull ahead, showing the benefit of scale. Robometer Architecture: A Vision-Language Backbone. Robometer is built on top of a large pretrained video-language model. Specifically, it “instantiates a causally masked VLM, Qwen3-VL-4B-Instruct,” which is an autoregressive vision-language model with about 4 billion parameters (Qwen is a Chinese gen AI model similar to GPT-type architectures, extended to accept images). The idea is to leverage modern multi-modal transformers that already know a lot about language and images, and fine-tune them for reward tasks. To do that, the authors insert new special tokens into the model’s input sequence. The core idea is:. For a single input video (during inference), Robometer will attend to frames and output a predicted progress (reward) for each frame, as well as a “success” prediction at the end. For a pair of videos (during preference training), the model sees them concatenated, and outputs which video is preferred as a whole. In practice, they format the input like:. Each ⟨|prog_token|⟩ is meant to predict the reward at its corresponding frame (progress score), and the single ⟨|pref_token|⟩ is a special classifier token for preference judgment. The text instruction is included in the prompt (the snippet doesn't show it, but likely it goes before or in the context). The trick is a causal mask so the progress tokens only see frames up to that point (so they can be decoded sequentially), while the pref_token sees both video sequences to compare them. By inserting tokens rather than altering the model deeply, they preserve the pretrained weights underneath. Only new heads (small MLPs) are added to the token outputs. In other words, the vision-language encoder is mostly intact, and the extra tokens carry the new tasks. Concretely, from [15]:. “We insert new learned tokens into the sequence. We interleave progress tokens within the first video sequence and a single preference token at the end of the multi-video prompt.”. Because of the causal masking, at time t attends only to the first t frames of video o^1, letting it predict frame-level progress. The , sitting after both videos, attends across the entire sequence and learns to output which of the two videos is better. Notably, they always put the progress tokens on the first video of the pair. During inference (single video case) that video is the one being scored. During training, both videos might have progress tokens attached to the first one; which is why when making a preference pair, they randomly assign one sequence as video 1 and the other as video 2 each time to avoid bias. They also fix both videos to have equal length (by subsampling in advance), so that the model can’t cheat by picking the shorter video as the “success” (since shorter might correlate with faster success). On top of the hidden embeddings from those tokens, they attach small MLP heads. Specifically:. Progress head: One MLP for each that predicts a categorical distribution over progress bins (they discretize progress 0–1 into N bins). Then they recover a continuous estimate by expectation. This distributional approach (inspired by “C51” in distributional RL) helps stability compared to regressing a scalar. The ground truth progress for a labeled demo is set to t/T at frame t, then linearly interpolated onto the bins. Training uses cross-entropy on this distribution. At final frame, they also force a binary “success” label. Success head: They also have each also predict a binary success value, though in practice only the last frame gets labeled positive. This is to help the model know when the task is actually done versus just keeping the arm still. (They note some demos keep recording after success, so they actually post-process to find the actual end of task for labeling.) The success label is 0 before the end and 1 at the end. Preference head: The hidden state goes into a binary classifier MLP that predicts which of the two videos is preferred. A label y∈{1,2} says whether trajectory1 or trajectory2 is better (relative to the given instruction). This is optimized with a standard cross-entropy loss on the two-way choice. So the overall loss is a sum: $$\mathcal{L} = \mathcal{L}{\text{prog}} + \mathcal{L}{\text{succ}} + \mathcal{L}_{\text{pref}}.$$ Intuitively, the progress+success terms anchor the reward output to actual progress on clean data, while the preference term shapes comparisons. During inference, given a single test video with instruction, Robometer can output a dense progress curve and a final success probability. The hope is that this reward curve will be smooth, increase steadily, and plateau near 1 when done, unless something bad happens (in which case it might dip or stagnate). Augmentations and Sampling Strategies. With the model set up, the next question is: how to generate those preference examples? Since human annotating pairwise robot trajectories would be expensive, the authors leverage either natural pairs in the data or easy synthetic constructions. They describe three complementary strategies:. Progress-based comparisons (different expertise): Whenever RBM-1M contains two trajectories for the same instruction, one clearly better than the other (e.g. one is an expert demo with $p=1$ and another is a failure with $p=None$ or lower progress), they form a pair. If one has a higher known progress or is labeled success, that one is preferred. Concretely, they sample $\tau_1,\tau_2$ with the same language $l$ but either one has higher progress label or one is expert ($p=1$) and the other is not. Then $y=1$ if $\tau_1$’s progress > $\tau_2$’s, else $y=2$. This contrasts real failures against successes. Instruction negatives (different tasks): To ensure the model ties rewards to the language, they also construct “wrong task” examples. They sample two trajectories $\tau_1, \tau_2$ that have different instructions $l^1 \neq l^2$. Then they randomly pick one of those instructions as the conditioning instruction $l$, and set the progress target of the other trajectory to zero. Effectively, “this video was doing some other task, so reward should be zero.” The preference is that the trajectory matching the instruction is better (and the mismatched one is treated as a failure). This teaches the model that doing the wrong thing yields no reward. Video rewinding (synthetic failures): Even if we only have expert demos, we can manufacture “failure” versions of them by churning time backwards. This is the “rewind” trick (used in Zhang et al 2025 ReWiND). They take a single good demo $\tau$ and chop out a segment of frames, then splice them in reverse order to simulate the robot undoing progress. Concretely, pick times $t_1