Latest / Embodied AI 101 / Teaching Robots What "Better" Means

Transcript

Teaching Robots What "Better" Means · Embodied AI 101

• Teaching Robots What “Better” Means. Freeform Preference Learning for Robotic Manipulation • Marcel Torne, Anubha Mahajan, Abhijnya Bhat, and Chelsea Finn • Stanford University • Conference on Robot Learning (CoRL) / arXiv • 2026. Publication note: Although this episode was prompted by discussion around September 8, 2026, the paper first appeared on arXiv on June 30, 2026, the project page is dated July 1, 2026, and the current arXiv version was posted on August 1, 2026. As of September 9, Stanford lists the work as a CoRL 2026 publication. The paper lists the first three authors as equal contributors. The Problem With Asking Which Trajectory Is Better. Suppose a robot is setting a table. In one rollout, it moves quickly, places the small plate in the formal location to the left of the main plate, and finishes most of the task—but knocks over the cup. In another, it moves more slowly and uses a less formal arrangement, but handles every object carefully. Now ask a human annotator a seemingly simple question: which trajectory is better?. There is no objectively obvious answer. The first is better with respect to speed and formality. The second is better with respect to care and avoiding damage. An overall binary preference forces the annotator to invent an implicit exchange rate between these qualities. How much speed compensates for dropping a cup? Is formality more important than smooth motion? Does partial task completion dominate hygiene or safety?. This is more than ordinary label noise. The two trajectories may be only partially ordered: one is preferable on some dimensions and worse on others. Yet conventional preference learning demands a total ordering, compressing that structure into a single bit. Freeform Preference Learning, or FPL, starts from the observation that this compression is unnecessary. Instead of asking for one overall judgment, it lets the annotator name the relevant comparison axes in natural language—speed, placement quality, smoothness, damage, cleanliness, carefulness—and then compare the trajectories independently along each axis. The result is not merely a more detailed label. It is a different representation of what human feedback means. For robot learning, this is particularly important because task completion is rarely the whole objective. A household robot should not just get the toast onto the plate; it should preferably use the spatula rather than touch the food with its gripper, avoid damaging the toast, move smoothly, and finish in reasonable time. A manipulation system can achieve the nominal goal while still producing behavior that no user would want deployed. FPL is therefore best understood as an attempt to turn human feedback from an opaque ranking signal into a learned coordinate system for robot behavior. FPL in One Pass. The method contains three main moves. First, annotators view pairs of complete robot trajectories. For each pair, they provide one or more natural-language axes and indicate whether trajectory A, trajectory B, or neither is preferable on that axis. The set of axes can vary between trajectory pairs. In some experiments, the researchers provide a task-specific rubric; in the open-ended plate-toast experiment, annotators can write their own criteria. Second, FPL trains one language-conditioned reward model. Given a trajectory and an axis description such as “quality of plating” or “smoothness of motion,” it outputs a scalar score for that trajectory along that particular axis. Because the model consumes the axis as text, semantically related labels can, in principle, share representations rather than requiring a separately trained reward head for every spelling or phrase. Third, the system scores every trajectory in the robot dataset along a selected set of representative axes. Those scores are attached to the trajectory as conditioning information when fine-tuning a vision-language-action policy. At deployment, the desired axis scores are changed to steer the policy toward the requested behavior. This last step is easy to misread. FPL does not simply average the learned rewards and run standard policy-gradient RL against the result. Nor does the policy query the reward model while acting. Instead, it is trained as a reward-conditioned generative policy: reproduce this trajectory given the observation, task instruction, and the multidimensional reward profile associated with that trajectory. At test time, the robot is given a more desirable reward profile. The policy has learned correlations between reward coordinates and action sequences, so conditioning it on high scores biases it toward behavior associated with those scores. This is closely related to return-conditioned supervised learning, upside-down RL, and Decision Transformer-style policy extraction. The reward vector functions both as an optimization interface and as a structured behavior label. Learning an Axis-Conditioned Trajectory Reward. The reward-learning objective is a language-conditioned extension of the standard Bradley–Terry preference model. For one trajectory pair, suppose the annotator says that A is faster, B is smoother, and B causes less damage. FPL evaluates A and B three times, once under each axis description. Training pushes the predicted speed score of A above B while pushing B above A for smoothness and damage avoidance. The important architectural choice is that all these comparisons train a shared model. The reward is not represented as a fixed vector whose coordinates must be declared before data collection. It is represented as a function of both the observed behavior and a text description of the question being asked. That creates a useful separation:. The trajectory says what happened. The language axis says which property of what happened should be evaluated. The scalar output says how well the trajectory performed with respect to that property. For the real-world experiments, the paper uses a pretrained Qwen 3.5 VL 4B backbone. The model receives strided frames from both the wrist and third-person cameras, along with a textual question asking for the score of a specified axis. The vision encoder is frozen while the remainder of the model is fine-tuned. A causal attention mask prevents each point in the sequence from seeing future images. A linear projection converts the relevant visual tokens into per-time-step scores, which are summed to obtain the trajectory score. The causal construction is important. The score assigned at a particular point may depend on the full visual history up to that point, not just on a single frame. Whether a plate has been placed correctly, for instance, can depend on where it started, how it was moved, and whether it remained stable after release. It also creates the possibility of temporal localization. Even though humans label complete trajectories and never mark explicit subtask boundaries, the model can raise the “large plate placement” reward when the large plate is placed and separately raise the “cup placement” reward when the cup reaches its destination. There is, however, an identifiability caveat. Pairwise supervision constrains the total score difference between two trajectories; it does not uniquely determine how that score should be distributed over time. The causal mask prevents future leakage, but it does not mathematically force a reward spike to coincide with the causally relevant event. The temporal traces reported in the paper are therefore an interesting emergent result rather than a guaranteed consequence of the loss. Policy Improvement Without Conventional Online RL. Once trained, the reward model evaluates the complete dataset along a selected collection of representative axes. These policy-conditioning axes may be chosen manually from the annotations or obtained by summarizing synonymous labels. The rewards are standardized independently for each axis using the dataset mean and standard deviation. In the real-world implementation, the resulting values are rounded to one decimal place and inserted directly into the policy’s text prompt. A prompt might specify the manipulation task and then provide target scores for speed, smoothness, placement quality, and damage avoidance. The policy itself begins from open-source π0.5 weights. π0.5 uses a flow-matching action head; in FPL, the model predicts action chunks of 16 and executes eight actions before replanning. The researchers fully fine-tune it for 30,000 steps with batches of 32, providing both camera views, the task instruction, and the textual reward conditioning. Crucially, FPL trains on the entire dataset, including unsuccessful and undesirable behavior. A dropped plate is not simply removed. It becomes a valid example of what actions look like when the “carefulness” or “damage avoidance” coordinate is low. A slow but precise trajectory teaches another region of the behavior space. During inference, the policy is conditioned toward the high-scoring region. This explains why keeping the reward dimensions separate can be more useful than averaging them into one weight. If speed and target placement remain distinct, the policy can potentially learn how each property changes behavior and then combine the two. If they are averaged, a fast wrong-target trajectory and a slow correct-target trajectory might receive indistinguishable scalar returns. Reward conditioning also explains the method’s steerability. The system is not training one final policy that always maximizes a permanently fixed utility function. It is training a family of behaviors represented by one conditional network. Changing the requested reward profile selects a different member of that family. The obvious risk is extrapolation. Pairwise rewards provide rankings, not naturally calibrated cardinal utilities, and the policy only sees a limited distribution of reward combinations. FPL’s per-axis standardization makes the interface manageable, but asking for extreme or unsupported combinations can still push the policy outside its training distribution. Freeform Feedback as an Automatic Curriculum. FPL can operate as a single offline training pass, but the paper emphasizes an iterative version. The process begins with an offline dataset or a pretrained policy. Humans compare trajectories, the reward model is updated, and the policy is retrained using the resulting reward coordinates. The updated policy then collects new rollouts, which are added to the dataset for another round of comparison and training. One appealing consequence is that the feedback rubric can evolve with capability. Early in training, annotators evaluating shorts folding mostly care about whether each fold happens at all. Once the robot reliably completes the folds, attention shifts toward wrinkles, alignment, and the final shape of the garment. Because annotators can introduce criteria while labeling, this shift does not require someone to redesign a formal reward function between iterations. This is a form of human-generated curriculum learning. The reward specification becomes more demanding as the policy improves. The annotation interface also amortizes video-viewing time. A conventional preference session yields one overall label after the annotator watches a trajectory pair. FPL can yield several axis labels from the same viewing. The authors report roughly a 1.85-times improvement in annotation speed per label. That does not mean an entire FPL annotation session is necessarily half the length; it means that each unit of video-review overhead produces substantially more structured supervision. What Was Actually Tested. The real-world experiments use a Franka arm in the DROID configuration, with wrist and third-person cameras and joint-velocity control. These are meaningfully dexterous tasks in the general manipulation sense—tool use, deformables, variable object configurations, and multi-stage execution—but this is not dexterous-hand learning with an anthropomorphic multifingered hand. The four tasks cover different failure modes. Plate Toast requires moving toast from a tray to a plate. The robot may use a spatula or pick up the toast directly. The relevant axes include speed, smoothness, cleanliness, plating quality, and damage to the toast. This is the most explicitly open-ended annotation setting: the authors report 295 trajectory-pair comparisons, more than 1,477 axis-level judgments, and 41 distinct raw labels spanning several broader topics. Setup Table is the primary long-horizon task. The robot places a large plate, small plate, cup, and cutlery according to a specified arrangement. Feedback covers the quality of each placement, formality, speed, carefulness, smoothness, and environmental damage. Fold Shorts requires three consecutive folds under variations in the garment’s initial pose. The axes distinguish the quality and wrinkle level of each fold, final alignment, speed, smoothness, and damage. Place Cube into Bowl is primarily a steerability diagnostic. The same policy must place a cube into a blue, orange, or yellow bowl depending on the requested reward profile. The task-specific datasets are modest. Plate Toast starts with 148 offline demonstrations; Setup Table with 100; and Fold Shorts with 138. The cube task is reported with no task-specific offline demonstrations and 400 rollouts used for preference learning. The real-world evaluation uses 20 rollouts per method. Two Robomimic-derived simulation environments provide more controlled diagnostics. Object Rearrangement requires putting bread and a can into their correct boxes, in the correct order, without dropping them. Bimodal Square deliberately creates a missing combination in the dataset: it contains fast and slow left-peg trajectories, but only slow right-peg trajectories. The desired behavior is a fast right-peg placement. Simulation uses three seeds. The simulation implementation is deliberately simpler than the real system. It operates on state rather than images, uses one fixed reward head per axis rather than a language-conditioned VLM, and passes the reward vector numerically rather than through text. This makes the compositionality experiment clean, but it also means that the strongest compositionality result does not itself test generalization over freeform language. The Results, With the Headline Properly Interpreted. Across the four real-world tasks, FPL obtains an average reported score of 0.75. Filtered behavior cloning, the next-best method on average, obtains 0.37. That is the paper’s headline 38-percentage-point improvement—an absolute difference, not a 38-percent relative gain. The per-task FPL results are 0.94 for Setup Table, 0.70 for Plate Toast, 0.55 for Fold Shorts, and 0.81 for Cube into Bowl. The strongest baseline varies by task, but no baseline approaches FPL’s overall average. Plain behavior cloning averages 0.31; the two scalar-preference variants average 0.33 and 0.34. Plate Toast is especially revealing. Filtered behavior cloning reaches 0.30, while the scalar-preference methods reach only 0.00 and 0.10. The paper’s explanation is intuitive: one trajectory may use the spatula but drop the toast, while another may complete the transfer by touching the food with the gripper. An overall preference must decide whether task completion outweighs cleanliness. Axis-specific feedback can supervise both independently. In Setup Table, sparse-reward methods often learn where items should go but not how they should be placed. They may drop plates from above rather than lowering them carefully. FPL can assign positive supervision for location and negative supervision for carelessness instead of allowing successful placement to erase the damage dimension. In simulated Object Rearrangement, FPL reaches 0.84 success, compared with 0.79 for a scalar-preference baseline matched for the number of judgments. The numerical gap is modest, but the observed failure mode matters: scalar preference policies often achieve the correct arrangement while dropping objects from excessive height. Multi-axis feedback separately represents task order, object placement, and dropping. The more distinctive result comes from Bimodal Square. FPL produces fast right-peg trajectories despite that combination being absent from the demonstrations. It reaches perfect reported success and substantially higher throughput than the baselines. When the requested target is inverted at test time, the same policy also performs fast left-peg placements without retraining. This is real compositional behavior, but it should be described precisely. The model is recombining factors—target choice and speed—whose individual variations are represented elsewhere in the data. It is not evidence that the policy can synthesize arbitrary skills whose components have never been observed. Finally, the table-setting reward traces provide qualitative evidence of improved credit assignment. Axis-specific rewards concentrate around the placement of the corresponding objects, despite the absence of temporal annotations. The scalar preference model instead produces a large late-episode spike unrelated to a major completed subtask. This is promising, though the paper does not provide a quantitative benchmark for temporal reward localization. What the Paper’s Central Contribution Really Is. FPL’s novelty is not simply “use more labels.” One baseline is given approximately the same number of scalar preference judgments, requiring many more trajectory pairs to be viewed. It still performs substantially worse. The useful information lies in the factorization of those judgments. Nor is the main contribution just language conditioning. Several experiments use predefined axes, and the policy ultimately conditions on a selected fixed set of representative axes. The language interface is important because it lets the reward model share structure across labels and allows annotators to expand the rubric, but the larger contribution is preserving multidimensional supervision all the way through policy extraction. This preservation changes the role of undesirable data. In ordinary filtered imitation learning, low-quality trajectories are liabilities. In FPL, they help define the conditional behavior manifold. They tell the policy what slow, rough, wrong-target, or partially successful behavior looks like, provided that the reward model labels them correctly. The result resembles a learned Pareto coordinate system. FPL does not first identify one universally correct trade-off between speed, precision, safety, and style. It models those qualities separately and postpones the trade-off decision until the reward profile is supplied to the policy. That is a particularly natural design for deployed robotics. Different users and environments may want different operating points. A warehouse might prioritize throughput until fragile objects are detected. A home user might trade speed for quiet or careful motion. A surgical or food-handling system might treat cleanliness and damage as dominant dimensions rather than soft details. The present experiments do not establish that degree of generality, but they provide a concrete mechanism for pursuing it. Where FPL Sits Relative to Related Work. The conceptual starting point is the preference-learning paradigm popularized by Christiano and colleagues, where humans compare behavior segments and a reward model learns the implied ranking. That approach demonstrated that rich behaviors could be learned without an engineered environmental reward, but represented human judgment as one scalar preference. FPL retains pairwise comparison while asking what that preference was about. PEBBLE attacked a complementary bottleneck: feedback efficiency. It combines unsupervised exploration, active preference queries, off-policy learning, and repeated relabeling of stored experience as the reward model changes. FPL instead concentrates on the structure of each judgment. A natural combination would use PEBBLE-style active query selection to identify trajectory pairs that are informative for particular FPL axes. Promptable Behaviors is perhaps the clearest multi-objective predecessor. It trains one policy over a spectrum of reward weightings and infers user preferences from demonstrations, trajectory comparisons, or language. Its objectives are fixed and its evaluation centers on embodied navigation. FPL brings a related promptability idea to manipulation while allowing the supervision axes themselves to be expressed in open-ended text. GRAPE also aligns VLA policies from trajectory-level preferences and can target objectives such as safety or efficiency. GRAPE decomposes manipulation into stages and uses VLM-proposed keypoints and spatiotemporal constraints to guide preference modeling. FPL deliberately avoids explicit stage boundaries, instead trying to recover subtask-sensitive rewards from trajectory-level axis labels. Physical Intelligence’s RECAP and π*0.6 are especially relevant to FPL’s policy-extraction mechanism. RECAP trains an advantage-conditioned VLA using demonstrations, autonomous experience, and expert corrections. FPL can be viewed as replacing a single success- and time-derived advantage signal with a vector of learned, human-specified trajectory attributes. The two approaches share the idea that a large generative policy can learn from both good and bad data when execution quality is supplied as conditioning information. RoboReward and Robometer address another complementary problem: scaling broadly reusable robot reward models. RoboReward trains vision-language reward models using large robot corpora plus synthesized failures and near misses. Robometer combines frame-level progress prediction with trajectory-comparison supervision over more than a million trajectories. FPL is narrower in scale but richer in local human intent. A compelling future system would begin with a broadly pretrained reward model and adapt it through a small amount of task- or user-specific freeform preference data. Finally, How to Peel with a Knife illustrates why subjective quality cannot always be reduced to binary success. Its preference-based refinement targets contact-rich manipulation where success depends on how well the task was performed. FPL generalizes that motivation into a multi-axis interface spanning tool use, deformables, placement, speed, and style. How I Would Adapt FPL in a Robotics Stack. For a team implementing this idea, the first priority should be the feedback schema, not the reward architecture. Axis descriptions need consistent polarity. “Damage” and “avoid damage” are semantically related but imply opposite desirable directions. I would phrase axes as explicit questions—“How well did the robot avoid damage?”—and enforce that higher scores always mean more desirable behavior. Second, collect comparisons over genuinely diverse outcomes. Reward conditioning only becomes useful when the dataset contains behavioral variation along each coordinate. If every successful trajectory is also slow and every failure is fast, the system cannot robustly disentangle speed from success. Deliberate strategy variation, controlled failures, policy rollouts, and active query selection can be more valuable than another batch of nearly identical expert demonstrations. Third, retain raw freeform labels for reward-model training but create a canonical policy vocabulary. The FPL paper already selects representative axes for policy conditioning. In practice, I would cluster semantically related annotations while preserving uncertain or conflicting clusters rather than automatically merging everything that looks synonymous. Fourth, validate each axis independently. Pairwise validation accuracy is necessary but insufficient. Evaluate counterfactual pairs that differ primarily along one property, inspect whether unrelated scene variables influence the score, and plot temporal responses around known events. For safety-relevant attributes, probe adversarial and out-of-distribution failures before allowing the score to influence a policy. Fifth, train the conditional policy on the full range of scores. Filtering out low-reward data removes the contrast that makes steering possible. But choose deployment targets from empirically supported quantiles rather than requesting arbitrary extreme values. A small target-selection controller could search over safe, in-distribution reward vectors using held-out evaluation rollouts. Finally, keep genuine constraints outside the learned preference system. Collision avoidance, force limits, workspace boundaries, and human-proximity rules should not depend solely on whether a learned VLM reward assigns a high “safe” score. Preference learning should shape behavior inside a verified safety envelope, not replace that envelope. The released repositories include simulation and reward-learning code, the real-world training pipeline, four real-task datasets, and a separate DROID preference-collection interface. The latter supports live A/B policy rollouts, comparisons of previously recorded trajectories, free-text axes, ties, teleoperation, and OpenPI policy serving. The paper reports using one H100 for real-world reward-model and policy fine-tuning, with smaller RTX-class GPUs for simulation. Limitations and Open Questions. The first limitation is that “freeform” applies more strongly to reward learning than to policy deployment. Annotators may introduce arbitrary text axes, but the current policy is conditioned on a fixed canonical set chosen during training. It cannot yet be given a completely novel axis at deployment and reliably change its behavior. The authors explicitly identify variable-axis policies as future work. Second, natural-language generalization is not directly isolated. The architecture should allow labels such as “quick,” “fast,” and “efficient” to share representations, but the paper does not report a dedicated held-out-synonym or unseen-axis evaluation. And because simulation uses fixed numerical heads, the cleanest compositionality results validate multidimensional conditioning rather than open-vocabulary reward semantics. Third, pairwise rankings do not supply calibrated utilities. Standardization makes test-time prompting practical, but a target of two standard deviations on “smoothness” does not have an inherent human meaning. Choosing target vectors becomes increasingly difficult as the number of axes grows and as the axes conflict. Fourth, decomposing reward reduces one route to reward hacking but cannot eliminate misspecification. An axis-conditioned model can still exploit visual shortcuts, misunderstand an unusual failure, or assign high scores outside its training support. Learned dimensions such as safety, hygiene, and carefulness require negative examples rich enough to expose the actual concept. Fifth, the empirical scope remains relatively small: four real tasks, one Franka-based setup, modest task-specific datasets, and 20 evaluation rollouts per real-world method. The qualitative credit-assignment analysis is compelling, but it is based on representative traces rather than a quantitative temporal-localization benchmark. There is also an unresolved human-factors question. Freeform axes can expose disagreements rather than erase them, which is good, but they do not automatically reconcile them. Different annotators may use the same word differently or introduce incompatible notions of quality. A production system will need annotator modeling, uncertainty estimates, provenance, and perhaps user-specific reward coordinates. Finally, human feedback is still required. FPL makes each trajectory viewing more productive and allows the rubric to evolve organically, but this remains supervised self-improvement rather than autonomous reward discovery. The most promising scaling path may combine human-defined axes with pretrained robot reward models, automated failure mining, and active selection of the small fraction of rollouts where human judgment is most valuable. The Bottom Line. Freeform Preference Learning makes a simple but consequential argument: asking which robot trajectory is “better” is often the wrong question. Long-horizon manipulation produces trade-offs among completion, speed, precision, safety, cleanliness, damage, and style. A single preference bit hides those trade-offs. FPL preserves them as natural-language reward axes, trains a shared trajectory scorer, and uses the resulting reward vector to condition a VLA policy. Its strongest result is not any one success rate. It is the demonstration that multidimensional feedback can simultaneously improve annotation efficiency, policy performance, behavioral composition, test-time steering, and the interpretability of learned rewards. The present system still relies on fixed deployment axes, modest datasets, and weakly identified temporal rewards. But it points toward a more flexible model of robot alignment: not one scalar definition of good behavior, but a language-addressable space in which users can say what kind of good behavior they need.