Latest / Embodied AI 101 / Freeform Preference Learning (FPL) for Robotic Manipulation

Transcript

Freeform Preference Learning (FPL) for Robotic Manipulation · Embodied AI 101

• Freeform Preference Learning (FPL) for Robotic Manipulation. Training robots to perform complex, multi-step tasks autonomously is still bottlenecked by reward design. A good reward should capture all aspects of desirability (e.g. accuracy, speed, safety, comfort) and be dense enough to guide the agent over long horizons. In practice, naive rewards are either sparse success signals (e.g. only give +1 when the task is fully complete) or hand-tuned shaped rewards, both of which have serious drawbacks. Sparse signals place an enormous burden on the reinforcement learner for exploration and credit assignment, while handcrafted dense rewards rarely capture all nuances of human intent and can induce unwanted behavior when optimized. One way to address this is to learn the reward from human judgments. Prior work on human feedback (e.g. binary preference learning) shows promise, but those methods still force humans to collapse all task-relevant criteria into a single “better/worse” label for a pair of trajectories. In a long-horizon manipulation task, this is inherently ambiguous: for example, if one robot execution is faster but riskier and another is slower but safer, it may be unclear which should be preferred overall. Freeform Preference Learning (FPL), a method introduced by Torne et al. (Stanford) in 2026, overcomes this by allowing humans to provide multi-dimensional feedback in natural language. In FPL, an annotator compares two trajectories along each criterion of interest. They can write axes like “speed”, “smoothness”, “preciseness of placement”, “carefulness”, “safety (e.g. avoiding collisions)”, etc., and then indicate, for each axis, which execution is better (or if they are equal) on that axis. This “freeform” feedback is used to train a single language-conditioned reward model that can score any trajectory with respect to any axis description. The trained model thus yields a separate reward value for “speed”, “accuracy”, etc., rather than a single bellicose score. A policy is then trained to optimize this multi-axis reward, and crucially can be steered at test time simply by choosing which axes to emphasize (for example, one could run the policy to maximize speed or to maximize precision, or some weighted combination). In experiments on four real-world robotic manipulation tasks (block-in-bin placement, table-setting, plating toast, folding shorts) and two simulated tasks, FPL consistently outperforms all baselines by a wide margin. The authors report an average absolute improvement of about 38 percentage points over the next-best method on real tasks. Compared to training on a sparse success reward or on single-axis (binary) preferences, FPL produces much denser and less ambiguous learning signals. Qualitatively, the learned reward truly “lights up” when subtasks are accomplished: for instance, the FPL reward for the table-setting task produces spikes whenever the robot places a plate, cup, or knife, even though the model was never given explicit segment boundaries. In contrast, a binary-preference-trained reward spikes only at the very end of the episode (wrongly assigning the entire credit to “the end of the demo”), illustrating how much crisper the multi-axis supervision can be for credit assignment. Below we unpack the FPL method in detail. We first review the limitations of standard reward learning and preference-based RL. Then we explain how FPL collects “freeform” annotations and turns them into a language-conditioned reward predictor. We describe how the policy is then trained with these multi-dimensional rewards (in effect as a kind of goal- or reward-conditioned policy). Finally, we summarize the key experimental findings: improved success rates, emergent composition (solving new variants by recombining axes at test time), sample efficiency of annotation, and richer interpretability of the learned rewards. Throughout, we link to relevant prior ideas in RL from human feedback, multi-objective RL, and vision-language-action models. The challenge of reward design and preference learning. Designing a good reward function is notoriously hard in robotics. The ideal reward would give immediate feedback about every aspect of the behavior, but without teaching the robot exactly how to act (so as to still allow discovery of potentially better strategies). For a toy example like stacking blocks, a “sparse” reward that only gives +1 when all blocks are perfectly stacked is simple, but completely uninformative until the task is done. In a long-horizon task like setting a table, a sparse success reward (say +1 when all items are correctly placed, 0 otherwise) means the learner might have to mess up all subtasks before ever seeing a positive signal. This sparsity makes credit assignment exceptionally difficult. By contrast, a human supervisor could easily say “this tabletop execution was 80% correct” or “trajectory A was much safer but slower than B”, but encoding all that into a programmatic reward is intractable. One appealing idea is preference-based RL: let humans compare two rolled-out trajectories (or segments) and say which is better. Since people have a holistic sense of success, it simplifies their job—just judge “overall which is better.” The RL agent then learns a reward function (via e.g. a Bradley-Terry model) that is consistent with those preferences. This can produce a smoother reward-like signal because it learns to score any trajectory in a way that aligns with human choices on examples. Christiano et al. (2017) famously showed that even a handful of such comparisons could drive policy learning in games. However, preference feedback still has a serious flaw: ambiguity. A single “better” label must conflate all factors (speed, precision, smoothness, etc.) into one number. For long tasks, different segments may require judging entirely different things, so collapsing them into an “overall” preference often confuses the annotator or yields noise. Consider two robot table-setting demos: one is very safe but slow, the other is fast but occasionally knocks a plate. Which is “better”? A single binary label forces the human to make an arbitrary call or guess the intended priority. In practice this hurts both the labeling process and the learned reward. Section 1 of the FPL paper notes: “Binary preferences collapse many competing notions of quality into one ambiguous signal. In long-horizon tasks this can make preferences difficult to provide and the resulting supervision ambiguous.”. (Equivalently, a single reward dimension is being asked to do the job of multiple objectives.). What if instead the human could give separate judgments on each relevant objective? That is exactly FPL’s key insight: instead of one preference per pair, ask the annotator to indicate *along multiple axes which trajectory is preferred on each axis. Crucially, the axes are in free-form language. The human might be given some guidance axes (e.g. “speed” and “smoothness”) or could write their own (“hygiene of plate”(!), “carefulness of placement”, etc.). For example, when comparing two table-setting runs, a human might mark that “Trajectory A is faster than B” on the speed axis, but “Trajectory B is safer and avoids spills, so B is better on safety.” Correspondingly, the collected data might say: on “speed” axis, A ≻ B; on “safety” axis, B ≻ A. Learning from this richer feedback is what gives FPL its name and power. In summary, FPL replaces a single ambiguous human label with multiple specific labels, each paired with a textual criterion. As one slide in the paper’s Figure 1 (Fig. 8 in the arXiv text) illustrated: instead of getting a single “A is better than B” for two trajectories, they ask for judgments like “speed: A”, “smoothness: B”, “placing precision: A”, etc. This fine-grained feedback turns out to be far more informative. Not only does it provide a denser training signal (every axis yields a bit of reward information), it also avoids conflating orthogonal concerns (so the reward model doesn’t have to “magic up” weights between speed vs safety). Collecting freeform preference data. To implement this idea, the FPL procedure starts with an offline dataset of robot trajectories (in experiments these come partly from human demonstrations). Then human annotators are presented with pairs of trajectories (videos of robot rollouts) and asked to compare them. But unlike standard preference labeling, the annotator can provide multi-axis feedback. In practice, the annotation interface works as follows:. The interface shows two side-by-side videos (Trajectory A and B) of the robot performing (or attempting) the task. The annotator first specifies the axes of judgment. For some tasks the designers predefine a few relevant axes (e.g. “formality of table setup” or “bread placement accuracy”), but the key feature is that annotators can EDIT or add new axis labels in natural language if desired. For instance, for a table feeding task someone might add an axis “care taken not to break plates”, or for folding clothes an axis “wrinkle-free” (see, where 41 distinct axes were used in a toast-making task). For each chosen axis (e.g. “speed”, “visibility of handle”, “precision”), the annotator picks which trajectory they prefer on that axis (A vs B vs “tie”). In effect, one viewing of the pair yields multiple binary labels — one per axis. In this way, one pair of demonstrations yields many bits of supervision instead of one. A practical result observed in the paper was that freeform multi-axis labeling was actually more time-efficient than collecting single-axis labels. Because the annotator can give, say, 3 or 4 judgments after one viewing, the average time per label was cut roughly in half compared to standard methods. Torne et al. report about a 1.5–1.85× speedup in annotation when using multi-axis labels. This makes sense: the only overhead is loading videos, and once the videos are up you can answer many questions at once. Collecting 100 multi-axis labels is thus much quicker than 100 separate “overall-preference” labels. Concretely, Table 3 in the FPL appendix summarizes their collected data. For example, in the “Plating Toast” task they gathered 295 trajectory pairs and a total of 1477 axis-level comparisons across 41 distinct preference labels (see Appendix Fig.14). Some data was collected with fully freeform axes (annotators writing them out), and some tasks used partially pre-defined axes. In the Setup Table and Fold Shorts tasks, there were sets of given axes (like “knife orientation” or “sharpness of folds”), whereas in Plate Toast and an object-bowl-placement task, annotators freely typed their criteria. Across all tasks, FPL crowdsourced or solicited enough preferences to train the reward model. At the end of this process, we have a dataset $\mathcal P$ of freeform preferences. Each element is one pair $(\tau_i, \tau_j)$ of trajectories, together with a set of labeled axes. For trajectory pair $(\tau_i,\tau_j)$ the annotator has provided labels ${(l_k, y_k)}{k=1}^{K{ij}}$, where each $l_k$ is a text description of an axis (e.g. “speed” or “delicateness of placing plate$”) and $y_k\in{0,1}$ says whether $\tau_i$ or $\tau_j$ is preferred on that axis. Notice that $K_{ij}$ (the number of axes labeled) can vary per pair. The ground truth company line is: “These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward.”. Learning a language-conditioned multi-axis reward. With the comparison data $\mathcal P$ in hand, FPL next trains a reward model $r_\phi(\tau,|,l)$ parameterized by neural network weights $\phi$, which takes as input a full trajectory $\tau$ and a text axis label $l$, and outputs a scalar reward score for that trajectory with respect to that axis. Crucially, $l$ is free-form text (the same “speed”, “safety”, etc. used in the annotation). In other words, we learn one reward function that is conditioned on language. How exactly is $r_\phi$ trained? The approach is to extend the standard binary preference model (Bradley-Terry) in a simple way. Recall that Bradley-Terry says if a human prefers segment $s_i$ to segment $s_j$, then $P(s_i \succ s_j) = \sigma(r(s_i)-r(s_j))$ where $\sigma$ is the logistic function. Minimizing negative log-likelihood over human pairs learns a reward $r_\phi$ that assigns higher value to the preferred ones. FPL generalizes this: for one preference datum on axis $l_k$, if the annotator preferred $\tau_i$ to $\tau_j$ on $l_k$, we want $r_\phi(\tau_i | l_k) > r_\phi(\tau_j | l_k)$. We simply plug in $r_\phi(\tau|l_k)$ in the Bradley-Terry loss. Concretely, if pair $(\tau_i, \tau_j)$ has labels ${(l_k,y_k)}$, the loss (Eq.3 in the paper) sums over axes:. [ L(\phi) = -\sum_{(\tau_i,\tau_j),,k} ;\log \sigma\bigl((2y_k-1),[,r_\phi(\tau_i|l_k) - r_\phi(\tau_j|l_k),]\bigr). ]. In practice they use mini-batches etc., but the idea is simple: each axis yields a usual preference loss. Because $l_k$ is fed into the network, the model can share information across related axes (“speed” and “velocity” would both activate similar understanding, for example). Importantly, we do not train separate reward models per axis; a single multi-modal network takes the axis as input, leveraging pre-trained vision-language representations to understand the semantics of each label. In the implementation, the reward model $r_\phi$ is a transformer or similar network that consumes the sequence of observations (video frames, or images) and the text of the axis prompt, and then outputs a per-timestep score. These per-timestep scores are summed (like a cumulative reward) to yield the trajectory-level score. This prefix-summing allows the model to capture temporal credit (for example, if “safety” depends on a subtle avoidance early in the exec). By the end of reward training, we have a model that can answer questions like “how safe was this trajectory?” or “what was its precision of placement?” on an absolute scale. Because it was trained on preferences, it only cares about ranking consistency, but the scales end up meaningful enough to train a controller. In fact, the FPL model produced reward curves that, when plotted over time, unsurprisingly fire when subtasks are being completed. The authors show an example (Fig.6 in the paper) on the table-setting task where the reward for a “formality” or “placement” axis peaks right when a plate is placed, another peak when a fork is placed, etc. So even though we never explicitly segmented the task into “place big plate, place small plate, place cup”, the freeform axes effectively taught the model those events. This means FPL is providing a much denser supervision: The reward model is essentially segmenting the task by itself by virtue of the multi-axis labels, something a flat binary preference model could not do. Another way to see the difference is: if you train a reward model only on a single “overall” axis, it has to treat the entire 10-second video as a single data point. In FPL, each axis provides a comparison at whatever time point it was relevant. This means the learned multi-axis reward model "knows" that after the plate is placed, the “placement quality” axis should jump, etc. The effect is that FPL yields qualitatively better credit assignment on long tasks. As the authors state: “FPL learns dense progress signals without explicit task segmentation”, whereas a binary-preference reward heavily weights the very end. Training the reward-conditioned policy. Once the reward model $r_\phi(\tau|l)$ is trained, we want a policy $\pi$ that achieves high scores on the desired axes. Crucially, FPL does not collapse all axes into one scalar; instead the policy is also conditioned on axes and target scores. In other words, they become part of the policy’s “goal specification”. The insight is similar to goal-conditioned reinforcement learning: feed the text and desired reward values into the policy network along with the state, and train the policy to output actions that would realize those scores. Specifically, the authors choose a set of axes $L = {l_1,\dots,l_{K_\pi}}$ to use at policy-training time (for example, a representative basis of relevant criteria seen in the dataset). Then for every trajectory $\tau_i$ in the dataset, they compute the predicted reward $r_\phi(\tau_i|l_k)$ for each axis $l_k$. Now consider the tuple $(s_t, l_1,\rho_1,\dots,l_{K_\pi},\rho_{K_\pi})$ where $s_t$ is the current state and $\rho_k = r_\phi(\tau_i|l_k)$ is the trajectory’s total score on axis $l_k$. In words: we tell the policy “the current trajectory has these scores on each axis” and then train it to reproduce the actions that actually happened (from the data) under these conditions. Mathematically, the policy learning objective (Eq.5) is simply supervised learning over the demo data:. [ L_{\rm pol}(\theta) ;=; -\mathbb{E}{(s_t,a_t^i,\tau_i)\sim D}\Biggl[\sum{t=1}^T \log \pi_\theta\bigl(a_t^i ,\big|, s_t^i,;l_1, r_\phi(\tau_i|l_1),\dots,l_{K_\pi},r_\phi(\tau_i|l_{K_\pi})\bigr)\Biggr]. ]. In other words, each training example tells the policy: given you are in state $s_t$ and the trajectory $\tau_i$ (from which $r_\phi$ gave certain axis scores) needs to get these reward values on each axis, choose the action $a_t^i$ that was in the data. Although the formula looks like a GAN, it is really imitation/regression: they use off-policy or offline RL techniques to fit the policy. In practice, they rely on a “reward-conditioned” formulation built on top of recent vision-language-action models (VLA models) that can accept language inputs. The advantage of this approach is that we never lose the axes decomposition. The policy learns to map $(s,l_1,\rho_1,\dots,l_K,\rho_K)$ to actions. Hence at test time, one can steer the policy by choosing different $\rho_k$ values (or even different subsets of axes) without retraining. For example, the user could run the policy with $\rho_{\text{speed}}=+100$ to make it go fast, or $\rho_{\text{carefulness}}=+100$ to make it slow down. This yields at least two big benefits. First, it retains a notion of compositionality: because the policy saw examples of a task under various axis conditions (both high-speed and low-speed), at test time you can combine them in new ways. Second, it prevents reward hacking from mixing everything into one scalar: you keep a kind of “Pareto frontier” of behaviors. As the paper notes, collapsing multi-axis feedback into one scalar can lead back to shaping problems (where optimizing the wrong mixture of axes causes unintended behavior). By contrast, preserving the axes allows, for instance, learning a policy that can do both the “fast placement” and “careful placement” tasks if instructed accordingly. In summary, policy learning in FPL is: (1) run the offline RL data through the learned reward model to get per-trajectory scores on each axis, (2) train a policy network conditioned on those axis labels and scores to imitate the data. The training pipeline can be done fully offline once the reward model is learned, or iteratively with new rollouts. Indeed, the authors outline an iterative algorithm (Algorithm 1) where they collect freeform preferences in rounds, retrain the reward model, then retrain the policy, then roll out new trajectories and repeat. In this paper’s experiments the focus was mostly on one-pass learning (since they had offline data), but the loop is general. Crucially, because the reward model’s outputs are unbounded, in practice at test time they standardize or normalize each axis so you know reasonable ranges. That way, a user doesn’t have to guess crazy signal values. The policy effectively learns an implicit ordering: “if you want high speed, here’s what to do; if you want high smoothness, here’s how.”. Baselines and alternative approaches. To appreciate FPL’s effect, it’s useful to compare what happens with simpler methods:. Sparse Success Reward: Here the robot only gets positive reward at the very end if the task is completed. This baseline represents the traditional “reward engineering” scenario where the designer only knows how to judge final success. The problem on long tasks is obvious: with, say, 10 steps and low success probability for a random policy, seeing any success at all may be too rare to learn from. In practice, this baseline often fails to solve the task within the allotted data. The FPL paper shows cases (e.g. object rearrangement) where every algorithm using the sparse signal falls over because no one learns to succeed even once. Single-Axis Preferences (Binary Preferences): This is the classical RL-from-human-preferences approach. It collects pairwise comparisons but only one overall axis. One can even do it via offline data (compare trajectories from data and label them). The learned reward is scalar. In principle it gives denser feedback than pure success. But in practice, if the human labeler is uncertain or inconsistent about how to balance axes, the label noise is high. The FPL experiments demonstrate that the single-axis preference baseline learns much less effective policies. In fact, the withheld figure (Figure 3) shows that on every real task, the single-preference policy achieves 30–40% success, whereas FPL is around 70–90%. The text commentary notes “The signal from binary preferences is too ambiguous to learn performant policies”. Behavior Cloning (BC): This baseline ignores rewards and just does supervised imitation on the demos. It’s a strawman showing how well the offline demonstration alone solves the task. If the dataset has imperfect or partial data, BC may struggle. The paper adds a variant Filtered BC that augments the offline set with any trajectories that are labeled “successful” by the preference data. This attempts to boost success examples while still doing BC. Unsurprisingly, on tasks with few successes this helps a bit, but still FPL greatly outperforms it. The charts show filtered BC only slightly above plain BC in most cases. Advantage Conditioning / Weighted Regression: These are recent methods for vision-language-action models. One competitor they use is the “Advantage Conditioning” trick from “Steerable VLA” (the π0 paper presumably). That method learns a value function from success/time-to-go and then conditions the policy on a binarized advantage. It’s effectively a single-scalar conditioning too. Another is a variant of advantage-weighted regression (AWR) that uses the multi-dimensional reward from FPL’s model but then simply collapses it (e.g. averages the axes) to train the policy via weighted supervised learning. These hybrid baselines show that even if you give the policy the FPL reward signals, you lose a lot by not using the full multi-axis apparatus. In all cases, FPL’s full method was best. In short, FPL is compared to a wide range of methods, and it wins decisively. The improvement is not marginal: across tasks FPL’s success rates jump tens of points above the next best. For example, Figure 3 in the paper shows that on “Plating Toast” the next best solver (advantage conditioning) only gets about 30% completion, whereas FPL exceeds 80%. On “Fold Shorts” FPL hits ~70% vs ~30% for single preference and even lower for sparse-success. The paper sums it up as an average gain of 38 percentage points over the best non-FPL method. These are very strong numbers indicating that the richer supervision translates directly into better robot behavior. Real-World Manipulation Tasks. The authors evaluated on four real-world robotic manipulation setups. Each is a long-horizon task with visual observations and actions executed on a real robot arm (via a pre-trained vision-language model controller). Here is a sketch of them:. Putting a Block in a Target Bowl: The robot must pick up a cube and drop it into the correct bowl (out of two). This seems simple, but the offline data is messy: sometimes it drops the block, or puts it in the wrong bowl, or misses. Success requires picking correctly and not dropping. Axes might include “accuracy of placement” or “dropping frequency.”. Folding Shorts: Here the robot arm manipulates a fabric (a pair of shorts) to fold it. Subtasks include picking up the cloth, smothing it out, aligning edges, folding at the midpoint, etc. Axes include things like “crease formation”, “sharpness of folds”, or even “no wrinkles.” The demos vary in quality (some folds are sloppy). Ground-truth success is fully folded shorts. Plating Toast: The robot needs to pick up a slice of bread (from a plate), put it on a serving plate, then pick a butter can and put it on top. Subtasks: pick bread, place bread, pick can, place can (in that order). This is a made-up cooking scenario. Human preference axes here included “bread placed correctly”, “can placed correctly”, “time taken”, and something like “cleanliness of butter placement.”. Setting a Table: This complex task has many steps: pick up plates, forks, knives, cups from a cupboard and place them at dining table settings. The scene might have multiple settings (like for two people) so the robot must do several placements. There are lots of fine details (how to orient the knife, the spacing, not breaking any plates, etc.) and speed could also matter. They mention axes like “formality of setup”, “path smoothness”, “knife orientation,” etc. For each real task, they used a mixture of data collection and preferences. They had offline demonstration or prior data (some of which failed) and then asked annotators to provide preferences on many trajectory pairs. We saw earlier that in all of these, FPL performed much better than other methods. The bar chart labeled “Average real-world task progress” in Fig.3 (from the blog) shows FPL at ~75% completion vs ~30–35% for any baseline. That chart aggregates all four tasks: FPL (blue bar) far outstrips Behavior Cloning, Binary Preference, etc. Each individual task also showed similar gaps. It’s worth noting that the gains were not uniform across tasks. The improvement was largest on the most complex, hardest tasks (e.g. Set Table and Fold Shorts), and smaller on simpler ones. This matches intuition: if a task is short or data mostly correct, even BC can reach some success. But when tasks are long and data varied, only the dense reward can pull it through. In Slice: “the sparse reward is too weak to provide necessary supervision” for these long tasks. Looking at workloads: Table 3 (Appendix) shows each task’s data volumes. For example, Fold-Shorts had only 239 preference axes (not much) but about 130 preferences flagged as single comparisons and 10 as multi-axis (?) – anyway small. Plate-Toast had 295 pairs and 1477 axis labels as mentioned. The annotators ended up specifying dozens of distinct criteria. Because of this, not only did the robots succeed more often, but the actual behaviors were qualitatively better: in fold-shorts, the final folded cloth had far fewer wrinkles; in plating-toast, the butter can was well centered on the bread slice; etc. The paper mentions explicitly that FPL “improves both task completion and qualitative aspects of behavior.” In fact, on tasks like setup-table where the baseline with binary preferences often just dropped objects (giving up on care), the FPL-trained policy did them carefully to get a high score on each axis. Simulated Tasks: Compositionality and Steerability. Alongside the real tasks, the authors included two simulated “long-horizon” domains to test specific properties: compositional generalization and steerability. Both simulations are from the Robomimic (MuJoCo) benchmark domain:. Object Rearrangement: This is akin to “put block in bowl” but in simulation. The goal is to place two objects (e.g. a block and a nut) into their correct bins in order. The reward axes include things like “bread placed” (in real plate toast analogy) or “object order correctness”. We can think of it as testing whether the dense reward helps with the sequencing of multiple placements. Bimodal Square Task: This is the most interesting. The robot must place a small “nut” onto one of two pegs (left or right). However, there is a twist: the offline data was composed of two “modes” of demonstrations. All the demonstrations placing on the left peg were performed fast, while all the demos on the right peg were performed slow (and might even fail or drop sometimes). The actual target goal for training is right peg (the robot should put the nut on the right quickly). This creates a classic situation of missing mode in the data: the target behavior (“fast on right peg”) was never seen in any demonstration, because right-peg demos were slow only. The latter setup is intended to explicitly test compositionality. If the policy can learn “fast” as a concept from left-peg data, and “right” as a target from right-peg data, perhaps it can compose them to do “fast right” on test. Standard methods (BC, sparse reward, single-pref) fail miserably because they never see a fast right example; they tend to stay in the slow regime or ignore the difference. Multi-axes rewards, however, allow it to disentangle “speed” from “which peg”. Indeed, FPL’s axes might include “speed” and “right-peg placement” as separate labels. Then at test time one can set both to high. The results confirm this. On the regular “bimodal square” task (fast right goal), only FPL succeeded consistently. The others had near-zero success. Figure 5 (Tables 1 and 2 in the paper) shows that with the same policy (trained once on the base task), by simply changing the reward conditioning, FPL could also succeed at the inverted task (fast on left peg). In other words, the same policy that learned to place right quickly could be steered to place left quickly by swapping a language prompt. Again quoting, the authors say: “FPL can successfully solve with the same policy the bimodal square and bimodal square inverted benchmarks by changing the reward conditioning at test-time”. No other method could do both; at best, BC might do right-slow and even fewer methods could invert to left without retraining. This is powerful: it means FPL’s learned policy is not tied to one concrete objective; it generalized the idea of “fast placement” and “peg selection” so that inverts at inference. In their words, the FPL policy “exhibits compositionality of behaviors not present in the data”. The second valuable feature is steerability. Because the policy is conditioned on axes (in text + numeric form), a user can tune them. If the user during deployment wants the robot to prioritize speed over precision, they just bump up the “speed” axis score. If they want extra caution, they raise the “carefulness” axis instead. None of the baselines even allow this kind of online adjustment (you would have to retrain with a different reward). The FPL formulation makes it straightforward. The authors explicitly tested this by, for example, setting the “Fast” axis weight high and showing the robot tuned to speed. This kind of real-time preference steering is a unique byproduct of their method. Analysis of learned rewards and annotations. Beyond just reporting success rates, Torne et al. also examine what the reward models look like and how annotation played out. A few highlights:. Qualitative reward curves: As already mentioned, FPL produces temporally localized reward spikes aligned with subtasks. They show a figure for the table-setting task where the multi-axis reward clearly highlights each plate and utensil placement step. The binary baseline, however, has a frustrating large spike at the very end of the episode even when nothing significant happens there, a classic credit misassignment symptom. This visual comparison drives home the advantage of decomposing feedback. Evolving preference axes during training: Another observation is that annotations can adapt as the policy improves. In early iterations of training, annotators tend to focus on coarse criteria (“Did it fold at all?”). Later, when the robot is already good at the coarse job, annotators start caring about finer issues (“Were there wrinkles? Does the final fold align perfectly?”). FPL naturally accommodates this: each round of data collection can use whatever axes seem relevant next, building a curriculum of supervision. This iterative spirit (get basic behavior right first, then refine) is hard to program manually but flows naturally with freeform feedback. Annotation efficiency: We touched on this already, but it’s worth emphasizing with data from the paper. The authors measured that the cost (per preference-label) is about half with freeform vs single. They say “collecting freeform preferences is approximately 50% faster per label than collecting single binary preferences”. This is because each video pair yields multiple labels. In their experiments, they actually tracked the annotator time and found nearly a 2× speedup (¹.⁸⁵× faster) when using multi-axis. This is an important practical point: one might worry “doesn’t asking multiple questions at once overload the annotator?” In fact, it makes better use of the viewing time. The authors also allowed annotators to mark “equivalent” if they thought the trajectories were the same on an axis, further reducing noise if needed. Diversity of criteria: Another fascinating stat: in the Plate Toast task, annotators created 41 distinct axes across all pairs. This highlights that people naturally judge robots on many dimensions (short of enumerating all, but with FPL you don’t need to guess them a priori). Traditional methods would have to pick maybe 2–3 proxies manually. FPL effectively crowdsources the objective function. The paper remarks that this diversity “demonstrates the need for preference learning methods that preserve this structure rather than collapse feedback into a single overall score.”. Limitation – reward selection: The authors candidly note that one downside is now the user at test time has to pick numeric values for the axes. The reward model outputs are unbounded, so they normalize them for convenience. But effectively, steering the policy becomes choosing targets. This is an extra design choice. Automating that (e.g. via some higher-level scheduler or interactive loop) is future work according to the authors. Discussion and future directions. Freeform Preference Learning is a compelling advance for several reasons. It bridges ideas from preference-based RL, multi-objective RL, and language-conditioned control in a new way. By treating preferences as language-conditioned, it sidesteps the need to predefine a fixed set of objectives (many multi-objective RL works assume you know the axes). Here the axes can come on the fly. It also leverages modern vision-language transformer architectures to flexibly interpret those axes textually. One can view FPL as a step toward truly compositional reward learning. In the extreme, one could imagine chaining even more factors: not only multiple criteria, but preferences about possibly higher-level plans. Because the policy is VLA-based, it might accept even more complex instructions (e.g. “finish in under 10 seconds and be very careful”). The research question is how far this scales. Another positive is that FPL does not require segmentation or hand engineering of subtasks. Traditional dense reward design might say “+1 for big plate, +1 for small plate, +1 for fork, etc.” Here, the segmentation is naturally learned. That may simplify applying RL to new tasks: one could collect a generic demonstration dataset and then ask the human to describe whatever axes seem important. The robot then figures out the segmentation implicitly. This could be especially useful in domains where enumerating all criteria up front is impossible. On the other hand, FPL does require more human input than “unsupervised” methods. Any preference-based learning has this drawback: you need humans in the loop. But given that collecting that feedback is often inevitable for safety and alignment, FPL makes that feedback far richer. The authors report, for instance, that Fold Shorts required 239 axis-queries and 130 pairwise comparisons (probably from a handful of annotators). This is not trivial effort, but for a complex multi-step real task it’s still manageable, especially if one annotator can provide many labels in a session. Scalability will depend on how easy it is to gather such annotations at scale; perhaps with simulated environments this can be crowdsourced, while real-robot tasks might rely on experts. It’s worth noting that FPL’s use of an offline dataset + preference labeling fits well with modern offline-RL and VLM-based control trends. The policy is learned via offline imitation/regression with conditioning, which avoids the sample-inefficient on-policy loop. This design choice addresses a core practicality issue: on a real robot, doing millions of PPO rollouts (as in older preference RL demos) is impractical. By using offline data plus human labels, FPL sidesteps that. Related Work. FPL builds on two big threads: RL from human feedback, and language-conditioned skill learning. The first includes Christiano et al. 2017 and successive works on preference-based RL, including some robotics applications. Those works mostly used single-axis feedback; FPL’s novelty is explicitly multi-axis and language flexible. There is some precedent in multi-objective RL, which allows specifying a weight vector over objectives. FPL can be seen as learning those objectives on the fly via language, and even allowing the secondary ability to modify them in natural language. On the language side, there have been VLA (vision-language-action) models like VIMA, CLIPort, and especially newer ones like π0 (Physical Intelligence team’s work) to do multi-stage tasks from demos. FPL could actually plug into any VLA policy: it just needs the ability to accept a text prompt. The related insight in VLA research is that policies can be conditioned on textual goals; FPL uses this for the reward side. It’s a bit reminiscent of recent RLHF in NLP (e.g. training RL policies with learned reward models from text labels) but adapted to continuous control with vision, and with the twist of multiple reward dimensions. Another related area is task-specific learning from demonstrations via segmentation or hierarchical RL. Some methods explicitly split a long task into subskills and learn rewards or policies per subtask. FPL effectively automates this by having the human do the splitting implicitly via axes. So it sidesteps manual segmentation. It also relates to works on curriculum learning, which might choose simpler aspects first. In FPL, a rudimentary curriculum emerged naturally as axes changed from coarse to fine. Finally, one could mention “preference learning for offline RL” which was also a topic of interest among several concurrent works (e.g. something about reward-free representation and offline preference RL). The broad point is: FPL is part of a wave of trying to make RL require less manual engineering by using learned or demonstration-based supervision. Conclusion. Freeform Preference Learning marks a significant advance in how we teach robots what we want them to do. By letting humans give rich multi-dimensional feedback in plain language, it creates reward functions that truly capture the fabric of good behavior. The learned rewards are dense and interpretable, the resulting policies are both higher-performing and user-steerable, and the approach generalizes in creative ways beyond the training data. For practitioners, the takeaways are: if you have a complex robot task, consider collecting preference data not just as “better/worse” but with several evaluative criteria. Use a language-conditioned reward model that can understand these axes. Then train a reward-conditioned policy. In return, you get a policy that can be tuned on the fly (e.g. “be faster” vs “be safer”) and that can solve tasks perhaps beyond the original dataset by recombining the learned concepts. Of course, this approach is not magic — it needs careful implementation and still depends on human time for labeling. The quality of the axes and labels matters. And the current system still assumes the set of axes seen in training suffices for all future use; innovations could include having the policy learn to handle new axes without retraining. But the results are very promising: we see real robots doing complicated tasks more reliably, guided by human tastes given in language. Freeform Preference Learning is a powerful step toward robots that more fully understand all the little things that humans care about. References: (Citations in text link to the FPL paper and supplementary materials.).