Latest / Embodied AI 101 / HANDOFF: Humanoid Agentic Task-Space Whole-Body Control via Distilled Complementary Teachers

Transcript

HANDOFF: Humanoid Agentic Task-Space Whole-Body Control via Distilled Complementary Teachers · Embodied AI 101

• HANDOFF: Humanoid Agentic Task-Space Whole-Body Control via Distilled Complementary Teachers. In this episode we dive into HANDOFF, a fresh approach to controlling humanoid robots proposed by Lizhi Yang et al. (June 2026). The big idea in HANDOFF is to make a single unified whole-body controller that takes a small, human-friendly set of commands – a 10-dimensional “task space” input – and dexterously drives the robot’s 29 degrees of freedom (the legs, arms, torso, etc.) to carry out complex tasks. To achieve this, the authors train specialized “teacher” controllers for different modes (motion imitation, walking, and fall recovery) and distill their knowledge into one mixture-of-experts student policy. The result is a robust, generalist controller that can handle diverse loco-manipulation tasks without needing task-specific fine-tuning. Moreover, they show it working on hardware: a Unitree G1 humanoid robot follows high-level language commands (via a vision-language planner) to do things like pick-and-place, badminton swings, and even recover from pushes – all through the same 10-D interface. We’ll unpack how HANDOFF is built: the compact command interface, the complementary teacher experts and how they’re stitched together, and finally how this controller plugs into an “agentic” planning pipeline. Along the way I’ll cite key details from the paper and project page to illustrate their claims. By the end you should understand what HANDOFF does, why it’s an important step toward more versatile humanoids, and how its ingredients might be adapted in your own work. A Compact, Planner-Friendly Command Interface. A core insight of HANDOFF is that how you command a humanoid matters a lot. Traditional whole-body controllers often require very detailed references – for example, desired joint angles or dense full-body poses – which are hard for high-level planners (or humans) to supply. As the authors note, “the choice of command space (i.e., the interface between task planning and whole-body control) is crucial”. Handing a planner a full 29-dimensional trajectory or complicated spatial constraints can break the connection between planning and action. HANDOFF addresses this by inventing a compact task space: only 10 numbers per command, chosen to be intuitive and versatile. Specifically, each command vector contains:. Planar base velocity: commanded forward/backward and lateral speeds, plus yaw rate (turning speed) for the robot’s pelvis in the horizontal plane. Root height (z): a target height for the robot’s torso/pelvis above ground. Bilateral wrist targets: three coordinates (x, y, z) for each wrist, expressed in the pelvis frame. That sums to 3 (base velocity) + 1 (height) + 6 (two 3D wrist targets) = 10 dimensions. In other words, the planner (or human) only tells the robot how fast and where to move its body and hands in space, never its joints directly. Figure 1 in the paper (and the project page) illustrates this command: for example, a low root height combined with forward wrist targets naturally yields a “squat-and-reach” posture. In practice this interface is remarkably expressive. The authors emphasize four desired properties: it should be intuitive (easy for a human, classical planner, or even a vision-language model to produce), general (covering many types of loco-manipulation tasks), modular (so you can separately swap out the planner, perception, or controller), yet still whole-body expressive (even though it’s low-dimensional, it drives complex coordinated motions). As they put it, “Compact commands still elicit coordinated full-body behavior.” Indeed, the same 10-D input can make the robot walk, reach, squat, or do all of the above together. Planners don’t need to output a full-body trajectory – they just output desired pelvis velocities, height, and hand targets, and the controller figures out the joint-level gymnastics. To give an example, the interface might encode a command like: move forward at 0.3 m/s, strafe left at 0.1 m/s, rotate slowly counterclockwise, lower the torso to height 0.5 m, and keep both hands at two specific pelvis-relative points. That single static command causes the robot to stride and reach in a natural way, without any further specification. The project page highlights this with illustrations: a low (squat) plus forward hand targets yields a coordinated squat-and-reach. In short, HANDOFF’s 10-D interface acts like a universal joystick for the humanoid – much easier for a planner to feed into than raw joint goals. The interface has proven powerful. On hardware tests with a Unitree G1 humanoid, HANDOFF achieves “one of the largest robust manipulation workspaces” reported so far. In other words, given that compact command, it can reach and manipulate over a big region. And despite its simplicity, it still matches state-of-the-art performance in tracking velocity commands. This suggests that clever learning can bridge the gap: even though we restrict the planner’s prompt to 10 numbers, the learning-based controller can internally interpolate and coordinate all 29 joint motions to satisfy it. That’s the magic of the next ingredient: distilling complementary teacher policies into a single network. Distilling Complementary Teacher Controllers. Building one controller that can both walk, manipulate objects, and get back up after a fall is hard—so HANDOFF takes a divide-and-conquer approach. The authors train three specialist teachers (using whatever method suits each) and then teach one student policy to mimic them. The key teachers are:. WBC (Whole-Body Control / motion-tracking) teacher: This is a 29-DoF policy trained to imitate human demonstration clips. The training set comes from retargeted human motion data, capturing natural body movements for tasks like reaching or picking manipulation. Crucially, before using these clips, they apply a Control-Barrier-Function (CBF) filter to ensure the motions are physically feasible (keeping the center-of-pressure inside the foot support). This avoidance of unrealistic motions means the student never learns to topple itself. During training, the WBC teacher learns to follow these full-body trajectories, and its output covers both the arms and legs. Locomotion teacher: This covers walking and base movement, with a simpler 15-DoF policy (basically hips and legs). It is trained (likely via RL) on flat-terrain velocity-following tasks, often with random perturbations (pushing the robot’s arms) to encourage stability. Thus it learns stable walking gaits and dynamic balance when moving. Because it only controls the body slice, it’s focused on leg/torso motion for locomotion. Fall-recovery teacher: When the robot falls down, this 29-DoF policy is specialized to stand up again. It’s trained on episodes of “fall-recovery” clips: the robot starts prone or after a fall and learns to right itself. This teacher takes over when a recovery situation is detected. These three teachers are complementary: no one policy alone does all. You need the WBC teacher to coordinate arms during manipulation, the locomotion teacher to handle brisk walking, and the fall teacher for robustness if balance is lost. The novel step is how to merge them. Rather than manually switching between modes, HANDOFF sets up a mixture-of-experts student policy that learns from all three teachers at once. During training, the student network receives the same 10-D command and proprioceptive history (the recent 11 frames of joint states) as input, and outputs all 29 joint velocities. Instead of hard-switching, it has internal expert heads and a gating mechanism. A shared encoder processes the input, then a softmax-based gate mixes the three specialist networks (one per teacher) to produce the final action. In effect, the student learns three sub-policies (experts) in one model, each biased by one of the teachers. To teach this student, they use a multi-teacher KL-distillation approach with a twist: they match the student’s output to each teacher on different slices of the action. As [Yang et al.] explain (and the Figure 3 caption summarizes), “context-based action-sliced KL pulls the body slice toward a velocity-gated blend of WBC and locomotion teachers, the arm slice toward WBC, and [matches] the fall-recovery teacher onto the full action when recovery is active”. In plain terms: for the arms and torso, the WBC teacher (motion imitation) guides the student’s outputs; for the legs/feet, the policy learns from a blend of WBC and locomotion teachers based on how fast it’s commanded to go; and if a fall occurs, the student is trained to reproduce exactly the fall-recovery teacher’s signals for the entire body. The KL-divergence loss is applied separately on these “slices” of the output vector, so each part of the robot is shaped by the most relevant expert. The context gating is what makes this smooth. The gate takes contextual cues (for example the commanded speed magnitude) and softly interpolates between expert outputs. When the 10-D command calls for high translational speed, the gate automatically shifts more weight to the locomotion expert; when speed is low or targeted at manipulation, it favors the WBC expert. Crucially, as the authors note, this is a smooth blend: “no hard switching between regimes”. The robot doesn’t leap from “walk mode” to “manipulate mode” abruptly; instead the gating network continuously adjusts expert contributions. And when a recovery situation is detected (say the robot is on the ground), the gate routes entirely (or heavily) through the fall-recovery expert for that instant. The end result is that all knowledge fuses into one policy under the 10-D interface, with no runtime switching. There is still an internal notion of “expert experts,” but the robot always runs a single neural policy. The teacher experts are used only during training. At deployment, the student autonomously blends the learned behaviors. (The architecture is modular too: in principle you could add another teacher and corresponding expert if you had a new required skill, by adding another channel to the gating network.). Another detail: every teacher’s reference data was filtered for safety. The authors apply a CBF projection so that in all teacher demonstrations, the robot’s center-of-pressure (CoP) stays within its foot support. Figure 2 (in the paper) shows an example: an unfiltered imitation trajectory drifts outside the foot support, but after the CBF filter it stays inside. This means the student is never taught to prefer an action that would cause an immediate fall. In effect, the student “internalizes” balance constraints during training. As [Yang et al.] note, the filtered CoP (left) stays in support, while the unfiltered (right) would have left the foot – so the teacher data is made feasible before distillation. Overall, the distillation procedure is quite innovative in combining multiple behavioral teachers. By leveraging KL losses on action slices with a soft gating, they unify walking, manipulation, and recovery in one learned policy. The result: a controller that can track velocity commands like a trot or a reach with equal aplomb, and shift between them gracefully. Mixture-of-Experts Controller Details. Let’s step through what the student network actually looks like. This “HANDOFF” policy is a neural network that takes as input the 10-D command plus recent proprioception (the last few frames of joint angles/velocities). In practice the authors use 11 frames of proprioception, so it sees its recent motion. After encoding this input, the net has three expert bodies. Each expert has 29 outputs (one per joint DOF). A gating sub-network (also from the shared encoder features) produces a 3-way softmax vector of weights. The final action is the weighted sum of the three expert outputs. During training, each expert is matched to its teacher as described: for example, the first expert might focus more on arms (WBC teacher), the second on legs (loco teacher), the third on falls. The context-based gating ensures that at any time, only the relevant teachers strongly influence the gradients. The training losses also include a small “load-balancing” term to encourage the gate to use all experts over time, and a “recovery loss” to sharply activate the fall teacher when needed. In orignal terms [25†L36-L40] spells this out: “Context-based action-sliced KL pulls the body slice toward a velocity-gated blend of WBC and locomotion teachers, the arm slice toward WBC, and the fall-recovery teacher onto the full action when recovery is active.” That sentence from the caption captures the logic of the distillation. In implementation, “body slice” means leg/trunk joints; “arm slice” means the arm joints; and “full action” means the entire action vector. When recovering, the student simply copies the fall teacher entirely. Finally, once trained, this yields a single controller: give it any 10-D command at runtime, and it computes joint velocities (at 500 Hz) by blending its learned sub-experts. There is no mode switch or manual logic needed during execution. Even though we trained with three teachers, the live policy is monolithic. As the authors emphasize, “no runtime switching” is needed – it’s all baked into the learned policy. Agentic Task Planning via Vision-Language Models. HANDOFF is meant to be the backend controller. What about the frontend planner? A nice part of the paper is showing a concrete “agentic” planning pipeline that generates the 10-D interface from natural language instructions. They hook HANDOFF into a vision-language-agent style planner so that, in principle, you could simply say “Put the mustard bottle in the red plate” and the robot does it. The pipeline works as follows (see Figure 4 on the project page). First, a high-level reasoner parses the natural language instruction into atomic tasks. For their experiments, they use a simple parser (regex + LLM fallback) to break the sentence into steps, but the key is that the instruction is decomposed into target objects and poses. Next, they run a Vision-Language Model (VLM) on the robot’s RGB-D camera. The VLM recognizes and localizes objects mentioned in the task. For example, it sees the “mustard bottle” and the “red plate” and computes 3D coordinates for them in the world. These detected positions are transformed into pelvis-centered waypoints (so the robot knows “the bottle is 0.5 m in front, 0.1 m to the right”). Now comes the conversion into the 10-D command. A tracking module takes those pelvis-frame waypoints and produces planar base velocities. In practice, if the current pelvis is not at the target, the tracker issues (v_x, v_y, ω_z) commands to move toward it. Simultaneously, a “skill selector” chooses values for the remaining dimensions: the desired root height and the wrist target positions. For manipulation tasks, this might mean choosing a lower height and attacking the hand targets to the bottle’s location. For a locomotion-only instruction, the wrists might stay at some neutral position. The skill selector is a small logic/deterministic piece based on simple heuristics (for example lowering the torso during reach tasks). The result of this process is a time-indexed stream of 10-D commands at 50 Hz, ready to feed the controller. In other words, the entire perceptual planning stack—from language to vision to tracking—outputs a continuous sequence of those 10 numbers per timestep. HANDOFF then runs at 500 Hz to meet those commands with joint actions. All these details are sketched in the paper’s Figure 4 caption. In their words: a single natural language instruction is decomposed; the VLM (at 0.1 Hz) projects 2D detections onto the RGB-D point cloud to give pelvis-frame waypoints; a tracker yields (vₓ, v_y, ω_z); and near the target a skill selector emits and wrist targets – forming the 10-D stream that drives the controller at 50 Hz. (Notably, it says the controller’s 29-DoF actions are tracked on hardware at 500 Hz, though that just means the low-level loops run fast enough.). The practical upshot is that they achieve zero-shot task execution without any extra RL or data collection for each task. The planner and vision system can be classical or modern, but once they output the 10-D stream, the controller is ready. HANDOFF never sees task-specific demonstrations or rewards at train time; it only learned from the three generic teachers. Yet at test time it can carry out many different commands. In short, the controller is completely agnostic to what the task is – it only cares about the abstract command vector. Experiments and Hardware Demonstrations. How well does HANDOFF perform? The paper reports both quantitative and qualitative results, but since the focus here is on how it works, we’ll highlight the key outcomes. First, on the Unitree G1 hardware, HANDOFF matches state-of-the-art performance on velocity tracking. In their tests, giving it various speed commands, it follows them as accurately as or better than previous controllers. More impressively, it achieves one of the largest robust manipulation workspaces reported. In simple terms, using only that 10-D command interface, the robot could reach and manipulate objects over a very large area around itself, thanks to the full-body coordination it learns. This suggests the learned policy doesn’t “run out of mobility” despite only seeing sparse commands. The real eye-candy comes from the task rollouts. The authors actually put the Unitree G1 in assorted tabletop tasks driven from voice commands (through the VLM pipeline). They show the same controller doing all of these, without any controller tweaks between them:. Pick-and-place: “Put the mustard bottle in the red plate.”. Pick-transport-place: “Pick up the whisk bottle, turn around and put it in the grey bin.”. Squat-pick: “Pick up the Pringles chip bottle, turn right and put it to the left of the Cheez-It box.”. Bimanual pick-and-hand-off: “Pick up the blue box with both hands, turn around and hand it over.”. Bilateral pick-and-place: Two bottles placed sequentially. Push recovery and continuation: For example, if the robot is picking up shampoo and gets pushed, it uses the fall-recovery mode and then continues the task. These examples are illustrated on the project page and summarized in the paper (and captured in). Importantly, all of these were done by sending natural-language commands to the planner (no additional data collection or RL for each scenario). For instance, [25†L51-L56] points out that across these tasks — from simple teleoperation-like commands (like playing badminton using the controller directly) to semantically complex language instructions — the same 10-D controller handled them. They emphasize “No controller-side change, data collection, or model fine-tuning is required between tasks.” This truly demonstrates the flexibility of the approach. Watching the videos (if you have them), the robot exhibits surprisingly human-like coordination. It bends at hips to reach down when needed, keeps balance when lifting objects, can even hand an object with both hands while facing backwards. Playing badminton was one of their examples of direct teleoperation: twisting its body and swatting a shuttlecock by setting root rotation and high hand targets. All through the same CMD stream. These rolls underscore the core claim: HANDOFF is a general-purpose controller. It doesn’t overfit to one scenario. Whether commanded to walk, balance while reaching, or stand up, it responds appropriately, thanks to the distilled multi-expert training. The combination of metrics (tracking error, workspace coverage) and visuals suggests the authors’ multi-teacher approach paid off. Discussion and Takeaways. Let’s recap why HANDOFF matters and how one might build on it. The main achievement is showing that you can have a unified, learned whole-body controller driven by a planner-friendly interface, without separate tuning for each task. This is a valuable step toward generalist humanoid agents. Previously, many systems either hand-coded controllers and switched modes (walking vs pushing arm) or trained narrow RL policies for one task at a time. HANDOFF instead learns to interpolate across skills. It bridges high-level planning with low-level control: planners only need to worry about “where to go and what to reach” in an abstract sense, not about joint trajectories. This decoupling is powerful. As [Yang et al.] note, existing controllers often “demand dense kinematic or spatial references that planners struggle to synthesize”. By contrast, HANDOFF’s interface is explicit, intuitive, and modular. Could you use this in your work? If you have a humanoid or even a different robot, the recipe is: design a compact command space appropriate to your tasks, train specialist policies for core skills, and distill them into a single Mixture-of-Experts policy. The gating and KL distillation scheme described here (with action-sliced losses) is generic enough to apply beyond just Unitree G1. One could imagine, say, a SAR robot where teachers include “rugged mobility” and “manipulation” etc. The key is ensuring the interface is rich enough to express needed behaviors; 10 dims for a humanoid seems to cut it for many manipulation/imitation tasks, but other robots might need different sets. There are a few caveats. The paper’s evaluation is on one robot (Unitree G1) and mainly on table-mounted tasks. It remains to prove how well this method generalizes to drastically different platforms or more unpredictable outdoor environments. The authors themselves note the limitation of single-robot evaluation. Also, the zero-shot language-to-action pipeline here is fairly structured – it relies on reliable object detection and a simple task parser. In more open-ended language scenarios, a more sophisticated “embodied agent” might be needed (though the building blocks they use, like VLM waypoints, could scale up). The 10-D interface might also have limits; very precise fingertip tasks or delicate manipulation might require extending the interface or adding fine-grained controls. Nevertheless, HANDOFF is an impressive demonstration of the concept: one controller, one small interface, many skills. It shows that complementary teacher distillation can integrate locomotion, manipulation, and recovery into a single whole-body policy. The use of knowledge distillation and mixture-of-experts in real robot control is fairly novel – most multi-task RL works do the opposite (lots of tasks in one network); here we do multiple specialized networks into one. From an ML perspective, the “action-sliced KL” trick is clever: it’s like saying “teach different parts of the output with different gurus.” That could inspire other multi-modal control problems too. In summary, if you’re designing a humanoid system that must both walk and manipulate under high-level commands, HANDOFF is a strong blueprint. Define a concise interface that your planner can easily target, train teachers for each fundamental capability, and then distill them together. The result is a flexible controller you can use off-the-shelf for many tasks. And as language and vision models get better, that unified controller can be plugged into increasingly capable planners. HANDOFF demonstrates that approach works in practice today. Sources: The description above is based on the HANDOFF paper materials, especially the abstract and project web page, and the detailed captions in the arXiv “Paper Capsule” summary which explain how the teachers and planner are set up. These sources contain the key details (10-D interface, multi-teacher distillation, gating, and hardware tests) we’ve discussed.