Latest / Embodied AI 101 / UniTracker: A Universal Motion Tracking Framework for Humanoid Robots
Transcript
- UniTracker: A Universal Motion Tracking Framework for Humanoid Robots. Humanoid robots promise to perform a wide range of human-like tasks – from dynamic balancing and acrobatics to dancing or playing sports. But getting them to imitate rich human motion in a robust way is incredibly challenging. Robots have high-dimensional bodies and delicate balance, so learning to track a variety of human motions while respecting physics is an open problem. Traditional motor controllers or simple imitation often produce dull, averaged-out behaviors that drift or fall. For example, a policy trained under partial sensing may “forget” which way it’s facing or slip contacts, ending up with orientation drift and other inconsistencies. The UniTracker framework, introduced by Yin et al. (2025) and open-sourced as part of the Intern Robotics toolbox, tackles exactly this problem by building a universal motion tracker that can drive a humanoid through a huge repertoire of motions reliably – just as a single person’s motions are expressive and stable. At a high level, UniTracker uses a three‐stage learning pipeline combining a privileged “teacher” policy, a CVAE-based “student” policy, and a fast adaptation module. In simulation, the teacher policy (trained via reinforcement learning) uses full state information to faithfully mimic reference human motion trajectories. The student policy is then distilled from this teacher under realistic sensing limitations, but crucially it is implemented as a Conditional Variational Autoencoder (CVAE). The CVAE lets the policy learn a global latent embedding of entire motions, so the robot keeps track of overall intent. Finally, for motions the policy still struggles with, a lightweight residual module can quickly fine-tune the policy on-the-fly. Together, these stages produce a single control policy that – as demonstrated in both Mujoco simulation and on a Unitree G1 humanoid – can zero-shot track thousands of diverse motions (kicks, squats, dances, etc.) with high fidelity, stability, and generalization. Why UniTracker? Limitations of Naïve Imitation. Before diving into details, let’s motivate the design. A common paradigm for learning motion tracking is a teacher–student imitation framework. Typically, a “teacher” policy is trained in an idealized simulator to follow motion capture references given privileged state (e.g. exact target pose, velocities, terrain). Then a “student” policy is taught (via supervised learning or DAgger) to imitate the teacher but using only the limited observations available on the real robot (e.g. body pose without future references). This can work to some extent, but it has key pitfalls. In practice, a straightforward student (say a feedforward neural net) tends to lose a lot of diversity and global coherence. Yin et al. note that purely “MLP” students often produce bland, averaged motions that no longer capture the vibrant style in the data. Even worse, they frequently suffer from global drift – e.g. slowly rotating and falling out of sync with the reference. The authors report that existing MLP-based trackers “suffer from limited expressiveness and…poor generalization across diverse behaviors,” and specifically “drift in global attributes like orientation” when they lack global context. In short, a basic feedforward policy forgets where it is in the bigger picture of the motion. UniTracker addresses this by injecting a latent “scene” or intent variable into the student policy. Concretely, during student training they use a CVAE architecture so that the policy learns a compact latent embedding capturing the global motion intent. By encoding an entire reference trajectory (or summary) into a latent vector, the student retains awareness of which phase of the motion it is in, where it should end up, and so on. At run-time, a learned “prior” network infers this latent from the partial observations. The result is that the student policy no longer operates purely in myopic, moment-by-moment mode, but with a sense of overall direction – greatly improving stability and fidelity. The UniTracker authors emphasize that this CVAE stage is what unlocks the generality: “the CVAE structure allows the policy to learn a global latent representation of motion, enhancing generalization to unseen behaviors and addressing the limitations of standard MLP-based policies under partial observations”. In practice, this means a single policy can track very different motions – stretching, walking, kicking, dancing – without re-training each time, whereas a narrower MLP policy would quickly lose its way on sequences it hadn’t seen. UniTracker Architecture: Three Stages. UniTracker realizes these ideas in a three-stage training process. Each stage serves a clear purpose: generate high-quality supervision, build a universal policy, and then refine it. We outline them below. Stage 1: Train a Privileged Teacher Policy. First, the system prepares a dataset of reference human motions (from motion capture libraries) that we want the robot to imitate. Using a physics simulator (MuJoCo) of the target humanoid, they train a teacher policy via reinforcement learning to track these motions. Crucially, the teacher has privileged information: it sees the full state of the reference motion, future waypoints, contact forces, etc., as well as the robot’s true state. This makes the RL task easier and yields a high-fidelity “expert” controller. In effect, the teacher learns, for each time step of a reference sequence, what motor command best follows the target. Yin et al. use a goal-conditioned RL setup for the teacher. The goal at each step might be something like “reach this next pose in the reference” or “minimize tracking error to next frame.” The paper describes this as training “a teacher policy with privileged observations to generate high-quality actions”. The teacher policy ends up imitating the reference almost perfectly in simulation, producing smooth trajectories and respecting contacts. Importantly, this stage populates much of the training data: the teacher rolls out over thousands of motion clips, recording the (state, goal, action) tuples as training examples. (At this point, a reader might wonder “Why bother with a teacher? Why not train directly?” In practice, training with full state is much easier. If we tried to train the robot with only partial observations from the start, the problem would be a tough POMDP. The teacher is essentially a clever way to generate synthetic expert demonstrations at scale. UniTracker then uses these to supervise the real deployable policy.). Stage 2: Distill into a Universal CVAE Student Policy. Next comes the heart of UniTracker. The goal is to train one deployment-ready policy that can run on the real robot with only its onboard sensing (proprioception or vision) and track any reference motion. To achieve this, they distill the teacher’s expertise into a new network using imitation learning, but crucially the student is a Conditional Variational Autoencoder (CVAE) rather than a plain MLP. Here’s how it works: During training, for each demonstration trajectory the teacher walked through, the student receives the partial observations that the real robot would have (e.g. robot’s joint angles, joint velocities, maybe a small window of past reference pose). The student network has two components – an encoder and a “prior” – which together produce a latent code $z$ capturing the global motion context. The encoder actually uses the teacher’s privileged info (full reference motion) to produce an “ideal” $z$, while the prior takes only the partial obs to predict $z$. The CVAE is trained such that the prior’s distribution matches the encoder’s distribution. Then the decoder (the policy) takes $(z, \text{partial obs})$ and outputs the target action. This way, the policy is effectively conditioned on a learned latent which encodes where in the motion it should be. An important trick is that they align the partial-observation prior with the full-observation encoder. In practice this means the student learns to “imagine” the global intent. The result is that, once trained, the student policy given an unseen motion can sample $z$ from the prior (based on what it currently observes) and then produce coherent actions. The authors note that thanks to this latent channel, the student “incorporates global intent during training” which pure MLPs cannot do. In other words, the student no longer drifts off course because its neural hidden state explicitly remembers the overall motion plan. By using this CVAE distillation, UniTracker preserves the expressiveness and variety of the teacher demonstrations. Where a vanilla student might converge to the “mean” motion, the CVAE student can still produce the rich gesture-like quality of the original. Yin et al. report that, compared to a standard MLP imitation (even with DAgger), their CVAE-based student generates motions that look much more natural. In short, the CVAE stage yields a universal policy: one network that can reproduce thousands of different poses and styles under partial sensing. Stage 3: Fast Adaptation for Hard Sequences. Even with the CVAE student, there will be some very challenging motions that it still can’t track perfectly in one shot. Kicking a ball, doing a backflip, or reacting to an unexpected perturbation might need a bit more fine-tuning. For these cases, UniTracker adds a third component: a lightweight fast-adaptation module. This adaptation module is essentially a small residual network attached to the student. During training, when the universal student struggles on a sequence, the adaptation network is trained to provide a corrective tweak. Concretely, you can imagine feeding a particular motion sequence through the student, measuring where it falls short, and then backpropagating a quick update into a tiny “adaptor” network (a residual decoder) so that the outputs improve. Once this adaptor is trained, we now have a version of the policy that is specifically tuned to that motion or class of motions. The magic is that adaptation is very fast and can be done on the fly. The paper demonstrates two modes: single-sequence adaptation, where an online gradient step or two tailors the policy for exactly one target clip, and batch adaptation, where the student policy rapidly fine-tunes on a batch of new motion examples. This modular scheme means that if the robot is about to do a tricky routine (say a martial-arts combo it has never seen), you can quickly adapt the core policy for that situation. The authors emphasize that this stage shows UniTracker’s “flexibility and scalability”. In summary, after these three training stages we end up with a final policy that (1) learned from a strong teacher, (2) carries rich latent knowledge of motion context, and (3) can be made even more accurate via fast fine-tuning. Yin et al. demonstrate this on a 29-DOF Unitree G1 humanoid both in sim and on hardware. Experiments: Motion Diversity and Accuracy. The UniTracker paper evaluates the framework on thousands of human motion clips and on a real robot. They collect a wide variety of motions – squats, kicks, dances, sports moves, and more – drawn from large mocap libraries like AMASS. Importantly, they filter the raw data through a physics-consistency check (courtesy of a tool called PHC) to remove any poses that would require self-penetration or impossible contact. The clean set then serves as the motion reference corpus. In simulation, they train the teacher on these motions and then train the CVAE-student. The InternHumanoid codebase makes all of this configurable; for example, to train on the G1 (29 DOF) they give a single command like, which bundles together the virtual robot model and the imitation task. After distillation, they roll out the student on hundreds of held-out sequences. The results are impressive. UniTracker can track sequences it has never seen during training – creative combinations like a hip-hop dance or a karate kick. In these tests, the policy follows the reference with low error: the robot’s joints stay close to the target trajectory and its global orientation remains aligned with the intent. Crucially, UniTracker avoids the mistakes common to baseline policies. For example, whereas a basic MLP student might systematically under-rotate during a spin and lose balance, the CVAE-based policy preserves the fluidity of the spin by virtue of its latent awareness. Likewise, the blended latent lets the student keep its “head in the game” globally, avoiding the orientation drift and falling that bedevils simpler imitators. Quantitatively, the authors report metrics like mean per-joint error and orientation drift. UniTracker outperforms comparable imitation methods by a wide margin. In ablations, turning off the CVAE latent or the adaptation module causes motion quality to degrade noticeably. The gains are not just in raw numbers: motion videos (see their accompanying demo) show that trajectories from UniTracker look far more natural and human-like. The paper states that the final system achieves “strong performance in motion diversity, tracking accuracy, and deployment robustness” on the actual G1 hardware. In fact, they claim superior tracking accuracy and expressiveness compared to any baseline they tried. Practically, this means a Unitree G1 running UniTracker can, out of the box, mimic raw human motion clips in real time. The policy is robust to sensor noise and slight model mismatch: they train with randomized masses and friction so it can handle reality. In hardware tests, the G1 can do squat cycles, go standing, perform kicks and cartwheels, and even react to pushes – all driven by the learned policy. An example (Figure 1 in the paper) shows the robot doing a squat, a golf swing, a high kick, a lateral side step, a slow dance under gentle pushes, and a difficult acrobatic move after adaptation, all without any manual tuning. Another highlight is scale: the system was evaluated on over 8,000 distinct motion clips. That’s an order of magnitude more variety than most imitation papers. Despite this scale, the universal policy handles them with grace. The CVAE really pays off here – described in the paper as allowing the student to “retain expressive motion characteristics while improving robustness”. Downstream Applications: Text and Video Inputs. Beyond pure imitation, the authors explore using UniTracker in creative downstream tasks. Since the student policy treats arbitrary reference trajectories as input, you can in principle drive it from any source of motion data. They demonstrate two examples:. Text-to-Motion: Using a simple language model or keyword template, they create synthetic reference sequences described by text. For instance, the phrases “A person is punching forward” or “A person is dancing the waltz” are converted into rough reference trajectories (using an external motion generator). UniTracker then tracks those generated motions. The result is a demo where someone types a sentence and the robot performs the corresponding action. Video-based motion capture: In another demo, they feed a real video or skeleton capture of a human doing some action into an estimator, then pass the estimated poses as the reference. UniTracker then has the robot follow that video’s motion. For example, a camera filming a choreographed dance can be used to make the robot mimic the dancer. These demos underscore UniTracker’s generality. The paper’s Fig.3 shows UniTracker taking as input the sentence “A person is squatting down” or “A person is extending his body” and producing the matching motion in simulation. The same policy also successfully tracks poses inferred from raw video footage. The key is that once you have a reference trajectory (whether from motion capture, a generative model, or vision), the policy doesn’t care – it just executes it. This opens up interesting possibilities like natural-language control or human-robot interaction with video. How It Works: Technical Highlights. Few more technical points are worth noting. First, the latent representation is future-conditioned. During training, the CVAE encoder is allowed to see the upcoming frames of the reference, so it can pack future intent into $z$. The prior network, however, only sees current/partial observations, so it learns to predict that same latent without cheating. This alignment (encoder = $z \sim q(z|\text{full traj})$, prior = $p(z|\text{obs})$) is trained by a KL-divergence loss as in standard VAE training. By the end, the prior network essentially captures the conditional distribution of future motions given what the robot has seen so far. Second, the adaptation decoder is implemented as a small residual on the student’s output. Concretely, after the main policy network generates an action $a$, the residual module outputs a correction $\Delta a$ based on the student’s latent $z$ (and possibly some short history). Only a lightweight network (far smaller than the main policy) is needed. In practice, this means adaptation adds minimal computational cost or overfitting risk: it just tweaks the final joint torques or targets. Third, all training (teacher RL, student distillation, adaptation) is done in simulation using an off-the-shelf humanoid model. The Unitree G1 has 29 actuated DoFs (hips, knees, ankles, arms, etc.). The same codebase can handle other humanoids too (H1, H1-2, or custom models), thanks to a gym-like environment. The results shown are mainly on G1, but the authors emphasize in their code repo that the pipeline is multi-robot ready. Finally, the system supports a modular workflow. After training, policies are exported (they use ONNX format) so that real-time control is just policy inference. The InternHumanoid toolkit provides scripts to play back policies on new motion clips, to do sim-to-sim tests, and to deploy on hardware. For example, they document simply running Spain’s Hydra config to transfer the policy from Isaac Gym to a MuJoCo test or directly to the real G1. The first release of this toolbox, which includes UniTracker, came out in July 2025, indicating its practical pedigree. Prior Work and What’s New. It’s useful to place UniTracker in context. Many prior works have trained humanoid controllers from human motion. Classical motion tracking controllers (like Paden–Kim style) rely on careful tuning and can panics on unseen moves. Recent deep RL approaches have shown that policies can imitate mocap using dense rewards or reference-pose losses, but often one job at a time. Other humanoid control papers train one specialized skill (walking, flipping, etc.) or focus on locomotion on varied terrain. UniTracker’s ambition is different: truly universal whole-body imitation. The teacher–student distillation idea itself is not brand new – it’s a standard trick in motion imitation. What stands out is how it’s done. By formalizing the student as a CVAE, the authors break the bottleneck of MLP expressiveness. This is a key novelty: instead of distilling into a single fixed network, they allow a structured latent that preserves motion diversity. Previous distillation papers rarely added this much representational power to the student. The results confirm that the extra complexity is necessary: UniTracker handily beats a comparable one-stage DAgger approach with a plain policy network (their experiments call it MLP-baseline or “standard DAgger”) in every metric. The fast adaptation step is also relatively new in this domain. Some prior works fine-tune policies per task, but UniTracker integrates it seamlessly as a quick residual tweak. This is an example of a learning-to-learn style trick: a universal policy plus a fast-adapting add-on. It sharply contrasts with rigid policies that would need hours of retraining. In short, the combination of CVAE distillation plus residual adaptation appears novel for humanoid motion tracking. It yields a more scalable solution: a single neural pipeline handles thousands of motions and even permits extension (via fine-tuning) to new ones. Using and Extending UniTracker. For practitioners interested in trying this out, the authors have provided extensive code and documentation. The entire pipeline lives in the toolbox. This repo includes environments for multiple humanoids, pre-built training scripts, and configurations for imitation tasks. For example, to replicate a universal tracker on the Unitree G1, one would simply run the provided PPO training script with. After training, one can switch to play mode to evaluate on new motion clips. The repository also contains hardware deployment code: policies can be exported to ONNX and run on the G1’s onboard computer. Sim-to-real was engineered from the ground up (with dynamics randomization and noise models) so that zero-shot transfer works reliably. In fact, the GitHub notes that with these techniques the G1 can already resist human kicks and other disturbances out of the box. Because the system is open-source, researchers can extend it. For instance, one could add new motion datasets or experiment with different latent sizes. The CVAE formulation is modular enough to try other generative bottlenecks (Gaussian mixtures, flow models, etc.) if desired. The adaptation module too is configurable – one could swap in a small MAML-style network for on-line updates. The InternHumanoid code structure (using Hydra configs) is very flexible, enabling quick iteration. In summary, UniTracker gives a practically useful foundation for expressive humanoid control. Instead of training specialized controllers for each move, you get one big controller that generalizes across the board. The framework also highlights some valuable lessons: when imitating diverse data, inject capacity for variation (via a latent); and for the tough parts, allow light tuning on top. Conclusions. Getting humanoids to smoothly imitate arbitrary human motions has been a long-standing goal. UniTracker represents a significant step towards that goal by combining modern imitation learning tricks into a unified pipeline. Its key insight is to preserve the intent and diversity of motion during distillation: the CVAE-based student policy learns a latent embedding of whole-body behaviors, which anchors the robot to the global style of the reference. This overcomes the tendency of simpler policies to drift or oversimplify. The experimental results are convincing: one policy on a real Unitree G1, able to follow thousands of motions drawn from human data, with high accuracy and robustness. Furthermore, the open-source implementation makes this advance accessible. Roboticists can adapt UniTracker to other humanoids or customize it for their tasks – for example, by plugging in new text-to-motion generators or video pose estimators via its downstream interfaces. UniTracker thus offers a universal tracking capability: a humanoid that can do “anything” it was shown (and quickly learn new tricks), much like a human practicing new skills. In future work, one could extend this by incorporating vision for terrain-aware tracking (as in related works), or by integrating higher-level task goals. But as it stands, UniTracker delivers an impressive demonstration that a single neural controller can flexibly imitate a broad library of human behaviors. Sources: The above summary is based on Yin et al., UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots (2025).