Latest / Embodied AI 101 / Humanoid-GPT: GPT-Style Transformer for Zero-Shot Dynamic Humanoid Control

Transcript

Humanoid-GPT: GPT-Style Transformer for Zero-Shot Dynamic Humanoid Control · Embodied AI 101

• Humanoid-GPT: GPT-Style Transformer for Zero-Shot Dynamic Humanoid Control. Imagine a humanoid robot that can learn to perform whole-body tasks — from soccer and dancing to digging and disinfection — without ever being explicitly trained on those tasks. Traditionally, legged robot controllers have faced a fundamental trade-off: highly agile behaviors (think parkour-like motions or acrobatic dance steps) typically require specialized controllers and copious task-specific data, while generalist controllers that can handle a wide variety of motions tend to be conservative and slow. A recent CVPR 2026 paper by Zekun Qi et al., introduces Humanoid-GPT, a novel GPT-style Transformer model trained on an unprecedented 2 billion frames of motion data, that appears to substantially break this agility–generalization trade-off. By massively scaling both data and model capacity, Humanoid-GPT learns a generative motion tracking policy: given a target motion trajectory, it outputs the joint actions to track it. Remarkably, this single model achieves zero-shot whole-body control on tasks it has never seen in training, even transferring directly onto a physical Unitree G1 humanoid robot without any fine-tuning. In this article we unpack the core ideas of Humanoid-GPT, place it in context of prior work, and discuss how it achieves such extreme generalization. We’ll describe how the authors assembled a billion-scale motion corpus, trained many specialized expert policies via reinforcement learning, and then distilled them into one giant Transformer via imitation (DAgger-style) learning. Along the way, key concepts like Harmonic Motion Embedding (HME) for quantifying motion diversity and clear scaling laws will emerge. The result is a foundation-model-like approach to humanoid control – analogous to how large language models unlocked general-purpose language understanding, Humanoid-GPT points the way toward general-purpose humanoid movement. The Agility–Generalization Dilemma in Humanoid Control. Controlling a humanoid robot is notoriously hard. Legged locomotion and whole-body motions involve complex contact dynamics and high degrees of freedom. Reinforcement learning and control policies have made huge strides in tasks like walking or flips, but those successes are often narrowly tailored to the task at hand. A policy that spectacularly does one athletic maneuver (say, a backflip) will often fail to track even slight variations of that motion, and certainly won’t step onto an unseen walk or do a new dance. Conversely, a policy trained to mimic a wide variety of motions (e.g. using motion-capture data of walking, running, and simple gestures) will generally appear clumsy and slow when asked to perform highly dynamic or acrobatic tricks. In short, “current humanoid motion trackers face a fundamental agility–generalization trade-off”: they can either track agile motions in-domain or generalize to unseen movements, but historically not both. This trade-off shows up repeatedly in the literature. For example, UniTracker (Kangning Yin et al., CVPR 2025) set out to learn a single tracking policy for many human motions by training RL “teacher” policies with privileged information and then distilling them into a conditional variational autoencoder (CVAE) policy. The CVAE learns a latent representation of motion intent, improving generalization to new behaviors, but it was still built on tens of millions of frames of data and required an extra fine-tuning stage on hard examples. GMT (Zixuan Chen et al., arXiv 2025) took a different tack: they used a Mixture-of-Experts (MoE) policy network and an adaptive curriculum to balance easy vs. hard motions. Their single policy could indeed track a diverse spectrum of motions in both simulation and on a real robot, but again was trained on only a few million frames of data. EGM (Chao Yang et al., arXiv 2025) further refined this direction with a curriculum sampling strategy and a decoupled MoE architecture that separately handles upper-body and lower-body features. EGM showed impressive zero-shot robustness after training on just ~4 hours of motion data (hard-won by synthetic generation), but required careful design to reach generality. Meanwhile, NVIDIA’s SONIC project (Luo et al., 2024) was one of the first to describe a "foundation model for motion tracking". SONIC scaled up motion tracking by increasing model size and data: it trained policies with up to 42 million parameters on 700 hours of motion-capture, over 100 million frames, demonstrating that performance “improves steadily with compute and data diversity, and learned policies generalize to unseen motions”. SONIC showed that, given enough scale, even full-body humanoid control could start to look like a foundation model: general-purpose and highly robust. However, SONIC’s largest policy was an MLP with a few dozen million parameters, and it was still limited to the 100M-frame dataset it had. The result was impressive whole-body motions and even VR/teleoperation interfaces, but the question remained whether one could push even further. Humanoid-GPT takes the next step: it combines a transformer architecture with truly massive training data, available because the authors aggregated every major human motion dataset they could (and then retargeted everything to one robot skeleton). The result is claimed to “track highly dynamic behaviors while achieving unprecedented zero-shot generalization to unseen motions and control tasks”. Let’s see exactly how they did it. Building the Billion-Frame Motion Corpus. The cornerstone of Humanoid-GPT is its billion-scale motion training corpus. Most prior work in humanoid control has been starved for data. Commonly used motion datasets like AMASS, LAFAN, Human3.6M, etc., total only a few million frames, covering mainly walking, running, and mundane activities. To overcome this scarcity, Qi et al. went on a data acquisition binge. They took AMASS (a unified mocap dataset), LAFAN1, Motion-X++, PHUMOS, MotionMillion, and possibly others [*], and augmented them with large-scale in-house recordings. Critically, they retargeted all motions onto the Unitree G1 humanoid skeleton, making a homogenous data representation. The result is astonishing in scale: roughly 2 billion frames of motion data. For perspective, SONIC used 100 million frames; UniTracker and GMT were on the order of 6–7 million; even video-based mocap (Human3.6M, etc.) is under 10 million. Humanoid-GPT’s dataset is two orders of magnitude larger than anything before. The one-sentence summary of the project notes: “Humanoid-GPT is pre-trained on a 2B-frame retargeted corpus that unifies all major mocap datasets with large-scale in-house recordings”. In their “Highlights” section, the authors emphasize: “We assemble an unprecedented 2B-frame motion corpus by aggregating all major mocap datasets (AMASS, LAFAN1, Motion-X++, PHUMA, MotionMillion) with large-scale in-house recordings”. Why so much data? The answer is just like in language and vision: more data usually means more generalizable models. With two billion frames, the motion space is truly vast, covering many sports, dance styles, acrobatics, chores, and even subtle balance recoveries. To quantify and manage this diversity, the authors introduce Harmonic Motion Embedding (HME). HME is a method for embedding motions into a latent space that emphasizes their harmonic (frequency) content, essentially capturing how dynamic or periodic a motion is. In the paper they write: “Harmonic Motion Embedding (HME) is a novel metric for quantifying and categorizing motion data diversity directly from the motion data itself”. Using HME, they analyze the dataset and find that their 2B-frame corpus has roughly 4–5× the latent “volume” of AMASS alone. This means the new dataset spans a much wider variety of behaviors, from slow, deliberate chores to fast, cyclical sports actions. In practice, HME is used for more than analysis: it guides how the data is partitioned. The authors cluster the motion data based on HME features, effectively grouping similar motions together (e.g. all sprinting-like sequences, or all twisting/dance sequences). This clustering allows them to train specialized experts on each type of motion, which we describe next. Expert Policies via Reinforcement Learning. With the clustered dataset in hand, Humanoid-GPT’s approach is a two-stage training pipeline (illustrated in the paper’s Figure). In Stage 1, a collection of motion expert policies is trained via reinforcement learning (and/or supervised imitation) on the different motion clusters. Each expert is a standard RL controller (for example an actor-critic policy or similar) trained to track one subset of motions. Because each cluster is more homogeneous, the expert can become very good at those motions using far less data. For instance, one expert might be trained on dig/drill motions, another on dancing motions, another on various balancing/recovery motions, etc. The key quote from the Method Overview is: “Motion experts are trained via reinforcement learning on clustered motion data using [HME]”. In other words, each expert policy is banked knowledge for a subset of the motions (with the retargeted Unitree G1 dynamics). By distributing the problem across many experts, they sidestep the data inefficiency and specialization issues of training one monolithic policy from scratch. Training each expert in its own sub-domain is far easier (and cheaper) than training one policy on every possible motion. It’s a divide-and-conquer approach. Once these specialized RL experts are trained, they can generate trajectories for those motions. The authors then proceed to Stage 2: distilling all the experts into one unified policy. For this, they employ a technique like DAgger (Dataset Aggregation), which is an imitation-learning scheme where the experts label new data as the generalist is learning. Concretely, the GPT-style Transformer (“Humanoid-GPT”) is trained to imitate the expert controllers across all motions. As the Transformer policy improves, it is sampled on new states and the corresponding expert actions are added to the training set, iteratively refining the model. The paper briefly describes this as: “A GPT-style Transformer is trained via DAgger distillation to consolidate all expert knowledge into a single generalist tracker with causal temporal attention”. In simpler terms: they teach the transformer model to behave like each expert in its domain, so that the final model can perform any motion an expert can, plus generalize to new ones. The “causal attention” part means the Transformer operates auto-regressively in time: it takes as input the recent history of states and desired reference motion, and produces the next control actions. By training on all expert demonstrations, this single Transformer learns a giant repertoire of behaviors. The massive dataset and model capacity allow it to “stitch together” patterns in ways no single expert could. Humane Transformer Architecture. What is this Transformer policy under the hood? It’s essentially a GPT-like sequence model, but for robot dynamics instead of text. The exact model details (number of layers, heads, etc.) are left to the paper’s supplementary, but conceptually it follows the GPT recipe: given a sequence of past observations (e.g. joint angles, velocities, and reference pose from the motion capture) and actions, the Transformer uses layers of multi-head self-attention (with causal masking) to predict the next action in the sequence. In effect, it treats a motion trajectory as a “sentence” and the control outputs as the “next word”. Because it is generative, Humanoid-GPT can sample entire motion sequences: starting from an initial state and a target motion, it simply iterates forward, each time predicting the next action. It can be used open-loop (just feed it the past frames of a trajectory and let it predict the rest) or closed-loop (constantly updating the reference as it goes). The authors note that this model is a “generative Transformer that tracks highly dynamic behaviors”, meaning it can generate smooth action sequences for even very agile motions. One key advantage of the Transformer is its ability to incorporate long-range context: it can, in principle, look back at many past frames via self-attention. This contrasts with the shallower MLP policies used in most prior work (including SONIC or UniTracker) which have much shorter memory. This temporal context can help with coordinating whole-body balance and timing complex actions. Additionally, because Transformers scale well with model size, the authors can simply enlarge the network (embedding dimension, number of layers, etc.) when training on more data. Indeed, they report clear scaling laws: “enlarging both the motion corpus and the model capacity yields consistent and substantial gains in tracking accuracy and stability”. In other words, just as in language models, bigger is better here too. Finally, they optimize the Transformer inference for real-time use. Using ONNX and TensorRT, they achieve end-to-end latencies under 1.5 ms per timestep on a high-end GPU (RTX 4090) — roughly 5× faster than prior motion-tracking policies like TWIST. This means the controller can run in a tight control loop on a robot, enabling dynamic movements. Results: Zero-Shot Control on Unseen Tasks. The real test of Humanoid-GPT is its zero-shot performance. After all this scaling, can it really handle brand-new motions? The authors put it through a battery of unseen tasks in both simulation and the real world. According to their videos and supplemental, the policy robustly tracks motions excluded from training, including complex dances, acrobatic flips, athletic maneuvers, and everyday tasks like disinfection or digging holes. For example, in demonstrations the Unitree G1 robot controlled by Humanoid-GPT performs soccer kicks, various dance routines, shovel motions for digging, and even wiping a surface for disinfection – none of which were explicitly in the training corpus. In all cases it does so at full speed and with stability. While we can’t play the video here, the quantitative and qualitative results speak volumes. The authors compare directly against the previous state-of-the-art SONIC controller (100M frames, MLP). They note in their supplemental: “We compare Humanoid-GPT against SONIC, the strongest prior generalist tracker, across four categories (daily motions, dances, high-dynamic, balance) — in every case, Humanoid-GPT produces smoother, more expressive, and more stable behavior” (figure in project page). In tabular form, they summarize that SONIC used ~100M frames and still suffered in some cases, whereas Humanoid-GPT uses 2.0B frames and achieves both agility and generalization. Critically, the policy transfers to the real robot without any additional training. In fact, social media buzz highlights this achievement. For example, the Twitter account @robotsdigest notes: “The real-world results are the strongest part. Humanoid-GPT transfers directly from simulation to a Unitree G1, tracking unseen dances, teleoperation commands, athletic movements, and recovery”. This means the physics and dynamics differences between simulator and the Unitree hardware were handled well enough that no fine-tuning was needed — a rare feat in legged robotics. The Unitree G1 (a standard research humanoid) simply executed the commands from Humanoid-GPT and succeeded at all those tasks. In their experiments, the authors also systematically measure performance. They report that on held-out motion test sets, Humanoid-GPT achieves lower tracking error and higher success rates than any prior method. They show scaling plots where training on 2B frames yields much better zero-shot tracking error than training on, say, 500M or 100M frames. Similarly, increasing model size consistently improves accuracy and reduces drift. Thus, the final model occupies a “new performance frontier”: it simultaneously handles highly dynamic motions and generalizes to new ones, something earlier methods could not do. To illustrate the scale gap: the table in their appendix lists prior whole-body trackers with their data size. Most use single-digit millions of frames and simple MLPs. SONIC was noted as using 100M frames. In contrast, Humanoid-GPT uses 2000M frames and a large Transformer. This massive scale is what makes the zero-shot leap possible. Discussion and Implications. Humanoid-GPT is, in essence, a foundation model for whole-body control. Just as GPT-3 or PaLM became generalists for language by training on internet-scale text, Humanoid-GPT is aiming to be a generalist for humanoid motion by training on internet-scale human motion (plus lab data). By distilling knowledge from many specialized experts and vast data, it yields one unified policy. The results suggest we can indeed break old trade-offs by “going big”. For practitioners and researchers, Humanoid-GPT offers some new possibilities. In principle, one could fine-tune or adapt the model to new robots or tasks, or feed it high-level commands (the architecture already supports latent tokens, etc.). One could imagine coupling it with vision-language models (like SONIC’s token interface) to steer the motions with language. Even without modification, it can already serve as a robust baseline controller: give it any new trajectory (even if quite different from training), and it will attempt to execute it dynamically. Of course, there are practical considerations. Pretraining on 2B frames is a massive compute effort (likely thousands of GPU-hours). Running a big Transformer in real time requires a powerful GPU, though the authors show that TRT-accelerated inference can hit ~1.5ms per step. As of now, the model is tied to the Unitree G1 dynamics (through retargeting). Transferring to a different robot would require either additional fine-tuning or retraining on motions for that morphology. And while 2B frames is a lot, it still comes from human motions; ever more exotic motions (say, incorporating vehicle-like behaviors or extremely novel tasks) might still challenge it. Nevertheless, Humanoid-GPT clearly sets the bar for what’s possible in whole-body control: agility and generality together. It validates the intuition that data and model scale can overcome many traditional bottlenecks in robotics. This opens the door to more ambitious “robotic foundation models” — the authors note that their scaling laws are “clear”: adding more data and layers keeps improving zero-shot performance. We may soon see even bigger motion models, multi-agent or multi-robot versions, or models that integrate perception and language (a full vision-language-action Transformer for humanoids, similar to RT-2/GR00T for manipulation). In summary, Humanoid-GPT demonstrates that by combining a GPT-style Transformer with billions of frames of human-derived motion data, one can achieve highly dynamic, agile whole-body robot control out-of-the-box for tasks never seen in training— effectively breaking the old specialization-generalization compromise. For roboticists, this work is an exciting proof-of-concept: it suggests that the path forward to versatile humanoid skills may indeed be through ever-larger “robotic foundation models,” much like the path in NLP and vision. References: Qi et al., Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking (CVPR 2026); Luo et al., “SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control” (NVIDIA, 2024); Yin et al., UniTracker (CVPR 2025); Chen et al., GMT (arXiv 2025); Yang et al., EGM (arXiv 2025). (Supplementary references in main text.).