Latest / Embodied AI 101 / Humanoid-GPT: A Foundation Model for Zero-Shot Humanoid Control
Transcript
- Humanoid-GPT: A Foundation Model for Zero-Shot Humanoid Control. In the quest to endow humanoid robots with general, agile whole-body skills, a new generation of approaches is emerging that mirror the rise of large language models. A recent CVPR 2026 work, “Humanoid-GPT” by Qi et al., takes this idea to heart, training a single Transformer-based controller on an unprecedented scale of motion data. The result is a demonstration of the Unitree G1 biped robot performing highly dynamic behaviors – from complex dance routines to sports-like maneuvers – straight out of the box, without any task-specific tuning. In other words, this model can track zero-shot new motion targets with surprisingly high fidelity. This article unpacks the Humanoid-GPT paper, exploring how it was built, what it achieved, and why it matters for embodied AI and robotics. The Challenge of Humanoid Motion Control. Humanoid robots are inherently high-dimensional and under-actuated, making robust whole-body control a grand challenge. Traditional methods tackle one skill or task at a time, using either hand-engineered controllers or reinforcement learning (RL) for each motion (walking, running, sitting, etc.). Even with RL, we typically train specialist policies in simulation or via teleoperation for a narrow set of motions. This “one skill per policy” approach is cumbersome and brittle: each new action sequence or style of motion requires new training or expert demonstrations. Moreover, most earlier methods relied on relatively small motion datasets and simple networks (like shallow MLPs or small recurrent networks) that had to trade off agility (can you move fast and precisely?) against generalization (can you handle new, unseen motions?). In practice, a controller trained for agile jogging might completely fail on an unfamiliar dance move. The Humanoid-GPT project was motivated by this very trade-off. The authors observe that prior humanoid trackers often plateaued when given more data or complexity – they simply could not grow to absorb the rich diversity of human motion. Their key insight was to scale both the data and the model simultaneously. In essence, they proposed treating robot motion control as a “foundation model” problem: gather a vast dataset of human motions, train experts, and distill them into a single large Transformer that can predict actions like GPT predicts words. In doing so, they aimed to eliminate the agility-generalization trade-off – the model, by virtue of its scale, would handle both subtle balancing acts and wild new maneuvers with the same ease. Building a Billion-Frame Motion Corpus. Central to the approach is an enormous training dataset. To reach human-level diversity, the authors aggregated virtually all relevant human motion capture (MoCap) data they could find. They combined four major public datasets (AMASS, LAFAN1, MotionMillion, and PHUMAN) – each of which contains hundreds to thousands of motion sequences of various activities – with large proprietary recordings from Galbot itself. This unified corpus spans virtually every kind of human behavior imaginable: locomotion, acrobatics, martial arts, dancing, sports plays, and more. In raw form the data is heterogeneous – different skeletons, sensors, sampling rates – but the team retargeted everything into the Unitree G1 humanoid’s 29-DOF joint space. In other words, every captured human pose (which might have mysterious numbering of bones) gets mapped to the same 29-joint representation that the physical G1 robot actually uses. Having merged the sources, the authors still needed to clean and balance the data. They filtered out any motions involving explicit object interactions or scene complexities that wouldn’t make sense on a bare-floor testbed. For example, sequences of sitting on chairs, swimming, climbing stairs, or other contact-rich actions were removed. The goal was to ensure the robot could physically execute the motions without needing extra environment modeling. After filtering, the core dataset covered hundreds of thousands of seconds of motion (scaled by 5× through simple time-warp augmentation). Time-warping means they took each motion clip and uniformly accelerated or decelerated it to create variants, effectively multiplying temporal diversity. The final result was a gargantuan 2 billion frames of robot-compatible motion data – roughly a million hours at 30Hz – far more than any prior humanoid tracker had ever seen. (By comparison, earlier tracking models usually trained on just a few million frames.). Importantly, this training set isn’t just large; it’s balanced to cover the full spectrum of motions. Without care, common actions like walking or running might dominate the data and bias a model to only do those well. To address this, the authors devised a novel diversity embedding called Harmonic Motion Embedding (HME). The idea is to measure each sequence by its rhythmic content: they train autoencoders to extract the dominant oscillation frequencies and amplitudes of each joint over time. For each sequence, they record the mean and standard deviation of those per-joint harmonic features into a compact vector. This “HME vector” effectively fingerprints the motion (e.g. “this is a bouncing leap, versus a slow choreographed pose”). They then run K-means clustering on these vectors to partition the dataset into roughly 300 motion clusters. Each cluster contains about 1,000–2,000 sequences that are locally similar in style (e.g. one cluster might be leaps and flips, another might be punching and blocking moves), while collectively covering the broad global distribution. By clustering in this way and then sampling uniformly from each cluster, the training process ensures no single style overwhelms the model. This balanced sampling (combined with the 5× time warps) means that a rare but dynamic action – say, a breakdance move – has comparable presence as everyday walking. In short, the 2B-frame corpus is curated for diversity. As the authors point out, this data expansion made their motion embedding space about 4–5× larger in “volume” (a proxy for diversity) than the largest public dataset alone – essentially capturing the majority of human whole-body motion priors. This enormous, balanced dataset is what allows the model to generalize so broadly. Stage 1: Training Motion Experts via RL. With the dataset prepared, the next stage was to translate it into robot behaviors. The authors adopted a “divide-and-conquer” strategy: instead of training one monolithic RL policy on all 2B frames at once, they first train specialist expert policies, one per cluster from the HME step. In each cluster, all sequences are quite similar, so a single controller can learn to mimic them well. Concretely, for each cluster (say cluster i of ~1.5K sequences), an RL policy $\pi_i$ is trained to track those motions. Training is done in MuJoCo simulation. Each episode samples a target motion sequence from the cluster, and the goal is for the robot to follow that sequence. The state fed to the policy at time $t$ includes a privileged state of the robot plus the desired target pose. Specifically, the authors used a state vector containing the G1’s full joint positions and velocities, the root (pelvis) angular velocity, the gravity vector, and the previous control action. The reference signal is the target pose $q^{ref}_t$ (the 29 joint angles of the motion clip at time $t$). Thus overall the policy $\pi_i(s_t^{priv}, q^{ref}_t)$ outputs an action $a_t$, which is simply a set of per-joint target values for PD control (i.e. the policy’s output goes into a low-level PD controller on the robot’s joints). The reward is designed to make the robot’s body match the reference motion. The authors compute a keypoint-level tracking reward: for a set of critical body points (pelvis, feet, arms, etc.), they penalize the differences in position, orientation, and velocity between the robot and the target. Formally, each keypoint gives exponential reward terms $R_{\text{pos}}$, $R_{\text{rot}}$, and $R_{\text{vel}}$ that decay as the error grows. They weight these by importance (for example, penalizing the feet more stringently than the wrist). In addition, small penalties discourage self-collisions and encourage smoothness (avoiding jerky motions). Summed together, this reward incentivizes stable, balanced imitation of the reference: the robot must keep up with the fast-moving keypoints (agility) while not falling over (stability). To improve robustness, the experts are trained with domain randomization in simulation. The paper mentions perturbing environments (like uneven terrain), external forces, and even varying physical parameters (joint friction, masses, slight jitter in target poses) during training. This prevents the experts from overfitting to one perfect model of the G1. In practice the authors only keep the experts that achieve sufficiently high fidelity – that is, if some cluster’s sequences are too hard, they might discard that expert. By the end of this phase, there is a library of about 300 motion-specific RL policies, each an expert at one part of the motion space. These experts act as “teachers” for the next stage. Stage 2: Distilling into a Unified Transformer Controller. Having trained a hundred-odd expert policies, the key innovation of Humanoid-GPT is to distill their knowledge into a single generalist controller. Instead of running 300 separate agents in parallel or hard-switching between them, the authors employ a GPT-style Transformer as a universal motion tracker. To train it, they frame the distillation as a sequence modeling problem in a DAgger (Dataset Aggregation) style. Concretely, a Transformer model $G_{\theta}$ is trained so that, given the robot’s recent history and a motion target, it outputs the same actions that the experts would have output. At each time step the network takes as input a concatenation of the current proprioceptive state $s_t$ and the target pose $q^{ref}t$ (just like the experts see). They build an input token $e_t = [s_t, q^{ref}t]$. During training, a sequence of $H$ such tokens $e{t-H+1:t}$ is fed through the Transformer with a causal mask (so that it can only attend to the current and past tokens, not the future). The model produces a sequence of $H$ actions $\tilde{a}{t-H+1:t}$, one action for each input token. We then compare this output sequence to the “teacher” sequence of actions $\hat a_{t-H+1:t}$ that the experts would have produced. In practice, for each history the system identifies which expert (or combination of experts) covers the motion target, and uses that expert’s action for each time step as the ground truth. The loss is a Smooth-L1 (Huber-like) distance between the whole predicted action sequence and the expert actions. This multi-step supervision is a big advantage: by training on entire chunks of trajectory at once, the Transformer learns temporal consistency and avoids drift. It also means that early in an episode the model implicitly learns to produce sensible actions even without much history seen (because it is trained on random-start segments). At inference time, the Transformer is rolled out autoregressively: at each control loop it maintains a fixed-length queue of the last $H$ state-reference tokens, feeds them through the network, and uses only the last output action (for time $t$) to command the robot. The earlier outputs in the sequence are effectively just intermediate predictions during training. Why a Transformer? The authors argue that GPT-style models elegantly exploit parallel sequence training (speeding up training over RNNs or other online methods) and capture long-range dependencies across time. The causal attention mechanism fits the online control constraint (no peeking at future frames) and generally scales to larger model sizes better than an MLP or temporal convolutional network. In effect, the Transformer learns to predict the next best joint targets given long windows of what it just did and what the reference motion is doing. Because of the large dataset and the diversity of examples, the network can learn complex “style” sequences – it isn’t just doing one-step tracking, but modeling how sequences of poses evolve. When distillation is complete, we have one model $G_\theta$ with tens of millions of parameters that can, in principle, imitate any of the experts’ skills. At this point the original RL experts are no longer needed for control. Everything at test time is done by querying the Transformer on the fly. In practice the authors tested several model sizes; their largest (Humanoid-GPT-B) is around 80 million parameters, which is quite large for a robot controller, but tractable with modern hardware. Experiments and Results. The paper presents extensive experiments, both in simulation and on the real Unitree G1 humanoid, to validate that the Transformer tracker can indeed generalize. A key theme is scaling laws: as they increase the amount of data and the model size, performance keeps improving, without the earlier plateaus seen in small models. In simulation, they first show that increasing the number of motion clusters and the length of history $H$ both raise tracking success rates. Using longer history windows allows better anticipation of long-horizon effects (like landing after a jump). Similarly, exposing the model to more randomized training variations (more environments, more physics tweaks) improves its robustness. Crucially, when they compare different backbone architectures, Humanoid-GPT consistently beats alternatives. MLP-based and TCN (temporal convolution) controllers quickly saturate: adding more data doesn’t help them much beyond a point, and they exhibit unstable jitters on complex maneuvers. The Transformer, on the other hand, keeps getting better as data/model grow. Metrics like Mean Per-Joint Position Error (MPJPE) and velocity error drop significantly with the Transformer. In particular, the authors compare to recent baselines like “GMT” (a prior General Motion Tracker by Chen et al.) and “TWIST” (a teleoperation-based whole-body system), as well as an approach called Any2Track. Across all these, Humanoid-GPT-B has the lowest tracking error and highest stability. For example, the Transformer’s outputs show smoother matching of reference and maintain balance even when quick changes happen, suggesting it has truly learned underlying body dynamics to some degree. A particularly striking result is how well the model carries over to the real robot. The same controller trained entirely in simulation was deployed on an actual Unitree G1 without any fine-tuning. The team reports that tracking performance on the hardware was close to simulation results: the robot executed dance sequences, jumps, and even operator telecommands with fluidity. For instance, in one demo a person could move their own limbs, and the robot mirrored the motion live (a kind of whole-body teleoperation). Another demo throws soft balls at the dancing robot – instead of falling, the controller adapts and the G1 absorbs the disturbance while continuing to perform the choreography (see the authors’ video). These show that the Transformer’s policy had implicitly learned balance and contact reaction strategies. Despite the model’s size, it still meets real-time requirements. The authors ported the Transformer to C++ with ONNX and TensorRT optimizations. Running on an NVIDIA RTX 4090 GPU, the full inference (running the 80M-parameter network) takes under about 1.5 milliseconds per step. This meets or exceeds the control loop requirements of the robot’s PD controllers. Hence the model can issue joint commands at full speed (they test at 30–60Hz loops) without latency issues – a crucial engineering point, since big networks sometimes struggle in real-time control. What Makes It Work? Scaling and Diversity. Two main factors explain the success of Humanoid-GPT: massive scale and smart data handling. By training on 2 billion frames from hundreds of hours of motion, the model simply sees more variety than anything before. The authors demonstrate explicitly that this diversity is key. For example, they measure the “volume” of the harmonic embedding space: the 2B-frame set covers 4–5× the log-volume of the AMASS dataset (a large public collection) along at least one dimension of the HME embedding. In plain terms, many motions that were exotic or missing in smaller datasets – from martial arts flips to odd sport stances – become routine in training. The HME-based sampling further ensures that this expanded space is covered evenly. On the modeling side, the Transformer architecture readily absorbed the scale. The authors experimented with different model depths and sizes; in every case, doubling model capacity reduced tracking error, and no “regime” was hit where more size hurt performance – quite unlike the MLP/TCN baselines that flatlined. This matches what one expects from “foundation models”: bigger is better, at least up to the tested 80M parameter scale. The temporal causal setting (only using past frames) ensures the model doesn’t cheat by looking ahead, and the multi-head attention learns to focus on the most relevant limb cues (in essence learning an implicit dynamics model). The combination also breaks the old agility-generalization trade-off. Previous humanoid controllers had to be relatively conservative if they were trained to handle many scenarios; the huge Transformer, however, attains both. In simulation the G1 could perform very dynamic actions (like high-leg kicks and aerial spins) while still smoothly handling new, previously unseen motions. Anecdotally, the authors note that the Transformer can handle things like dances and martial arts moves it never encountered – and it does so without losing posture or stability. In short, by treating motion tracking as a sequence completion problem (akin to language modeling) and feeding it a breadth of sequences, the model learns an implicit “world model” of humanoid dynamics. It can then generalize to produce sensible output in novel contexts. The fact that it needed no additional tuning for any test scenario is emphasized: they literally took the model trained in sim, plugged it into the hardware, and it worked on a variety of tasks. This is what one means by zero-shot generality in robotics: no per-task fine-tuning. Limitations and Considerations. Of course, the approach has some caveats. First, the model is only as good as the reference motions it sees. It tracks given target motions – it is imitating arbitrary human-like sequences, not planning them from scratch or reasoning about the world. So to make it do a task like “kick a ball” or “dance to a song,” one must still provide an appropriate motion example (perhaps from a dancer’s capture or an animator’s trajectory). The network doesn’t generate brand-new creative motions on its own, but it can mirror any example accurately. Second, the model operates purely from proprioception and target poses. It has no visual input or goal reasoning, so it won’t, say, aim a foot at a moving ball by itself – you would need a higher-level system to set the reference trajectory. Similarly, it doesn’t handle contact-rich interactions beyond what was in the dataset (in fact, object interactions were mostly filtered out of training). In the real demonstrations, the ball-throw was just a disturbance, not a deliberate “playing soccer” task. So while the controller is agile, it still lacks any explicit understanding of the environment or objects. Finally, the method required enormous compute to train. Collecting 2B frames, training 300 RL experts, and then feeding sequences through a Transformer is non-trivial. Only labs with significant resources can attempt this scale. However, the fact remains that the resulting model (80M params) is in fact deployable in real time, which suggests that future “foundation controller” efforts are feasible as hardware continues to improve. Implications for Robotics and AI. Humanoid-GPT is part of a broader trend: applying “big model, big data” thinking to robotics. In natural language and vision, we’ve seen evidence that huge pretraining yields emergent generality (GPT-4 can do arithmetic, translate, answer questions zero-shot). Here, Humanoid-GPT shows a analogous effect for motion: a single model that can handle many tasks without retraining. It suggests that as we collect more diverse robot experience data (either real or simulated), we may similarly build universal controllers. This work also blurs the line between model-based and data-driven control. The Transformer isn’t given any explicit physics or equations of motion, yet by seeing enough examples it learns to encode balancing, momentum, and kinematics implicitly. It’s a learned “world model” for humanoid dynamics. In principle, one could incorporate other modalities (vision, language) in future extensions, e.g. to transform a language command ("do a roundhouse kick") into a reference motion that this controller then executes. The authors hint that richer inputs and interactive scenarios are next steps. For robotics practitioners, this paper suggests new possibilities. Rather than handcraft or learn one skill at a time, we might eventually train an all-purpose humanoid “motion engine” offline and deploy it to any humanoid platform. Of course, transfer to other hardware would require retargeting or some adaptation. The current work was tailored to Unitree’s 29-DoF G1 robot; a different body or actuator setup would need either retraining or smart adaptation layers. But the general recipe – collect a broad motion dataset for the target body and apply large-scale training – is clear. In summary, Humanoid-GPT marks a milestone in embodied AI. It demonstrates that whole-body humanoid control can benefit from the same scaling laws as language models: more data + bigger models = better generalization. A single Transformer-based controller, trained on billions of motion frames, can mimic wildly varied human-like movements on a real robot with no per-task tuning. As the authors conclude, they have established “a new performance frontier” for motion tracking. This line of work opens the door to truly generalist robot athletes, dancers, and helpers, powered by the next generation of motion pretraining. Looking ahead, it will be exciting to see how Humanoid-GPT evolves. Integrating perception (vision) or higher-level reasoning could turn such a model into an autonomous humanoid agent that plans and executes whole-body strategies. For now, it stands as a proof-of-concept that the “GPT paradigm” can indeed extend into the physical world of joints and motors, enabling robots to exhibit an unprecedented breadth of zero-shot agility. References: The above analysis is based on Qi et al. “Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking” (CVPR 2026), augmented by summaries of their method and results and secondary sources detailing dataset construction and experiments. All figures and factual claims are derived from that work and related technical discussions.