Latest / Embodied AI 101 / LATENT: Teaching a Humanoid to Play Tennis from Imperfect Data

Transcript

LATENT: Teaching a Humanoid to Play Tennis from Imperfect Data · Embodied AI 101

• LATENT: Teaching a Humanoid to Play Tennis from Imperfect Data. Imagine a humanoid robot rallying tennis balls back and forth with a human player. This was long considered science fiction, but a new system called LATENT by researchers at Tsinghua University (and partners) has inched us closer to that reality. LATENT is a multi-stage pipeline that starts from limited human motion capture fragments and ends with a trained policy that can volley a ball with surprising success on a real humanoid (the unitree G1). The key insight is to learn a latent action space of tennis primitives from noisy, fragmentary human data, and then use high-level reinforcement learning (RL) to compose these primitives on the fly. The result: a robot that can return forehands ~90% of the time and sustain multi-shot rallies with a person. Below we unpack how LATENT works, why it matters, and what this means for embodied AI. The Challenge: Playing Sports with Robots. Playing tennis is a stiff test of perception, planning, and control. A tennis athlete must track a fast-moving ball, predict its trajectory, and coordinate an agile whole-body swing — all in real time. For a robot, that is extremely hard. Wearable motion-capture systems (like Vicon suits) struggle to track a player running the length of a court or capturing subtle wrist motions, and teleoperating a humanoid to hit a tennis ball is practically impossible. Incorporating video footage into learning is also fraught: translating 2D broadcast videos into precise 3D motions is a “complex pipeline” requiring massive vision-and-physics engineering. In short, there is no “perfect” dataset of human tennis kinematics out there. Most robotics approaches to dynamic tasks either rely on carefully scripted motions or reinforcement learning in simulation without real-world demos. NVIDIA’s recent work (2023) did learn a simulated tennis player from thousands of broadcast clips, using a hierarchical policy to chain together strokes. But that approach did not control a real robot. The LATENT system, by contrast, aims squarely at real-world execution. It starts with imperfect data and a physical robot, harnessing simulation and modern RL techniques to bridge the gap. The name LATENT itself emphasizes extracting a latent action space from limited demonstration data. Like LEGO blocks for tennis, the system uses small human motions as building blocks. In their experiments, the team collected only about five hours of motion capture data of humans performing basic tennis actions: forehand swings, backhand swings, lateral shuffles, crossover steps, and so on. They did this in a tiny 3×5 meter area (17× smaller than a real court) with a few players, not full matches. This fragmented dataset is far from a polished demonstration of actual rallies – it’s “imperfect human motion data” capturing primitive skills rather than complete point-by-point footage. In Tek Notícias’s words, tiny gestures like a forehand or a side-step become modules that the robot later stitches together in real time. Crucially, the researchers hypothesized (and proved) that even this incomplete data contains valuable priors about correct tennis techniques. The question is how to turn those priors into a robot controller. LATENT’s answer is a three-part pipeline, roughly: (1) Pretrain primitive skill policies from the human fragments, (2) Distill those skills into a latent action model (or generalist controller), and (3) Train a high-level tennis policy in simulation that chooses among and corrects these skills to hit balls reliably. We walk through each step below. Pretraining Primitive Skills. The first step is to teach the humanoid basic tennis motions from the human demonstrations. For each fragment of human motion (e.g. a forehand stroke), LATENT trains a separate “tracking” policy that reproduces that motion on the robot in simulation. This phase heavily leverages the OpenTrack framework developed by the same group. OpenTrack uses MuJoCo (with JAX) and PPO to mimic motion capture clips: given a sequence of desired poses, a policy learns to output joint torques that make the robot imitate the reference. These policies act as teacher networks for each fragment. Training is done offline, in parallel on multiple GPUs, possibly with slight domain randomization (e.g. varying friction, mass) to encourage robustness. The result of this stage is a collection of motion-tracking policies: each one is skilled at performing a particular canned tennis move on the humanoid. For instance, one policy might have learned a human forehand swing (weighted to the robot’s 23-joint body), another learns a two-handed backhand motion, another learns how to shuffle sideways while holding a racket, etc. The GitHub code calls this (and also a variant with domain randomization). By the end of pretraining, the robot has built a library of motor skills grounded in real human motion. At this point the humanoid can play back those gestures on demand – but only in isolation. Each policy is narrow: “do a forehand” or “shuffle”. The robot is not yet deciding when to shuffle, or how to modify the swing if the ball is off. To hit a real ball, skills must be sequenced and adapted. More importantly, there are too many specific clips to train a direct RL from scratch. So LATENT’s innovation is to distill this skill library into a latent action space. Distilling a Latent Action Space. Rather than using dozens of discrete skill policies, LATENT compresses them into a smooth latent representation. The idea (as tweeted by the authors) is: “We learn a latent action space from imperfect human motion data and train a high-level policy to sample from it for task accomplishment.”. In practice, this means training a policy network that takes a compact “skill vector” (a latent code) as part of its input and produces the robot action. During distillation, the multiple tracked motions are used to shape this latent space. One way to think of it: imagine an embedded manifold where nearby points correspond to similar tennis moves (a little forehand variation here, a wider swing there). By mapping many demonstrated motions into such a space, one can interpolate between them continuously. The high-level controller can then pick points in this space on the fly. This turns planning into choosing a latent code (which then decodes into a full body motion via the pretrained skill network) rather than generating torques directly. The paper mentions using DAgger-like online distillation to merge teacher policies into a single student. Essentially, the robot runs its pretrained trackers on example states, then a student policy learns to imitate all of them together. The outcome is a unified “latent-action” network (let’s call it a latent motor policy) that can reproduce the trajectories of all primitive skills. This yields a continuous action space of tennis skills. Crucially, the latent actions are anchored by the human data. Regularization can be added to keep predictions close to human-like behavior, preventing wild RL drift. In fact, the Portuguese report notes that LATENT biases exploration toward human-like motions to avoid unrealistic maneuvers. The learned latent space thus encodes the essence of forehands, backhands, footwork, etc., but abstracts away exact timing or context. High-Level Tennis Policy via RL. With a latent skill controller in hand, the system’s third stage is to learn how to play tennis in simulation. Now the robot is set loose on a court in MuJoCo and uses standard reinforcement learning (PPO optimization) to train a high-level policy. The state includes the ball’s position and velocity (and perhaps the robot’s base and joint states), and the action is the latent skill code plus maybe a corrective offset. In practice, each policy step the high-level agent might output a latent code that triggers a motion, then wait for the swing to execute, then possibly generate another latent key to return to a neutral stance or hit again. The simulation environment throws tennis balls toward the robot from various angles and speeds. The goal is to hit the ball back into a target region on the other side of the net. Rewards are simple: successful returns yield points (e.g. ball landing in bounds), and misses yield nothing. Over many simulated rallies, the RL learns which skill to use and when. For example, if a fast ball comes to the robot’s right side, the policy might select a forehand latent action; if it comes to the left, maybe a one-handed backhand latent. Crucially, the agent can also learn timing, footwork, and “shop around” for the right posture – by controlling the latent space, not by re-discovering everything from scratch. The paper emphasizes two novel design elements for this stage. First, the latent space itself is pre-trained, so the agent doesn’t have to learn low-level balance or swing mechanics anew. Second, they include mechanisms to correct and compose imperfect primitives. For instance, if the demonstrated forehand wasn’t timed perfectly for a given ball, the high-level policy can tweak it or switch to a similar latent code mid-motion. The experimental results suggest this works: the robot learned to “step out of” a prelearned shuffle if needed or to amplify a swing angle slightly to drive the ball farther. During policy training, they also applied domain randomization and other sim-to-real tricks. In simulation, physical parameters (masses, frictions, even air drag) were randomly perturbed each episode, so that the learned policy remains robust when deployed. They trained the policy using Brax/JAX, taking advantage of very fast parallel simulation to churn through thousands of tennis scenarios. (The GitHub code even includes a step to convert the Brax checkpoint to ONNX for final use.). Sim-to-Real Transfer. A key innovation is that LATENT is explicitly designed for real-world transfer. The final policy isn’t just tested in MuJoCo; it is deployed on an actual Unitree G1 humanoid. Several careful design choices make this possible:. Domain Randomization and Regularization: As noted, they randomized mass, friction, and aerodynamics during sim training. They also likely tuned the rewards to penalize extreme joint angles (maintaining “near-human” postures as in). The net effect is that the joints don’t overextend in ways that the real robot can’t replicate, and the dynamics mismatch is mitigated. Robust Tracking: Remember the initial skill experts? Those teacher trackers themselves were trained in Mujoco to very faithfully match the human reference motion. The real robot can follow those controllers remarkably well thanks to the Unitree hardware (23 motors, large range of motion) and high-rate PD control. Before deployment, the team tuned the simulated model (loco-mujoco) to closely match the G1’s actual mass distribution and kinematics. Large Training Data: Incredibly, the team spent about three weeks with a 50+ camera Vicon system to gather the training fragments. Those 50 motion-capture cameras ran at 120 Hz, covering a 19×15 m area, at a rental cost of roughly 350k RMB (≈50k USD). In other words, despite calling the data “imperfect,” it was still high-dimensional and precise. This investment ensured the simulation-to-reality gap was mainly dynamic, not kinematic. Policy Adaptation in Hardware: Finally, once the policy was loaded onto the G1, the researchers likely fine-tuned PD gains and verified the controllers using kinesthetic tests. They report that the robot, in the real world, performs a stabilizing sidestep before swings and adjusts joint torques in real time. Small delays and sensor noise inevitably appear, but the agent had learned to be fairly reactive and stable. The trajectory predictions of the ball were computed onboard (likely with a simple linear extrapolation given the ball’s flight), and the racket trajectory was preprogrammed up to a scale factor set by the RL mission, so latency was minimized. The end result was a system that robustly transfers a highly dynamic skill into reality. Results: Human-Level Volleys. The outcomes speak volumes. In simulation, LATENT’s player can hit an incoming ball accurately 96% of the time for forehand strokes. (One can see this as similar to the NVIDIA results, but here the character is a full humanoid with a multijoint arm and leg coordination.) When actually tested on the G1 robot with a human partner, the performance was nearly as impressive: about 90% return success on forehands and ~80% on backhands. In plain terms, the robot was able to keep pace with a human who was feeding it volleys without great effort, landing the ball consistently in bounds. Perhaps the most striking claim is that the G1 can sustain multi-shot rallies. Both the arXiv abstract and New Atlas report note that the robot can play “stably sustained multi-shot rallies with human players”. In videos accompanying the paper, the G1 is seen running back to serve position, adjusting its posture, and then smoothly swinging the racket to catch fast balls. Even Elon Musk took notice – an X (Twitter) post of the demo drew a “yeah” from him. Of course, it’s not Wimbledon-grade play; the ball speeds tested were on the order of 15–20 m/s (given their note that over 15 m/s was handled). However, this capability of “world’s first whole-body whole-racket humanoid tennis” has clear implications. As one commentator quipped, a $10K robot may soon be a credible tennis training partner. Summarizing the key numbers: after training, the policy on real hardware achieved nearly 90% success on forehand returns and ~80% on backhands (in practice, a few volleys with different angles). In controlled trials, performance did degrade if strains became extremely dynamic (very high-speed topspin balls remain hard), but the robot still handled a wide range of angles and speeds gracefully. It also notably preserved natural human-like motion style – the swings look fluid rather than robotically rigid, suggesting the latent priors were effective. Innovations in Context. LATENT combines ideas from imitation learning, hierarchical RL, and sim-to-real transfer in a creative way. Learning from fragmented demonstration data is not new (see e.g. latent motor primitives or skill embeddings), but the twist here is using imperfect, limited data on an athletic skill. The system is reminiscent of motor skill transfer: You could view the latent action space as a set of motor primitives akin to a tennis coach’s repertoire, from which a player chooses moves. It differs from purely model-free methods (like learning everything from scratch) by offloading low-level control to the pre-training stage. And it differs from purely data-driven imitation (which would require perfect match recordings) by allowing RL to adapt and refine. The closest prior work is probably NVIDIA’s tennis-from-video system. That work also uses hierarchical policies and motion embeddings to chain complex tennis shots. However, NVIDIA’s agents remained in perfect simulation (with idealized dynamics) and did not face real physics uncertainty or hardware constraints. LATENT’s real-world demonstrations are thus an important advance. It also required no manual stroke annotation or labeling of video shots – only raw mocap clips of generic actions. Another relevant line is “motion tracking” or “motion imitation” research (like DeepMimic and OpenAI’s humanoid projects). LATENT pushes these into the sports domain. The OpenTrack codebase showed that you can train a single policy to imitate many human motions concurrently; LATENT builds on that by composing the outcomes for a task. The use of DAgger-style distillation to merge policies also recalls recent works on generalist motion models. In the robotics community, domain randomization and sim-to-real are standard techniques, but they rarely tackle such a high-speed task. It’s instructive that LATENT randomized physical parameters (mass, friction, aerodynamics) during training, not just visual or simple parameters. This highlights the reality gap challenges in dynamic tasks: if your sim has slightly wrong mass or drag, a tennis policy will fail catastrophically. The Russian report confirms they “changed mass, friction, aerodynamics randomly so reality wouldn’t be a shock”. This seemingly mitigated sim mismatch, as evidenced by the strong real performance. Finally, it should be noted that the software and (some) data have been released. The authors open-sourced LATENT (code and a subset of motion clips) on GitHub. This is valuable for embodied AI: we now have a benchmark that sits between purely simulated locomotion tasks and full industry hardware work. In fact, the ease of data collection (just 5 h from a few people in a small lab) is purposely advertised: it suggests other labs could replicate or extend this with modest effort. Conclusion. In summary, LATENT demonstrates that realistic, high-speed whole-body behaviors can be learned from sparse demo data when combined with clever policy design. The system’s three stages – motion imitation pretraining, latent skill distillation, and high-level RL – form a cohesive pipeline. It shows that even “imperfect” fragments of human tennis can guide a robot to athletic skill. This is perhaps the closest we’ve seen to a humanoid engaging in a dynamic physical sport alongside a human. Beyond tennis, the take-home message is broad: for any complex task where collecting full demonstrations is hard or costly, LATENT’s approach suggests a way forward. You gather rough chunks of behavior, train low-level controllers, combine them into a latent repertoire, and then let RL polish the strategy in simulation. As the authors note, success in tennis“implies the same principle could apply to other demanding physical activities”. We might soon see humanoids learning other sports or even industrial skills the same way. For now, LATENT stands as a milestone in embodied AI – a $10K robot playing tennis with a human, guided by hours of messy human data and smart learning. It’s not Rod Laver yet, but it’s a convincing prologue. Citations: The description above is based on (and cites) the LATENT preprint and its related public materials. The open-source implementation and project page also contain details of the pipeline. The system leverages prior work on motion tracking and hierarchical policies (e.g. NVIDIA’s tennis player) but uniquely achieves real-world humanoid tennis.