Latest / Embodied AI 101 / LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data
Transcript
- LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data. Building human-level tennis skills on robots has been a long-standing challenge in embodied AI and robotics. Unlike static manipulation or simple locomotion, tennis requires highly dynamic full-body motions: players sprint across courts (often exceeding 6 m/s), react to balls flying at 15–30 m/s, and make precise contacts within a few milliseconds. Capturing or teleoperating their motions for a humanoid is essentially impossible. Even passive motion-capture is hugely difficult: tennis motions span a half-court area, involve long multi-shot rallies, and demand very fine wrist action when striking the ball. In practice, there is no cheap dataset of continuous human tennis–motion on which to train a robot by imitation. The LATENT system (by Zhang et al., CoRL 2024) confronts this data scarcity head-on. Instead of trying to record perfect, end-to-end tennis plays, they collect only motion fragments of primitive tennis skills – like a forehand swing here, a lateral shuffle there – from amateur players. This radically lowers data collection cost: in their setup, a small 3×5 m capture area suffices to record 5 hours of forehand/backhand strokes, shuffles, crossover steps, etc. These clips are “imperfect” in two ways: the racket swings are not very precise (especially the wrist joints are noisy), and the clips are unanchored — they contain no ball, serve, or complete rally context. But crucially, they do encode how humans naturally move their legs and torso during tennis strokes. The insight motivation of LATENT is that even imperfect, fragmentary data provides useful priors about feasible athletic motion. Given these primed motions, LATENT then proceeds in a three-stage pipeline (Figure 2 in the paper) to build a working tennis policy for a simulated humanoid, and finally transfer it to reality. In Stage 1, primitive tennis moves are collected from humans and retargeted to the robot. In Stage 2, those fragments are distilled into a latent action space that smoothly encodes natural tennis skills. And in Stage 3, a high-level policy is trained (by reinforcement learning) to pick and correct motions from that latent space to actually return balls to targets. Throughout, special design tweaks ensure the policy stays stable and transfer-ready. The end result is a Unitree G1 humanoid that can volley balls back and forth with a human player, tracking fast incoming shots and sending them to precise landing zones while looking surprisingly natural. Below we unpack each part of this pipeline, explain the technical cruxes, and summarize the results. We also compare to prior work in latent‐action humanoid control and other robot sports, as well as highlight the impressive sim-to-real success of LATENT. The key takeaway is that highly dynamic, whole-body skills can indeed be learned from cheap, partial data, provided one carefully structures the representation and learning. 1. The Data Challenge: Primitive Motion Fragments. The core obstacle is data. We would love to imitate human tennis players, but recording them on a robot is nearly impossible. Chess moves are simple by comparison, but a robot tennis player would need to learn footwork, balance, swings, and timing. Direct motion capture of real tennis matches (say from broadcast video or special setup) is daunting: “large-scale human movements that span the half court, sustained multi-rally interactions, and subtle and highly precise wrist motions during ball striking… make it extremely difficult for motion capture systems to collect precise and complete human-tennis motion data”. To sidestep this, the LATENT team took a more humble approach. They invited five amateur players to perform basic tennis actions (forehand and backhand groundstrokes, lateral shuffles, crossover steps, etc.) inside a small lab. The motion capture area was only 3 m by 5 m – a tiny subset of a tennis court – and no actual balls or volleys were used. This meant they captured motion fragments: short clips of a single stroke or footwork pattern. In total they recorded about 5 hours of such fragmentary data. Crucially, they did no sophisticated filtering or editing; the data remains “imperfect” (for example the wrist motion is not professionally precise). However, these clips do cover the essential ingredients of how humans swing rackets and move their bodies in tennis. As one summary puts it, this data “captures the primitive skills used when playing tennis rather than precise and complete human-tennis motion sequences, [thus] significantly reducing the difficulty of data collection”. Why even bother with imperfect data? It turns out that even an imprecise forehand stroke carries a lot of useful information. The Agent Times commentary explains the intuition: “imperfect human motion fragments still encode useful priors about how humans move in athletic contexts”. In other words, how the hips rotate, how the weight shifts, how the legs brace – those patterns don’t require perfect tennis-to-human reenactment. By feeding these as hints, the robot can learn to generate natural movements and then adapt them to the actual game physics. In practice, the LATENT system took these human clips and retargeted them onto the humanoid. They used the Locomujoco framework to map human joint trajectories to the 29-DoF Unitree G1 body. Because the raw data and mappings are noisy (especially the wrist), the result is not immediately robot-ready. That is why Stage 2 is needed: to turn these imprecise, possibly infeasible actions into a structured latent space that the robot can execute reliably. 2. Building a Latent Representation of Tennis Skills. With imperfect motion data in hand, the next step is to learn a latent action space that captures the repertoire of primitive skills. The idea is to train a continuous embedding (via a neural network) where each latent vector decodes into a full-body robot action. This is a common trick in humanoid RL – pre-learned action primitives make learning new tasks much easier than letting the policy output every joint torque directly. But here the novelty is that the primitives themselves come from noisy, incomplete human data, so the latent space must be deliberately made correctable. LATENT proceeds in two sub-steps. First, it trains a motion-tracking policy to imitate the raw clips on the robot. Think of this as a teacher: for each timestamp of a captured motion, it gives the robot a target pose (via state $\tilde{s}{t+1}$) and asks the policy $\pi\mathrm{tracker}$ to produce torques that reach that pose. This is done with a traditional tracking framework (Any2Track) using reinforcement learning. The tracker takes as input the robot’s current state $s_t$ (joint positions, velocities, etc.) and the next desired pose $\tilde{s}_{t+1}$ from the human data, and outputs joint torques. By optimizing these to minimize pose error, the tracker learns to physically realize the captured motions. However, one twist is that the collected tennis data had unreliable wrist swings, so they omit the right-wrist actions during tracker training. Concretely, they remove the control commands for the racket-holding wrist from the tracker’s action space and add random perturbations there. This forces the tracker to focus on getting the body and arm trajectories right while ignoring the exact wrist angle, treating it as an unknown disturbance. As a result, the tracker learns to faithfully reproduce all joints except the racket wrist, essentially leaving the wrist degree of freedom open for later correction. The tracker also sees random noise on the wrist so it learns to stay balanced even if the wrist is off. In the paper’s words, the right wrist motions were “highly inaccurate and cannot be directly utilized”, so this design makes the lower-body controller robust to arbitrary future corrections. Once the tracker policy is trained, it can roll out all the primitive skills on the robot. But instead of freezing that policy, LATENT next distills it into a latent-action model (a “student network”) with an information bottleneck. They use an encoder–decoder pair as follows: the encoder takes the current state $s_t$ and the reference next pose $\tilde{s}{t+1}$ and outputs a distribution $q(z_t|s_t,\tilde{s}{t+1})$ (a Gaussian with mean $\mu_e$ and std $\sigma_e$). The decoder then takes a sampled latent $z_t$ and state $s_t$ and produces a body action $a^\text{body}t=D(s_t,z_t)$. In training, they do DAgger-style online distillation: the teacher tracker policy is rolled out to generate ($s_t,\hat a_t$) pairs, and the student is trained to imitate those outputs. The loss is $\mathcal{L}\text{action} = | \hat a_t - a_t|^2$ plus a KL term to keep $z$ distribution close to a (learnable) prior. This ensures the latent space is compact (via the VAE bottleneck) but sufficient to reconstruct the teacher’s actions. The upshot is a continuous latent skill space: each point $z$ in this space decodes to a joint action pattern that was seen in the demonstrations. Crucially, because of the removal of wrist signals earlier, this latent space is constructed to be compatible with later wrist corrections. In other words, the decoder never learned to depend on the exact raw wrist velocity, so one can safely tweak the real wrist without breaking the gait. As the paper notes, this lets LATENT “bridge the gap” between precise tennis swings and imprecise data. Importantly, the latent policy is conditional on the robot’s state: the encoder $E(s_t,\tilde s_{t+1})$ and prior distribution are functions of $s_t$. One can think of this as a state-dependent embedding of all feasible short-horizon moves (like a conditional action prior). In the end, Stage 2 yields two things: a trained latent decoder $D$ and a conditional prior distribution $(\mu_p(s_t),\Sigma_p(s_t))$. Sampling $z\sim\mathcal{N}(\mu_p,\Sigma_p)$ and decoding it gives a natural-looking action to take from state $s_t$. This forms the basis for the next stage of learning. 3. High-Level Policy Learning with Corrections and Constraints. With the latent action space prepared, Stage 3 is to actually learn which primitives to use and when, so that the robot plays tennis. This is done via reinforcement learning in simulation using PPO. Each episode is a sequence of eight balls tossed toward the robot at 2-second intervals, from random positions and velocities (see Figure 3 of the paper). The robot starts in a neutral ready pose. Its task is to return each ball to a designated target location on the opponent’s side. Returning means the hit ball must land within some tolerance of the target, and certainly within the court boundaries. Rather than output raw torques, the high-level controller now outputs in the latent space. At each decision timestep (50 Hz), the policy network observes the robot’s full state and the ball’s state, and outputs a vector $(a^\text{latent}_t, a^\text{corr}_t)$. The vector $a^\text{latent}_t$ is interpreted as a proposed offset in latent space, while $a^\text{corr}_t$ are direct correction torques for the right-wrist joints of the robot. Concretely, LATENT’s policy interprets $a^\text{latent}_t$ via a special latent action barrier (LAB) mechanism. The policy network actually predicts an unconstrained $u$, and then forms the final latent code as $$z = \mu_p(s_t) + \lambda,\sigma_p(s_t),\tanh(a^\text{latent}_t),,$$ where $\mu_p,\sigma_p$ are the prior mean and std from the latent model at the current state. Here $\lambda$ is a scalar hyperparameter setting the allowable range. Using $\sigma_p$ and $\tanh$ ensures that $z$ stays within a “radius” (a Mahalanobis distance bound) around the prior mean. In effect, the policy can only stray so far from the demonstrated distribution – it cannot pick an arbitrarily wild latent code that decodes to untrained behavior. This is the latent action barrier. It serves as a trust-region in latent space, keeping exploration close to the manifold of human-like movements. Meanwhile, $a^\text{corr}_t$ is a low-dimensional vector of direct torques on the wrist joints (the racket arm). Adding this injects non-imitative corrections: since the human data had imprecise wrist swings, the high-level policy must learn to adjust the wrist to actually hit fast balls. By handling the wrists separately, the latent embedding (which was trained without wrist information) remains stable. As the authors explain, “the high-level policy can output arbitrary wrist motion adjustments without compromising the stability and agility of the lower-body movements”. In practice, the final action sent to the robot at time $t$ is computed by decoding $z$ to get most joints, and then applying the correction torques to the right arm joints. The reinforcement learning reward is carefully crafted. Important terms include: (1) an approach bonus for moving close to the ball (so the robot learns to run toward serves); (2) a large hit success reward for actually making a valid return; (3) a landing distance reward for sending the ball near the target. On the other hand, many regularizers penalize unrealistic behavior: high torques, jerky motions, self-collisions, leaving bounds, etc (see Table 1 in the paper). For instance, falling or missing the ball incurs a heavy -200 penalty to terminate the episode early. All together, the reward pushes the policy to find a trade-off between “winning” (getting balls back successfully) and “natural style” (smooth joint movements and respecting safety limits). The high-level RL then picks latent codes that represent running strides and swing movements, while also fine-tuning the wrist to meet the trajectory of each ball. This hierarchical formulation helps immensely: the policy does not need to discover raw low-level gaits from scratch (thanks to the latent embedding), but can instead focus on sequencing and adjusting known motor primitives. 4. Sim-to-Real Transfer: Handling the Reality Gap. Training in simulation is only half the battle. The LATENT team knew that throwing a racket and ball at a robot in the real world faces huge uncertainties: friction, mass, aerodynamics, sensor noise, etc. They therefore incorporated thoughtful randomization and noise modeling to close the sim-to-real gap. First, dynamics randomization was applied to virtually every physical parameter that could vary. Table 2 of the paper lists how they perturb both the robot and ball physics. For the robot, they randomize foot friction (uniform in [1.0,2.5]), joint damping, motor armature constants, total body mass, and center-of-mass offsets. They also slightly randomize the racket’s mass and COM. For the tennis ball, they randomize its mass (±5%), ground restitution (bounce) [0.71,0.79], tangential damping [0.4,0.7], and especially the air-drag coefficient [0.01,0.04]. This last one models aerodynamic uncertainty (spin and wind have complex effects) by varying drag in simulation. The goal of all this is to ensure the learned policy is not brittle. Indeed, the authors remark that with “sufficiently varied domain randomization in dynamics, we obtain effective sim-to-real transfer and greatly improve robustness” without even modeling all spin forces explicitly. In practice, during training each episode is drawn from a fresh randomization of these parameters, so the policy must succeed under many plausible physical conditions. Second, they injected observation noise and latency to simulate real sensing. The robot’s proprioceptive sensors (joint angles, velocities, etc.) get small random noise. Crucially, the ball’s state is made noisy and sometimes dropped: since in reality the ball is tracked by motion capture or camera, its readings have jitter. To mitigate that, LATENT adds uniform noise and occasional dropouts in the ball’s observed position. They also include a short latency. Observing the high ball speeds, they note that directly differencing noisy position estimates can yield huge velocity errors. So instead they keep a four-step sliding buffer of recent ball positions and use the averaged velocity for decision-making. This clever smoothing allows the policy to see roughly the true ball speed despite raw measurement noise. Finally, for the real setup they equipped the physical Unitree with a real racket and used a motion-capture system to track the ball and robot base. The MAVIG commercial mocap tags on the ball and robot gave global state with sub-centimeter accuracy. Thus at runtime the policy still “thinks” it is in simulation, getting as input a state vector of robot pose and ball trajectory – but now that state includes real-world imperfections that it trained for. Taken together, these sim-to-real measures paid off. When deployed, the policy – unmodified from training – was able to cope with variations in footing and ball physics that the sim didn’t know exactly. In fact, the team emphasizes that they could “deploy our policy on the Unitree G1 humanoid and stably sustain multi-shot rallies with human players”. In one test they even had human opponents softly volley balls against the robot, and it responded correctly with both forehand and backhand. 5. Results: Simulation Benchmarks. Before talking about the hardware deployment, let’s look at the simulated performance. LATENT’s policy is evaluated on a “tennis return” task in MuJoCo: each episode has a randomized ball, and success means returning it. They compare against several baselines:. PPO from scratch (no motion data) and MotionVAE (a kinematic VAE version from Vid2Player3D) – both fail entirely (N/A entries in their table). AMP (Adversarial Motion Prior, a known RL-with-motion-prior method) achieves only ~40–50% success. ASE (Adversarial Skill Embeddings) gets ~55–70%. PULSE (an online-distilled latent-space approach similar in spirit) gets ~63–74% success depending on stroke/court. In contrast, LATENT hits 96.5% success on forehands and 82.1% on backhands when averaged across targets, which is dramatically higher. (For reference, ‘success’ means the robot returns the ball within 2.5 m of the goal.) The distance error to target (DE) is also much smaller (around 1.3–1.9 m versus 3–5 m for others). Importantly, LATENT produces smoother and lower-torque motions (smoothness ~25 vs 30–37 for PULSE/ASE, and torque ~7.4 vs ~10). Lower smoothness/tortuque numbers are better here – meaning the robot moves more stably and lightly. This indicates that our constrained latent exploration pays off: the robot not only succeeds more often, but it does so with more natural, safe movements. An abridged summary of these numbers (Table 3 in the paper) is:. | Method | Forehand SR↑ / DE↓ | Backhand SR / DE | Forecourt SR/DE | Backcourt SR/DE | |:-------|:------------------:|:----------------:|:---------------:|:---------------:| | AMP | 41% / 4.71m | 37% / 5.22m | 49% / 4.20m | 41% / 4.85m | | ASE | 63% / 3.62m | 55% / 4.01m | 71% / 3.28m | 68% / 3.90m | | PULSE | 72% / 3.01m | 63% / 3.15m | 74% / 2.97m | 73% / 3.02m | | LATENT (ours) | 96.5% / 1.32m | 82.1% / 1.89m | 86.4% / 1.73m | 89.8% / 1.55m |. Compared to PULSE (prior latent-AE method), LATENT’s success is roughly +25% higher. The distance errors are cut by more than half. These gains show that the additional “corrections + latent barrier” designs make the skill space truly usable for a real task. Indeed, ablating those components hurts performance. If they remove the wrist correction outputs entirely, success rates drop to ~82% forehand and 68% backhand. And if they remove the Latent Action Barrier (LAB), the policy still manages high success (~93% forehand) but at the cost of much higher acceleration (smoothness 37 versus 25) – i.e., it resorts to jittery exploits of the latent space. This confirms the authors’ claim that LAB is needed to keep behavior natural. In summary, every piece of the pipeline ablation clearly degrades results, indicating they all contribute. 6. Results: Real-World Tennis on Unitree G1. Perhaps the most striking achievement is that LATENT’s policy, trained entirely in simulation, actually works on a real humanoid. After outfitting the G1 robot with a tennis racket and motion-capture markers, they ran real-ball tests. In one evaluation, they performed 20 human-robot rally matches (one ball at a time). Each trial throws a ball (random position/velocity) to the robot, and we measure whether it returns it into the opponent’s court. Because pinpoint accuracy is harder on hardware, they consider a shot successful if any valid return lands in bounds (within the far half-court). They record how often the robot succeeded on forehands vs backhands, in forecourt and backcourt regions. The results (Table 5) were impressive, but also illuminate the importance of their sim-to-real measures. The full LATENT system (with all randomizations) achieved about 90.9% success on forehands and 77.8% on backhands in the forecourt, and similarly high 88.9%/81.8% in the backcourt. In practical terms, the robot rarely missed. Its average distance error to target was around 3–3.9 m, which is larger than in sim (unsurprisingly), but still kept most returns in play. The ablations tell the story: without ball dynamics randomization, success plummeted to ~17% on forehands (0% on backhands!). Without observation noise modeling, performance was also much worse (50% forehand, 0% backhand). In other words, if the system was trained on a “clean” sim, the real ball behaved different enough that the policy failed. Once both randomization and noise were included, it began working well again. The claims in the paper are that “adequately randomizing ball dynamics and observation can enable effective sim-to-real transfer” – and the numbers verify it. Video evidence (linked on the project page) shows the G1 robot running for balls, switching between forehand/backhand swings, and sustaining multi-shot exchanges. It even managed to chase balls near the net and smash them back. In light of other robotic sports achievements – e.g. simulated humanoid badminton (21 consecutive hits in sim, 10 m/s shuttle on hardware) or table-tennis bots (over 100 hits) – LATENT represents a major step. This is the first time we’ve seen a bipedal robot keep up with real-timed tennis volleys using purely learned, vision-free control. The fact that it runs on a standard lab rig with motion capture (no special ball launcher target tracking, though) means it’s reproducible if others have a similar setup. The broader implication is: one does not necessarily need perfect demonstration data. As the Agent Times article summarizes, the key lesson is that “imperfect, fragmentary human motion data – the kind most readily available in the real world – can be corrected and composed into capable whole-body policies.”. If this approach generalizes, it suggests athletic robot skills can be bootstrapped from everyday amateur videos or simple recordings, rather than requiring expensive motion-capture suits on elite athletes. 7. Relation to Prior Work and Future Directions. LATENT builds on and extends the emerging literature of latent-action humanoid control. Past works (like PULSE and others) have also distilled human motions into VAEs for reuse, but usually on smoother tasks (walking, dancing, some manipulation). Sports tasks add new wrinkles of speed and precision. Other systems — e.g. Vid2Player3D from Stanford/NVIDIA — have focused on simulated tennis from broadcast footage, but did not tackle a real robot. In adjacent racket sports, Legged Robotics and others demonstrated RL-based badminton and table tennis, but often with simpler kinematics (e.g. fixed-paddle robots or no full legs). LATENT’s novelty is combining highly-dynamic whole-body control with very limited human data. For the roboticist, the take-away is that a hierarchical RL pipeline with a correctable latent space can handle severe data imperfections. The design lessons — remove low-quality signals during training, allow high-level corrections, and constrain deviation via a latent barrier — might apply to other areas (soccer goal shooting, pistol aiming, even humanoid parkour where people only give small snippets of motions). The authors acknowledge limitations and future work. Their real robot still relies on motion capture to track the ball; replacing that with onboard vision would be a next step. Also, their scenario is still singles return drills with static targets, not a true live opponent match. Extending training to a full game (possibly multi-agent RL) is an open challenge. But even at the current, first-attempt level, LATENT shows a promising path: the data bottleneck can be loosened. 8. Conclusion. In summary, LATENT is the first system to make a humanoid robot play dynamic tennis volleys with any success using only partial human data. By cleverly converting low-quality motion clips into a robust latent skill space, and then learning a policy that corrects and sequences those skills, it achieves human-level rally performance. The impressive sim-to-real transfer (90% hit rate on hardware) underscores that smart learning and randomization can overcome the reality gap. As the project page emphasizes, this approach “paves the way for future research” in athletic humanoid control. For practitioners, the lesson is clear: even “messy” data can serve as a springboard if the training pipeline compensates properly. The era when every new robot skill required pristine motion-capture demos might be ending. With algorithms like LATENT, we may soon train robots for many sports or physical tasks using the kinds of motion footage that are easy to gather. And that could dramatically accelerate development of physically capable AI agents across industries. Key References: Zhang et al., Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data (LATENT), CoRL 2024. (See also the project page and videos，and related work on humanoid sports control.).