Latest / Embodied AI 101 / Joga: Why a Soccer Humanoid Needs to Control Its Gaze
Transcript
- Joga: Why a Soccer Humanoid Needs to Control Its Gaze. Joga: Full-Stack Humanoid Soccer with Active Onboard Vision • Nolan Fey and collaborators • MIT CSAIL / Improbable AI Lab affiliation verified for Fey • Public research announcement; paper described as forthcoming • 2026. The interesting part is everything between the kicks. Imagine a humanoid receiving a rolling pass. Before making contact, it needs to estimate where the ball is going, move into a useful stance, and keep watching while its own body changes position. After contact, it must determine whether the ball actually slowed down, decide how to approach the next touch, and remain balanced throughout. Now ask it to turn sharply, maintain possession, and shoot. The interesting capability is not a collection of impressive poses. It is the continuous coordination of observation, locomotion, and contact. That is the attraction of Joga. The supplied description identifies a Unitree G1 platform, while Nolan Fey’s announcement reports dribbling, sharp directional changes, and linked receiving, dribbling, and shooting or passing. It identifies two technical ingredients: a motorized neck for visual ball tracking, and residual models addressing perception and actuation discrepancies between simulation and hardware. There is an important evidence boundary. As of September 24, 2026, I could verify the announcement but could not locate a public Joga manuscript or implementation. The linked Chris Paxton post was not directly retrievable; Fey’s announcement was recoverable through an indexed mirror and explicitly described the paper as forthcoming. Consequently, this episode is a research preview—not a reconstruction of an unpublished methods section. Architecture details, training budgets, quantitative results, and ablations remain unverified. We can nevertheless examine the systems problem in depth, connect it to documented research, and identify what would make the eventual technical contribution compelling. One distinction should remain clear throughout: onboard sensing, onboard computation, and autonomous decision-making are different properties. A robot can obtain images from its own camera while another computer processes them. It can execute a sophisticated learned controller while a person selects the desired direction or initiates a shot. A useful technical report should identify those boundaries explicitly. My reading of the announcement is that Joga’s most interesting promise is closing the entire perception–action loop during athletic object interaction. That is a more precise—and more defensible—interpretation than treating a soccer demonstration as evidence of general-purpose humanoid intelligence. Soccer exposes two different kinds of uncertainty. For analysis, consider a minimal soccer controller. It needs some representation of the robot, the ball, and the intended outcome. That representation might be explicit state estimates or distributed features inside a recurrent policy. Either way, the information has to support the same physical decisions: where to step, when to touch the ball, and how much momentum to impart. The difficulty is that the robot changes the thing it is trying to estimate. Before a kick, a short history of ball positions might support a reasonable velocity estimate. At contact, that history becomes partly obsolete. The controller must recognize a discontinuity and update quickly. Meanwhile, camera motion changes the image even when the ball’s world-frame motion remains simple. Here is an illustrative calculation, not a Joga measurement. A ball moving at three meters per second travels eighteen centimeters in sixty milliseconds. If a controller acts on an uncompensated observation that old, even a geometrically perfect detection describes a substantially different contact situation. This makes observation age a control variable in everything but name. Image frequency alone is insufficient: the useful question is how old the information is when an action takes effect. Now add balance. Suppose the robot wants to touch the ball with its left foot. That foot may also be needed for the next stabilizing step. Moving it sooner could improve interception and worsen support. Waiting could preserve balance and lose possession. A dribbling controller must resolve that conflict repeatedly rather than once. DribbleBot previously demonstrated this coupling on a quadruped: controlling the ball required coordinated locomotion and manipulation under onboard visual sensing, with terrain-dependent ball dynamics included in the problem. Soccer therefore provides a concrete setting for studying more than velocity-command tracking. For Joga, the useful conceptual split is between observation error and transition error. Observation error means the controller has the wrong picture of the current situation: perhaps the ball appears farther away, arrives late in the estimate, or disappears during a turn. Transition error means the controller’s prediction of what an action will do is wrong: a commanded motion produces a different response, or contact transfers an unexpected impulse. Those errors can compound. An apparently late interception might result from stale vision, sluggish actuation, or both. A policy can even compensate for one error using another, producing acceptable behavior until a component is recalibrated. That is why I would resist evaluating perception and actuation only in isolation. Accurate images do not guarantee useful contact timing. Accurate free-space joint tracking does not guarantee a successful touch. The relevant object of evaluation is the closed-loop interaction. An actuated neck changes the sensing problem. Active perception is not a new concept. Ruzena Bajcsy’s foundational formulation treated information acquisition as something an agent controls, including models of sensors and their processing, rather than as a passive stream delivered to a decision-maker. The important connection here is that moving a sensor can change which decisions are possible. To see the design implications, compare three hypothetical arrangements. With a fixed camera, the robot may need to rotate its torso to keep a nearby ball visible. That makes looking and moving partly inseparable. With a sufficiently wide camera, some of that coupling can be reduced, but the system must still satisfy its resolution, geometry, and processing requirements. With an actuated camera, gaze becomes another controllable degree of freedom. The third arrangement creates an opportunity: turn the body toward a useful next stance while continuing to observe the ball. It also creates a new coordination problem. The controller must decide how to distribute motion between the camera and the rest of the robot. A head that tracks successfully while the body moves into an unrecoverable configuration has not solved the task. There are several ways to implement the gaze controller. Classical visual servoing can regulate an image-space feature, such as the ball’s location in the image, or operate on an estimated three-dimensional target position. Chaumette and Hutchinson’s visual-servoing tutorial distinguishes these image-based and position-based formulations. Neither requires that gaze itself be learned through reinforcement learning. A different design would include neck commands in a learned action space. Another would let a task policy choose a gaze target while a conventional controller tracks it. The available Joga announcement does not establish which formulation it uses. That distinction matters scientifically. A successful motorized tracker establishes the value of controllable sensing. A learned gaze policy could additionally establish that the robot discovers task-dependent sensing strategies. Those are related, but different, contributions. For a concrete implementation thought experiment, consider camera geometry. If the head moves between exposure and inference, the current neck angle is not necessarily the correct angle for interpreting the image. Transforming an estimated ball location into a body-relative frame requires a camera pose consistent with when the observation was acquired. Camera motion and object motion must not be conflated. This follows directly from the relationship between image features and camera motion underlying visual servo control. I would therefore treat synchronization as part of the perception model, not as middleware housekeeping. The design should make it possible to reconstruct which image, neck configuration, and body estimate produced each control input. There is also a strategic question: should the robot always look at the ball?. For a narrowly defined tracking task, perhaps. For soccer, not necessarily. Imagine a robot that maintains excellent ball visibility but never updates the goal direction. Or one that watches the ball during every touch but cannot detect an approaching obstacle. Perfect fixation can still be poor sensing behavior. A useful future experiment would compare ball-centering gaze with gaze chosen for downstream task success. Another would test what happens when the neck reaches its motion limit: does the body reposition to restore visibility, or does the estimator simply lose its target?. The broader lesson is not that every humanoid needs a particular neck mechanism. It is that sensing geometry belongs inside the task design. The robot should move not only to create useful contacts, but also to preserve the information needed for the next contact. Residual models: identify what is being corrected. “Residual” describes a relationship to a baseline, not a complete algorithm. A learned correction might modify an action, adjust a state estimate, perturb a simulated observation, or change a simulator’s response. Those choices place learning at different points in the loop. Residual policy learning, for example, can combine a conventional controller with a learned action correction. Johannink and colleagues’ Residual Reinforcement Learning for Robot Control is a documented example of that approach. It should not be confused with learning a model correction that is used only during simulation-based training. For interpreting Joga, I would separate three questions: what quantity the residual predicts, what data supervises it, and where it runs. Correcting simulated actuation. A particularly relevant precedent is Fey and colleagues’ Bridging the Sim-to-Real Gap for Athletic Loco-Manipulation, published at RSS 2025. Its Unsupervised Actuator Net, or UAN, learns corrective simulated torques from histories of joint-position and velocity errors. Training compares simulated motion with recorded hardware trajectories without requiring explicit output-torque labels. Task policies are subsequently trained using the corrected simulator. The useful idea is selective learning. Rather than replace rigid-body simulation wholesale, learn a correction in the part of the model that is failing to reproduce hardware. That paper also connects model fidelity to task optimization. A policy trained to maximize an athletic outcome can exploit incorrect simulated dynamics. Improving the actuator model helps make aggressive task optimization meaningful on the physical machine rather than merely successful inside the simulator. Contact-UAN, a subsequent project by Fey, Khai Nguyen, and collaborators, addresses the additional difficulty of learning such corrections for a floating-base humanoid in contact. It first establishes contact-consistent replay using proprioceptive estimates, then learns an actuator-command residual. Its project page reports learning from 6.5 minutes of walking data and using the frozen correction during downstream policy training, with only the task policy deployed on hardware. The contact-consistency point is important: if a replay starts with an inappropriate support interaction, a learned correction may compensate for initialization error rather than actuator mismatch. Contact-UAN explicitly targets that confound. These are documented details of Contact-UAN, not verified implementation details of Joga. For anyone adapting this research direction, that suggests a diagnostic principle: before asking a neural network to explain trajectory disagreement, check whether the two trajectories begin from physically comparable conditions. Correcting perception is a different problem. The phrase “perception residual” leaves more possibilities open. One option would correct a deployed estimate. Start with a geometric or learned estimate of ball position, then predict a context-dependent adjustment. Another option would correct the training observation model. Start with simulated ground truth, then learn how real observations differ from it so that the policy trains on more realistic measurements. These operate in opposite directions. One tries to improve the robot’s interpretation of hardware data. The other tries to make simulated inputs resemble hardware data. They also require different validation procedures. The announcement does not resolve that ambiguity, so it would be premature to attribute either design to Joga. As an engineering proposal, I would begin by asking what error remains after ordinary calibration and synchronization. Suppose ball range is biased near the image boundary but reasonable near the center. That is a candidate for a conditional correction. Suppose errors occur in bursts immediately after rapid turns. That suggests testing motion-dependent and temporal explanations rather than fitting a static offset. A useful residual should also have a defined failure behavior. When the ball is not visible, does the system predict forward, mark the estimate as uncertain, or provide a special missing-observation state? Returning an apparently precise location in every frame can conceal exactly the uncertainty the controller needs to respect. The two residuals must remain identifiable. Consider a hypothetical debugging trap. The vision system consistently reports the ball too far away. A learned controller responds by taking an extra step. Meanwhile, an actuator discrepancy makes that step shorter than expected. The combined system appears well calibrated. Now fix the actuator model. The robot overshoots, even though one component became more accurate. This is why I would collect separate evidence for sensing and dynamics corrections before judging their combined task performance. A successful rollout is necessary evidence, but it does not uniquely identify what each model learned. The same reasoning argues for held-out trajectories and operating conditions. A residual that explains one repeated motion may be compensating for a task-specific artifact. A correction that remains useful across distinct motions is stronger evidence of a reusable model. The transferable principle is not “add another neural network.” It is: preserve useful structure, isolate a consequential mismatch, and measure whether the correction generalizes beyond the data that made it look good. Receiving, dribbling, and shooting must agree on boundary states. The appeal of a linked soccer sequence is that each behavior creates the initial conditions for the next one. Receiving is not merely touching an incoming ball. In a useful sequence, the result should leave the ball within a region from which the robot can continue acting. A technically successful interception that sends the ball behind the robot may be a poor setup for dribbling. Likewise, dribbling should do more than preserve proximity. If the eventual objective is a shot, the last few touches need to produce a useful approach, support configuration, and ball position. Shooting then has two outcomes: what happens to the ball and what happens to the robot. A strong strike followed by an unrecoverable fall is not interchangeable with a strike from which play can continue. This is an analysis of the task, not a claim about Joga’s reward functions or skill architecture. The public material does not reveal whether its behaviors are produced by separate policies, a state machine, a command-conditioned controller, or another arrangement. For a prospective implementation, I would focus first on the interfaces between behaviors. What conditions indicate that receiving has finished? What state does the dribbling controller expect? Is the shooting behavior trained on states actually produced by dribbling, or on conveniently reset states?. Those questions often matter more than whether the software is described as modular or end-to-end. A simple calculation illustrates why composition deserves its own metric. Under an optimistic independent-stage model, three skills that each succeed 95 percent of the time produce a complete-sequence success rate of about 86 percent. That is before introducing additional distribution shift at their interfaces. This is only a toy example, but it shows why individual skill scores cannot substitute for sequence evaluation. My preferred test would begin with a distribution of incoming passes and end only after the shot and recovery. Intermediate success conditions would help diagnose failures, but the headline result would cover the entire episode. I would also inspect unsuccessful transitions. Did the ball leave the reachable region? Did the policy switch before the state estimate settled? Did a change of command produce a discontinuity in body motion?. The strongest evidence of integration is not an uninterrupted successful sequence alone. It is a system that tolerates imperfect intermediate outcomes without requiring the experimenter to restore a favorable state. Where Joga sits relative to documented work. Three nearby research directions help prevent both underestimating and overstating the announcement. First, DribbleBot, by Yandong Ji, Gabriel Margolis, and Pulkit Agrawal, established an important precedent in 2023. It combined onboard sensing and computation with quadrupedal ball manipulation across natural surfaces, including sand, gravel, mud, and snow. Its demonstrations also included recovery after falling. The official project provides a code link. The relevant comparison is therefore not simply “does the robot dribble using its own camera?” That capability has precedent. The question for a humanoid system is how its sensing and contact-control design handles the different embodiment and the desired sequence of skills. Second, Dhruva Tirumala and colleagues’ Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning, released in 2024, directly addresses onboard visual humanoid soccer. The work maps egocentric RGB observations to joint-level actions with onboard computation and sensing. It combines physics simulation with NeRF-based realistic rendering and teacher-based multi-agent training. Crucially, that work also studies active head movement. In a probe that fixes the other joints, the learned policy moves head pan and tilt to keep a moving ball visible. The authors report that tracking behavior emerges without a dedicated ball-localization reward and observe corresponding behavior on hardware. Joga should therefore not be presented as the first demonstration of active visual humanoid soccer. A meaningful comparison would instead examine embodiment, task coverage, training assumptions, robustness, and the specific benefit of residual calibration. Without Joga’s technical report, those remain comparison axes rather than an established ranking. Third, Jipeng Kong and colleagues’ Learning Soccer Skills for Humanoid Robots: A Progressive Perception-Action Framework, released in February 2026, studies soccer skills on the G1. Its PAiD framework separates human-motion-based skill acquisition, lightweight perception–action integration, and physics-aware transfer. The paper describes ball and goal positions as policy inputs, with deployed ball localization obtained through fused visual and radar-based estimation. That makes PAiD a useful comparison for staged learning and perception-guided kicking, but its observation interface should not be silently equated with an RGB-to-action policy. Together, these works suggest a better way to position Joga. There is no single ladder on which “soccer robot” systems can be ordered. One study may contribute realistic visual training, another terrain robustness, another dynamic skill acquisition, and another actuator fidelity. The interesting possibility is that Joga integrates several of those concerns into a capable humanoid system. The eventual paper will need to show where that integration produces a measurable advantage—and which parts are responsible. What I would measure before trying to reproduce it. Because the implementation is not yet verifiable, the most useful adaptation advice is an experimental design, not a guessed training recipe. I would start with a compact factorial study: active versus fixed gaze, perception correction enabled versus disabled, and actuator correction enabled versus disabled. That gives eight configurations before considering interactions with the skill architecture. The controls matter. Freezing a neck only at test time measures sensitivity to a sudden intervention. Training a fixed-camera baseline measures what an alternative system can learn under that constraint. Those experiments answer different questions, and I would report them separately. The same applies to residuals. Removing a correction from a policy trained to depend on it is not equivalent to retraining without that correction. A convincing study should distinguish dependency from benefit under comparable training conditions. Measure information quality where it affects action. I would evaluate ball-position and velocity errors near consequential events: interception, foot contact, sharp turns, and temporary loss of visibility. Average detection accuracy over an easy sequence could hide a system that fails at precisely those moments. Similarly, mean position error could hide temporally correlated mistakes that repeatedly push the controller in the wrong direction. The logging design should connect each action to its information history. Record image acquisition times, estimator outputs, neck state, control inputs, and relevant skill transitions. The objective is to answer a practical debugging question: what did the robot believe when it chose the action that failed?. External measurement can be useful here without becoming part of the deployed policy. An onboard-control claim should specify whether an external system supplies runtime inputs; it need not prohibit independent ground-truth instrumentation used only for evaluation. For gaze, I would measure time without usable observations, reacquisition time, and task performance after occlusion—not just how closely the head points at the ball. Measure the task without rewarding the wrong shortcut. For receiving, evaluate the resulting ball state, not merely whether contact occurred. For dribbling, combine progress with possession and balance. A robot that keeps the ball nearby by barely moving should not receive the same interpretation as one that completes a commanded route. For turning, compare retention at matched motion conditions. For shooting, report accuracy and ball motion together with post-shot recovery. For composition, use complete episodes rather than multiplying isolated benchmark scores. I would include both controlled tests and longer, less curated runs. The former identify causes; the latter expose accumulating state-estimation errors, repeated recoveries, and interactions that short trials may miss. Crucially, the analysis should distinguish misses, lost tracking, lost possession, balance failures, and protective stops. Aggregating everything into one success rate obscures which research component actually needs improvement. Make the data split reveal generalization. For model fitting, I would hold out complete trajectories rather than randomly separating adjacent samples. Neighboring frames and control steps can be so similar that a sample-level split becomes an optimistic interpolation test. Then I would vary conditions according to the claim being tested. Changes in lighting and ball appearance test visual robustness. Changes in surface and ball behavior test interaction robustness. Different motion patterns test whether an actuator correction is useful beyond its collection policy. These are proposed tests, not reported Joga results. My implementation sequence would also be deliberately staged. Establish synchronized sensing first. Validate estimates during controlled camera and body motion. Validate dynamics corrections independently. Only then combine them in contact tasks and expand to skill transitions. That order is not a claim about the team’s development process. It is a way to keep a replication interpretable. If the first experiment simultaneously introduces a new camera mount, a new residual, a new policy, and a new contact task, even an impressive result may be difficult to explain—or reproduce. What this demonstration could change. The strongest interpretation of Joga is not that soccer automatically produces general-purpose humanoid competence. It is that an athletic robot may need to manage the reliability of its own information as carefully as it manages balance and contact. That principle extends naturally to other tasks as a research hypothesis. A mobile manipulator approaching an object could benefit from maintaining a useful view while repositioning. A humanoid preparing a contact could benefit from a dynamics model that accurately reflects its hardware. A multi-stage task could benefit from training on the states that preceding behaviors actually produce. But those are transfer opportunities, not demonstrated Joga outcomes. Evidence for general-purpose capability would require evaluations beyond the soccer setting, with clear accounts of what transfers unchanged and what requires retraining. I would also avoid framing the work as a contest between learned and engineered systems. A motorized camera mount, calibrated timing, a structured simulator, learned corrections, and task policies can all contribute to the same result. The scientifically useful question is which structure makes learning more effective and which learned component resolves a genuine limitation. For researchers considering this direction, the main takeaway is methodological: do not ask a policy to compensate blindly for every discrepancy in the system. Identify which uncertainty comes from sensing, which comes from dynamics, and which can be reduced by choosing a better action or viewpoint. Joga’s announcement is interesting because it puts those concerns in the same demanding task. The next step is to turn the reported capability into an inspectable account of mechanisms, data, and failure rates. A successful kick is the visible event. The deeper capability is arriving at that kick with the right information, executing it under realistic dynamics, and remaining ready for whatever happens next. Research note: This article draws on the recoverable team announcement and the named primary research sources. A public Joga methods paper and code release could not be verified as of September 24, 2026, so its architecture, training details, quantitative performance, and component-level contributions remain unresolved.