Latest / Embodied AI 101 / Digit Leaves the Lab: The Real Lesson of 500 Demonstrations
Transcript
- Digit Leaves the Lab: The Real Lesson of 500 Demonstrations. Digit Whole-Body Mobile Manipulation at IROS 2026 (research demonstration) • Agility Robotics research team; public reporting by Chris Paxton • Agility Robotics • IROS 2026 • 2026. A small task with a large systems question. Picking objects from a tote and carrying them across a booth does not sound like a frontier robotics benchmark. That is precisely why it is interesting. Consider what that apparently ordinary task asks of a humanoid. The robot must approach an object without making the eventual grasp awkward, position its hands while maintaining balance, accommodate the change in loading after contact, transport the object, and finish somewhere useful. A successful grasp is only one event in a longer feedback loop. The relevant capability is not manipulation plus locomotion as two independent demonstrations, but their coordination under changing physical constraints—a central challenge in humanoid loco-manipulation research. Agility’s IROS demonstration offers an interesting systems-level answer: learn the manipulation behavior from real demonstrations, while delegating whole-body execution to a controller trained in simulation. An attendee’s account reports approximately 500 teleoperated demonstrations and roughly 80–90% success in the unfamiliar conference setting. Those are promising numbers, but their meaning depends heavily on the boundary between the two learned components and on what counted as success. The most useful question is therefore not whether 500 demonstrations are enough to “solve humanoids.” It is what the demonstrations were responsible for teaching, what competence already existed underneath them, and which forms of generalization the deployment actually tested. That is the focus of this episode of Embodied AI 101: the architecture suggested by the public evidence, the engineering significance of its interface, and the experiments needed to turn an impressive field demonstration into a reproducible research result. What was actually reported. First, the timeline. IROS 2026 took place in Pittsburgh from September 27 through October 1, 2026. This article assesses the public material available as of October 4, 2026. A contemporaneous post by Chris Paxton identifies the demonstrated platform as Digit v4 and describes the initial mobile-manipulation run as working immediately in the new setting. The supplied X page was not directly retrievable. The evidence available for this analysis includes first-person posts preserved in indexed social feeds, Agility’s earlier technical publications, and closely related research. That makes this a demonstration analysis, not a review of a fully specified IROS paper. The central numerical account comes from Jonathan Stephens, who describes discussing the demonstration with Paxton. Stephens reports that Digit autonomously removed objects from a tote, carried them across the booth, and repeated the process. His account attributes the manipulation policy to approximately 500 real-world teleoperated demonstrations, paired with a simulation-trained whole-body controller. He reports approximately 80–90% success despite unfamiliar surroundings, lighting, and background activity, and explicitly says Paxton characterized the work as research rather than the system currently running in production. A separate first-person account from Agility researcher Jemuel Premkumar reports approximately 12 hours of demonstration operation over three days, with several stretches continuing for hours without a reset. That is useful additional evidence of sustained operation, but it is not a claim of twelve uninterrupted, intervention-free hours. These accounts support a narrower—and still significant—conclusion than “80–90% success across arbitrary unseen environments.” They describe transfer to a previously unseen conference venue, not a standardized evaluation across a distribution of independently selected sites. There is also a terminology correction worth making. The meaningful claim is zero-shot deployment at the venue, or deployment without venue-specific policy fine-tuning. It cannot mean learning without real-world experience, because the reported manipulation training explicitly used real robot demonstrations. We should preserve both sides of that distinction. The deployment is interesting because new surroundings did not require collecting another local dataset. It is not evidence that the entire robot’s competence emerged from simulation alone. “End-to-end” does not mean one network. The reported training split already tells us something important: “end-to-end” should not automatically be read as a single jointly trained network mapping camera pixels directly to motor torques. At the systems level, the description contains two learning problems. One concerns task behavior: what manipulation action should happen next? The other concerns execution: how should a floating-base robot realize the requested movement while remaining dynamically viable? Agility’s earlier controller documentation explicitly describes downstream manipulation behaviors being built above a reusable whole-body control layer. That separation is not an architectural compromise we need to apologize for. It is a hypothesis about where to place reusable competence. Related work on the Digit platform makes this boundary explicit. The Humanoid Hanoi project uses independently trained high-level skills that issue motion directives to a shared whole-body controller, which then produces joint-level targets. It is a concrete example of how learned task behavior and learned motor control can compose without being one monolithic policy. For the IROS demonstration, however, the public evidence does not identify the complete observation space, action representation, task-conditioning mechanism, network backbone, or inference schedule. We should not fill those gaps with whichever architecture is currently fashionable. In particular, there is insufficient evidence to call this specific task policy a vision-language-action model, a diffusion policy, or an action-chunking transformer. Those are possible implementations, not established facts. Why does this restraint matter technically? Because the action interface determines what the upper policy must learn. Predicting joint commands, Cartesian hand targets, body motion, or a latent motor command produces different learning problems, even when the resulting videos look similar. Temporal representation matters too. Diffusion Policy, for example, demonstrates an approach in which a visuomotor policy predicts action sequences and executes them through receding-horizon control. That is relevant background for understanding modern policy interfaces, but it does not establish that Agility used that method. For a useful technical discussion, “end-to-end” needs an endpoint specification. End-to-end from which observations to which commands, with what state estimation, execution controller, and supervisory logic still outside the learned mapping?. Until those details are published, the strongest interpretation is integrated learned task behavior over a separately trained motor-control substrate. The controller is an interface, not just a stabilizer. Agility’s most informative public background document is Training a Whole-Body Control Foundation Model, published on August 28, 2025. It describes a recurrent LSTM controller with fewer than one million parameters, trained in NVIDIA Isaac Sim over three or four days representing decades of simulated experience. The controller accepts position-and-orientation objectives for the arms and torso, with training targets sampled across the manipulation workspace. The article also emphasizes position-aware control rather than relying exclusively on velocity commands. These are specifications of the earlier published controller, not independently verified specifications of the exact IROS 2026 checkpoint. The architectural significance is the interface. A task policy can request a meaningful movement without explicitly producing every joint trajectory needed to realize it. My interpretation is that this can change the effective difficulty of imitation learning more than a modest improvement in the task-policy backbone. Instead of asking real demonstrations to teach both object interaction and the detailed mechanics of balancing through that interaction, the system can reuse a motor layer across demonstrations and tasks. But the interface must deliver more than “the robot does not fall.”. Suppose a hypothetical policy requests a hand movement toward an object. If body motion changes the achieved hand trajectory, then grasp geometry can become wrong even while the robot remains upright. Conversely, excellent hand tracking is insufficient if the robot cannot accommodate the external forces produced by lifting or pushing. Work such as FALCON explicitly treats force adaptation as part of humanoid loco-manipulation, rather than assuming that accurate free-space motion alone resolves interaction. This suggests two different tests for a reusable controller: does it maintain physical viability, and does it preserve a sufficiently predictable relationship between commands and task-space outcomes?. That distinction becomes important when the controller is placed underneath a learned policy. An interface with inconsistent tracking errors may force the upper learner to compensate for low-level execution details. The intended abstraction then starts leaking. There is a second, less glamorous part of Agility’s controller story: simulation fidelity. In its earlier Crossing the Sim2Real Gap With NVIDIA Isaac Lab article, Agility describes policies that walked well in simulation but slipped on hardware. Its investigation identified problems involving collision geometry, actuator and transmission dynamics, and constraints associated with Digit’s closed-chain leg mechanism. The company emphasizes correcting those discrepancies rather than repeatedly patching undesirable behavior with additional reward terms. The practical lesson I draw is that “trained in simulation” is not a shortcut around physical understanding. It relocates some of that understanding into simulator construction and validation. For an analogous system, I would therefore treat the controller’s operating envelope as a first-class artifact: reachable targets, tracking accuracy, load tolerance, command-rate limits, and behavior under estimation error. A task policy can only exploit the abstraction reliably if that envelope is understood. And none of this, by itself, establishes general obstacle avoidance or safe operation around people. A motor interface and a complete deployment safety case are different engineering objects. What five hundred demonstrations really purchase. The demonstration count is compelling because it sounds small relative to the complexity of a humanoid. But the relevant accounting question is: small relative to which learning problem?. In a hierarchical system, task demonstrations need not teach the entire motor repertoire from scratch. They can supervise behavior through an already capable execution layer. HOMIE, for example, explicitly combines a learned loco-manipulation controller with teleoperation hardware, then demonstrates that the collected data can support autonomous imitation learning. That makes the real-data budget easier to interpret. It is primarily a budget for teaching task-conditioned behavior through an existing interface—not necessarily a budget for discovering walking, balance, reaching, and object interaction simultaneously. We also should not convert demonstrations into training hours without knowing their duration. As a purely illustrative calculation, 500 thirty-second episodes sampled at twenty observations per second would yield 300,000 observation-action records. Those are not Agility’s reported collection parameters. More importantly, those records would not represent 300,000 independent situations. A long approach sequence can contribute many nearly redundant samples, while an unusual grasp failure might contribute only a handful. For an experiment built around this data budget, I would care more about coverage than raw frame count: different approach geometries, object presentations, reach heights, lighting conditions, and recoverable mistakes. Five hundred carefully varied episodes and five hundred repetitions of one narrow setup are different datasets, even if their file sizes match. This connects directly to the distribution-shift problem formalized by DAgger. A policy trained on expert trajectories can make an error and then encounter states poorly represented in its training data. DAgger addresses that mismatch by collecting expert supervision on states visited by the learner. The relevant lesson here is the importance of the policy-induced state distribution, not a claim that Agility used DAgger. For a mobile manipulation dataset, I would consequently want examples of recoverable deviations: approaching slightly off-axis, reacquiring an object after a missed grasp, or adjusting after an object moves. These would be deliberate additions to coverage, not an assumption that random visual augmentation can substitute for physical recovery experience. The logging interface deserves equal attention. I would record both the operator’s requested commands and the robot’s achieved motion, with accurate timestamps. If the controller tracks imperfectly, those two signals carry different information. One records intent; the other records execution. Finally, the collection controller and deployment controller should be versioned together. Otherwise, a task policy may be evaluated through a different command-to-motion relationship from the one that generated its demonstrations. The reported episode count is therefore best treated as the visible task-data budget. It does not measure all the simulation, physical modeling, software integration, or possible pretraining that made those episodes useful. Three kinds of transfer, only one conference venue. The term “generalization” is doing several jobs in this story. Separating them makes the result more informative. The first is dynamics transfer: whether a controller trained in simulation behaves effectively on hardware. Agility’s simulator work addresses that problem directly. Differences in physical contact and robot dynamics can matter even when the task and visual scene remain unchanged. The second is behavioral transfer: whether an imitation policy continues to make useful decisions when execution deviates from its demonstrations. That is the familiar sequential distribution-shift problem highlighted by DAgger. The third is environmental transfer: whether the frozen system remains effective when moved somewhere visually and spatially unfamiliar. The conference-floor account provides evidence relevant to this third category, although it does not isolate environmental variation from the other two. Here is how I would separate the possibilities experimentally. A new background with familiar objects and familiar relative geometry mainly tests appearance robustness. Moving the source and destination while preserving object identity adds geometric variation. Changing object mass or friction adds physical variation. Requiring a different manipulation sequence adds task variation. These are useful evaluation axes, not interchangeable meanings of “unseen.”. A conference can look dramatically different to a person while preserving much of the task’s local structure. Conversely, a scene that looks almost identical can become mechanically difficult after a small change in object pose or friction. I would therefore avoid using visual unfamiliarity as a proxy for the full distance between training and deployment conditions. The same caution applies to crowds. Background people, partial visual occlusions, people crossing the robot’s intended path, and physical contact with the robot are different perturbations. An account mentioning a crowded venue does not establish that all four were tested. None of this diminishes the value of a successful deployment. It tells us which follow-up experiment would be most informative. My next step would be a held-out-site evaluation with the policy frozen before site selection, coupled with a record of any calibration, configuration changes, or operator setup. That would make “zero-shot” an auditable protocol rather than an adjective attached to a video. Reading 80–90 percent like an experimentalist. A success percentage is only interpretable when its unit is clear. Did an attempt mean one grasp? One pick-and-transport cycle? One complete tote-clearing episode? Was a retry allowed? Did the robot need to return to its starting configuration? Did an operator replacing a dropped object count as a failure or simply as preparation for another trial?. The available first-person numerical report does not provide that protocol. Consequently, the percentage should remain an attributed field estimate, not be presented as a benchmark score or a confidence interval. This matters especially for repeated workflows. Consider an illustrative system—not an estimate of Digit—with an 85% probability of completing each full cycle independently. The chance that all ten consecutive cycles succeed would be about 20%. For twenty consecutive cycles, it would be about 4%. That calculation is deliberately simplified. Real failures need not be independent, and a robot may recover autonomously. The point is that per-cycle success and uninterrupted workflow completion are different quantities. Recovery changes the interpretation substantially. A failed initial grasp followed by a successful autonomous retry may be a minor throughput penalty. A superficially similar failure that requires an operator to enter the workspace may dominate the operational cost. For that reason, I would report first-attempt success separately from eventual success, and eventual success separately from intervention-free completion. Collapsing all three into one number hides the behavior most relevant to deployment. The duration report adds a different kind of evidence. Premkumar’s account of extended runs without resets suggests that the system was not limited to one isolated successful rollout. But total demonstration hours still do not provide the number of attempted cycles, the distribution of failures, or the amount of operator assistance. For a serious evaluation, I would want the numerator and denominator, a precise success definition, cycle-time distributions, intervention counts, and a failure taxonomy. I would also separate manipulation errors from locomotion interruptions, perception failures, and supervisory stops. There is a final distinction: task reliability is not a safety argument. Even a high task-success rate cannot establish how a system behaves during sensor degradation, communication loss, unexpected contact, or an unrecoverable action request. My preferred interpretation is therefore neither dismissive nor promotional. The reported performance is encouraging field evidence. Its missing denominator and protocol prevent stronger quantitative conclusions. Related work: the same boundary, different choices. Several closely related systems help explain why the interface between task learning and body control deserves so much attention. Mobile ALOHA, by Zipeng Fu, Tony Z. Zhao, and Chelsea Finn, develops whole-body teleoperation and imitation learning for bimanual mobile manipulation. Its wheeled platform avoids the bipedal balance problem, but it still requires demonstrations that coordinate movement of the base and arms. The work also investigates co-training with static manipulation data. The useful comparison is methodological: mobility must be represented in the data and action interface, rather than treated as an unrelated prelude to manipulation. I would not compare its success rates directly with the Digit field estimate. The embodiments, tasks, datasets, and evaluation conditions differ. It is more useful as evidence that collecting the right coordinated demonstrations can make mobile manipulation tractable. HOMIE, by Qingwei Ben and collaborators, is a closer humanoid comparison. It combines a reinforcement-learning-based loco-manipulation policy with an isomorphic exoskeleton, gloves, and a pedal. Its controller supports walking and height changes while accommodating upper-body configurations, and the authors demonstrate imitation learning from data collected through the system. The project is publicly released. The connection is the role of the controller in making demonstrations possible and useful. A teleoperation system is not merely a recording device; it shapes which behaviors can be demonstrated accurately. My takeaway is that better data-collection controllability can be as important as the downstream learning algorithm. Boston Dynamics and Toyota Research Institute’s Atlas behavior-model work offers another decomposition. Their published system uses a VR teleoperation interface together with model predictive control, and the learned policy uses the same control interface as teleoperation. The behavior model processes images, proprioception, and language, with a diffusion-transformer architecture trained using a flow-matching objective. This is an important counterweight to an overly simple “RL replaces classical control” narrative. The shared principle is a capable execution interface underneath learned behavior. The execution machinery can be optimization-based, learned, or hybrid. A demonstration of the complete system does not by itself establish which implementation is best. HERO, by Runpei Dong and collaborators, approaches visual humanoid loco-manipulation through a modular system combining open-vocabulary visual understanding with accurate end-effector control. Its residual-aware tracking design brings together inverse kinematics, a learned forward model, goal adjustment, and replanning. HERO is particularly relevant because it treats task-space accuracy as an enabling capability for generalization. It also reminds us that modularity and generalization are not opposites. A system can gain robustness by choosing useful representations and interfaces, rather than by making every component part of one learned mapping. Together, these works suggest a better comparison question than “Which robot has the most impressive clip?”. What task-relevant structure is encoded in the controller, what remains for the policy to learn, and how much real data is needed once that boundary has been chosen?. How I would adapt and stress-test the idea. Suppose the goal is not to reproduce Agility’s undisclosed implementation, but to build a comparable research system. I would organize the work around interface validation before scaling the task learner. First, define the control contract. Specify command units, coordinate frames, update rates, expected latency, reachable target regions, and behavior when a request cannot be tracked. Do not let those details remain implicit in a teleoperation application. I would then test whether the interface has consistent semantics across operating conditions. A nominally identical hand command should not produce unexplained changes in task-space behavior after a small variation in body pose or carried load. Where behavior does change, the upper policy needs either observations that explain the change or a narrower command envelope. Second, measure execution directly. Record requested and achieved hand poses, body motion, estimator outputs, and timing. Evaluate tracking during standing, stepping, reaching, and representative object interactions before attributing downstream failures to the visuomotor network. This would also help diagnose latency. I would log the age of the observation used for each action, not merely the network’s inference duration. A fast model operating on stale images can be a different control system from the same model receiving fresh observations. Third, collect demonstrations through the deployment interface and preserve controller versions. If the controller changes, run a compatibility evaluation before assuming the existing dataset still defines the same learning problem. I would distribute the data budget deliberately. A useful allocation might emphasize distinct approach configurations and object arrangements, then reserve a portion for recovery demonstrations. The exact allocation should follow measured failure modes rather than an assumption that every additional nominal success contributes equal value. Fourth, freeze the evaluation design early. Hold out complete sites, not just frames or neighboring episodes. Record what changes between sites and what stays fixed. If a new venue requires camera calibration or task configuration, document it instead of treating all non-gradient setup as irrelevant. For the research question here, I would run three kinds of ablation. One would vary the demonstration budget while keeping the controller fixed. Another would vary the execution controller while keeping the task interface as comparable as possible. A third would compare integrated mobile behavior with a constrained approach that separates travel from stationary manipulation. Those comparisons need careful task matching. A controller that cannot reach part of the workspace should not be evaluated as though it had the same capabilities. I would report both performance within a shared feasible region and performance on the complete intended workflow. Fifth, make failure recovery explicit. Predefine which failures the robot may retry, how completion is detected, and when the system must stop or request assistance. Evaluate recovery states directly rather than hoping that a high nominal success rate makes them rare enough to ignore. Finally, release enough information to make the architectural claim testable: observation and action schemas, controller versions, dataset composition, site splits, raw trial counts, and representative failure logs. Full proprietary source code is not the only possible contribution to reproducibility. The central hypothesis I would want to test is straightforward: does a more reliable whole-body action interface reduce the real demonstration data needed for robust task behavior?. That is a stronger and more reusable research question than whether one particular policy can repeat a booth demonstration. The real takeaway. Agility’s IROS demonstration is best understood as evidence that a learned manipulation system can survive a meaningful change in deployment setting—not as proof that general humanoid autonomy is solved. The promising architectural idea is the division of learning effort. Real demonstrations can focus on task behavior, while a reusable execution layer handles much of the robot-specific coordination. Related systems show several ways to implement that division, including learned controllers, model predictive control, and hybrid task-space tracking. My main takeaway is that the quality of this boundary may be one of the most important determinants of real-data efficiency. A good interface lets demonstrations express useful intent. A poor one forces the task learner to absorb execution quirks that should have been handled elsewhere. The next decisive evidence would be frozen-policy tests across multiple held-out sites, with explicit trial definitions, recovery accounting, and controller ablations. That would turn a compelling conference result into something the rest of the field can reason about quantitatively—and adapt. Research boundary: The search recovered first-person field reports and Agility’s earlier controller documentation, but not the supplied X page directly or a matched technical paper, dataset, checkpoint, or task-policy repository for this exact demonstration. The numerical claims remain attributed reports; the proposed mechanisms and experiments above are engineering analysis, not undisclosed implementation details.