Latest / Embodied AI 101 / Digit's New Motor Cortex: Sim-to-Real RL for Whole-Body Control
Transcript
- Digit’s New Motor Cortex: Sim-to-Real RL for Whole-Body Control. Humanoid robots like Agility Robotics’ Digit promise to automate the most strenuous warehouse and logistics tasks – things we expect humans to do, but that unstructured world is tough for machines. Tiny bumps, shifting loads, uneven floors and complex manipulation can easily confuse a rigid, hand-coded controller. Agility’s latest announcement shows how they’re using modern reinforcement learning (RL) techniques to give Digit a kind of “motor cortex” – a neural controller that learns in simulation and transfers to the real robot without fine-tuning. The result is an always-on whole-body controller that can react to pushes, adjust foot placement, and carry heavy objects, all while being commanded simply by desired hand and torso positions. In other words, instead of prescribing every move, the system learns how to move Digit robustly so a human or planner need only say “go here” or “place the object here.”. In this article we’ll unpack how Agility built that capability. We’ll start with the problem of whole-body humanoid control, contrast model-based vs learned approaches, and then dive into Agility’s solution: a small LSTM “motor cortex” trained purely in NVIDIA’s Isaac Sim environment. We’ll describe how they covered the robot’s entire workspace with randomized targets (rather than relying on limited human-motion datasets), conditioned the policy on end-effector positions (not just velocities), and even incorporate demonstration data via teleoperation. Finally, we’ll see the impressive real-world results – from step-recovery to heavy-object manipulation – and discuss what this means for future humanoid robots. The Challenge of Whole-Body Humanoid Control. Unlike a fixed-base robot arm, a walking humanoid must constantly juggle many degrees of freedom while balancing on a moving support. Each time Digit swings a leg or plants a foot, a whole new contact mode is created. These contact transitions mean the robot’s dynamics are fundamentally hybrid: part of the time a foot is off the ground (swing phase), and part of the time it is on the ground applying forces (stance phase). Planning or controlling such a system in real-time is notoriously complex. Model-based controllers often simplify things by assuming, for example, that feet stay planted while the arms reach or manipulate. That turns the problem almost into a fixed-base one (like an arm on stilts), which is easier but limiting. If the robot must take a step – say, to reach an object far away or to recover from a push – that simplification no longer holds. As Agility’s blog explains, “preventing the robot from dynamically adjusting its foot placement will limit its manipulation workspace and prevent it from intelligently and naturally reacting to disturbance forces”. In everyday language: we want Digit to behave like a human who can step when needed. When a person reaches forward and is about to lose balance, we naturally shift our feet or lean to compensate; programming that by hand requires enumerating every weird combination of falls, which is nearly impossible. The complexity of foot contacts is a “heavy computational cost,” the company notes. Fixed-base robots don’t have to worry about this, so giving a fixed-base-style interface to a humanoid is usually done by force – but it severely restricts the robot’s capabilities. The ideal interface, Agility argues, is one where a user or planner simply specifies hand/torso target motions in 3D space and the robot “simply tracks the hand motions desired by the user while taking steps on its own when necessary, avoiding collisions […] and trying really hard not to fall over”. Achieving that requires solving both balance and manipulation simultaneously – a notoriously hard problem with contact dynamics that change at every step. Traditional vs. Learned Control. Historically, Agility (and most legged robotics teams) have relied on model-based control for balance and walking. This means the onboard controller uses a simplified robot model, fused with sensor data (IMU, joint sensors, cameras), to decide where to put the feet or how to shift center of mass. In practice they run an optimization or inverse-kinematics solver in real time: “the model is a simplified physics model of the robot, running on the computer inside the robot, calculating things like where to place a foot to balance the robot”, and then using inverse dynamics to compute torques. This can work extremely well when you know the environment (smooth floor, no surprises), and it can run quickly onboard. But it also means you have to know that floor is flat, that the mass distribution of the carried object is known, etc. Model inaccuracies or unforeseen pushes can easily break it. Reinforcement learning offers a completely different paradigm. Rather than solving balance with a hand-tuned model in every scenario, one can simulate Digit in software many times and gradually learn a controller that succeeds most of the time. Agility explains: “Instead of using the robot and world models onboard in real-time to calculate the motor torques… we simulate worlds to learn a control policy for the robot ahead of time, on off-board computers that are much more powerful than the onboard robot computers”. In practice, they build a very faithful virtual Digit in NVIDIA Isaac Sim (including expected bumps, slopes, pushes, heavy objects, etc.), and then run millions of simulated steps where the policy tries different torques. At each step, a cost function judges success (e.g. did it stay balanced, did hands reach targets?), and the RL algorithm iterates. Over “decades of simulated time” (all crunched down into a few days of real wall-clock training) the network learns how to coordinate all 28 joints of Digit to walk, shift weight, and reach targets robustly. The payoff of this approach is impressive. For one, the learned controller can be robust to disturbances that a model-based controller isn’t. Agility notes that in tasks like step-recovery – recovering from a stumble or unexpected push – traditional controllers often fail because the exact perturbation wasn’t anticipated. But by training in simulation with many random pushes, slips, or catches, the RL controller becomes surprisingly strong: “we can train a controller to be robust to many of these disturbances with reinforcement learning, training it on many possible ways that the robot might fall”. Indeed, the company showed side-by-side experiments where a model-based controller falls while the RL-based controller recovers after a big shove. In short, the RL “motor cortex” can discover coping strategies (like stepping further out or bracing in a certain way) without anyone coding them explicitly. At the same time, the tradeoff is that training in sim requires careful setup and compute, and success hinges on closing the sim-to-real gap. Agility reports tackling this with realistic physics in the simulator and domain randomization, but the real test is whether the learned policy works on the real Digit without tweaking. Spoiler: it does – by design, the policy is deployed zero-shot. Every learned motor command goes straight to Digit’s hardware once training is done. Building a Whole-Body “Motor Cortex” Foundation Model. The core of Agility’s new approach is what they call a “whole-body control foundation model” – or colloquially, a learned motor cortex. At the software level it’s a fairly small neural network (an LSTM) that sits at a low control layer and acts continuously to stabilize Digit. Agility emphasizes that the model is tiny: under one million parameters. Tiny by modern deep learning standards, but still big enough to capture the complex dynamics of a whole human-shaped robot. Because it’s recurrent (LSTM), it has memory of past states and can produce smooth, temporally coherent behavior. This learned controller is trained “purely in simulation”. Specifically, Agility uses NVIDIA’s Isaac Sim physics engine, setting up Digit with all its sensors and joints, then running simulations that each simulate years-worth of motion. In real time the training took “three or four days” of wall-clock time, but that corresponds to decades of simulated time thanks to massive parallelization. During training, Digit is repeatedly given random tasks: we’ll dive into the details below, but the short story is that the target motions or goals (like where the hands or torso should go) are all randomly sampled across the robot’s workspace. The policy’s job is to make the robot achieve those targets safely and efficiently. Over thousands of such tasks, the network learns a generalized notion of balancing while moving the limbs. After training, this LSTM becomes Digit’s motor cortex. In operation, higher-level planners or human operators simply give it desired end-effector positions (in space) and orientations for Digit’s hands, arms, or torso. The motor cortex then generates joint torques to achieve those targets – stepping or leaning as needed to stay balanced. Crucially, this learned controller transfers zero-shot to hardware. That means whatever the network learned in sim, it can be deployed on the actual Digit robot immediately, without any real-world retraining or heavy calibration. In practice, what can this motor cortex do? According to Agility, pretty much any free-space limb motion. By prompting the network with a sequence of hand (or foot or torso) waypoints, Digit can navigate and manipulate the world. For example, they demonstrated it walking to pick up heavy crates and place them on a shelf, reacting fluidly if the crates shift. At NVIDIA’s GTC 2025 conference, an early version of the controller even enabled Digit to “go shopping”: detecting grocery items via vision, handing (picking and placing) them into a cart, all while maintaining balance. Because the controller truly coordinates feet and hands, Digit doesn’t rigidly lock its legs in place when reaching – instead it can take an extra step if an item is just out of reach. This mimics how a human would do heavy manipulation: reposition feet for stability rather than stretching an arm to its limit. Agility hopes to stack more on top of this core. The motor cortex is intended to be “always on” as a base safety-and-balance layer. Then more complex behaviors can be learned or programmed on top of it. For instance, one could fine-tune or separate a network for specific dexterous tasks (like screwdriving or assembly) using, say, imitation or smaller RL loops, while relying on the foundation to keep the robot upright and responsive. They even mention linking it with vision and large language models: for example, an LLM could produce a high-level plan (“place the box on the conveyor”), which then gets converted to hand/arm targets, all executed via the same learned motor cortex. The idea is that this foundation model makes every other layer of control simpler, by guaranteeing robust whole-body behavior under the hood. Training the Controller in Simulation. Let’s peek under the hood at how Agility actually trained this whole-body controller. The key is that they needed the policy to handle anything you throw at it — not just a few scripted motions. Other recent research in whole-body humanoid RL (“GMT”, “TWIST”, etc.) has often used large motion-capture datasets of human movements (like AMASS or LaFan1) to define target motions. Those human motions produce realistic walking or gestures, but they have a big limitation: they only include the poses humans naturally make. If you need Digit to reach a far corner or pick up something off the floor, it’s unlikely a human mocap dataset covered that. In Agility’s words, mocap trajectories “often don’t cover the entire desired manipulation workspace”. That means the robot might not know how to use its full reach, especially at the fringes. To solve this, Agility decided to uniformly cover the workspace with random targets. In training, both hand positions and torso positions are sampled randomly and uniformly across the allowable volume. For each sample, a small trajectory is constructed (with random velocities and rotations thrown in) that the robot must follow. In essence, they pick two random points (with orientation) and have Digit’s hands and torso move between them. Then the reward function encourages the robot to minimize the error between its actual hand positions and the target. By doing this thousands of times in all combinations, the policy eventually learns to hit any point in space with the hands or torso, to whatever orientation is needed. In more detail: the company writes that they generate “time-indexed trajectories for both hands and the torso” by picking random poses and movement speeds, then train the motor cortex to reach those poses with minimal error. This thorough coverage means Digit won’t run into blind spots. If later you ask it to reach something at the edge of its reach, the policy has already practiced something similar in simulation. One design choice worth highlighting is that the network is prompted by positions and orientations in task space, not by joint angles. Many other works parameterize upper-body targets as joint configurations, which forces any operator (human or planner) to know Digit’s joint-level kinematics. Instead, Agility’s controller expects something like “left hand go to (x, y, z) with orientation R.” The LSTM then figures out the inverse kinematics plus balance. This greatly simplifies the interface: you can use computer vision or language models to get a 3D point for an object, without needing to convert that into motors and angles. As Agility notes, this is closer to how we programmers think – it’s easier to say “pick that cup” than to specify every joint’s angle. (In fact, they explicitly cite the human analogy: the model “functions much like the motor cortex in the human brain,” accepting high-level movement commands.). This random-target training contrasts with velocity-based locomotion training. Historically, many legged-RL controllers were given a desired velocity rather than a position goal (e.g. “walk at 0.5 m/s forward”). That simplifies learning because you just match speeds, and it produces periodic gaits. Agility, however, wanted Digit to navigate to specific locations or hold a position – crucial for precise tasks. Thus they conditioned their policy on target positions. They observed that most literature (e.g. AMP, Falcon) takes the velocity approach because “tracking velocities rather than positions provides a much richer and more forgiving reward signal”. But velocity-only controllers require another layer (a planner or human teleop) to tell them where to go. Agility flips this: the user can now simply tell Digit “go there” in world coordinates, and the controller will handle it, even recovering from pushes. It’s more challenging for RL training, but it pays off in usability: a higher-level planner or even a large language model can output concrete coordinates (perhaps of an object detected in 3D), and Digit just follows. Speaking of training, what RL algorithm was used? Agility doesn’t specify in public (likely some form of PPO or TD3 in Isaac Gym style), but the principles are clear. They run parallel simulations, apply domain randomization (for friction, etc.), and accumulate rewards for tasks like reaching or stabilizing. They mention cost terms for reaching accuracy, presumably with penalties if Digit drifts too far from a target or falls. The blog notes that they invest significant compute “on off-board computers” to evaluate millions of potential action sequences. In practice, thanks to modern GPUs and Isaac’s optimizations, this training is over in a few days. In that time, the policy learns not just one skill, but a rich repertoire of stable motions covering Digit’s whole body. Data Sources: Teleoperation, Simulation, and More. Creating the training data for Digit’s AI brain is a multi-pronged effort. As Agility’s “Agility and AI” primer explains, they have three primary data sources: teleoperation (real demonstrations), simulation, and public data. Public datasets of robot motion simply don’t exist (yet), so the first two count the most. Teleoperation / Demonstration: Digit is teleoperated by humans in the real world to perform tasks over and over. This can be done either by direct remote control or with motion-capture suits translating human motions. Every time a human shows Digit how to do something (walking up to a tote, reaching for a bin, etc.), that provides rich observational data. Using learning-from-demonstration (LfD), Agility can feed this data into the training pipeline. In practice, they teleop the robot to do a task numerous times, then record all the state and action trajectories. This helps especially for tasks that would be tedious to simulate (like complex manipulation sequences) or to seed the network with plausible motions. The company notes they gather such teleop data both at their own labs and from client deployment sites – an advantage of shipping robots to real warehouses is that every failure and every human-led correction becomes a data point. Simulation: This is the workhorse. Agility uses Nvidia Isaac and related tools to create high-fidelity virtual environments. In sim, they can replay teleop trajectories or simply throw script commands to generate more training episodes. They mention using NVIDIA’s ecosystem and even MuJoCo families to “play out an unlimited amount of parallel scenarios in rapid succession, compressing months of effort into hours”. By combining teleop seeds with random exploration in sim, the policies learn far beyond what human teleops can do alone. For example, to make Digit strong against heavy loads, they can simulate picking up crates of varying mass hundreds of times – something practically impossible on the real robot due to time and wear. Public Data (Limited): Unlike language or vision models which train on vast internet datasets, robots have almost no publicly available motion corpora. Agility acknowledges this: “This is how the modern AI frontier labs have made such rapid progress training their LLMs. But that data does not yet exist for robot movement”. In short, they can’t mine YouTube for robot motion (yet!). So they rely almost entirely on #1 and #2 above. By fusing teleop and simulation, Agility can create training scenarios specific to industrial tasks. For example, suppose they want Digit to learn how to pick a heavy tote from the floor. They might begin with a teleop demonstration of how a human expert would do it at lighter weight. Then they replay and augment that scenario in Isaac – adding noise, increasing weight, introducing slight pushes – to train robustness. This way, new capabilities (like “pick up this type of box”) can be taught relatively quickly: gather a handful of demonstrations, run sim RL for a day or two, and deploy the refined model to the robot overnight. (Indeed, the Twitter announcement even touts that these skills are “deployable overnight on hardware.”) The foundation model makes this faster: because Digit’s basic balance is already learned, new tasks only have to specialize the existing motor skills rather than learn from scratch. Deploying on the Real Robot. All the training and simulation would be moot if the real Digit robot couldn’t use it – but here comes the payoff. Thanks to careful sim-to-real design, Agility reports that Digit can use its learned motor cortex immediately. The policy runs onboard and takes as inputs the same sensor data Digit normally has (IMU, joint encoders, etc.) plus the commanded end-effector positions. At every control step, the LSTM outputs joint torque adjustments that in normal operation steer Digit’s limbs towards the targets. Because it was trained on walks with pushes and foot slippages, it can handle those gracefully now. In demonstrations and early deployments, some behaviors stood out:. Heavy Object Manipulation: The new controller is robust enough that Digit can pick up very heavy items – even beyond what previous model-based methods dared. In videos from Agility, Digit lifts a large crate and walks with it without losing balance. It’s one thing to hold a small cup; carrying a 20-30kg box while walking is elite. This capability is credited to the RL training including heavy loads and perturbations. Agility explicitly notes that “the controller is also robust to picking up very heavy objects”, which is critical for warehouse use. Step Recovery and Sprite-like Robustness: If the floor gives way or someone bumps Digit, the new controller reacts by stepping out or bracing. In the step-recovery tests mentioned earlier, the RL controller easily climbed back to standing, whereas the old model-based one fell. There’s footage (and apparently robotreport coverage) of Digit’s new gait looking surprisingly fluid even when dealing with obstacles or unexpected contact. General Mobile Manipulation (The Grocery Demo): Possibly the most attention-grabbing demonstration was the “shopping” task at NVIDIA GTC. There, an early motor-cortex policy was combined with an open-vocabulary 3D object detector. The detector identifies an item (say, a box of cereal) and produces a 3D point for its location. The high-level plan says: move the right hand to that point and then carry the item to a bin. The motor cortex takes these hand/torso targets and executes them. A simple state-machine logic orchestrates the sequence (detect item, reach out, grasp, move hand back to bin, release). According to Agility, everything worked without falling over: the learned controller stayed balanced even as Digit held out its arms and carried groceries around. This is exactly the kind of dynamic mobile manipulation scenario that was previously out of reach. Integration with AI Tools: In one teaser, Agility even showed Digit responding to a prompt from Google DeepMind’s new Gemini model. The idea is that you could ask an LLM “Make Digit pick up that small package with its right hand,” and Gemini could translate that into a 3D coordinate target (via the vision system) and hand it to the motor cortex. The blog mentions being able to prompt the controller with Gemini “in research preview”. We have yet to see the full details, but it suggests they envision the motor cortex as an interface layer between high-level AI and low-level controls. In all these tests, the key is that the new skills were ready to go on the real robot almost immediately after training. The tweet announcing the work stresses “deployable overnight on hardware,” implying that once you finish your sim training (a few days), you can flash the policy and it works. This is a huge improvement over older workflows where any new skill might have required hours of tuning on the real robot. The small size of the network also helps – it can run at high control rates on Digit’s onboard computer without taxing it. Position-Control and Interface Design. It’s worth repeating the nice user interface that this approach enables. For many robots, moving in a human-intuitive way is tricky. Agility points out that earlier systems often took stick inputs or specified joint angles. Instead, digit users now can think in terms of Cartesian space: “go to this 3D point,” “keep your arm pointing that way,” or “move the torso a bit to the left,” and the controller does the inverse kinematics moment-to-moment. This frees the operator (or the planner or LLM) from ever worrying about balance or walking speed. If the hand target is too low, Digit simply steps down. If it’s leaning, Digit shifts its stance. This design choice — conditioning on 3D targets rather than joint trajectories — greatly simplifies building high-level behaviors. As the Agility team notes, many imitation-learning works parametrize the output as joint poses, which means any higher-level system (like a human operator or an LLM) would need to know Digit’s joint mapping. By contrast, Digit’s motor cortex accepts “free-space positions and orientations”, which aligns with how vision and planning systems naturally see the world. For example, they mention that using open-vocabulary vision (Detectron or Gemini-based 3D detection) allows detection of a “banana on the shelf” and outputs a 3D coordinate, and Digit’s controller can take that directly. Likewise, if a human teleoperator just gestures with a VR-hand, that can map to a position target. This setup also means the system avoids a big weak spot of velocity-based policies. In the older approach (“velocity-conditioned”), the network might require a separate path planner to slowly walk Digit to a target. Here, the policy can be told a position goal once, and it will internally figure out how to approach it and stop there, including how to adjust if shoved. The blog spells it out: if you only control velocity, “you need a higher-level planner or a human teleoperator to provide constant guidance” to avoid drift. Instead, Digit’s controller can reach a fixed position on its own and stay (like leaning against a wall or following a chair) even under disturbance. That is a much more intuitive interface. Implications and Future Directions. This whole-body controller is a major step for Digit and similar humanoids. It effectively turns Digit into a single, unified mobile manipulator rather than two separate systems (a balancing base + an arm). The “motor cortex” can instantly trade off walking vs reaching. Importantly, it does this in a learned way, so if Agility improves their simulation or collects more demonstration data, they can refine the policy further without redesigning the whole system. It also lays the groundwork for foundation-model-style intelligence on robots. Agility even calls this work their first “motion foundation model” for real-world robots. In AI lingo, a foundation model is a single, large network trained on a broad distribution of tasks that can then be used as a base for many applications (think GPT for text, or Stable Diffusion for images). Digit’s motor cortex is analogous: it’s a base controller trained on millions of pose tasks, and you can build on it. For instance, you might fine-tune a smaller policy for specific tasks (e.g. fine manipulation of objects in hand) or chain it with a planner to solve a factory LP. You could also imagine a shared controller among multiple robots (transfer learning), or even continuous learning. In the bigger picture, this shows that sim-to-real humanoid RL is finally practical. Just a few years ago, it was an open question whether a human-scale biped could be trained in sim and work at all. Now Agility says it’s deployable in an afternoon. They join other recent work (from academic labs like Berkeley’s real-world humanoid project or Google Robotics’ humanoid attempts) showing humans trainable. The difference is Agility’s focus: they emphasize mobile manipulation value (lifting, carrying, stocking) rather than flashy acrobatics. While other research demos shout dances or backflips, Agility’s demo is scanning shelves and grocery tasks – arguably more directly relevant to warehouse jobs. Challenges remain, of course. The sim-reality gap must be continuously addressed (sensors, contact friction, unmodeled stiffness all differ). The policy still needs safety checks and kill switches. And there’s the thorny issue of “intelligence” above base control: translating a warehouse order into a sequence of target positions is still nontrivial. But with this learned controller, the hardest part (balancing while reaching) is now handled by data-driven AI, leaving planners free to focus on high-level logic. In conclusion, Agility’s new work turns Digit’s control philosophy inside out. It trades handcrafted modeling for a bit of compute and data, yielding a versatile whole-body controller that can safely move the robot– anywhere the demands might go. With it, Digit can take steps, hold poses, and carry loads all by itself, driven by simple end-effector commands. As Agility puts it, their motor cortex is “an ‘always on’ safety layer that enables reactive and intuitive control”. It truly feels like we’ve taught Digit a fundamental skill: how to coordinate its whole body in service of any task. Sources: Agility Robotics’ technical blog posts and press releases provide the details of this approach and its results.