Latest / Embodied AI 101 / Training a Whole-Body Control Foundation Model
Transcript
- Training a Whole-Body Control Foundation Model. Humanoid robots are designed to work in the human world, and few advantages are as compelling as their ability to use the environment we built. Staircases, cupboards, switches and factory floors – all assume human geometry and motion. To match this, a humanoid must walk, balance, reach with its hands, grasp and manipulate objects, all while keeping its bulk upright and not toppling over. This is a huge challenge: unlike a fixed-base robot arm, a legged “whole-body” robot has to reason about lifting its feet as part of every reach, handle contacts and disturbances, and coordinate dozens of joints across arms, legs and torso. Until very recently, researchers often took shortcuts – for example, assuming the feet stay planted when manipulating – because planning truly dynamic whole-body motions in real time is daunting. The field has made impressive strides, but many solutions focus on dynamic tricks (like dancing) rather than the precise, forceful lifting and carrying that practical tasks demand. A recent effort from Agility Robotics tackles this problem head-on by training an end-to-end learned “motor cortex” – a single recurrent neural controller – to handle Digit the humanoid’s entire body and incorporate perception. This controller acts as a foundation model for whole-body control, a low-level layer that always runs, keeping Digit upright and reacting to surprises, while taking high-level commands about where to put the hands or feet. Trained entirely in simulation with massive randomization, this small LSTM-based policy generalizes zero-shot to the real robot, handling everything from walking to heavy-object lifting as long as you give it a task in Cartesian space. In this episode we unpack how it works, why it’s a milestone, and how it relates to other trends in embodied AI foundation models. Why Humanoid Control Is Hard. For fixed-base industrial arms, whole-body coordination is largely solved: given a desired end-effector pose, inverse kinematics or inverse dynamics quickly compute the joint commands needed. The operator or planner never has to worry about balance or foot placement – the robot is bolted in place. In contrast, a biped must juggle two modes of dynamics: when a foot is in the air versus on the ground. Every step introduces making/breaking contact, contact with friction and impact, and a high-dimensional balance problem. Dealing with that discontinuity explicitly is expensive and unreliable. As Agility Robotics notes, a common hack is to simply assume the feet never leave the ground while the arms do their work, effectively treating the biped like a wheeled robot during manipulation. This fixed-foot assumption has powered many advances, but at a cost. In particular, it cages the robot’s capabilities: it cannot instinctively step to catch itself if shoved, nor can it dramatically extend its reach by taking a step. Limiting foot motion shrinks the reachable workspace and leaves the robot unable to react naturally to disturbances. What we really want is an interface as simple as for an arm: “move my hand here,” and the robot just does whatever whole-body motion is required, including stepping as needed, while never falling. Ideally, it would track hand motions (or torso motions) in task space, independently plan safe foot placements, avoid collisions, and remain robust if someone or something nudges it. Historically this was practically impossible: computing a truly whole-body motion plan with contact changes in real time was intractable. Even recent trajectory optimization and model predictive control methods struggle with the combinatorics of contact. In short: whole-body humanoid control is hard because the robot’s free-floating base makes inverse dynamics a hybrid, non-smooth problem, and manual controllers face severe tradeoffs between reactivity and safety. This is where modern machine learning, especially reinforcement learning (RL), comes in. Instead of trying to analytically solve the switchy physics every cycle, we can learn a policy that implicitly handles it. Recent work has shown that a neural network can be trained in simulation to produce joint torques or desired accelerations so adeptly that the resulting behaviors on real hardware rival or exceed hand-crafted approaches. Crucially, deep RL lets us dodge hard assumptions. We don’t need to force the feet to stay down or explicitly predict contact; the network can discover stepping behaviors and balance strategies on its own in a high-dimensional Markov decision process. This approach has enabled a leap in performance: teams now routinely control simulated humanoids that lie, crawl, jump, or dance with flair. However, much of prior RL work on humanoids has emphasized agility and style (think dynamic dancing or backflips) more than task precision. For example, many policies are conditioned on target velocities (walk at this speed and direction) rather than precise positions, because velocity-conditioned rewards are smoother and easier to learn. That makes intuitive sense: to make a stable walker, rewarding a velocity target is forgiving – overshoot or drift between steps is fine in a dance. But for manipulation tasks, leftover drift is not acceptable. Moreover, using velocity commands means at deployment you must have a higher-level planner or human continuously steering the robot; the low-level controller by itself won’t go anywhere on its own without that guidance. Agility noticed this gap. In modern mobile manipulation, one often needs the robot to “go here and stay there,” not just move at some speed. Yet almost all existing legged RL work still targets velocities. As they point out, root-velocity policies (like DisneyResearch’s VMP or AMOR, or the ExBody series) are trained to match speed/direction commands. These are fine for rejecting pushes or following a moving target, but require odometry or an outer loop to nail a world position. By contrast, Agility conditions their policy on target positions directly: “go to this 3D point.” That lets the controller itself try to hold position; if the robot is shoved, it will attempt to return to the goal rather than drift away. Of course, training on absolute positions is harder (rewards are sparser and require good localization), but Agility felt the payoff was worth it. They want Digit to autonomously navigate to waypoints or reach spatial goals even under disturbance, without a separate planner constantly correcting it. Another important design choice is what form the high-level commands take. In some previous RL controllers, especially those imitating human data, the target motions might be given as joint-space trajectories (e.g. replaying recorded hand and joint angles). But that has a big downside: whoever or whatever is commanding the robot needs to know Digit’s full kinematics. Either you wear a motion capture suit and map from human joints, or you run an inverse kinematics planner to convert tasks into joint angles. This makes integration with higher-level systems hard. Agility observed that it would be far more convenient if the interface were simply Cartesian commands: “move this hand to this (x,y,z) pose” with a given orientation. In other words, task-space targets. This is exactly what a human might naturally specify (e.g. “grab that object on the shelf”), and it aligns with vision-based planning or LLM-based reasoning. For example, if an object detector finds a box, it can hand its 3D location directly to the controller rather than translating it into complicated joint commands. Fingering to Cartesian targets means downstream planners or teleoperators don’t need an internal human-readable model of Digit’s kinematics. Concretely, Agility trained their model with hand and torso positions and orientations in space as the goal variables, which turned out to make the robot much easier to drive by scripted behaviors or language-based planners. Deep RL Takes the Stage. With these interfaces in place (position commands for hands and torso, velocity or footstep implicitly handled), Agility adopted a straight-up deep RL solution. The key idea: model the whole system in a physics simulator (NVIDIA’s Isaac Sim) and let an RL algorithm train an end-to-end policy that maps observations to joint-level actions (or torques). No manual inverse dynamic equations, simplified contacts or task-specific controllers – just sample interactions. Over billions of simulated time steps, the network sees countless random scenarios and learns what actions keep Digit balanced and on target. Because the policy is an LSTM, it can keep a sense of motion history and timing for smooth trajectories, as well as implicitly learn to shift legs against terrain changes. As Agility’s blog explains, the resulting controller plays the role of a “motor cortex” for the robot. All the lower-level reflexes and balance behaviors are embedded in this continuously running network. The network continually reads out signals from Digit’s state (joint angles, velocities, contact sensors, IMU, etc.) and also any other sensor input (the NVIDIA case study even says “perception data stays in the loop” so that Digit can learn how vision and motion interact). High-level planners or deciders (human teleoperators, rule-based scripts, even GPT-like systems) can then “spark” this motor cortex by giving it new goal points. In neural terms, the goal coordinates are part of the observation or conditioning vector. In effect, this foundation policy translates-between the user’s goal and the limbs’ torques. Training this way has big advantages. First, it naturally learns to make or break contacts whenever needed. It has seen picking a hand target that’s far away, and learned to step instead of over-extending the torso. If a shove happens, it has practiced balancing out pushes. It can stride, crouch, or swivel in fluid ways that would be tough to program manually. In practice, prior works on RL-driven humanoids (dancing bipeds or quadrupeds with arms) have shown very lifelike, dynamic behaviors that are easier to get working on hardware than classical model-based controllers. Agility notes this trend: deep RL lets “the robot quickly and efficiently position itself to achieve [end-effector] targets” without explicit contact modeling. Indeed, modern RL whole-body controllers often outperform simple PID-like schemes for complex tasks. That said, Agility also recognizes that pure-performance RL can overshoot its remit. Many groups who pioneered legged manipulation with RL focused on flash: agile running, flips, running on uneven terrain, or following motion-captured dance patterns. These are fun demos, but bustle with energy rather than precision. What Agility needs is fine control under load. Digit’s job includes lifting heavy boxes or awkward parts, so it must regulate contact forces in its hands and feet. A dancing robot might not mind 10cm of hand position error; Digit cannot. So while Agility adheres to the same RL paradigm, it tailors the training to a different niche: the policy must learn to apply stable forces with the hands and feet, not just fling them around. Precisely for this reason, Agility’s controller focuses on position-tracking rewards (hands and torso reach their targets closely) and forceful manipulation. In their words, the goal is for Digit “to lift and maneuver heavy things.” In the training environment, they explicitly simulate heavy objects for lifting and include disturbance pushes. The policy not only minimizes positional error, but also avoids falls. In a sense, it learns a “don’t fall” safety objective implicitly as much as any other goal – serving as a constant fallback if something unplanned happens. Covering the Workspace: Random Targets Everywhere. A key insight from the Agility article is how they generated training tasks. Many prior humanoid RL controllers are trained on second-hand human motion data (like the AMASS or LAFAN mocap databases). These datasets contain lots of realistic human arm/hand trajectories, which make the learned motions look natural. But the downside is that they only cover the regions humans typically reach into. Imagine all the motions of a person sweeping their arm in front of them or picking from waist height – the dataset will be dense there. But what about reaching overhead, or directly behind, or far to the side? Those can be very sparse or absent. Consequently, a policy trained just to imitate those motions might not know what to do if given a craftily placed target outside the “folk dance” region. Agility wanted Digit to be able to reach any point in its functional workspace (for both hands and torso), not just those humans usually hit in everyday living. So instead of i.i.d. sampling from mocap, they uniformly sample target poses. Concretely, the training loop repeatedly picks random 3D positions and orientations for each hand (and likely the torso) every episode. Then it constructs a time-indexed trajectory from the robot’s current state to those random goals (with random speeds). The policy is rewarded for minimizing the distance between its current hand pose and the sampled target over time, as well as maintaining balance. Over millions of episodes in simulation, this ensures that Digit’s arms learn to sweep the entire reachable volume, not just a subset. In effect, it self-generates the demonstration dataset by exploration. The blog explains it simply: “we randomly pick positions and orientations uniformly from the workspace, and generate random translational and rotational movement speeds between those points to create time-indexed trajectories”. This deliberate covering of extremes contrasts with the human-data approach. Agility argues that the mocap trajectories, while gorgeous, leave the corners of the workspace untrained. For true mobile manipulation, you might need the hand at the edge of reach or even beyond a human would go. By enforcing uniform coverage, their “motor cortex” can reliably handle any target in the workspace. It never has to crack open some outer planner just to figure out how to extend the arm that far; it has already practiced in simulation to do exactly that. Position Commands vs. Velocity Commands. As mentioned, they condition on target positions rather than velocities. One way to see why this matters is reward shaping. Matching velocity is an easy, rich signal: if you ask the robot to go at 1 m/s forward, it can get small dense rewards by adjusting its pace. But if you say “go to point X,” the robot only receives reward when it’s getting closer to X, and must deal with the integrator drift. It’s a harder objective to optimize, but directly relevant to the real task. The Agility paper notes that velocity-target policies provide a more forgiving reward (so many papers – e.g. VMP, AMOR, FALCON, ExBody – used that). They also remark that an absolute position target requires good legged odometry or localization to execute on hardware (essentially, you have to know exactly where you are in the world to reach a specific point). These are challenges they accepted. Importantly, the drawback of velocity control is that it forces reliance on something else for positional accuracy: “a higher-level planner or a human teleoperator” must correct the drift, since the controller won’t hold a point if pushed. Agility’s solution is to train for position anyways. They observed that even though it’s technically harder, the benefit is an autonomous controller that seek and hold goals on its own. As they put it: ideally, “we would tell the controller where it should be in free space, and it would navigate there and return naturally to that position even if shoved”. That’s exactly the behavior their LSTM learned: you feed it a coordinate target for the hand or body, and it walks and adjusts the legs to get there, compensating for pushes along the way. Task Space vs. Joint Space Targets. Another innovation in their training setup is the type of targets used for the upper body. Because they sample random end-effector poses, the policy learns to interpret targets as Cartesian coordinates (a 3D position and orientation for each hand/arm and perhaps the torso). Many earlier humanoid controllers instead trained on joint-space targets. For example, if you had MoCap data, you’d use a sequence of joint angle configurations for the whole skeleton as the target trajectory. This means the policy’s input or output must be in joint angle terms, which complicates high-level control. If you want to move Digit’s arm to a new location, you either need a huge joint-space table or solve inverse kinematics first. Agility correctly points out that describing motion in task space (end-effector pose) is far more convenient. In a sense, it abstracts away the exact limb geometry. You can simply say “move right hand to (x,y,z) with orientation R” and the policy will figure out how to step around and reach there. This interface aligns much better with vision and language too: vision models or LLM planners naturally output object locations or positions in space, not raw joint angles. In the blog they note that using free-space positions avoids forcing the planner to “have a model of the robot’s kinematic state”. It also “has the advantage of being much closer to the representation used for state-of-the-art imitation learning methods” and lets the robot take cues from predicted object poses (Gemini-style). In practical terms, this means if a detection network finds a box at (2.3 m, 0.5 m, 0.9 m), Digit can simply be commanded to put its hand there; no intermediate joint-space mapping is needed. This choice (task-space prompts, velocity outputs) makes the foundation model a very flexible substrate for higher-level behaviors. The Learned “Motor Cortex” Policy. Putting it all together, the end result is a compact LSTM network – dubbed Digit’s “motor cortex” – with under a million parameters. It takes as input the robot’s proprioceptive state (joint angles/velocities, inertial measurements, contact sensors, any vision-based observations) along with the desired hand/torso pose targets, and outputs joint commands. (The exact output – torque vs. velocity targets vs. direct joint angles – isn’t spelled out, but presumably they use something like desired accelerations or torques per joint.). The key is that this single network was trained end-to-end in simulation, with no hand-coded sub-controller. Training ran for “decades of simulated time over three or four days” across a GPU cluster. Because they used NVIDIA’s Isaac Sim, they could run millions of parallel episodes: each episode randomizes the task as described and runs until the robot settles or fails. Over and over, the LSTM network collected experience of wildly different scenarios. The blog highlights that Digit’s policy was exposed to incredibly broad variations: they “script thousands of stress-test scenarios (varying aisle widths, floor friction, lighting, and shove forces) to probe every balance-breaking edge case”. They also run a parallel MuJoCo simulator to catch contact physics differences, hardening the policy to modeling errors. (This cross-check step, running the same policy under a different simulator, is a clever engineering trick: any motions that only work in one sim but not physically will show up as failures.). In addition, as [20] reports, because the training was entirely in simulation, the data was enormous and highly varied. They could crank the simulation faster than real time, randomize every possible physical parameter, and ignore wear-and-tear or experimental cost. In fact, it mentions that the team “generated diverse and high-volume data sets, including variations in terrain, object dynamics, and disturbances,” ensuring the LSTM learned robust policies that generalize well beyond exact training conditions. It’s a striking advantage: they essentially did “simulation to infinity” training. The NVIDIA case study even touts “training across billions of simulation steps” to improve Digit’s stability. At the end of this, the policy does transfer directly to the real robot with zero additional training. In the lab, the same network that perfected these tasks in Isaac Sim is run on Digit and behaves sensibly. It obeys the same free-space requests, walks and balances, and coordinates arms and legs just as it learned. Critically, the team reports that it remains safe and stable under loads – for instance, Digit could pick up and manipulate very heavy objects without toppling. (A reward function that covers forceful lifting likely helped here.) In their demos, Digit carried grocery items and even water jugs with little issue. In short, the network trained in the netherworld of simulation becomes Digit’s always-on safety and coordination layer, precisely as a human motor cortex keeps us upright and responsive. In fact, Agility calls it a constant “safety layer,” always correcting and refining motions to prevent falls. This foundation controller isn’t meant to replace all higher-level planning – rather, it sits underneath it. You can imagine the stack like this: at the bottom, Digit’s motor cortex handles the nitty-gritty of balance and primitive motion. Above it, various methods can specify goals: you might have an LLM deciding to “pick up object A from shelf B,” a vision system locating the shelf and giving a 3D target, and a small task planner breaking it into subtasks. These high-level decisions translate into the desired end-effector positions, which become inputs to the motor cortex. The cortex then integrates everything over time, stepping and adjusting as needed to achieve those goals. Effectively, Agility’s system allows you to treat Digit as if it were an IK-controlled arm, letting the learned policy figure out the legwork. The team demonstrated this integration: at NVIDIA’s GTC conference, a live demo had Digit doing grocery shopping. An open-vocabulary object detector watched the scene, sent the 3D pose of a target item up (e.g. a can on a shelf) to the planner, which then triggered Digit to reach there with its arm via the motor cortex. Digit walked to the shelf, grasped the can, and brought it back, all under one neural policy’s domain (other than the vision and planning modules). They even showed it being prompted by what apparently was a chatbot interface (Gemini research preview) – presumably, the LLM interpreted “go get that” or similar into goal positions for the robot. In each case, the core whole-body network handled stability and coordination. In practice, Agility notes that any method of providing targets can be used. You can script them, teleoperate them like a 3D joystick, use a planner or motion capture input, or have an LLM/vision pipeline provide them. The key is that the interface is task-space for end effectors, which any high-level system can compute. Then the motor cortex bridges from that to joint actions. As Agility puts it, they can “prompt our model using various methods to accomplish a diverse set of tasks”. Simulation to Reality (and Back Again). A critical piece of this success story is how much simulation variation they exploited. Both Agility’s own writeup and independent coverage emphasize that they wildly randomized physics, sensors, and scenarios in sim to cover all the “edge cases” of the real world. For example, they randomized floor friction across dramatically different surfaces; threw lighting changes; applied random pushes mid-episode; varied object weights and materials; even tested Digit on moving platforms or uneven ground. Each such random factor was an extra dimension of training data. Over billions of steps, the policy effectively saw every situation the developers could conceive – plus the disciplined exploration meant it was forced to handle whichever the environment threw at it. NVIDIA’s own case study highlights this: they describe feeding the robot thousands of simulated “stress-test” scenarios built from customer floor plans and CAD data. These stress tests included narrow aisles, obstacles, slippery patches, irregular force disturbances – anything to break the robot. Millions of RL episodes then refined the policy under these varying conditions. Moreover, the policy wasn’t just locked to one sim engine. They ran the final controller in a MuJoCo “digital twin” environment as well; differences in contact physics between Isaac and MuJoCo exposed corner-case failures and helped harden the network. Because of this intense domain randomization, the controller became inherently robust to model error and real-world unpredictability. This “simulation-first” strategy paid off. Digit’s motor cortex generalizes to multiple environments out of the box. Agility reports deploying it in two pilot customer sites: a GXO warehouse where Digits pick and sort totes, and an automotive plant (Schaeffler) where Digits load/unload heavy engine parts. In both cases, the same foundation model handles material-handling tasks reliably. For instance, Digit in Schaeffler’s plant is now routinely loading and unloading washing-machine housings – a precise, high-tolerance job that prior robots (or even custom automation) found challenging. The only controller used is the one learned in simulation, just moved to the new site. This real-world success underscores what massive sim training bought them: a controller that can “shrug off bumps, avoid obstacles, and get to work – without costly trial-and-error learning on the factory floor”. One way to see why this is novel is to compare it to earlier sim-to-real attempts. In many robot learning projects, the sim is just for prototyping or initial learning, and some hardware finetuning (or carefully tuned domain randomization) is needed for deployment. Here, by contrast, Agility claims “zero-shot transfer” of this controller. That means no real-world retraining was required; they simply ran the policy and it worked. This is a testament to the breadth of their simulation. It also justifies calling it a foundation model: like GPT-4 is a model trained on massive text to generalize to many tasks, this network is a model trained on massive simulated experience to generalize across environments and tasks. Comparing to Other Approaches. Agility’s “motor cortex” fits into a growing trend of generalist controllers for robots, though it has its own flavor. It is closest in spirit to work like Meta’s Meta-Motivo, which also calls itself a “foundation” model for humanoid control. In Motivo’s case, researchers at Meta (Facebook AI) used an unsupervised RL approach to train a large humanoid agent on motion-capture data and found it could be prompted to do various tasks (walking, goal-reaching, etc.) out of the same policy. The goals there were driven by motion imitation and reward signals. Agility’s model similarly can be prompted with different targets (though Agility explicitly sampled its own task distribution rather than only offline mocap). Both aim for one-policy-fits-all intelligence, but Meta’s is largely for a virtual humanoid and emphasizes fine motion style, while Agility’s is for a physical robot and emphasizes robustness under load. In terms of broad RL philosophy, Agility’s approach echoes the mantra of many: skip the elaborate splitting of locomotion vs. manipulation. For example, prior work by Fu et al. demonstrated a unified policy controlling both the legs and an attached arm of a custom quadruped: they trained end-to-end and the resulting robot could do mobile manipulation (tracking AR tags, coordinating leg bending and arm motion). Similarly, the Agility solution does not stratify footstep planning vs. arm control – the single LSTM learns to do both simultaneously. This trend of “whole-body learning” is also seen in systems like ExBody or CLONE, though each has its quirks (ExBody prioritized agility, CLONE prioritized mobility, etc). The key novelty in Agility’s stack is sparing assumptions: they don’t hard-code dictating foot stays down or separating tasks, whereas many classic controllers do offload locomotion to an outer loop. There are also interesting connections to very new work on multi-robot or multi-embodiment learning. For instance, Dornheim et al. (2024) introduced a “unified morphology” architecture (URMA) that can train one policy to control any of a set of legged robot designs by encoding the robot’s shape into the network. Park et al. (2025) explored taking a foundation policy trained on one robot and fine-tuning it to work on another, by adding sensory modalities (like forcing it to use depth cameras or foot contact sensors differently). These works share the same spirit: build a general, adaptable controller. Agility’s model isn’t explicitly multi-robot, but it could in principle be extended. For example, one could imagine running the same training with random variation in Digit’s limb lengths or mass distribution to make it even more general. The foundation policy could then be “universal” across Digit variants. The idea of a learned “basal layer” for robotics is gaining currency. We see it in builders like CLONE, OmniH2O, HOVER, etc., for legs+arms, and in the notion that future humanoid brains might use large language models at the top controlling a huge repertoire of learned skills (as alluded by Agility’s mention of LLMs). Indeed, Agility explains that the motor cortex is just the beginning: on top of it, one can stack planners, LLM-driven behavior modules, or further dexterous fine-tuners. In this layered vision, Digit’s learned motor cortex is like the spine and reflexes of a human: always active, handling the low-level physics, while higher reasoning modules focus on what to do. In fact, Agility references a survey article (“The Emerging Humanoid Motor Cortex”) by Alan Fern which catalogs RL-trained controllers and hints at exactly this architecture. As the field moves forward, it’s clear that combining robust low-level controllers with flexible high-level AI will be key, and Agility’s work is a concrete step in that direction. Demos and Real-World Impact. It’s worth dwelling briefly on what this enables in practice. A humanoid with such a controller can be given novel tasks on the fly. For instance, if a warehouse manager points out a random crate on a high shelf, a camera-LLM pipeline could compute the 3D coordinates of the crate’s handle. That becomes the target fed to Digit’s motor cortex: “move hand to (x,y,z), then grasp.” Without this foundation model, one would need to solve leg stepping and arm triangulation in every loop. With it, Digit simply walks and reaches in an almost humanlike way. Agility showcased robustness too. In lab tests, Digit could pick up a 10 kg box with one hand without stumbling, because the network had practiced lifting heavy loads. If someone gives it an unexpected shove while it’s holding something, the robot stumbles but then plants a foot and recovers equilibrium. The policy learned exactly those reflexes. Even the planning dancer often noted – “try really hard not to fall over” – became part of the reward during training, instilling in the policy a fear of 0-g. One publicly recorded example was Digit shopping at NVIDIA’s GTC. Vision and object recognition told Digit where the items were, and Digit’s interface received a series of 3D hand targets. It confidently walked aisleways, avoided people in simulation (and thus in reality), and lifted items off shelves. The crowd saw a robot doing human-like grocery tasks, but behind the scenes it was a single neural loop making completeness decisions at every moment. Additionally, in heavy-duty trials, the foundation model was shown lifting large appliances and boxes in manufacturing settings, tasks that previously would have required custom end-effectors or guards. The fact that the same policy handled, say, a plastic milk crate at a grocery demo and a steel pipe at an assembly line, is testament to its generality. Finally, Agility and NVIDIA reports emphasize the ecosystem point: once you have this simulation-driven controller, you can scale humanoid use. They mention robots-as-a-service companies deploying fleets of Digit in warehouses, all using essentially the same learned controller for day one operations. If a new workflow comes along – rearranging zip-tie harnesses on a car assembly line, or unloading cartons in a mall – they expect they only need to provide new target waypoints or put new objects in front of the vision system. The motor cortex remains constant. This is the real promise of a foundation model: not just research demos, but something you install on many robots to bootstrap a whole application domain. Conclusion. Agility Robotics’ new work on a learned “motor cortex” highlights an important path forward for humanoid control. By leveraging massive simulation variation and end-to-end RL, they built a foundation model that anchors Digit’s whole-body in a robust, adaptable control policy. This removes many classic limitations – the feet can now adjust when needed, the reaching range is fully exploited, and disturbances are handled reflexively. Meanwhile, the interface to this foundation is deliberately kept simple (Cartesian hand/torso goals), making it broadly usable by vision systems, LLMs, and planners on top. In effect, Agility has created a general-purpose locomotor-manipulator that can be prompted to do low-level tasks much like higher-level “foundation models” in language and vision are prompted to do cognition tasks. Of course, this is just Step 1. The policy is under 1M parameters and designed for reaction-level control. On top of it, Agility and others will build specialized behaviors: plan a complex assembly, hand off tasks between robots, further refine dexterous grips, etc. They’ve also hinted that future work will use this cortex as a basis for learning more in simulation – for example, letting each deployed robot watch a new demonstrative run of a task, then practice it internally before execution. The same idea could apply: the foundation locomotor stays fixed, while a higher-level net addresses the novel object manipulation. In robotics terms, this is akin to “finetuning the brain for each job” but in simulation. In summary, Agility’s foundation model is an exciting result because it shows that general-purpose humanoid control is within reach through learning. The field has seen many specialized controllers and policies; this is a demonstration that a single unified policy can handle a broad span of tasks and situations. For roboticists, it means thinking differently about control design. Instead of careful mode-switching logic, we can rely on a versatile learned controller and focus on better ways to set its goals or adapt it to new bodies. We’ll be watching closely how this line of work evolves – it could well be that future humanoid fleets run essential parts of their “brains” on networks like this, just as modern AI stacks increasingly rely on pre-trained foundation models. For those who want to dive deeper, Agility suggests checking out a recent survey by Alan Fern on RL-trained humanoid controllers, “The Emerging Humanoid Motor Cortex,” which catalogs many of these ideas and their trajectory. The idea of treating a robust learned policy as an embodied “motor cortex” is catching on. As Agility puts it, Digit’s motor cortex is “an always-on safety layer” – something that every future human-centric robot will need before venturing out into our world. That’s the path we’re on: giving robots a learned body intelligence that feels more natural, more human-like, and ultimately more capable. References: The ideas in this episode come from Agility Robotics’ August 2025 blog post “Training a Whole-Body Control Foundation Model”, related conference summaries, NVIDIA’s case studies, and recent papers on humanoid RL and foundation models. These sources detail how small LSTM controllers, trained across billions of simulated frames with rich perception inputs, are emerging as a “motor cortex” for whole-body robot intelligence.