Latest / Embodied AI 101 / Learning Unified Force and Position Control for Legged Loco-Manipulation

Transcript

Learning Unified Force and Position Control for Legged Loco-Manipulation · Embodied AI 101

• Learning Unified Force and Position Control for Legged Loco-Manipulation. Imagine a legged robot with an arm pushing open a heavy door. It doesn’t simply follow a predefined path in space – it must lean its body, adjust posture, and apply just the right amount of force without tip-over. In such loco-manipulation tasks, the robot must integrate contact forces with its motion. Traditional RL-based policies excel at moving limbs and bodies to follow positions or trajectories, but they typically ignore direct force control. The new work by Zhi et al. (2025) proposes a single RL policy that handles both position tracking and force application together for legged robots, without any dedicated force sensors. In simulation, the policy encounters a variety of position and force commands (and random disturbances), and learns to infer the needed contact force from its own motion history. The result is a versatile controller that can switch between pure position control, pure force control, impedance (compliance), or any hybrid smoothly. On real robots – a Unitree quad with an arm and even a humanoid – it enables behaviors like pushing with a precise force, yielding compliantly when bumped, and robustly executing contact-rich tasks. Beyond direct control, the learned “virtual force” also makes the robot a better teacher. Because the policy internally estimates contact forces, the researchers use it to collect demonstrations for imitation learning in tasks like wiping a board or opening drawers. Incorporating the estimated force into the demonstration data yields a robot that learns these tasks far better than vision-only baselines – on the order of 30–40% higher success rates in their tests. In short, this work shows how a legged manipulator can perceive and use contact force purely through proprioception and RL, granting it a kind of touch-sensitivity without extra hardware. Background and Motivation. Modern legged robots (quadrupeds and humanoids) have started to combine running, climbing, and even using arms to interact with the world. Research labs have demonstrated impressive locomotion and even some manipulation – jumping obstacles, body-assisted object pushes, or simple pick-and-place with arms while walking. But most of these learned policies were really just controlling positions or velocities; they implicitly hope that any force interactions work out, rather than explicitly commanding forces. In contrast, many real-world tasks like opening doors, turning cranks, or wiping surfaces demand explicit control over contact pressure and compliance. If the robot treats a heavy door “as air” and only cares about moving its arm to the handle pose, it will either fail to open it or slip and fall over. As the authors note, “in the physical world, interacting forcefully with the environment requires force control” – something that ordinary trajectory-following RL policies simply don’t do. Classical robotics has a well-known remedy: impedance or hybrid force/position control. These controllers adjust the target position dynamically based on the desired force, essentially treating the end-effector as if attached to a virtual spring or damper relative to the task target. Early work from the 1980s already described hybrid-position/force approaches and impedance control as a way to reconcile motion and contact forces during manipulation. More recent learning-based controllers have also used this insight: some RL systems allow commanding a springiness (compliance) or track forces, especially in fixed-base manipulators. In mobile manipulation, there have been whole-body controllers (some model-based, some learned) that implicitly handle forces, but often they require precise dynamics models or force sensors. On the learning side, only a few recent papers explicitly tackled force control in legged robots. For example, Portela et al. (2024) learned an RL-based force controller for a quadruped with an arm, allowing a human operator to command a target contact force. That policy could, for instance, grip a heavy object by commanding a force upwards or assist in balancing, all without any force-torque sensor on the wrist. However, it treated force control separately from positional goals: you could command a force to be applied but not simultaneously specify a desired position. In that sense, Zhi et al.’s unified policy goes a step further. It can take both a motion target and a force target as inputs, and the single neural policy will achieve them together. Meanwhile, many teams have focused on learning whole-body loco-manipulation policies (e.g. controlling legs and arm together). These approaches typically mix locomotion and manipulation by, for instance, first learning to walk and then coordinating the arm to reach anywhere in space. RoboDuet (Pan et al. 2024) is one such multi-stage system that enables a quadruped to roam and use its arm broadly by training in phases. But again, those works care about following poses all over the body’s workspace, not the force at those contacts. On the imitation learning front, the robotic vision-and-trajectory datasets (like RT-2, RX, BridgeData) are growing massive, but they generally record just joint angles and end-effector poses. They lack force feedback because most robots don’t have wrist sensors. The usual supervised learning pipelines simply clone the motions, ignoring contact forces. For many tasks – say dragging a marker on a board – that can be enough. But for others, like ensuring steady contact or avoiding applying too much pressure, vision alone simply doesn’t capture the needed data. The authors explicitly show that without force information, even a task as simple as wiping a blackboard fails most of the time. By teaching the robot to estimate force, this work fills that data gap. In short, this paper bridges two gaps: (1) it brings force control into learned loco-manip policies for legged robots, and (2) it uses the result to generate better imitation data for forceful tasks. It does so via a clever blend of classical impedance ideas and modern RL. Unified Force/Position Control via Impedance. The key insight is to view force and position control in one unified “impedance” framework. In classical impedance control, one assumes a virtual spring-damper between the robot’s end-effector and a desired position. If the robot tries to move toward the target but is blocked or pushes, the spring law $F = K(x - x_{\rm des}) + \dots$ means a restorative force is generated. Zhi et al. use this idea to translate a commanded force into a commanded displacement. Concretely, if the policy is told to apply a force $\mathbf{F}^{cmd}$ at the end-effector, it doesn’t literally output torques from a force sensor. Instead it adjusts the target position so that a spring of stiffness $K$ would produce that force. Mathematically one derives a “virtual target” position: $$ \mathbf{x}^{target} = \mathbf{x}^{cmd} + \frac{\mathbf{F}^{ext} + (\mathbf{F}^{cmd} - \mathbf{F}^{react})}{K},, $$ where $\mathbf{x}^{cmd}$ is the nominal pose command, $\mathbf{F}^{react}$ is the passive reactive force (from environment contact), and $\mathbf{F}^{ext}$ are any other disturbances. In words, the target is offset from the commanded pose by an amount proportional to the net desired force (divided by stiffness). If the commanded force is zero, this formula reduces to pure position control. If the position command is fixed while $\mathbf{F}^{cmd}$ is nonzero, the target moves so that the controller will push as needed. By varying $\mathbf{F}^{cmd}$ one slides between pure motion and pure force control; setting both simultaneously yields true hybrid behavior. Importantly, Zhi et al. apply the same idea not only to the arm but also to the robot’s base velocity. The base is commanded a velocity $\mathbf{v}{base}^{cmd}$, and we may also want the robot base to exert or resist an external push horizontally. They model the base with a damper: $$ \mathbf{F}{base} = D(\mathbf{v}{base} - \mathbf{v}{base}^{des}),, $$ and similarly derive a velocity target that includes force compensation: $$ \mathbf{v}{base}^{target} = \mathbf{v}{base}^{cmd} + \frac{\mathbf{F}{base}^{ext} +(\mathbf{F}{base}^{cmd}-\mathbf{F}_{react,base})}{D},. $$ In practice, this means if someone pushes the robot’s body, the controller will command a velocity to move exactly enough to counteract the push. As noted in their experiments, this can cause a humanoid to “halt and lean” when pushed, or make a quadruped start walking forward if knocked, even without any user velocity command. Rather than programming these calculations by hand, the team embeds them into the RL policy’s reward structure and training setup. Effectively, the policy learns to respect the above dynamics equations in its behavior. By training under diverse commanded positions, forces, and disturbances, the RL algorithm internalizes the coupling of $(x,F)$ in the learned controller. Policy Architecture and Training. The learned policy runs end-to-end in the low-level controller. Its inputs are purely proprioceptive and the high-level commands: at each timestep $t$, the observation $\mathbf{o}_t$ includes the robot’s current base orientation and angular velocity, all joint positions and velocities, the previous control action, and the current commands ($\mathbf{c}_t^{cmd}$) issued by the user or high-level planner. The command vector $\mathbf{c}_t^{cmd}$ bundles together things like the desired base velocity, the desired end-effector position, a desired end-effector force, and a desired base force. (For a quadruped with an arm, all these are present; for a humanoid without a gripper, only the base velocity and base force commands are used). The network architecture has three parts. First, an encoder compresses the recent history of observations (they use 32 timesteps of memory) into a latent feature. This temporal context is crucial: without any force sensors, the robot must infer contact forces from observed deviations and motions over time. The encoder’s output is fed to two heads. The state estimator head predicts the unobserved external force acting on the end-effector (and optionally base) as well as the end-effector’s true position and the base velocity. Think of this as a learned “virtual force sensor.” If the arm was commanded to move and suddenly doesn’t, the estimator learns to say “Ah, an external force must be opposing me.” That estimated $\mathbf{F}^{ext}$ is then treated as if it were an input to the controller. The other head is the actor which directly outputs the target joint positions (which will be converted to PD motor commands). Training uses Proximal Policy Optimization in high-speed batches. The team used NVIDIA’s Isaac Gym simulator with 4096 parallel environments to gather data. In each simulated rollout, the robot gets randomly sampled position/velocity and force commands (within a range). For instance, arm position commands are sampled in a sphere around the torso, force commands up to ±60 N in each axis, and base velocity/force commands similarly randomized. On top of the commanded inputs, they also inject random external disturbance forces on the end-effector and base (again up to ±60 N). These forces are applied via a schedule: they ramp up, hold for a bit, then ramp down, forcing the policy to handle transient contacts and releases. Domain randomization is applied too: friction, body mass, motor strength, etc. are jittered (Table A.2 in the paper). The reward explicitly encourages two things at once: precise tracking of the effective targets (end-effector should go to $\mathbf{x}{ee}^{target}$ as computed above, and base to $\mathbf{v}{base}^{target}$ under force) and accurate force estimation. Concretely, they reward exponential penalties on errors in end-effector position and base velocity relative to their compensated targets (see Table A.1). They also penalize collisions, joint limits, and excessive torques for safety and smoothness. By training in two phases (first without force tasks to learn basic mobility, then with the full variety of commanded forces), the final policy becomes adept at hybrid behaviors. Critically, no real force sensor is used anywhere. All “knowledge” of contact comes from joint movements, body accelerations, and the learned estimator. By covering such a wide range of commands and disturbances in sim, the policy gains the ability to, for example, automatically back off if pushed too hard or maintain a given force if commanded. Behavior and Control Modes Demonstrated. Once trained, the unified policy exhibits a spectrum of useful skills. By simply changing the inputs, we can see it perform both traditional and novel behaviors:. Pure Position Tracking: With zero force commands and in free space, the robot behaves like a normal position controller. In simulation tests, when asking the arm to follow random trajectories (and not applying any external push), the end-effector stays within about 0.1–0.15 m of the target at most (with most error along the slightly constrained Y-axis). The internal state estimator’s position error stays under ~5 cm. So the policy learned good positioning as a baseline. Force-Control to Hold Constant Force: When commanded to apply a non-zero force in a fixed direction, the robot will move until the actual contact force matches the command. For example, the paper shows a heavy 2.5 kg dumbbell hanging from the arm. By issuing a +25 N upward force command (with no explicit position goal), the end-effector drifts upward until gravity is counteracted – effectively holding the dumbbell up. Without the force command, the same weight would simply make the arm droop. This demonstrates gravity compensation: the robot “imagines” a target above the dumbbell to pull up and match the commanded force. Zero-Force Tracking (Compliance): A special case of force control is setting the commanded force to zero. In this mode, the arm will just follow whichever external force is pushing it. If you push on the end-effector, the robot moves along with you; when you let go, it stays in the last pushed position instead of snapping back. In other words, it yields softly to any external push, like a stiff spring that’s been released. The result is a compliant mode where the end-effector floats when undisturbed but yields under load. Hybrid Impedance Control: If both a position target and a nonzero force command are given, the policy blends them. For instance, it can hold an end-effector at a point while resisting pushes as if it were attached by a spring. Think of rigidly standing at the door handle but giving away a bit if you pull too hard. They demonstrate this by a kind of tug-of-war: the robot keeps trying to reach a fixed target, but when a human pushes or pulls the end-effector, the robot’s resistive force increases with displacement (like a spring). Further from the goal, it pushes back harder – classical impedance behavior. Base Force Response: The unified scheme isn’t limited to the arm. If the robot’s body (base) is pushed, the controller can cause the robot to move or adjust posture. In one test, giving a base force command causes the robot to walk forward to apply an opposing force. More interestingly, if an external push is applied to the robot’s body, the policy can generate a base velocity to resist it. For example, they show the robot’s body being nudged: the quad starts shuffling forward even with no velocity command, and the humanoid stops and leans back when one tries to push it (essentially countering the push). This “force tracking” on the base enables intuitive human guidance – you can push the robot around and it will compliantly follow. Each of these behaviors arises automatically from the one trained policy: by simply setting the desired $\mathbf{x}^{cmd}$ and $\mathbf{F}^{cmd}$ inputs appropriately, the same neural controller realized all of the above. The experiments include real-world demonstrations too: for instance, in the lab they had the robot hold up a weight (force control), walk forward when the robot body is kicked (body force tracking), and play tug-of-war with a person (impedance). The supplemental videos from the authors show clean execution of these skills, confirming the single learned policy can smoothly switch modes without mode switching logic. Quantitatively, the policy’s force estimation proved good enough for the tasks. In sim it tracks commanded positions with and without forces to similar accuracy. On real hardware, they tested by commanding e.g. 0–60 N along each axis and measuring the actual force with a dynamometer. The estimated vs true forces stayed within about 5–10 N consistently (a small fraction of the scale), even though they never trained on real data. There is some sim-to-real gap (the Y-axis performance was slightly off), but for most loco-manip tasks that level of force error is acceptable. Force-Aware Imitation Learning. A particularly novel aspect of this work is how the learned policy is used to improve higher-level imitation learning. In many practical robotic systems, one might want to record a human demonstration and then train a “generalist” policy to imitate those behaviors (potentially in response to vision). Since force is critical but usually unmeasured, Zhi et al. use their policy as a data curator:. Teleoperated Data Collection: A human operator remotely drives the robot through tasks using a joystick or smartphone interface. Critically, the learned low-level policy is in the loop: the human issues mixed position/force commands to the arm while the two-stage low-level controller executes them. The system records sensory data along with the estimated contact force from the policy’s internal state estimator. In short, each demonstration trajectory now has not only joint angles and images, but also an estimate of the contact force at the end-effector at each time. Train a Force-Aware Policy: They then train a diffusion-based imitation policy (a recent generative-policy technique) on this teleoperated dataset. The imitation network takes the robot’s current state, camera images, and the recorded force estimate as input, and outputs both a position command and a force command for the low-level policy to execute. In effect, the imitation policy learns to suggest how much force to apply at each step, guided by visuals (and now by the force hint). They evaluate this on four contact-rich tasks on the real robot: wiping a blackboard (sliding the arm while maintaining pressure), opening a push-to-open cabinet (pressing a door just enough to trigger the latch), closing the same cabinet, and opening a drawer with an occluded handle. Each task requires nuanced force; without knowing the forces, a vision-only imitating policy tends to fail. The results are striking. Across all tasks, the force-aware imitator vastly outperforms the baseline that ignores force. In the wiping task, the success rate jumped from only ~22% (vision-only) to 58% with force. In opening the cabinet, it went from ~36% to 70%, and closing from 30% to 72%. The hardest case was the occluded drawer: baseline vision alone succeeded only 30% of the time (it simply doesn’t see the handle at all past the edge), whereas the force-aware policy shot up to 76% success. Overall, this is a ~39.5% absolute increase in success (nearly doubling performance in many tasks). Why such large gains? The examples help illustrate: when wiping, a vision-only policy often loses contact and fails to clean off ink, or else it forces too hard and scratches the surface. The force-aware policy, however, can sense when it’s pressing too little or too much and adjust to keep the brush on the board with consistent gentle pressure. For the cabinet, the latch moves only a few millimeters under force – a visual system usually misses this tiny motion. But the learned contact force estimator can “feel” when the door resists and stops pushing (the policy then backs off at just the right time). In the drawer case, the handle is entirely hidden behind the robot’s gripper partway in. The baseline blindly pushes and likely overshoots or sticks. The force-savvy policy, however, feels the contact bump and knows it’s hit the front of the drawer even with no sight, then applies and releases just enough force to pop it open. These improvements clearly highlight the value of having contact information during training. By using the RL policy’s estimator as a “virtual force sensor,” the team generated a dataset that taught the imitator when and how much to push. The authors argue this illustrates a general pipeline: use learned force-position policies to collect force-labeled demonstrations, enabling downstream tasks in the real world even when no physical force sensors are present. Cross-Robot and Robustness Tests. Though the policy was trained on one particular quadrupedal robot (Unitree B2 with Z1 arm), Zhi et al. also tested it on a very different robot: a new Unitree G1 humanoid. Without retraining, the same unified policy (with a human in the loop) managed to control the humanoid in analogous ways. For example, the base force-feedback behavior emerged equally well: a lateral push to the humanoid’s body caused it to halt and lean to balance, exactly as predicted by the base impedance rule. Likewise, the quadruped would step forward against pushes (Fig. 5d), and similar compliance appeared. This cross-embodiment success suggests the approach captures a general physical principle rather than a robot-specific gimmick. In terms of robustness, the experiments show the policy handles a wide range of conditions. Because training included significant domain randomization (friction, mass, motor strength, even simulated pushes on the robot), the learned controller tolerates reasonable differences between sim and reality. In the real force tests, errors in estimation were relatively stable. Of course, the authors note some limits. The force estimator degrades when interactions are either very fast/high-frequency or when the contact point is near the edge of the manipulator’s workspace. There is also a small sim-to-real mismatch: along one axis the force was slightly under-estimated, likely due to unmodeled hardware details. These are acknowledged as issues to refine. Finally, the paper clearly identifies future steps. One exciting direction is multi-point force control. Currently the policy treats the arm and base contacts independently; but real robots can brace with one contact while pushing with another. For example, a quadruped might press its body against a wall while using the arm to push a lever. Coordinating multiple force points would require extending the model, perhaps estimating forces on several contacts and commanding them jointly. Zhi et al. leave this for future work, but it is a natural extension of the unified view. Conclusion. In summary, this work presents the first learned whole-body controller for legged robots that unites force and position control in the same policy. By casting force commands as adjustments to target positions (like a virtual spring), and embedding that in an RL framework, the resulting policy smoothly does position tracking, force control, compliance, and their hybrids. Remarkably, all this is achieved without any hardware force sensor. The policy’s embedded “state estimator” learns to feel the world through proprioception, estimating contact forces just from the robot’s state history. This development has two major implications. First, it greatly expands the range of manipulatable tasks for legged robots. The same quadruped or humanoid can now approach a panel, wipe it gently, push open a door, or yield safely to a collision – by simply choosing different force/position inputs. Second, it enhances data-driven learning: by generating force-annotated demonstrations, the system allows high-level policies to learn delicate manipulation tasks that vision alone could never master. The ~40% boost in task success is a testament to this. Of course, challenges remain. Force estimation at the limits of control remains noisy, and adapting to sharper impacts or very fast events could use further work. The sim-to-real gap is still a concern, though the shown results are encouraging. And multi-contact scenarios will demand richer models. But by teaching a robot to “feel” through its actions, Zhi et al. have effectively given it a new sensory modality. As the authors conclude, this lays ground for robots that interact not just by seeing, but by sensing their environment in a more human-like way. Future robots may thus not only see where they go, but also touch as they go, all learned through simulation and policy optimization rather than extra hardware. Key References: The reader can view Zhi et al.’s full paper for details, including reward terms and network architectures. For background, see the related works on whole-body mobile manipulation and learned force control (e.g., Portela et al. 2024 for an earlier force-commander RL approach, and Pan et al. 2024 for staged whole-body policy training). The broader context includes classical impedance control and many recent “vision-language-action” policy works, which this paper extends by adding a force dimension.