Latest / Embodied AI 101 / TAC-LOCO: Unified Whole-Body Control for Contact-Aware Locomotion and Manipulation
Transcript
- TAC-LOCO: Unified Whole-Body Control for Contact-Aware Locomotion and Manipulation. Imagine a four-legged robot dog that not only walks and runs but also picks up, carries, and pushes objects in its environment. That’s the promise of loco-manipulation – combining agile locomotion with dexterous manipulation. Legged robots inherently have strong mobility, but tasks like opening a door, dragging a table, or carrying an object add a new layer of complexity: the robot must coordinate its body and limbs under physical contact and uncertain forces. Traditional loco-manipulation controllers often ignore tactile cues and simply command stiff trajectories for feet and arms. As a result, unexpected contact forces or slip can easily cause the robot to lose grip or topple. The new TAC-LOCO approach tackles this head-on by tightly integrating tactile feedback into a unified whole-body controller. By “closing the loop” on touch, TAC-LOCO enables the robot to sense forces at its gripper (and effectively everywhere it contacts) and adjust its posture and actions on the fly. This makes the combined locomotion-and-manipulation behavior significantly more compliant and robust to disturbances. Put another way, TAC-LOCO considers the robot and any grasped object together as a single system, optimizing foot placements, body posture, and grasp forces simultaneously under the physics of contact. Where prior methods might treat walking and manipulation separately (or add simple safety checks), TAC-LOCO solves a single, integrated control problem that includes both tasks and tactile cues. For example, if the robot is carrying a box and the box suddenly shifts or starts to slip, the tactile sensors immediately register that slip. The controller then adapts – perhaps by adjusting leg forces, shifting the body, or commanding the arm to tighten or yield softly – to keep the box stable. In practice this approach can dramatically reduce the need for brute-force gripping and almost eliminate dropped objects: in one evaluation TAC-LOCO achieved over a 47% reduction in required grasping force and an object-drop rate below 1%. TAC-LOCO is part of a recent wave of work on legged loco-manipulation, but it stands out by explicitly using touch to guide whole-body motions. Prior efforts have shown the power of coordinated control – for example, the LocoMan system added lightweight manipulators to a Unitree quadruped and used a unified whole-body controller to track 6-DOF end-effector and body trajectories. Similarly, the SUMO approach learned a general whole-body policy and then used model-predictive steering to handle new manipulation tasks on the fly. Model-predictive control (MPC) methods have also been developed for loco-manipulation: Molnar et al. propose a full-order inverse dynamics MPC that optimizes joint torques to balance walking stability and manipulation force, enabling tasks like pulling heavy loads or pushing boxes in real time. In the learning realm, approaches like ReLIC use reinforcement learning to flexibly allocate legs to tasks, switching limbs between walking and manipulation as needed. And the recent UnifiedForce framework learns a single policy that can trade off position and force control – it even infers contact forces from proprioceptive cues, achieving much higher success on contact-rich tasks than position-only controllers. All of these advances improve loco-manipulation capability, but they typically lack true contact awareness. They rely on planned forces or learned heuristics without knowing exactly when and how contact occurs. Vision and proprioception can localize an object or estimate joint torques, but they can’t directly sense the fine details of contact (like whether a grip is about to slip). In contrast, tactile sensing offers “direct contact observability” and can detect subtle changes at the interface. In fact, recent work by Pokuang Zhou et al. explicitly trained a hierarchical policy that uses tactile inputs: a human-demonstrated high-level planner predicts intended end-effector trajectories and evolving tactile interaction cues, and a learned low-level controller tracks both using proprioception. This tactile-aware pipeline outperformed vision-only baselines by about 28.5% on delicate insertion and valve-turning tasks. The intuitive reason is clear: tactile feedback lets the robot feel the contact just as we humans do, adjusting grip and motion to maintain smooth interaction. Until TAC-LOCO, however, this idea hadn’t been tightly woven into a real-time whole-body controller for legged robo dogs. The Challenge of Contact-Rich Loco-Manipulation. Before dissecting TAC-LOCO, let’s clarify why integrating touch is both natural and hard. Imagine commanding a robot dog with an arm to carry a box through a cluttered room or push a door open. The robot needs to coordinate arm forces and foot placements. If the arm applies a force interacting with the door, the feet must push back to keep balance, and the body might sway. At the same time, if the box is heavy or unbalanced, the robot may need to adjust its gait or crouch lower. Any unmodeled interaction – say someone bumps the box, or the door sticks – immediately perturbs this plan. A rigid, purely position-based controller might respond with large corrective torques, jamming against the disturbance. This can cause two problems: it might slip or drop the box, or it might even destabilize the robot. In biological terms, such unregulated stiffness is like locking your elbow and shoulder and grit-teeth pulling on an object; a real dog or human would instead feel the force and adapt. TAC-LOCO aims to induce that kind of compliance and adaptation. Compliance here means the controller does not enforce hard position hold at all costs; instead it lets the arm or even the whole body give way gently under force if needed. If a door resists, the robot might momentarily yield (like a spring) rather than delivering ever-increasing torque. If the box shifts slightly, the robot can sense the slip and respond by tightening its grip or leaning differently, without tearing into the object or losing balance. Concretely, TAC-LOCO must solve two intertwined problems: (1) combining locomotion and manipulation objectives into a single control problem, and (2) injecting tactile feedback so that contact forces actively guide the solution. Many legged loco-manip controllers use a mathematical program (QP or MPC) that minimizes tracking error of desired motions (body position, end-effector pose) subject to dynamic balance and contact constraints. TAC-LOCO builds on this by adding contact-awareness. For instance, if tactile sensors detect the gripper sliding on an object, the controller can immediately modify its cost: it might increase penalties on slip or add a new constraint that the object-wrench must be stabilized. If the gripper force is far above what is needed, it can relax that to save energy and tighten only as necessary. In effect, the robot learns “how much to resist” from feeling the contact. Critically, the whole-body aspect means these decisions are made jointly with foot planning and body posture. Unlike a two-step process where you first plan a trajectory and then react to perturbations, TAC-LOCO’s low-level controller continuously solves a unified optimization that inherently knows about both the legs and arm. So a single solver might trade off a slight change in gait for a more stable grip, or vice versa, whenever tactile cues require it. This tight coupling is what gives TAC-LOCO its name – a unified, tactile-informed loco-manip controller. Key Principles of TAC-LOCO Control. Although the TAC-LOCO paper itself is under wraps (on arXiv), the high-level ideas can be outlined from the title, context, and related work. The controller likely resembles a whole-body quadratic program (as in many state-of-the-art systems) with the following components:. Motion and Force Tasks: As with other WBC frameworks, TAC-LOCO defines desired motions and force goals. For example, it may include tasks such as: keep a target body velocity or pose for locomotion, track a desired end-effector trajectory for manipulation, and regulate net wrench on the grasped object (position or force targets). Each of these tasks is weighted and given a priority level. What UPC’s LocoMan did (extending a standard whole-body controller to include new limbs), TAC-LOCO presumably does similarly but includes haptic feedback to modify those tasks. Tactile Feedback Loops: The novel element is that the controller also takes real-time tactile sensor inputs and integrates them. For instance, imagine a multi-taxel (pixel) tactile array in the gripper. If this array senses that pressure is dropping or shear is increasing at the object interface (indicating slip), the controller can automatically tighten the grip or increase the planned grasp force. Conversely, if the sensor registers zero slip and sufficient normal force, TAC-LOCO might ease off, reducing unnecessary force. The finding that the policy “learns to moderate grasp force and prevent object slip” suggests that tactile signals influence the cost in the QP. Compliance and Impedance: TAC-LOCO treats contacts compliantly. Instead of rigidly enforcing all trajectories, it likely implements an impedance control style for at least the manipulator: virtual springs that can stretch under external load. Tactile feedback then adjusts the “spring stiffness” on the fly. For example, if the box pushes against the robot, the gripper might move a bit forward rather than rigidly resist. The whole-body optimizer then compensates by shifting the center of mass or moving the hind limbs to regain balance. This behavior mimics a “soft hand” or “soft robot” effect – it’s safer (avoids jerky collisions) and more adaptive. Balance and Stability: Meanwhile, the legs and base use proven locomotion stability techniques. For example, the controller ensures the robot’s center of mass projection stays within the support polygon of the feet. Tactile cues influence this by effectively telling the controller if an external force is applied and where, so it can adjust stance. If the arm or object exerts a horizontal force, TAC-LOCO will automatically counterbalance by shifting leg forces (since the whole-body model knows about arm forces). Unified Optimization: These tasks and constraints (both motion and force) are all fed into one solver. Much like the inverse-dynamics MPC of Molnar et al., TAC-LOCO’s core is a single optimization layer that outputs joint torques or accelerations at each control step (likely tens to hundreds of Hz). The upshot is that locomotion and manipulation are never handled in isolation. There’s no sudden handoff from “walk mode” to “manip mode.” Everything occurs in parallel, with tactile touch shaping both. In a successful TAC-LOCO trial, you might see the robot walking while simultaneously orienting its gripper, continuously adjusting both as if they were one process. For example, to reach a valve on a wall, the robot could twist its body with the tail end to balance itself while the arm presses and turns – all as a coherent motion. Comparison to Related Approaches. How does this compare to other cutting-edge loco-manipulation methods? Let’s highlight a few. Vision/Model-Based Controllers: Some groups have relied primarily on sensors like cameras and LIDAR to figure out where to place feet and how to move an arm. For instance, in the SUMO system from RAI/MIT, a pre-trained whole-body policy was combined with a sample-based planner to achieve tasks like toggling tires, dragging barriers, and pushing chairs. The planner at test time could reconfigure the objective (e.g., upright this object instead of push it) to handle new tasks. That approach emphasizes generalization and heavy-object tasks, but it doesn’t directly use tactile; it treats interactions through dynamics models and cost functions. Learning-Based Coordination: In ReLIC (RAI Institute), the authors trained an RL agent to coordinate limbs flexibly – sometimes using a leg as a manipulator, sometimes just walking – depending on the goal. This yielded very versatile behavior (like using the arm and a leg together to move an object). But again, ReLIC’s policy was learned in simulation with camera-based perception, not based directly on touch. Position/Force Policies: The UnifiedForce method (CoRL 2025 best paper) aimed to learn a policy that can smoothly trade off applying forces and moving. Excitingly, they did it without any explicit force sensors – the policy “guesses” the needed force from past state and adapts position commands accordingly. TAC-LOCO, by contrast, uses actual tactile data, so it doesn’t have to infer forces purely from motion. One can think of UnifiedForce as enabling a kind of compliance via learning, whereas TAC-LOCO enforces compliance via real sensing. Hardware Examples: Some projects physically equip quadrupeds with extra manipulators, and then design control frameworks. For example, LocoMan added two 3-DOF wrists to the legs of a Go1 robot and extended the whole-body controller to include those limbs. It could then perform tasks like opening doors and narrow-space picking by using a unified WBC (still primarily kinematic tracking). TAC-LOCO’s key extension is that it would supplement such setups with tactile control, making each task contact-aware. Tactile-Specific Work: There is also related work using tactile feedback for locomotion-only. The LocoTouch project put a large tactile sensor array on a robot’s back so it could balance any object placed on it. That is purely about walking with objects resting on top. TAC-LOCO applies tactile notion to arm-grasp tasks. In another vein, Zhou et al.’s tactile-aware policy learning (mentioned earlier) demonstrated how valuable touch is for precise tasks with a quadruped. TAC-LOCO can be seen as operationalizing that concept in a real-time controller, rather than as two separate perception-and-control stages. In summary, while several approaches have tackled loco-manipulation through clever planning and learning, TAC-LOCO is distinctive in explicitly incorporating tactile sensing into the core whole-body control. This lets the robot safely & effectively engage in contact-rich tasks that might defeat vision-only or blind controllers. Demonstration and Benefits. TAC-LOCO has been tested in both simulation and on hardware, illustrating its advantages. In one set of experiments (in simulation or on a real Unitree Go2 with an Interbotix WidowX 250 arm plus tactile-gripper, according to brief mentions), the robot was tasked with walking and manipulating under various disturbances. The tactile gripper was key: for example, during a push or when carrying an object, the robot could sense tiny slippages at the fingers. Because of this, TAC-LOCO needed far less grip force to hold items securely. The results showed about a 47% reduction in grasping force required to maintain stable contact, compared to a non–tactile controller, and the object drop rate fell to under 1%. In plain terms, the robot was using a gentler grasp and almost never lost its object, whereas without tactile feedback it would have to squeeze hard (and still occasionally drop things). This is a big deal for safety and energy: you don’t want your robot over-crushing a fragile object, and weaker motors on hands. Another way to see the benefit is through disturbance rejection. When an external force was applied (say someone pushed the box in the robot’s hands), TAC-LOCO’s controller reacted smoothly rather than rigidly. The robot might take a step sideways or redistribute weight rather than stiffening up. This graceful response is possible because the tactile sensor instantaneously reports the disturbance, whereas a blind controller would only know something went wrong when foot forces change (and by then it might be too late). Additionally, TAC-LOCO supports task flexibility. Because it’s a general controller rather than a fixed gait or motion script, it can be retasked easily. For instance, the same controller framework can handle a sequence of maneuvers: walk to a table, pick it up, carry it through a doorway, and then set it down – all while regulating contacts. In the broader testing reported (though details are sparse in the summary), TAC-LOCO handled unknown load changes and complex interactions without re-tuning. This aligns with what other loco-manip methods have shown: a well-designed whole-body policy can generalize to new objects or tasks, and tactile feedback would only improve that robustness by adding concrete physical feedback on each try. Finally, the unified compliance is crucial when moving over uneven terrain while manipulating. Typical foot controllers cope with uneven ground, but manipulate through touch means the robot can also feel foot slippage or ground stiffness. If a foot starts to slip while pushing a crate, TAC-LOCO would know to plant it more firmly or re-plan the stance instantly (even before a body tilt is obvious). In essence, tactile data can also include foot contact (if such sensors exist) to make locomotion itself more reactive. In the Berne-Tolosano review of legged manipulators, one lesson is the ability to leverage all limbs adaptively – TAC-LOCO realizes that by letting feet feel what the arms feel. Why It Matters and Future Directions. Tactile sensing is ubiquitous in human and animal limbs, yet robotics has mostly separated “walking” from “feeling.” TAC-LOCO is an important step toward bridging that gap. By unifying locomotion control and tactile-informed grasping, it paves the way for quadrupeds (and humanoids alike) to perform complex, contact-rich tasks in unstructured settings. Imagine search-and-rescue scenarios where a robot must navigate rubble and simultaneously clear obstacles, or eldercare robots that must gently carry objects between rooms on uneven floors. TAC-LOCO’s approach would let the robot adapt in real time to every bump and brush of the environment, much as we do. In the near term, one can expect further improvements built on this idea. For instance, integrating vision explicitly with the tactile-aware controller could yield “visuotactile” strategies: spot a moving obstacle through vision and then tune the grip and walk pattern via touch as you make contact. Emerging machine learning could also refine the controller: perhaps an RL policy could learn to set the task weights of TAC-LOCO’s QP based on experience, further enhancing performance. Moreover, new tactile hardware (like high-resolution skin or force-torque arrays) will give even richer signals for control. Ultimately, TAC-LOCO reminds us that sense-act loops are vital. In loco-manipulation, feeling the world is as important as seeing it. This work echoes findings from other domains – even simple robots with finger sensors carry unsecured loads much better – and extends them to whole-body agility. As legged robots become ubiquitous, controllers that are both unified and contact-aware will be key. TAC-LOCO’s success in reducing drop rates and unnecessary force shows that when a robot “feels” what it’s doing, it can act more deftly and confidently, even in the face of surprises. Sources: TAC-LOCO’s development builds on a growing body of loco-manipulation research. For example, Zhou et al. demonstrated end-to-end tactile-aware policies for contact-rich tasks; Zhang et al. and Sorokin et al. (SUMO) combined whole-body motion policies with planning to handle heavy objects; Lin et al. (LocoMan) extended whole-body control to leg-mounted manipulators; and Molnar et al. gave a full-body MPC solution to loco-manipulation. TAC-LOCO distinguishes itself by explicitly feeding tactile returns into the controller, yielding dramatically safer and more efficient interaction under disturbance.