Latest / Embodied AI 101 / Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding

Transcript

Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding · Embodied AI 101

• Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding. Dexterous manipulation with multi-fingered hands remains one of robotics’ toughest challenges. Success in tasks like unscrewing a jar lid or flipping an object inside the palm hinges on managing rich, multi-point contacts that constantly change. Yet most learned policies don’t explicitly reason about these contacts. Vision-driven approaches often fail when an unanticipated slip or collision occurs, and even tactile-enhanced policies typically treat touch signals as just extra inputs. In contrast, Contact-Grounded Policy (CGP) explicitly grounds its actions in predicted contact outcomes. CGP uses a generative model to forecast how the robot’s state and tactile readings will evolve, and then actively chooses motions that realize the intended contact plan. This “planning in contact space” leads to far more robust behavior on contact-intensive tasks than naive visuomotor control. In CGP, a conditional diffusion model imagines the immediate future of the robot’s configuration and sensor feedback – effectively predicting where and how its fingers will touch objects. A second learned module then converts these imagined contacts into concrete commands for a compliance controller, which physically executes the motion. By explicitly predicting the coupled evolution of pose and touch, CGP bridges high-level intent and low-level control. In experiments with a four-finger Allegro hand outfitted with high-resolution Digit360 tactile sensors (as well as a simulated five-finger Tesollo hand with whole-hand tactile arrays), CGP could perform very challenging tasks. For example, it successfully unscrewed sealed jar lids and flipped boxes in-hand – tasks that notoriously defeat vision-only policies – achieving significantly higher success rates and robustness than comparable baselines. In short, CGP’s key insight is to stop reacting to tactile events as they happen, and instead predict and plan around them in advance. The Challenge of Contact-Rich Dexterity. Contact-rich manipulation is hard because tiny differences in surface geometry or friction can completely change the ideal motion. The CGP authors frame it succinctly: “task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip”. Imagine spinning the lid off a jar: if the contact forces shift just a bit, the lid might start to slip or the fingers might jam. Conventional policies typically treat tactile feedback simply as additional sensory inputs, without a structured way to reason about how their outputs will alter those contacts. As a result, many vision-driven policies blindly push or twist until a slip is detected, often too late to recover. Recent work has begun to show that including tactile information can help – but usually only implicitly. For example, DAC-D-PO from Columbia or other visuomotor policies add a force/torque reading to the state vector. DexTac (Zhang et al. 2026) goes further by imitating human demonstrations with high-dimensional tactile data, achieving ~91.7% success on delicate syringe insertion tasks. And the emerging VisuoTactile Local (ViTaL) framework (Zhao et al. 2025) decomposes tasks into “reach” and “local contact” phases, reaching ~90% success on diverse contact tasks by leveraging tactile cues. These results underline a common theme: tactile priors and rich touch sensing are critical for dexterous manipulation. CGP takes this idea to its logical conclusion by building an explicit model of future contact. Rather than merely reacting to touch, CGP plans with it. In related spirit, Contact-Anchored Policies (CAP) from NYU replace high-level language goals with explicit contact points, training modular skill models around those contacts and showing remarkable generalization from very limited data. CGP similarly emphasizes contact as a central element, but instead of user-provided contact anchors, it uses a learned diffusion model to predict all relevant contacts directly from the sensor streams. In short, CGP belongs to a new wave of methods that embed contact information at the core of policy learning, rather than treating touch as a secondary afterthought. Overview of CGP. At a high level, the Contact-Grounded Policy combines two neural components. The first is a conditional diffusion model that forecasts short-horizon trajectories of the robot’s own state and its tactile readings. In practice this means the model imagines how the hand will move and what touch it will feel a few steps into the future. Crucially, this prediction lives in a learned latent space that jointly represents joint states and sensor feedback, so it captures the correlations between pose and contact. The second component is a contact-consistency mapping: a neural module that takes the predicted robot state and tactile signals and outputs a suitable target for the actual low-level controller. In other words, if the diffusion model predicts that “the index finger pad will press with force X at location Y,” the mapping converts that into a goal hand configuration that will realize that contact (via a compliance controller). Putting it together, the policy loop works roughly as follows. At each decision step, CGP observes the current visual and tactile inputs along with proprioception. It encodes them into a latent and then uses the diffusion model to sample an “imagined” future latent trajectory. The decoder part of the model then turns this latent sample into a sequence of predicted joint states and tactile readings. The final (or intermediate) joint-state prediction might not exactly correspond to the real physics of contact if executed naively, so the contact-consistency network adjusts it: outputting a modified joint-target that a standard compliance controller can attempt to reach. The compliance controller then moves the hand softly toward that target, physically interacting with objects. As long as each step’s target respects the predicted contacts, the real hand succeeds in mimicking those imagined contacts. From a bird’s-eye view, CGP distinguishes itself by this prediction-before-action paradigm. Many prior methods would simply take the current observation and output an immediate action command or joint velocity. In contrast, CGP first imagines what will happen (both in terms of motion and touch) and then realizes it. This creates a stronger coupling between intent and outcome. Predicting Contact with a Diffusion Model. The core of CGP’s contact reasoning is its generative forecasting model. The authors adopt a conditional diffusion framework to predict the coupled robot state and tactile feedback. Diffusion models have recently been popular for robotics policies (e.g. Diffusion Policy by Chi et al.) because they can learn complex, multi-modal action distributions. In CGP, the diffusion model is conditioned on the current encoded observations and then iteratively denoises from a noise vector to produce a latent representation of the future. This latent implicitly captures both pose and contact features. After sampling this latent, a decoder network recovers an explicit prediction of the robot’s future joint angles (or end-effector poses) and the corresponding tactile sensor outputs. Working in a latent space has two advantages. First, raw joint + tactile space can be very high-dimensional (especially with vision-based tactile sensors that produce rich images). By compressing into a latent, the model can focus on the underlying structure of contact dynamics. Second, diffusion allows for multi-horizon planning – one can generate a sequence of future states by conditioning on partial trajectory segments. In practice, CGP predicts a short trajectory chunk, selecting the last frame as the immediate target. We can think of the diffusion model as a learned simulator of sorts. Given “memory” from the current visuotactile observations, it tries to generate the most plausible future up to some horizon. Because it is stochastic, it captures uncertainty as well; in principle, the policy could sample several futures, though the CGP paper focuses on the one-step-ahead execution target. In summary, this module is effectively a predicted contact trajectory generator. By forecasting “robot state + touch” simultaneously, the policy knows how a particular motion sequence would feel. Contact-Consistency Mapping and Compliant Control. Imagining contacts isn’t useful unless the robot can enact them. That’s where the second ingredient comes in: the contact-consistency mapping. After the diffusion model outputs a predicted future robot state plus tactile reading, CGP passes this pair through a small network that produces an actual robot target state. Why is this needed? The predicted robot state from the diffusion model is like a thought-experiment – it assumes the robot could just go there. In reality, contact forces might require a slightly different pose to achieve the same tactile outcome. For instance, if the prediction says “finger is pushing with this force,” the mapping might adjust the target so the finger will actually press with that force under the robot’s compliance behavior. Technically, this mapping can be seen as learning the inverse contact model of the low-level controller. Recall that CGP does not directly output torques; it uses a compliance (impedance) controller as the low-level actuator. The compliance controller takes a desired joint state (or end-effector pose) and softly attempts to achieve it, yielding some contact forces in the process. The contact-consistency network learns how to “nudge” that target so that the eventual realized contact matches the diffusion plan. In effect, it ensures consistency: the intended contacts (from the diffusion) actually happen under compliance control. This two-stage approach – imagining contacts and then realizing them – decouples high-level contact planning from low-level actuation. The diffusion model focuses purely on “what should happen to the hand and touch,” while the consistency mapping aligns that plan with the actual dynamics. Together, they form a visuotactile policy that grounds its decisions in real contact outcomes. The authors summarize: CGP “grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller”. Learning From Demonstration. CGP is trained from demonstrations of the target tasks, collected via teleoperation. The authors teleoperated both a simulated hand and a real Allegro V5 to produce example trajectories of state, vision, and tactile data. These demonstrations provide the “expert” distribution for the diffusion model: at training time, the model learns to denoise noisy future trajectories toward the ones seen in the data. Crucially, the diffusion policy is trained on successful demonstrations, so it learns the correlations between state and touch that lead to task completion. The contact-consistency network is also learned from data. Intuitively, one could collect pairs of “(predicted state, actual contact forces)” by comparing the diffusion model’s output with what the compliance controller achieved. The network is then trained to map the former to the latter context. The paper indicates that both simulation and real data were used in training, although details are in the supplementary material. In any case, CGP relies on a rich multi-modal dataset where humans guide the hand through the tasks of interest. This is essentially an imitation learning approach. Once the models are trained, the policy can be rolled out autonomously. The paper notes that CGP was tested on entirely new trials (in both sim and real) to verify generalization. The combination of vision, proprioception, and high-dimensional tactile arrays in the training data gives CGP a heavy sensory input, but the payoff is that it learns to anticipate how those inputs will evolve in contact-rich motions. Hardware and Tasks. The core experiments for CGP involve two hardware setups: a real Allegro Hand V5 (a four-fingered dexterous hand) retrofitted with Digit360 vision-based tactile sensors on each fingertip, and a simulated Tesollo DG-5F (a five-finger humanoid hand) equipped with dense tactile arrays covering the entire hand. This demonstrates CGP’s versatility across different tactile modalities. The Digit360 sensors produce high-resolution images of the contact geometry at each fingertip, while the simulated Tesollo provides per-cell pressure values across the palm and fingers, mimicking a tactile web. Critics might worry about the fidelity of the simulation, but Tesollo is a real commercial hand, so the physics in sim is at least plausible. The demonstration and evaluation tasks span classic contact-rich manipulation benchmarks. In simulation, the authors tested a variety of scenarios: in-hand object reorientation, delicate grasping of slippery or soft objects, and tool-use tasks that require maintaining stable contact (though the paper does not detail each task in the abstract). On the physical Allegro Hand, they highlight two particularly challenging tasks: Jar Opening and In-Hand Box Flipping. In the Jar Opening task, the robot must grasp a sealed jar and unscrew the lid – a task requiring precise force distribution and continuous finger adjustments to overcome static friction. In the In-Hand Box Flipping task, the robot holds a small box in its palm and flips it completely over (imagine turning a small cardboard box lid from topside to bottom without dropping it). Both tasks were done “blind” under CGP control, meaning the policy had to rely on its sensing (vision and touch) to carry out the motion. The results on these tasks were impressive. On the Allegro hardware, CGP was able to successfully unscrew lids and flip boxes in a way that other baselines could not. As one report notes: the addition of high-resolution tactile sensing and the CGP framework “demonstrated significant improvements in success rates and robustness for complex manipulation tasks — such as jar opening and in-hand box flipping — compared to existing vision-motion baselines”. In other words, tasks that were essentially impossible for a vision-only policy became reliably solvable when using CGP. The figure in that report highlights that CGP was fully validated on the Allegro Hand V5 Plus (with Digit360) as part of a Meta Reality Labs research initiative, underscoring that this is more than just simulation hype. For quantitative evaluation, the authors compared CGP against two main baselines: a visuomotor diffusion policy (which takes only images and proprioception) and a visuotactile diffusion policy (which adds tactile inputs but does not explicitly model contact). Both baselines were trained on the same demonstration data. Across all tested tasks – in-hand manipulation, delicate grasps, tool use – CGP consistently outperformed the baselines. The abstract succinctly reports: “Across a range of dexterous tasks (in-hand manipulation, delicate grasping, tool use), CGP outperforms visuomotor and visuotactile diffusion-policy baselines.” This indicates that simply feeding tactile data into a standard policy is not enough; the explicit contact-grounding really made the difference. Unfortunately, the abstract and news write-up do not give exact numbers, but the qualitative phrase “significant improvements in success rates” suggests quite a gap. For instance, if a vision-only policy might succeed 20–30% of the time on Jar Opening (due to slippage), CGP might push that above 80–90%. In the absence of hard numbers, we trust the paper’s claims and the fact that it won an Outstanding Paper Award at CVPR’s Sense-of-Space workshop. Why CGP Excels. What makes CGP so effective? One intuitive way to see it is by contrasting it with what a baseline policy does. A typical diffusion-based policy (as in recent work) might generate a sequence of actions or end-effector commands conditioned on current observation. If an unmodeled contact event happens, the policy only learns to react with those present sensors. CGP, however, has the luxury of foresight. By predicting tactile outcomes, it can preempt dangerous events. For example, during jar unscrewing, CGP’s model might foresee that continuing the twist motion will cause the lid to slip. Knowing this, the policy can proactively adjust the grip or apply just a touch more pressure on certain fingers before the slip occurs. Without this planning, vision-only controllers would have to wait until the slip is already happening (maybe detected by high tactile velocity or force change) and then react, which often fails. The compliance controller also contributes to robustness. Unlike stiff position control, compliance allows the hand to yield slightly under force. The contact-consistency mapping ensures that the compliance-target produces the intended tactile forces. This means the robot can maintain stable pushes without fighting the object too hard. Many dexterous tasks (think gently twisting a lid or sliding a box on the palm) require this softness. The news report on the Allegro demo emphasizes “seamless hardware integration” with Digit360 and a compliance-based execution, implying that CGP outputs were indeed implemented as impedance goals. In essence, CGP’s innovations focus the learning capacity on the truly relevant part of the problem – contact dynamics – instead of on raw motor commands. The diffusion model learns the manifold of feasible contact trajectories from the demonstrations, and the consistency map learns how to achieve them. The system as a whole forms a closed loop: predict-then-act-then-perceive-and-predict-again. This loop allows it to use tactile feedback not just as a sensor but as part of a predictive control pipeline. The result is a “visuotactile policy” that is grounded in the physics of wall and finger interactions, leading to orders-of-magnitude better reliability on contact-laden tasks. Relation to Prior Work. CGP builds on two major threads in recent robot learning. The first is the use of multisensory tactile feedback. As noted, methods like DexTac and ViTaL have shown that tactile information is crucial for finesse. DexTac, for example, encodes contact force distributions (center of pressure) from human demonstrations and achieves >90% success in sophisticated insertion tasks, far outperforming vision-only or force-only baselines. ViTaL demonstrates that training a specialized touch-aware policy can yield high performance across new contact scenarios, achieving around 90% success on unseen tasks and highlighting that “tactile sensing significantly boosts performance in contact-rich tasks”. CGP differs in how it uses touch: rather than simply feeding tactile data to a policy, it uses touch as a prediction goal, explicitly planning over it. In that sense, CGP can be seen as a more structured evolution of the visuotactile paradigm championed by these works. The second thread is generative diffusion policies for manipulation. Diffusion-based action generation (as in Diffusion Policy and its variants) has become popular because of stability and expressiveness. Researchers have already started blending diffusion models with tactile cues (for example, reactive diffusion policies that adjust planned trajectories when slip is detected). However, most of these still train the policy to directly propose actions. The very recent DPTG method (diffusion policy with tactile guidance) takes a different approach: it trains a vision-driven diffusion policy on successful trajectories, and separately learns a tactile “feasibility” classifier from failure cases to bias the sampling process. That is somewhat akin to pruning out bad contact plans. CGP also stems from this diffusion tradition but goes further by generating the contact predictions themselves. Another class of related work includes methods like Contact-Anchored Policies (CAP) that reframe manipulation with contacts as guiding anchors. CAP replaces natural language or spoken commands with spatial contact points as conditions, and learns modular utility policies around those contacts, demonstrating strong generalization and data efficiency. Like CAP, CGP recognizes the primacy of contacts in dexterity; but CGP does not require external annotation of contact points. Instead, it “discovers” the critical contacts through its generative model and uses them autonomously. Conceptually, both CGP and CAP underscore that physical contact should be an explicit part of the policy vocabulary. In summary, CGP is at the intersection of visuotactile learning and generative planning. It shares with DexTac and ViTaL the goal of tactile-grounded dexterity, and with diffusion-policy approaches the idea of sample-based trajectory generation. Its novelty lies in explicitly predicting contact outcomes and translating them into control targets. This places CGP as a unique contribution: a learned visuotactile world-model for robot hands. Discussion and Future Directions. CGP marks an exciting step, but it naturally comes with caveats and open questions. First, it relies on high-quality demonstrations. Ferr it learns future contact trajectories only as good as the data it sees. Tasks not covered by the demos would require fresh data collection. Moreover, diffusion models can be computationally expensive to sample, so real-time performance depends on optimization. The paper doesn’t detail inference times, but practical deployment would need ensuring the diffusion prediction can run at a suitable rate (likely slower than, say, a 1 kHz torque loop, but possibly on the order of tens of hertz which may suffice for manipulation). Second, CGP uses a point-goal compliance controller underneath. This mix means it is not “model-free”—it explicitly leverages knowledge of how a compliance servo behaves. If the controller gains or the contact stiff­ness change, the contact-consistency mapping may need retraining or adaptation. However, this modularity could also be a strength: one could swap in other low-level controllers (force control, hybrid control, etc.) and retrain the mapping to suit them, while keeping the diffusion model the same. Third, CGP’s training is offline. It would be interesting to integrate online learning or adaptation. For instance, if a predicted contact doesn’t happen as expected at run-time, a future extension could involve correcting the model on-the-fly. Reinforcement learning techniques or online latent embedding adjustments might further improve robustness. Additionally, CGP currently focuses on short-horizon contact prediction. Extending the horizon (multi-step ahead planning) or chaining multiple diffusion steps could be natural future work, akin to model-predictive control in the latent space. Finally, CGP’s success on examples like jar opening and box flipping suggests potential for more ambitious applications. Since it explicitly reasons about contact, it might adapt well to tasks like tool manipulation (e.g. using a screwdriver, handling deformable objects, or bimanual tasks where both hands contact an object). Combining CGP with visual planning—for example, using a top-down camera to decide which task to attempt and CGP to figure out how to perform it—could yield a highly capable system. In conclusion, the Contact-Grounded Policy presents a compelling proof-of-concept: by letting a robot “imagine” its own touches before acting, we can achieve far more robust dexterous manipulation. The approach highlights the value of treating touch as something to plan over, not just sense and react to. For roboticists building next-generation manipulation systems, CGP offers a new tool: a visuotactile world-model that can be interrogated and controlled, bringing us closer to human-like dexterity. References: The CGP framework and results are detailed by Xu et al. (2026), with supporting documentation on the project website and Allegro Hand news release. Related works include tactile imitation learning like DexTac and ViTaL, and diffusion-based policies augmented with touch or contact conditioning.