Latest / Embodied AI 101 / ViTacFormer: Dexterous Manipulation via Active Vision and High-Resolution Touch Sensing
Transcript
- ViTacFormer: Dexterous Manipulation via Active Vision and High-Resolution Touch Sensing. Dexterous in-hand manipulation remains a grand challenge in robotics. Vision alone often cannot handle the subtleties of contact-rich tasks – imagine trying to thread a wire through a tiny hole or twist off a bottle cap without any feeling. Tactile feedback, by contrast, provides the fine-grained contact information necessary for precision and adaptation in unstructured environments. The new ViTacFormer system by Geng, Heng, Abbeel, Malik, and colleagues tackles exactly this problem. It unifies vision and high-resolution touch in a deep learning framework to enable very precise, long-duration manipulation. Remarkably, the system completes up to 11 sequential steps (on an anthropomorphic robot hand) in tasks like making a hamburger, running for about 2.5 continuous minutes without human intervention. That is far longer and more complex than any prior real-world dexterous demo. Across a suite of benchmarks, ViTacFormer boosts success rates by about 50% over previous state of the art. In the following we’ll explain how it works and why it matters. The Need for Touch in Dexterous Robotics. Traditional robot manipulation has often relied on cameras. Vision is great for coarse tasks and gives global context. But for subtle in-hand operations – small insertions, twisting nuts, feeling friction – relying on vision alone can leave a robot “blind to touch” once objects occlude the view. The ViTacFormer team argues that to really reach human levels of dexterity, robots must feel what they touch. As they write, “tactile sensing remains crucial for fine-grained control, particularly in unstructured or visually occluded settings”. In practice this means fusing camera images with fingertip sensors so the robot knows not just what it looks like but what it feels like. Simple approaches to this fusion often suffer from bottlenecks. Naively concatenating high-dimensional images from a camera with, say, force readings from sensors can leave the network unable to relate the two modalities. Earlier work has started to use attention or other tricks to combine vision and touch, but mostly in simpler settings. ViTacFormer makes two key insights: (1) use cross-attention transformers to fuse vision and touch features throughout the policy, and (2) train the model to predict future touch signals and use that prediction in deciding actions. In short, the model doesn’t just see and feel now, it learns to anticipate how the fingers will feel in the next moment. This predictive, multi-modal latent space turns out to be much richer for control. Combined with imitation learning from real human demonstrations, it unlocks dramatically better dexterity. Before diving into the deep learning, let’s describe their hardware and data collection, which is itself impressive. Hardware Setup and Data Collection. ViTacFormer is tested on a real bimanual robot setup. At each arm’s end is a SharpaWave anthropomorphic hand (development version) with 5 fingers and 17 degrees of freedom each. This is a human-scale, human-like hand (SharpaWave is a newly developed platform), capable of coordinated movement across all fingers. To give the robot eyes, they use four cameras: two wide-angle fisheye cameras mounted on the wrists (one on each arm) for close-up views of the tasks, and a stereo pair from a camera (ZED Mini) on the ceiling for a global overhead view. This combination gives the system both local detail and overall context – an “active vision” setup in which the cameras move with the robot. Crucially, each fingertip is also equipped with a high-resolution tactile sensor (320×240 pixels) developed by Sharpa. These sensors can see the pressure pattern on the fingerpad in fine detail. (The raw sensor output is visual-like data, but in practice for control the researchers extract a compact format: three-axis force and torque readings from each fingertip.) This rich touch data lets the robot feel contact forces and incipient slip. To collect training data, the team built an exoskeleton teleoperation system. A human operator wears mechanical gloves that are physically linked to the robot hands, copying every finger motion. While teleoperating, the person sits in VR: they see through the robot’s perspective via the stereo and wrist cameras, and even get real-time overlays of tactile contact on virtual fingertips (for intuitive feedback). In this way the operator can perform complex tasks two-handed in real time, letting the robot gather high-quality demonstrations in its real hardware. All data streams (RGB images, joint states, and tactile force/torque readings) are time-synchronized and logged to become the expert trajectories for imitation learning. In summary, their experimental setup is quite comprehensive ─ two realistic arms, each with a dexterous anthropomorphic hand and 4 cameras, plus 5 high-res touch sensors. The human can “become” the robot using gloves and VR, allowing capture of complicated, contact-rich behaviors with ground truth actions. This left the team with hundreds of multi-modal demonstrations of tasks ranging from small inserts to kitchen prep. The ViTacFormer Architecture. The core of ViTacFormer is a transformer-based policy network that jointly processes vision, touch, and proprioception. It is effectively a form of conditional variational autoencoder (CVAE), similar in spirit to methods like ACT, but enriched with opt-in visual and tactile streams. Here is the high-level design:. Style Encoder (Left side): A transformer encoder ingests the expert actions and joint states from a demonstration and produces a latent “style” vector $z$. This $z$ acts as a global context or mode of behavior, enabling the network to capture multimodal action distributions. (During training, $z$ is drawn from the encoder’s distribution; at test time $z$ is fixed to zero, as done in prior work.). Vision+Touch Encoder-Decoder (Right side): Another transformer module takes in the robot’s current observations and predicts the next actions. The observations include: sequences of recent images from all cameras (encoded through a vision backbone), the robot’s joint angles/poses, and the processed tactile signals (force/torques from all 10 fingertips, over a temporal window). Critically, this module fuses vision and touch with cross-attention layers. In each cross-attention block, visual features and tactile features attend to each other: visual keys/values with tactile queries and vice versa, as illustrated in Figure 3 of the paper. The outputs from both cross-attentions are concatenated into a joint hidden state. This means the network can compare “where the object is in view” with “where it feels a push” at every level of abstraction. Tactile Prediction Head: Before generating an action, the model first predicts the future tactile signals. Concretely, given the current fused representation and style $z$ (and the robot’s current joint state), the network outputs a vector of what each fingertip’s force/torques will be in the next moment. Empirically, this prediction had low error (around 0.08 normalized L1 error) and strongly aided planning. The intuition is that anticipating contact changes – e.g. knowing that a fingertip is about to make strong contact – is richer than passively reading the current contacts. After predicting these future tactile tokens, the network appends them to the input and then runs another round of cross-attention fusion to finally produce the next action sequence. Action Decoder: The output is a short sequence of fine-grained control commands (100 frames at 10 Hz) for the two arms and hands. This distributed output allows issuing high-frequency motion while still planning a horizon. (The paper mentions they output 100 future steps and then smooth them to execute at 10 Hz.). The training loss combines several terms: a KL term on the latent $z$, an L1 imitation loss on the predicted actions, an L1 loss on the tactile forecast, and auxiliary supervision on the end-effector positions/orientations. In practice the best learning arose when they also explicitly trained the model to match the robot’s hand and arm poses (not just joints) to the expert. This multi-component loss encourages the latent space to encode nuances of both motion and contact. Lastly, they introduce a curriculum/training schedule for the predicted touch. Early in training, the tactile-forecast head is weak, so feeding its raw guesses back into the policy would confuse learning. Thus, for the first 75% of the training epochs they use the ground-truth tactile signals as input (like teacher-forcing). Then in the final 25%, they switch to using the predicted tactile outputs. This staged approach is akin to scheduled sampling: it lets the network first build a stable visuo-tactile mapping, then adapt to the model’s own prediction errors. According to the authors, this two-phase strategy was crucial – without it the policy overfits to perfect touch and fails to handle small prediction errors at test time. In summary, ViTacFormer is essentially a CVAE policy network that (a) deeply fuses vision and touch via cross-attention, (b) forces the latent to represent “what will be felt” as well as “what is seen,” and (c) uses staged learning to tie it all together. At inference time, the style latent is zeroed and the camera/tactile inputs drive an autoregressive prediction of touch and action. The heavy lifting is done by transformers, but importantly the structure ensures multi-modal interaction. Active Vision and High-Resolution Touch. We emphasize the sensory setup because it’s unusual compared to most robot learning experiments. The vision system is active in the sense that it’s not a fixed camera on the ceiling, but moves with the robot’s arms. The wrist-mounted fisheye cameras provide close-up views of the immediate task (e.g. the object in hand or under manipulation), whereas the ceiling stereo gives a global overview (object locations, coarse scene geometry). This multi-view vision means the robot can reposition its hands and still keep track of the important features. The touch system is also notable. Each fingertip has a dense optically based sensor (320×240 shown in the figure). These might be GelSight-like images of contact deformation. Such high-resolution tactile data could potentially be processed by convolutional nets, but that would be very heavy to run at control speed. Instead, the authors convert each scan into a 3-axis force-torque vector, capturing the magnitude and direction of pressure. Thus at each time step the model sees a 30-dimensional tactile vector (10 fingers × 3 axes). These vectors over a short history (they use 18 past frames) form the tactile input. Even with this compression, the tactile information is rich and at 18 steps of history it encodes events like contacts and releases. By combining these, the robot essentially “sees” and “feels” everything. For example, when inserting a peg into a tight hole it can look down the barrel of insertion and gently advance; when twisting a cap it can see the cap and simultaneously sense the torque load on its fingertips. The ViTacFormer network is explicitly designed to leverage both: vision to know the geometry, touch to know the contact state. Benchmarks and Tasks. To evaluate ViTacFormer, the authors set up several real-world dexterous manipulation tasks, ranging from fairly simple to very long-horizon. Each task is performed by the robot’s two hands working together. Importantly, they only use 50 expert demonstrations per task for training – demonstrating a very data-efficient approach. Four short-horizon tasks were defined (illustrated in Fig.4 of the paper):. Peg Insertion. Insert a peg into a hole with 0.5 mm clearance. This requires very precise alignment. The hole is occluded (likely under a cover), so the robot must use touch to actually feel the insertion once the peg enters the hole. Cap Twist. Unscrew or screw a tight cap. The robot must grasp a small cap and apply rotational torque precisely, detecting via touch when the cap starts to loosen or tighten. Vase Wipe. A vase with ink marks; the robot must wipe around the vase surface to clean it. This requires maintaining contact along a complex contour under the camera’s view. Book Flip. Flip pages of a book one by one (or flip a tilted page). This needs very delicate touch to get under a single page and pry it up without ripping or grabbing too many at once. Each of these uses vision to locate the object but has critical thresholds where vision may fail (e.g. under the page). They challenged the system by slight spatial perturbations at test time to check generalization. The ultimate long-horizon task is a staged “make-a-hamburger” sequence of 11 steps. In successive stages, the robot: turns a lever (like a tap or spatula handle) to “open”, shovels meat onto the bun, flips that set of steps to "assemble," picks up lettuce and places it, adds top bun, and finally flips a closing sign back. Essentially it’s a mini assembly line with cloth-like and hard objects, requiring everything from careful grasping of a soft lettuce, to coordinating two hands to stack ingredients, to monitoring an indicator at the end. The entire sequence must run continuously (the figure shows it took about 2.5 minutes). Crucially, for evaluation they break the sequence into stages and assign scores, but they also report whether the robot can fully complete all tasks in one go. Overall, this benchmark suite is quite challenging. The tasks incorporate fine alignment (peg), rotational force control (cap), surface contact following (vase), micro manipulation (book), and a mix of those plus soft-body handling (hamburger) with in-the-wild elements. This ensures the learned policies are versatile, not overspecialized. Experimental Results. The core question is: does ViTacFormer actually improve over prior approaches? The answer from the experiments is a strong yes. The paper compares against a set of existing imitation learning methods: Diffusion Policy (DP) as a recent generative policy model, and ACT (a CVAE actor-critic model) both with and without simple tactile input. They also include HATO, a version of diffusion policy conditioned on (naively fused) tactile inputs. In short, the baselines are DP and ACT with vision only, and HATO/ACTw/T which add tactile by just concatenating or conditioning on the force inputs. Short-horizon results: On each of the four basic tasks, ViTacFormer nearly solved it. As reported, its success rates were 10/10 on peg and cap, and 9/10 on vase and bookflip. By contrast, the best baseline (ACT with touch) only managed about 6/10 for peg and cap, and even lower for the others. Vision-only models often scored just a couple of successful trials. For example, diffusion policy (vision-only) succeeded only 2/10 on peg insertion and 0/10 on cap twist. In terms of percentages, ViTacFormer was roughly 50–60% higher success on average than the best of the prior methods. This quantitatively confirms the intuition: the cross-modal predictive model dramatically boosts performance. The paper also notes that just adding naive touch data helps (ACTw/T beat ACT, HATO beat DP) – but ViTacFormer’s learned fusion gave it another big jump over that. Long-horizon (hamburger) results: Here the differences are even starker. ViTacFormer is the first system to complete such a prolonged sequenced task in one shot. In terms of human-normalized stage score (a custom metric that scores each sub-stage 0–3 and weights by tactile reliance), ViTacFormer achieved about 0.88 (normalized to 1.0 for perfect). The vision-only ACT model was only 0.61 on this scale, and the naive-tactile ACTw/T got 0.72. That means ViTacFormer raised the overall task quality from ~61% to ~88%. In success-rate terms, define “success” two ways: (a) completing every stage non-trivially, and (b) completing the entire sequence without any human correction. Under criterion (a), ViTacFormer finished all stages in 80% of trials, versus 40% for ACTw/T and 10% for ACT. Under criterion (b), they report 70% for ViTac vs 10% for ACTw/T and 0% for ACT. In plain terms: only ViTacFormer got the robot to do the whole hamburger reliably, and even then it still failed some runs. The others almost always blew up at some stage. This confirms that the predictive visuo-tactile representation was key for chaining together so many precise actions. The authors break down which parts improved most. Stage 5 (grasping delicate lettuce) for example is nearly impossible without feedback – ViTacPassed it cleanly, whereas the others often gripped too loosely or crushed it. They explicitly note that grabbing the soft lettuce “requires tactile sensing to detect contact and stabilize the grasp”. Similarly, stage 11 (flipping a hidden sign) needs the robot to know exactly when it’s touching the tab under occlusion, which again the touch input enables. In general, the stages involving fragile or occluded contact saw the biggest gains. They also did ablation studies (Fig.6) showing that removing either the cross-attention fusion or the tactile-prediction head noticeably drops performance. Without cross-modal attention, the policy could not relate vision to touch effectively; without the predictive head, it lacked foresight. The conditional VAE style variable and end-effector loss also contributed, but the largest effects were from the multi-modal design and curriculum. Some other technical notes: each task only used 50 demonstration trajectories, so the model had to generalize from limited data. Impressively, ViTacFormer learned robust behavior from those few demos – they mention other works with similar data sizes often didn’t reach such high success. Training was done on 2 GPUs: short tasks converged in about 12 hours, and the long task in around 2 days. So although the model is complex, it is feasible to train in a lab setting. For execution, the network runs at 10 Hz, outputting high-frequency action rollouts which are then smoothed into actual joint commands. Impact and Applications. ViTacFormer clearly demonstrates that richly integrating vision and tactile feedback in a learned policy pays huge dividends in manipulation. For roboticists, the takeaway is twofold. First, multi-modal representation matters. The cross-attention structure can serve as a blueprint: fuse the features from multiple sensors at every layer, not just at the last moment. Systems that only “peek” at touch infrequently or that simply concatenate signals struggle on delicate tasks. Second, predicting contact (instead of merely reacting to it) is powerful. By training the network to imagine what the touch sensors will read next, the latent space becomes organized around the task-relevant change in contacts. This foresight makes the policy more reactive and stable. (One can think of it as giving the robot short-horizon proprioception in the contact space.). The success on very long tasks is particularly significant. In robotics labs, closing the loop on even 2–3 steps of a long chain without resets is hard. This work shows 10+ steps are doable with careful design. It suggests that future home robot chefs or assembly-line workers with similar sensory suites could handle complex tasks end-to-end. From an engineering perspective, this also underscores the value of specialized hardware. Many existing robots have only coarse touch feedback (torque at joints) or low-res bump sensors. Here the fine-grained 320×240 tactile sensors, plus active cameras, created a rich dataset that the deep model could exploit. As such sensors become more common (and faster to process), similar ideas could apply broadly. Finally, ViTacFormer’s approach is still imitation learning; it could in principle be combined with reinforcement or offline RL to further refine policies or expand beyond demo coverage. The authors’ release of code and data (via RoboVerse) will let others experiment with these ideas. We might expect, for example, extensions that include more modalities (like audio of contact) or apply similar fusion to fleet data. In summary, ViTacFormer demonstrates a new level of dexterity by jointly leveraging “active” vision and touch. The key insight is that a transformer can learn to see with touch by attending between the two senses and planning with predicted tactile futures. The results speak for themselves: success rates jump ~50% over prior methods, and the robot can autonomously run through long assembly tasks (up to 2½ minutes and 11 steps) that were previously out of reach. For researchers in embodied AI, this is an exciting example of how rich sensory fusion unlocks new capability. Sources: The above description is based on the ViTacFormer paper by Heng et al. (2025) and related project documentation. Key results and methods (cross-attention, tactile forecasting, curriculum) are cited from the authors’ report and benchmarks. For more details see the ViTacFormer project page (RoboVerse) and full paper.