Latest / Embodied AI 101 / HapticVLA: Extending Vision-Language-Action Models to Contact-Rich Tasks Without Touch Sensors
Transcript
- HapticVLA: Extending Vision-Language-Action Models to Contact-Rich Tasks Without Touch Sensors. Vision-language-action (VLA) models have emerged as a powerful way to control robots using camera images and natural language. By jointly reasoning about what the robot sees and what the task is (in human language), a VLA can output robot motions to accomplish complex pick-and-place or manipulation jobs. These models excel at many tasks, including bimanual coordination or mobile manipulation, and they “speak the language” of human instructions. However, one striking limitation remains: most VLAs are essentially “touch-blind.” They rely on vision (and the robot’s joint positions) to plan motion, but they have no direct sense of force or contact. While vision captures an object’s shape and appearance, it cannot by itself tell you whether you are squeezing too hard or letting an egg slip. This lack of tactile awareness can cripple performance on objects that are soft, delicate, or require precise pressure. Haptic sensing (tactile or force sensors) is known to help robots handle fragile or oddly shaped objects safely. With a force sensor in the gripper, a robot can learn to squeeze lightly on an egg or detect when an object starts to slip. Indeed, recent VLA-based systems have begun to incorporate touch hardware, using camera-based tactile pads or force encoders on the fingers. These approaches show that adding a “sixth sense” let the robot adapt its grip to object properties during the task. But there’s a catch: specialized tactile sensors are expensive, fragile themselves, and hard to integrate on every robot. A robot design that works in one lab may fail in another because the finger sensors don’t match. In short, relying on real hardware during deployment hurts reproducibility and accessibility. HapticVLA confronts this challenge head-on. Its core idea is simple but powerful: Learn the benefits of touch during training, but remove the need for actual touch sensors at inference time. Concretely, HapticVLA trains a VLA with tactile feedback in the loop, using it to shape the policy with safety-oriented rewards, then distills that tactile knowledge into a purely vision-based policy. The result is a VLA that behaves as if it had seen touch, even though only vision and joint state are used at runtime. In experiments on a real robot, HapticVLA achieves an 86.7% average success rate on challenging pick-and-place tasks (handling a plastic jar of jam, fragile waffles, and a raw egg) – comfortably outperforming all baselines, including variants that do have touch sensors during inference. In other words, the “blind” student surpasses even “seeing” teachers. The HapticVLA framework unfolds in two stages: (1) Safety-Aware Reward-Weighted Flow Matching (SA-RWFM), which fine-tunes a VLA action model with tactile reward augmentations; and (2) Tactile Distillation (TD), which transfers the teacher’s haptic-aware behavior into a traditional VLA without touch. Below we unpack these ideas, explain how they are implemented, and discuss the results. Along the way, we cite the authors’ own statements and figures (hereafter in quotes like “[…]【‡L–L】”). From Vision to Feel: Why VLA Models Need Tactile Senses. Vision-language-action models operate on camera images plus robot state plus language, and they output motions in the robot’s own coordinates. These models have unlocked many tasks – e.g. “reach out and grasp that brown mug,” or “slide the screwdriver across the panel” – that would be hard to program by hand. In everyday manipulation, though, visual information alone can leave the robot blind to subtleties. A ripe tomato, a foam block, and a hollow plastic jar may look distinguishable by sight, but what about how much to squeeze or how gently to insert them? Without feedback, the robot might crush the object or fail to secure it. Researchers have long noted this gap. Several recent VLA works explicitly equip the gripper with tactile pads or force sensors. For example, some robots use finger-mounted cameras with special lighting to capture local contact pressures, treating those images as an extra input. Others use force-torque sensors on the wrist or motors, effectively giving the policy a sense of how hard it pushed. These modalities indeed improve dexterity – for instance they allow the VLA to modulate grip strength based on object hardness. But by requiring hardware at run-time, they sacrifice ease of deployment. _HapticVLA contrasts these approaches sharply. The authors ask: Can we learn what tactile feedback contributes, and then dispense with the hardware? Their thesis is that tactile-aware skills can be internalized offline. Instead of forcing every future user to install a tactile sensor, they collect (or simulate) data with those sensors once, distill the knowledge into a model, and afterwards run without sensors. As the authors summarize: “We argue that tactile-aware manipulation can be learned offline and deployed without direct haptic feedback at inference.”. In practice, this hinges on two sub-ideas – one from the world of imitation learning plus generative modeling, and one from policy distillation – which we turn to next. Stage 1: Safety-Aware Reward-Weighted Fine-Tuning. The first stage of HapticVLA is to train a VLA action expert that knows how to use touch. To do this, they start from a pre-trained VLA (the so-called “SmolVLA” model from prior work) and fine-tune it on a dataset of contact-rich demonstrations. Crucially, these demonstrations include rich tactile data (pressure maps on the fingers) as well as the usual camera view and joint states. But simply training on the raw demonstrations isn’t enough: the model should prefer safe, gentle grasps when possible. HapticVLA introduces explicit tactile safety rewards to bias learning. The idea is to penalize trajectories that use excessive force or risky motions, even if the final task succeeds. For example, if a robot is inserting an egg and squeezes too hard (cracking it) or drops it, that should yield a worse reward. The authors encode this via an episode reward that sums up successful outcomes and penalizes failure modes. Concretely (in their equations):. “The final episode reward combines the averaged step reward, task outcome, and safety risk penalty:(R_{\text{step}} = \tfrac{R_{\text{step scale}}}{T}\sum_{t=1}^T r_t,)(R_{\text{episode}} = R_{\text{step}} + R_{\text{succ}}\times \text{success} ;-; R_{\text{drop}}\times \text{drop} ;-; R_{\text{damage}}\times \text{damage} ;-; R_{\text{risk}}\times \text{risk}.). In words, they give a bonus term (R_{\text{succ}}) if the pick-and-place episode ended in a successful placement, and subtract penalties (R_{\text{drop}}), (R_{\text{damage}}), and (R_{\text{risk}}) for common failures (dropping the object, damaging it, or entering a high-force configuration). During fine-tuning, every training sample (i.e. each trajectory snippet) is weighted by its episode’s total reward. The policy is trained by “flow matching” – a generative approach where the model learns to output continuous action sequences (joint commands) under a flow-based distribution. Critically, Reward-Weighted Flow Matching (RWFM) multiplies the standard imitation loss by a weight (w_i) for each sample. This weight is essentially an exponentiated advantage score: higher reward episodes get more emphasis. In practice, the model normalizes and clips these weights, and also adds a small L2 regularizer to keep the fine-tuned policy close to the original (an “anchor” to avoid collapse). We need not detail the math, but the key point is clear: good, safe behaviors are amplified, bad ones are down-weighted. This safety-aware weighting indeed pays off. The authors note that simply training a VLA naively (even with sensors) tends to average over unsafe demos. By contrast, SA-RWFM focuses learning on the trajectories that keep forces moderate. In experiments, the SA-RWFM expert (still using tactile feedback) reached a hefty 75% mean success across tasks (compared to much lower baselines). In two of the three tasks, it even hit 85%. (By contrast, their uncontrolled baselines with no reward shaping failed completely – the “X-VLA” and “VLA-0” models got 0% on these tight tasks.) This shows that rewiring the training loss via tactile rewards substantially improves the robot’s grasping behavior. Once the SA-RWFM expert is trained, it can in principle be used directly (with sensors) as a strong performer. But HapticVLA’s goal is to remove the sensors at inference. That brings us to the second stage: distillation. Stage 2: Distilling Haptic Knowledge into a Language-Action Model. Tactile Distillation is the heart of HapticVLA. Here the goal is to transfer everything useful that the tactile-aware, reward-fine-tuned teacher policy has learned into a standard VLA model that will only see vision and joint state. The student VLA has no modifications in architecture and receives no touch inputs. How is this possible?. The authors use a classic teacher-student paradigm with a twist. In Stage 1: Offline Teacher Target Generation, they simply run the teacher policy on all the training observations to get its predicted actions. That is, for each sampled state (o_i) and actual robot state (s_{T,i}) (which includes the tactile embedding), they compute the SA-RWFM teacher’s output action sequence (\hat{a}^T_i). This yields a large dataset of “soft” action targets (51,251 rollouts in total). Because the teacher was optimized with tactile rewards, these predictions are already biased toward safe, precise behaviors. Importantly, this is done offline – they don’t need to query sensors at distillation time; they just use recorded observations and the saved teacher network. The teachers’ outputs effectively encode “what the policy would have done if it could feel touch.”. Next, in Stage 2: Teacher Backbone Initialization, they initialize the student network from the teacher’s weights. Since the teacher has a little tactile encoder internally, they copy every weight except those in the tactile branch. Practically, in the state encoder that originally took a 134-dimensional combined state [joint positions (6) + tactile embedding (128)], they keep only the first 6 columns corresponding to proprioception. In this way, the student starts with the same visual and proprioceptive feature processing as the teacher. The student now has a “head start” aligned with the teacher’s representation space, without requiring a sensor input. Finally comes Stage 3: Blended Target Training. The student is trained by flow-matching on special “blended” action targets (\tilde{a}) that mix the ground-truth action (from the human demo) with the teacher’s predicted action. Formally, they set [ \tilde{a}_i = (1-\alpha),a^{GT}_i + \alpha,\hat{a}^T_i, ] with (\alpha=0.5). In other words, the target is half the human-demonstrated action and half what the touch-savvy teacher would have done. This blending has two benefits: the ground-truth component keeps the student anchored to what worked (avoiding teacher mistakes), while the teacher component injects the tactile-optimized adjustments. The student’s flow-matching loss is then computed against (\tilde{a}). During validation (testing baseline performance) they set (\alpha=0), so that improvement is purely due to distillation rather than “peek” at teacher actions. By the end of this process, the student VLA has absorbed the teacher’s haptic reasoning – all without ever seeing real tactile data at inference. The student can predict the equivalent of a “tactile token” implicitly from vision and state. In production, the robot ignores any finger sensor; it just uses vision+state to output actions. Yet these actions reflect a “sense-of-touch” learned offline. Experiments: Tasks, Data, and Metrics. HapticVLA was tested on a suite of contact-rich pick-and-place tasks designed to stress fine manipulation. These tasks involve objects of varying fragility and hardness, each requiring precise control of grip force:. Jar of Marmalade (plastic) – “Pick a jar of marmalade and place it into the box.” The jar is squeezable and its deformability must be accounted for. Box of Waffles (fragile) – “Pick waffles and place them into the box.” The waffles are delicate (they can crumble or stick) and require a careful grasp and precise positioning. Egg (very fragile) – “Open the carton. Pick an egg and place it on the tray.” This is the toughest: the robot must apply extremely gentle and well-aligned pressure to lift the egg without cracking it. For each task, success means the object ends up intact at the target location. Cracks, breaks, excessive force, or drops count as failure. The authors collected a mixed dataset of 310 real-world demonstrations (with both successful examples and intentional failure modes) for these tasks. Faulty demos – say, an egg dropped or cracked – were assigned negative rewards based on measured contact forces and state. These negative examples are essential for the reward-weighting stage: they help the model learn to avoid unsafe trajectories. To augment scale, the same setup was replicated in NVIDIA Isaac Sim. There a digital twin of the robot and objects was used to generate 1,000 additional episodes by randomizing object poses and box locations (while keeping observations and actions aligned with the real robot). This larger synthetic set improves generalization to different orientations. All policies – HapticVLA and baselines – were fine-tuned on the same combined dataset of real + simulated episodes. During evaluation, each model was tested in 20 trials per task on the actual robot hardware. Some baselines had access to tactile input at test time (to see how important it is), while HapticVLA and certain baselines did not. Results: HapticVLA Outperforms All Baselines. The results are striking. Recall that HapticVLA’s student never sees tactile inputs during execution. Yet it beats even sensor-equipped baselines. Overall, HapticVLA averaged 86.7% success rate across the three tasks. By comparison, the base VLA model (SmolVLA without any safety weighting or distillation) did far worse, especially on the egg. In fact, with SmolVLA only about 11 out of 20 eggs were placed correctly, and its mean was only 75% (compared to 86.7% for HapticVLA) [from Table I]. Another strong baseline, X-VLA (a larger 0.9B-parameter foundation model), actually scored 0% on these ultrasensitive tasks. The emergent lesson: without haptic guidance, standard VLAs collapse on high-precision contact tasks. Digging deeper, two ablation results tell the story. First, using only the SA-RWFM teacher (with real sensors at test) yields about 75% mean success – far above the naive baselines. This confirms that safety-weighted training alone makes the policy substantially better. Second, turning on Tactile Distillation (with sensors off) further boosts performance. An ablation table (Table I in the paper) shows that the student with full SA-RWFM+TD reached 86.7% (15/20 jar, 18/20 waffles, 19/20 egg), whereas without TD it managed only 75.0%. In other words, the distillation stage contributed roughly an extra ~11 percentage points of success. The benefit was especially pronounced on the egg: HapticVLA was correct on 19/20 eggs, a 45% absolute improvement over the base SmolVLA (11/20). Another key finding is that HapticVLA even outperforms versions of SmolVLA that are given direct tactile feedback during inference. The authors remark on this in plain language: “HapticVLA achieves a mean success rate of 86.7%, consistently outperforming baseline VLAs — including versions provided with direct tactile feedback during inference.”. In other words, the student learned to anticipate the role of touch so well that it did better than cheating with real touch!. Why might this be? One possibility is that the teacher’s reward shaping had already filtered out overly aggressive behaviors, so even with real touch the baselines didn’t exploit the hardware fully. Meanwhile, HapticVLA gets the best of both worlds: it benefits from offline tactile training but isn’t jittered by noisy sensor readings at run-time. Throughout all these evaluations, HapticVLA’s performance remained robust. The authors note that asynchronous inference (the SmolVLA style of pipelining actions) slightly degraded performance, likely due to timing mismatches, but the overall trends persisted. Crucially, every metric – jar, waffles, egg, and mean – was highest for HapticVLA in its full configuration. Why It Works: Offline Learning + Distillation. Several insights emerge from HapticVLA’s success, as summarized by the authors themselves:. Offline Learning of Tactile Skills: By collecting data in a setup with touch sensing, the model effectively learns the physics of contact in a task-specific way. The safety-aware rewards ensure that only the “good” portions of that experience shape the policy. As the paper observes, “the ability to perform tactile-aware, contact-rich manipulation can be learned effectively during an offline training phase by integrating safety-aware tactile rewards into a VLA action expert”. In practice this means the dataset of demonstrations (with many failure cases) becomes a rich source of knowledge: successful pick-ups will have subtle pressure signatures that the model learns to output actions for, and failures will be down-weighted. Knowledge Distillation Without Hardware: Even with such offline learning, there’s no guarantee that the policy can act without sensors. HapticVLA’s key step is showing that complex tactile-aware strategies can be distilled into a sensor-agnostic policy. The blended target training means the student inherits the nuanced force-modulation from the teacher. The authors note this explicitly: “knowledge distillation can successfully transfer complex tactile-aware capabilities from a sensor-dependent teacher policy to a sensor-agnostic student policy, enabling inference without physical tactile sensors”. Thanks to the blended training, the student learns to mimic not just the “what” (the final action) but also the “why” (why that action was chosen over a more forceful alternative). Blended Targets Provide a Strong Signal: A subtle point is that the student is not trained solely on the teacher’s outputs; they mix the teacher’s actions with ground-truth demonstrations. This avoids the common pitfall of blindly imitating a teacher that could be wrong. In HapticVLA’s words: ground-truth anchoring “prevents the student from amplifying teacher errors, keeping actions grounded in demonstrated behavior. Teacher shaping injects tactile-aware force modulation learned by the SA-RWFM teacher”. In the extreme fragile-case, for example, if the human demo had a slight misalignment, the teacher would correct it; blending ensures the student catches that correction without drifting off course. Generalization to New Objects: Impressively, even though HapticVLA is trained on a fixed set of objects (jar, waffles, egg), it shows robustness to new initial poses and to the reality gap between sim and real robot (thanks to the Isaac Sim episodes). Achieving 86.7% on held-out trials indicates that the distilled model really internalized the “essence” of gentle manipulation. Taken together, HapticVLA demonstrates that the essentials of tactile intelligence can be learned without actual touch at runtime. The two-step approach – first infusing haptic reward into learning, then distilling into a perception-only model – effectively closes the gap between “touch-aware” and “touch-blind” VLAs. It sidesteps the hardware challenges while still yielding nearly-human-safe behaviors. Comparison to Related Work. HapticVLA builds on several threads of recent robotics research. It sits at the intersection of vision-language models for control and tactile/force-aware manipulation:. Vision-Language-Action (VLA) Models: HapticVLA uses as its backbone the SmolVLA architecture, a lightweight transformer-based policy tuned for high-frequency robot control. Like other VLA systems (ActLM, OpenVLA, etc.), it employs flow matching to produce continuous actions. Its novelty is to combine this with the above haptic techniques. Tactile-Enhanced VLAs: Many prior works have augmented vision-based policies with tactile streams. For example, the FD-VLA model (cited in the intro) also distills force information into latent tokens, but still assumed some force sensor. Others like OmniVTLA or VLA-Touch train huge models on visual+tactile datasets for specific tasks. The key distinction is that all those approaches require specialized sensors or extra modalities during execution. By contrast, HapticVLA is unique in achieving tactile-level performance sensorlessly at run time. Offline Reward Learning: The concept of reward-weighted imitation has precedents in probabilistic inference and reinforcement learning. It echoes ideas like Advantage-Weighted Regression or Reward-Weighted Regression, where trajectories are re-weighted by returns. HapticVLA pushes this into the flow-matching framework. Its safety-focused reward design (explicit terms for drops and damage) is reminiscent of “damage-aware” benchmarks like OopsieVerse, but here used to alter the policy distribution. Distillation and Teacher-Student: Using blended targets for distillation is a known trick (see e.g. literature on DAgger, Dataset Aggregation, or works like ActDistill). HapticVLA’s twist is to apply it for cross-modal transfer: teacher has extra input modality (tactile) that student lacks. In that sense it is analogous to audio-visual distillation or teacher-student setups for multi-sensor robots. The success in this paper underscores that such cross-modal distillation can work even for subtle physical interactions. Simulation Use: The use of an Isaac Sim digital twin to expand data is in line with the broader trend of sim-to-real. Unlike some works that focus only on sim, HapticVLA always tests on a real robot (two SO-101 arms with Realsense cameras) and closes the loop with real touch data. The mixed real+sim dataset (310 real episodes + 1000 sim episodes) ensures good coverage. Overall, HapticVLA pushes the state of tactile-via-vision models forward by showing it is not necessary to carry expensive touch hardware into production. It should inspire others: whenever a sensor is hard to deploy, see if its effect can be distilled away by clever training. Discussion and Future Directions. What are the takeaways for practitioners? First, if you have a VLA system struggling with contact tasks, consider a two-phase training pipeline similar to HapticVLA’s. Even if your final robot has no touch sensor (or for reasons you cannot afford one), you can temporarily outfit your rig with one during data collection. Then use reward shaping on offline examples to teach the policy about gentle/firm actions. Finally, distill that learned touch-awareness into the final network. The fact that HapticVLA outperforms even sensorized baselines suggests this strategy is highly effective. Second, the reward design is crucial. HapticVLA doesn’t just rely on raw tactile signals; it abstracts them into safety rewards. This makes the approach potentially adaptable: if your task has other notions of risk (e.g. bending the robot joints too aggressively, or stirring too violently), similar penalties can be baked in. In effect, any measurable criterion of safety or quality can be integrated offline. Of course, limitations remain. HapticVLA was tested on three pick-and-place tasks involving mostly rigid objects (one very fragile, one somewhat flexible, one deformable plastic). It remains to be seen how well it generalizes to other families (e.g. cloth manipulation, ultra-soft items, or tasks like screwing/unscrewing where torque is key). Also, the tasks are still fairly structured (the jar, waffles box, and egg cartons have known layouts); truly open-world scenarios might challenge the visual generalization. Fine-tuning on every new category might be needed, as with most robot learning. Finally, HapticVLA’s success underscores a broader theme in robotics: closing the “modality gap” through learning. As multimodal machine learning advances, we will likely see more cases where one sensor’s information can be predicted or replaced by others, once the model has been trained. In HapticVLA’s case, vision alone became a proxy for touch, thanks to clever offline training. In sum, HapticVLA builds a bridge between touch and vision in robots. Its blend of safe imitation learning and distillation offers a recipe for enabling “sensitive” manipulation on hardware that isn’t physically sensitive. For researchers and engineers in embodied AI, it’s a clear demonstration that some forms of perception can be achieved in two stages: first experience with the sense, then internalize it into a purely visual routine. References: The descriptions above are based on “HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing” by Gubernatorov et al., including its abstract, methodology, and reported results. The reader may consult those excerpts for the original technical details and experimental numbers.