Latest / Embodied AI 101 / When Touch Means Something: OmniVTLA and the Missing Contact Layer in Robot Foundation Models

Transcript

When Touch Means Something: OmniVTLA and the Missing Contact Layer in Robot Foundation Models · Embodied AI 101

• When Touch Means Something: OmniVTLA and the Missing Contact Layer in Robot Foundation Models. OmniVTLA: Vision-Tactile-Language-Action Models With Semantic-Aligned Tactile Sensing • Zhengxue Cheng, Yiqian Zhang, Anni Tang, Keyu Wang, Wenkang Zhang, Haoyu Li, Hengdi Zhang, and Li Song • Shanghai Jiao Tong University and Paxini Tech • IEEE Robotics and Automation Letters • 2026. A vision-language-action model can recognize a bottle, understand the instruction “put it on the plate,” and generate a plausible reaching trajectory. Yet it may still fail at the first physically decisive moment: when the fingers touch the bottle. At that point, the scene may barely change in RGB. But the control problem changes completely. Success now depends on normal force, tangential load, compliance, slip, contact geometry, and whether the grasp is balanced. These quantities are either invisible or only weakly observable through external cameras. The model knows what a bottle is, but not necessarily whether it currently has a secure hold on this particular bottle. OmniVTLA addresses that gap by asking a more specific question than simply, “Should VLAs use touch?” Its question is: What kind of tactile representation should a pretrained VLA consume? The paper’s answer is a dual-path tactile encoder combining a conventional pretrained visual transformer with a tactile transformer explicitly aligned to visual and linguistic semantics. That encoder is trained using ObjTac, a new collection of synchronized force, video, and object-description data. The resulting policy is evaluated on real pick-and-place and peg-insertion tasks using both a parallel gripper and a tactile dexterous hand. The headline results are strong: a two-finger pick-and-place success rate of 96.9%, up 21.9 percentage points from the vision-language baseline, and an 83.3% peg-insertion rate, up 33.3 points. But the more interesting story lies beneath those aggregate numbers. OmniVTLA is an unusually useful study of tactile initialization, semantic pretraining, dual encoders, and the sometimes weak relationship between offline imitation metrics and actual closed-loop robot behavior. The Real Problem Is Not Merely Adding Touch. Tactile sensing is exceptionally heterogeneous. An optical sensor such as GelSight or DIGIT produces camera-like images of an elastomer’s deformation. A force-torque sensor returns a small vector. Magnetic skins and taxel arrays produce spatial fields of normal and shear forces. Other systems expose vibration, impedance, temperature, or event streams. That heterogeneity creates a representation problem. Treating every tactile observation as just another image is convenient, but the statistics of a magnetic force field are very different from those of a natural photograph. Conversely, building a separate architecture for every sensor makes it difficult to reuse models and data. AnyTouch, one of OmniVTLA’s main foundations, attacked this problem by learning sensor-agnostic tactile representations through masked modeling, multimodal alignment, and matching between different tactile sensors. The challenge is amplified inside a VLA. The visual and language encoders are generally not blank slates. They arrive with rich representations learned from vast image-text corpora. Their latent spaces already connect visual patterns to concepts such as glass, fabric, soft, rough, bottle, and sponge. A tactile stream trained only from a few dozen robot demonstrations enters the model at a severe disadvantage: it has little data, no broad semantic organization, and a sensor-specific input distribution. A straightforward tactile encoder trained from scratch can therefore overfit the demonstrations. A generic image encoder offers a better initialization, but its features were learned from reflected light rather than contact mechanics. A tactile foundation encoder captures contact structure, but may not be naturally compatible with the VLA’s existing visual-language space. OmniVTLA’s central hypothesis is that these options are complementary. One tactile path inherits broad visual pretraining, while a second path is deliberately trained so that touch, images, and words referring to the same physical object occupy related regions of representation space. The policy gets both a generic image-derived prior and a specialized tactile-semantic prior. It is helpful to be precise about the word semantic here. OmniVTLA is not teaching the tactile encoder to perform open-ended language reasoning. Its semantic supervision concerns object identity and physical attributes such as material, roughness, and hardness. That is narrower than general language understanding, but potentially very relevant to grasping: the policy should approach a milk carton, rigid can, towel, and sponge with different expectations about deformation and required grip. OmniVTLA, From Sensors to Actions. OmniVTLA is built on the architecture of Physical Intelligence’s π0, a flow-matching VLA that combines a pretrained vision-language stack with an action expert for continuous robot control. π0 was designed to inherit semantic knowledge from a vision-language model while generating continuous action chunks rather than discretizing every motor command into conventional language tokens. The OmniVTLA platform supplies the policy with a third-person RGB image, a wrist-camera image, a natural-language instruction, robot proprioception, and tactile readings from the end effector. Both RGB views are resized to 224 by 224 pixels and encoded using SigLIP-style visual processing, producing 256 tokens per image. Language passes through the PaliGemma tokenization stack. The force-based tactile readings are converted into a three-channel image representation. The paper applies max-min force normalization, reshapes the force tensor, and, when multiple sensors are present, stitches their arrays into one composite image. That composite is also resized to 224 by 224 and processed through the tactile encoders. The approach gives a common image-like interface to the two sensors on the parallel gripper and the 11 tactile sensors on the dexterous hand. This serialization choice is pragmatic. It lets the researchers reuse ViT machinery without designing a new tokenization system for every taxel layout. But it is not physically neutral. The arrangement of sensor patches in the stitched image implicitly defines which taxels are treated as neighbors. Moving to a new hand morphology, changing sensor placement, or dropping one sensor would require careful remapping. Semantic alignment does not automatically solve that topological problem. The tactile image enters two parallel branches:. The pretrained visual path uses SigLIP initialization, exploiting filters and representations learned from large-scale natural-image training. The SA-ViT path uses the semantically aligned tactile transformer trained with tactile, visual, and language data. The branches’ tactile tokens join the camera, language, and state tokens in a Gemma 2.6B-class backbone. A π0-style flow-matching head then predicts a chunk of continuous robot actions. For the parallel gripper, each action step contains three relative translation values, a continuous six-dimensional rotation representation, and one gripper command. For the dexterous hand, the final scalar is replaced by 16 joint commands, producing a 25-dimensional action. Conceptually, this is modality-specific encoding followed by token-level fusion. Touch does not merely trigger a separate reflex controller, nor is it converted into a textual description before control. It becomes another token stream inside the model’s main reasoning and action pathway. The policy variants are fine-tuned for 30,000 steps with a batch size of 32, a peak learning rate of (2.5 \times 10^{-5}), and 50-step action chunks. The reported training configuration uses an 80-gigabyte A100. Unlike the Diffusion Policy comparison, the π0-based models receive no observation history, meaning the architecture does not explicitly encode a temporal window of tactile changes. That last detail matters. OmniVTLA incorporates touch, but it is not fundamentally a high-frequency tactile reflex architecture. It encodes the current tactile observation and uses it to condition an action chunk. The paper itself identifies temporally dynamic fusion as future work. ObjTac: Connecting Force Fields to Concepts. The semantic tactile branch depends on ObjTac, the paper’s new dataset. ObjTac contains synchronized tactile, visual, and textual information for 56 objects grouped into ten broad categories: plastic, glass, wood, brick or stone, metal, fabric, leather, ceramic, paper, and a miscellaneous category. The inventory ranges from bottles, ceramic tiles, and metal boxes to paper products, clothing, sponges, plush toys, and an apple. Data was collected using a Paxini Gen2 force-based tactile sensor. The sensor uses Hall-effect measurements to observe changes in embedded magnetic elements as the surface deforms. Onboard processing estimates the magnets’ pose changes and then solves an inverse mechanics problem to produce multidimensional force distributions over the sensor surface. OmniVTLA therefore receives processed spatial force fields rather than raw magnetic measurements. Each object was touched in two to five trials lasting between 10 and 60 seconds. Tactile data was recorded at 60 hertz, while first-person RGB video was captured at 30 frames per second and 720p resolution. Across 252 videos averaging approximately 18 seconds, the collection contains about 135,000 video frames and 270,000 force samples. After temporal synchronization, the authors report 135,000 paired tri-modal samples. Language annotations are applied at the object level. They include the object name, material, binary roughness class, binary hardness class, video metadata, and textual descriptions. Thus, the semantic signal is strongest for persistent object properties. It is less directly informative about transient interaction events such as incipient slip, a changing contact patch, excessive shear, or the onset of jamming. SA-ViT’s training follows the second-stage alignment recipe developed for AnyTouch. Within each batch, the model is asked to align vision with language, vision with touch, and touch with language in both directions. A separate cross-sensor matching objective encourages tactile observations carrying corresponding object information to remain similar even when produced by different sensor technologies. This produces an important conceptual bridge. The existing tactile pretraining ecosystem is dominated by optical sensors whose outputs already resemble images. ObjTac adds a force-field sensor whose appearance is more abstract. By aligning both to common images and descriptions, the model can use semantics as an intermediary between otherwise incompatible sensor distributions. The perception results show both the value and the limits of this approach. On the in-domain ObjTac evaluation, SA-ViT reaches 70.44% material accuracy, 82.21% roughness accuracy, and 93.91% hardness accuracy, outperforming CLIP, SigLIP, a tactile ViT without alignment, and the original AnyTouch encoder. On the separate Touch and Go benchmark, however, AnyTouch remains stronger: 79.39% versus SA-ViT’s 74.90% on material, 86.32% versus 85.46% on roughness, and 95.16% versus 92.10% on hardness. The right interpretation is therefore not that SA-ViT is a universally superior tactile foundation model. It is better adapted to the authors’ force-sensing domain while retaining respectable performance on an optical-tactile benchmark. That is still valuable, but it is narrower than complete sensor invariance. There is another important dataset caveat in the appendix. ObjTac’s completed collection is the Touch process. A separate Grasp process—intended to cover successful and failed grasps, stable holding, controlled releases, and slip events—is described as planned future work. The currently reported dataset is consequently much richer in object-touch associations than in manipulation outcomes. The Ablation Ladder Is the Paper’s Strongest Feature. Rather than compare only a vision-only baseline against the final model, OmniVTLA constructs a useful progression of tactile encoders. VTLA-FS trains a tactile encoder from scratch using only the robot demonstrations. VTLA-Pre starts from a pretrained visual encoder. VTLA-SA uses only the semantically aligned tactile encoder. VTLA-Pre-Pre is a parameter-matched dual encoder composed of two generic visual backbones. VTLA-Pre-Tac combines the visual branch with a tactile masked-autoencoder branch that lacks semantic alignment. Finally, OmniVTLA pairs the pretrained visual branch with SA-ViT. This design tests several distinct questions. Is touch useful at all? Does generic pretraining help? Does tactile pretraining help? Does semantic alignment add anything beyond tactile reconstruction? Does a second encoder merely win by adding parameters?. The parameter-matched dual-encoder controls are especially important. Both VTLA-Pre-Pre and VTLA-Pre-Tac have roughly the same architectural capacity as OmniVTLA, yet generally perform worse online. The evidence therefore supports the view that representation choice matters, rather than the result being solely a consequence of inserting another transformer. There is still a causal ambiguity. In the tactile perception table, SA-ViT is the only specialized tactile encoder marked as having seen ObjTac during pretraining. The non-aligned tactile ViT did not. The experiments convincingly validate the complete package—ObjTac exposure, multimodal alignment, and dual-path integration—but do not perfectly isolate semantic alignment from in-domain force-sensor pretraining. What the Robots Actually Did. The main platform is a UR5 arm equipped either with a two-finger tactile gripper or a four-finger tactile dexterous hand. Both configurations have a wrist camera, and the workspace is also observed by a base camera. The gripper carries two tactile sensors; the hand carries eleven. For parallel-gripper pick and place, the authors use a short can, square coffee bottle, gum tin, and milk carton. They collect 40 teleoperated demonstrations per object at 30 hertz. Evaluation consists of 32 rollouts over grid-based initial poses, with the demonstrator and policy allowed repeated grasp attempts. Two additional soft objects—a towel and sponge—are held out for generalization tests. The dexterous-hand experiment trains on a coffee bottle and milk carton, then also evaluates a tall plastic bottle and square bottle not present in that training set. The dexterous hand’s 16 joints are controlled directly by the action output. Peg insertion uses triangular, square, and round pegs with clearances between 0.9 and 1.8 millimeters. Demonstrations are collected for the triangular and square shapes, while the round peg is reserved as an unseen geometry. Each shape receives ten real-world evaluation trials. The robot must grasp the peg, move it above the hole, and use repeated adjustments to complete insertion despite visual occlusion and contact uncertainty. The online metrics are success, completion steps, end-effector smoothness, and gripper-command smoothness. Spatial smoothness is based on acceleration variability, while gripper smoothness uses the mean absolute change in gripper commands. The authors also report offline action-prediction mean squared error, which leads to one of the paper’s most revealing results. Reading the Results Beyond the Headline. On the parallel-gripper task, the vision-language π0 baseline succeeds on 75% of trials. OmniVTLA reaches 96.9%—equivalent to 31 successes out of the stated 32-rollout protocol, versus 24 for the baseline. Completion falls from an average of 657 to 498 steps, a reduction of 24.2%. The gains are not distributed evenly. The baseline already achieves 100% on the milk carton and gum tin. Most of OmniVTLA’s improvement comes from the short can, which rises from 62.5% to 100%, and the bottle, which rises from 37.5% to 87.5%. That concentration is encouraging: touch helps most where vision-only control is actually struggling rather than merely inflating performance on already-solved objects. The single SA-ViT variant is also revealing. VTLA-SA reaches an 87.5% success rate—below the full model—but completes successful trials in 484 steps, slightly faster than OmniVTLA. It is also the smoothest model according to both reported smoothness metrics. The dual path therefore appears to favor robustness and final success, while the specialized semantic path alone produces more economical and stable trajectories. On the dexterous hand, the baseline is already strong at 93.8%, and OmniVTLA reaches 100%. The two training objects are solved by both systems. The difference comes from the unseen bottle variants, where the baseline scores 87.5% and OmniVTLA scores 100%. Completion decreases from 343 to 322 steps. This is directionally positive, though the small number of failures and ceiling-level baseline make it less decisive than the gripper and insertion results. The peg-insertion experiment provides the clearest evidence for contact-driven generalization. The vision-language baseline averages 50%: six successes on the triangle, six on the square, and three on the unseen round peg. OmniVTLA succeeds seven, nine, and nine times respectively, for 25 successes out of 30 and an 83.3% average. The unseen round shape jumps from 30% to 90%. Yet insertion completion speed tells a more complicated story. OmniVTLA averages 802 steps, only slightly below the baseline’s 816. VTLA-SA finishes in 678 steps. The full dual encoder seems willing to spend additional adjustments to secure a successful outcome, especially on the unseen round peg, where it averages 1,058 steps. That may be an entirely reasonable control strategy, but it means “better” depends on whether the priority is success, speed, or force-efficient insertion. The Diffusion Policy comparison supports the broader claim that touch helps outside π0. Adding tactile input to the from-scratch Diffusion Policy setup increases average pick-and-place success from 59.4% to 78.1% and reduces completion from 851 to 682 steps. Because the Diffusion Policy and π0 systems use different image sizes, histories, training schedules, and state representations, this should not be read as a ranking between the two backbones. It is evidence that the benefit of tactile observation is not unique to one VLA implementation. Perhaps the most important result for policy-learning researchers is that offline action error does not predict online success. VTLA-Pre-Tac achieves the lowest validation MSE, while OmniVTLA is only second best. Online, however, VTLA-Pre-Tac reaches just 84.4% on gripper pick and place and 43.3% on insertion, compared with OmniVTLA’s 96.9% and 83.3%. This is a textbook example of why demonstration-regression metrics can be misleading. Multiple actions may be valid away from contact, while tiny errors at the moment of contact can determine success. A model can closely imitate the average trajectory yet fail to react to slip or jamming. Conversely, a policy that deviates from the demonstration to retry a grasp may incur higher offline error and still be much more competent online. The soft-object experiments expose another metric trade-off. VTLA-FS and VTLA-Pre achieve the highest success, averaging 93.8%, but close the gripper more aggressively. VTLA-SA produces the gentlest grasp according to the paper’s deformation-related gripper proxy, but succeeds only 62.5% of the time. OmniVTLA occupies the middle ground: 81.3% success with substantially less closure than the from-scratch model. That result is consistent with semantic tactile pretraining helping the policy recognize compliance. But it also demonstrates why success rate alone is inadequate for gentle manipulation. A controller that picks up every sponge by crushing it can dominate a success-only benchmark. Future evaluations should include peak normal force, deformation, damage, slip margin, and grasp stability rather than relying on gripper commands as an indirect proxy. What OmniVTLA Establishes—and What It Does Not. The paper provides strong evidence for three conclusions. First, tactile feedback substantially improves these real-world manipulation tasks. The gains appear with both π0 and Diffusion Policy, across two end effectors, and are largest in visually ambiguous contact states. Second, tactile pretraining matters. Training the tactile branch only on a small number of demonstrations is unreliable and prone to overfitting. Generic visual initialization helps, but an encoder exposed to tactile data and physical-property supervision generally transfers better. Third, the semantic and generic paths are complementary. Parameter-matched dual-encoder controls do not recover OmniVTLA’s online success, suggesting that the final gains are not simply due to model size. The evidence for broader sensor generalization is more limited. ObjTac and the perception evaluations connect optical and force-based sensing at the representation level, but the manipulation policy is not transferred between substantially different tactile hardware. A convincing sensor-agnostic control experiment would train on one end-effector and sensor family, then adapt or deploy on another with minimal policy data. The semantic labels are also coarse. Material, binary roughness, and binary hardness are useful priors, but contact-rich control often depends on state-dependent concepts: slipping, sticking, rolling contact, edge contact, underactuated motion, excessive shear, partial insertion, and stable seating. Aligning touch with those interaction verbs and contact states may be more valuable than aligning it only with object nouns and static properties. Temporal processing remains another open issue. OmniVTLA’s policy configuration uses no explicit observation history, despite ObjTac containing time series. An isolated force map can reveal load distribution, but slip and jamming are often best detected through derivatives and temporal patterns. The paper acknowledges this limitation and calls for temporally dynamic fusion. The normalization procedure also deserves clarification. Absolute force magnitude is often safety-critical. The paper reports max-min normalization and conversion to an eight-bit image representation, but does not make clear whether normalization statistics are global, per sensor, per episode, or per frame. Per-frame normalization could suppress absolute load information, whereas calibrated global normalization would preserve it. This detail matters when adapting the method to insertion, fragile-object handling, or force-limited interaction. Finally, the evaluation is still small. Pick-and-place percentages move in increments of 12.5 points per object, and insertion percentages move in increments of ten. No confidence intervals or repeated training seeds are reported. The results are compelling demonstrations, especially the 25-versus-15 insertion outcome, but they are not yet large-scale estimates of deployment reliability. Reproducibility is incomplete as well. The dataset is linked from the project site, but as of September 1, 2026, the code interface is still marked “coming soon.” Details such as the complete pretraining mixture, exact alignment weights, sensor-stitching layout, and normalization implementation will be difficult to reproduce faithfully until the training code is released. Where OmniVTLA Fits in the Tactile-Policy Landscape. The emerging tactile-policy literature can be organized around three questions: what tactile features should represent, where they should enter the policy, and how quickly they should affect control. AnyTouch focuses primarily on the first question. It learns static and dynamic tactile representations across multiple optical sensor families, using masked reconstruction for local detail and multimodal alignment plus cross-sensor matching for sensor-agnostic semantics. OmniVTLA adopts that alignment philosophy, extends it with force-based ObjTac data, and tests whether the resulting initialization improves an end-to-end VLA. The earlier VTLA insertion model emphasizes a different bottleneck. It encodes visual and short tactile sequences as multimodal tokens, then combines next-token supervision with Direct Preference Optimization to make discretized action predictions behave more like continuous control. Its central innovations are temporal token construction and the action-learning objective, whereas OmniVTLA’s central innovation is the tactile encoder. ForceVLA moves fusion toward the action decoder. It introduces a force-aware mixture of experts that dynamically combines visual-language features with six-axis force measurements during action generation. Tactile-VLA goes further toward explicit control structure, combining tactile-aware reasoning with a hybrid position-force controller. These methods ask the model to decide when and how force should affect control, rather than relying mainly on semantically improved input tokens. Reactive Diffusion Policy attacks the temporal-rate problem directly. Its slow pathway generates longer-horizon plans, while a fast tactile pathway modifies behavior at high frequency inside action execution. That division is attractive when tactile events happen faster than the main policy can afford to re-encode cameras and run a large language backbone. More recent work such as N0-VTLA points toward predictive tactile representations. Instead of only encoding the current contact map, it predicts latent tactile tokens describing expected contact changes over the coming action chunk and conditions the action expert on them. This suggests a likely next step beyond OmniVTLA: combine semantic tactile initialization with temporally predictive, contact-reactive control. How I Would Adapt the Idea. For a lab adding touch to an existing VLA, OmniVTLA offers a practical template, but several modifications would make it stronger. Preserve the sensor’s physics during tokenization. An image-shaped interface is convenient, but define channels and spatial layout according to physical meaning. Retain calibrated force magnitude, provide explicit no-contact references, and include sensor identity or geometry embeddings when combining multiple fingers. Align touch with interaction states, not only object properties. Material, hardness, and roughness are good starting points. Add descriptions and labels for stable grasp, edge contact, slip, shear direction, excessive compression, alignment, jamming, seating, and release. These are closer to the decisions the policy must make. Separate slow semantic interpretation from fast tactile reaction. The semantic encoder can tell the policy that it is handling a soft textile or rigid metal part. A smaller temporal pathway should detect millisecond-scale changes and modify grip or end-effector motion without waiting for a full VLA forward pass. Evaluate online with competing objectives. Report task success, attempts, completion time, peak and integrated force, object deformation, slip events, intervention count, and hardware safety violations. Offline MSE should remain a debugging metric, not the primary model-selection criterion. Use true domain holdouts. Split tactile pretraining data by object and collection sequence rather than by correlated frames. Hold out entire sensor models, finger morphologies, material families, and contact procedures if cross-sensor or semantic generalization is the claim. The dual encoder is most attractive in a moderate-data regime where neither generic visual pretraining nor a specialized tactile model is sufficient alone. For latency-sensitive deployment, however, I would consider distilling the two encoders into a compact tactile backbone or activating the specialized path only after contact. Before contact, the force map often contains mostly sensor background; after contact, it becomes central. I would also add modality dropout during training. A robot should degrade gracefully if one finger loses communication, a taxel saturates, or the tactile image is temporarily invalid. Cross-sensor alignment helps representation transfer, but robustness to missing and corrupted sensors must be trained explicitly. Final Takeaway. OmniVTLA’s lasting contribution is not simply the observation that tactile sensing improves peg insertion. Roboticists have known for decades that touch matters. The contribution is a concrete argument that touch should arrive inside a VLA with structure and prior knowledge, rather than as an anonymous tensor learned from a few demonstrations. The paper’s best evidence is the combination of parameter-matched encoder ablations and real-world closed-loop evaluation. Semantic tactile pretraining improves behavior where offline action prediction can be actively misleading, while the dual path raises success beyond either generic or specialized tactile encoding alone. The current system remains static-frame-heavy, sensor-layout-specific, and evaluated on a modest task set. But it establishes a valuable design direction: future embodied foundation models should not merely possess more sensors. Their sensory streams must share enough meaning to cooperate—while still preserving the temporal and physical information that makes each modality unique.