Latest / Embodied AI 101 / Before the Robot Feels It: How N₀-VTLA Predicts Contact to Control It

Transcript

Before the Robot Feels It: How N₀-VTLA Predicts Contact to Control It · Embodied AI 101

• Before the Robot Feels It: How N₀‑VTLA Predicts Contact to Control It. $N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens • NeoteAI Team & Fudan TEAI Team • NeoteAI and Fudan University TEAI • arXiv preprint • 2026. A robot can recognize a plug, localize an outlet, and move the plug into approximately the right place—and still fail in the final millimeter. The same is true for seating a circuit board, standing up a compliant bottle, separating stacked containers, or folding a towel without dropping it. In these tasks, vision gets the robot to the interaction, but contact determines the outcome. N₀-VTLA is NeoteAI and Fudan TEAI’s attempt to move tactile sensing from a task-specific accessory into the foundation-model stack. It combines three ideas: large-scale visuo-tactile pretraining on NeoData, a predictive tactile pathway that conditions a flow-matching action generator, and an offline policy-improvement method called ALTER that learns from demonstrations, autonomous failures, human corrections, and recovery trajectories. The headline results are strong: higher success than ACT and π₀.₅ on all nine evaluated NeoReal tasks, 63.8% mean success across twenty simulated tasks, and between 75% and 95% success after offline improvement on three long-horizon physical tasks. But the most interesting part is not the benchmark number. It is the paper’s claim about where touch belongs inside a VLA. N₀-VTLA does not simply append tactile images to the visual prefix. Nor does it feed the latest tactile frame directly into the action decoder. Instead, it predicts a compact latent representation of how contact is expected to change during the coming action chunk, then asks the action expert to generate motion conditioned on that predicted contact. That design is both clever and more limited than the phrase “predictive touch” might initially suggest. Understanding that distinction is the key to understanding the paper. Why Treating Touch as Another Camera Is Not Enough. Most contemporary VLAs inherit an architectural assumption from vision-language modeling: observations are information-rich tokens placed into a prefix, and actions are decoded from that prefix. This works naturally for cameras. A visual frame contains object identity, geometry, layout, appearance, and scene context even when the robot is not interacting with anything. A tactile image has a different temporal structure. Away from contact, it is largely a picture of the sensor’s undeformed surface. During manipulation, the useful information arrives in short bursts: initial contact, pressure redistribution, shear, slip, jamming, release, and impact. N₀-VTLA argues that simply placing these sparse tokens beside hundreds of visual tokens makes it too easy for the policy to ignore touch. Feeding only the current tactile state to the action expert is better, but still reactive: it reports what the preceding action has already done rather than what the next action should be prepared to encounter. The companion N₀-Foundation report provides some evidence for this architectural ordering. In a controlled study using a fixed π₀.₅ backbone, raw tactile-image concatenation increased mean NeoReal success only from 26.5% to 27.5%. Conditioning the action expert directly raised it to 30%, while conditioning on the physically grounded NeoForce representation reached 32.5%. Those are not the N₀-VTLA results, but they support the broader point that both representation and placement matter. N₀-VTLA’s alternative is to turn touch into a prediction target. The policy observes the current scene and current contact, but the latent supplied to the action generator is trained to match the tactile change that will occur over the next action horizon. The intended effect is to make contact an anticipated consequence rather than an extra image modality. There is an important nuance here. The tactile predictor does not receive a candidate future action and simulate its consequences. It predicts expected future contact from the current vision-language context and tactile state. In probabilistic terms, it is closer to learning the contact transition normally associated with a task and policy state than to learning an action-conditioned tactile dynamics model. My reading is that the latent therefore functions as a compact, policy-consistent contact intention or affordance: “this is the kind of contact that should happen next.” It cannot explicitly compare several candidate actions by forecasting the tactile outcome of each one. That places N₀-VTLA somewhere between reactive tactile conditioning and a full tactile world model. The Architecture: A π₀.₅-Style Policy with a Predictive Tactile Side Path. The base policy follows the π₀.₅ recipe. A PaliGemma vision-language backbone processes camera views, the language instruction, and discretized robot state. A separate action expert uses flow matching to denoise a continuous future action chunk. The public implementation describes a PaliGemma prefix built around a Gemma 2B language model and a roughly 300-million-parameter Gemma action expert; the released checkpoint is listed as approximately four billion parameters and about 8.25 gigabytes. The action horizon is fifty steps. Every embodiment maps into a canonical 32-dimensional container. The first ten dimensions represent one arm’s end-effector position, six-dimensional rotation representation, and gripper channel. The next ten repeat that structure for a second arm. The final twelve dimensions are reserved. This allows single-arm, dual-arm, and handheld collection embodiments to share the same model interface. Touch enters through a separate route. Each episode begins with the gripper open and contact-free for at least half a second. The system saves a tactile baseline during this period. At every subsequent observation, it subtracts that baseline from the current tactile image. This suppresses the static appearance of the gel, illumination, and some sensor-mount signatures, leaving an image dominated by deformation caused by contact. The paper is careful not to claim complete sensor invariance; baseline differencing reduces hardware-specific appearance but does not eliminate it. A frozen DINOv2 encoder processes each difference image. In the paper’s formulation, its patch grid is pooled to a class token plus a three-by-three spatial grid, yielding ten tactile tokens per active sensor view. A learned linear projection maps them into the width used by the vision-language model. Freezing DINOv2 preserves its representation, reduces optimizer memory, and, in principle, lets a new sensor be accommodated by retraining only the lightweight projection. The tactile predictor then combines those current-contact tokens with the contextualized visual and linguistic prefix. Learned queries distill the combined information into latent tactile tokens, denoted by the paper as (z). If tactile input is unavailable, the model uses a learned null representation, allowing the policy to fall back to vision-language control instead of failing at the interface level. What turns (z) into a predictive representation is its training target. For each tactile view, the system compares the tactile frame at the current instant with the frame fifty steps later. It encodes that future difference using the same frozen tactile encoder and averages the target tokens across the active tactile views. The predictor is trained to identify the corresponding future target using a symmetric contrastive objective. An auxiliary reconstruction head must also recover a coarse spatial map of future tactile change, discouraging the model from winning the contrastive task through a non-spatial shortcut. Notice what is—and is not—being predicted. The target is the net tactile difference between now and the end of the chunk. It is not a frame-by-frame tactile rollout, an explicit force trajectory, a probability of slip, or a contact-state machine. All of those quantities may be encoded implicitly, but the supervision itself is a compact endpoint change. The predicted tactile tokens are projected into the action expert’s width and placed before the noisy action tokens. The current tactile tokens never enter the action expert directly, and tactile information never enters the vision-language prefix as an extra camera. Action queries can attend to the predictive tactile block, while the tactile tokens themselves are removed before the action output head. This gives N₀-VTLA a clear modality separation. Vision and language describe the global scene and task. Current touch tells the predictor what contact state the robot is in. The predictor estimates the contact transition expected over the next chunk. The action expert generates motion conditioned on that estimate. Predictive, but Not Tactile Servoing. The deployment schedule deserves special attention. NeoData is synchronized at 30 frames per second, and a chunk contains fifty steps. A complete chunk therefore spans nominally about 1.67 seconds. The deployment protocol explicitly requires executing all fifty actions before requesting another prediction. Replanning early can repeatedly discard gripper commands that occur near the end of the chunk, leaving the arm at the object while the gripper appears never to close. That means N₀-VTLA is closed-loop at the action-chunk boundary, not at tactile servo rate. New tactile measurements arising midway through a chunk are not used until the next inference request. The tactile latent helps the model anticipate what will happen during that interval, and the chunk can contain a prelearned sequence of approach, contact, adjustment, and retreat. But genuinely unexpected contact cannot trigger an immediate high-frequency correction. This makes the paper’s predictive framing especially consequential. A long open-loop interval is exactly where a useful contact forecast can compensate for the lack of continuous feedback. It also means that anyone comparing N₀-VTLA with impedance control, force servoing, or receding-horizon tactile policies should not conflate the mechanisms. This is a tactile-conditioned chunk policy, not a learned high-bandwidth contact controller. Why the Tactile Path Is Trained in Three Stages. Adding a randomly initialized modality branch to a pretrained VLA creates two straightforward failure modes. The backbone may ignore the new branch because vision already predicts the demonstrations reasonably well. Or unrestricted joint training may allow the tactile latent to drift into an arbitrary appearance or task code rather than representing contact. The paper addresses this through staged optimization. In Stage 1, the entire base VLA is frozen. Only the tactile projection, predictor, and reconstruction head are trained. Contrastive future-target matching gives the latent discriminative contact information, while coarse reconstruction preserves spatial contact structure. Because the trainable surface is shallow, this stage is relatively inexpensive. In Stage 2, the tactile perception stack is frozen at its grounded checkpoint. The vision-language prefix is masked from the action expert, and only the latent-to-expert projection and action expert are trained. The expert cannot take the visual shortcut; if it wants to reduce action error, it must learn to read (z). In Stage 3, the mask is removed and everything except the frozen DINOv2 tactile backbone is jointly optimized with the normal action loss. Vision, language, and predictive touch can now specialize together. The public code adds another stability device: a zero-initialized gate on the tactile tokens, so the model begins exactly at the base-policy behavior and gradually opens the tactile channel. This sequence is arguably the paper’s most reusable contribution. It treats modality integration as an interface-alignment problem. First give the new representation physical meaning. Then force the consumer to use it. Only then permit full end-to-end adaptation. NeoData: Scaling Touch Beyond a Handful of Laboratory Skills. The training recipe rests on an unusually large tactile corpus. The full NeoData collection contains more than 30,000 hours, 1.4 million episodes, 3.3 billion timesteps, eight billion RGB frames, and ten billion tactile frames. It covers more than 450 tasks across five fixed robot platforms and a handheld TacUMI collection system. Nearly one hundred operators contributed data, and the handheld system accounts for 57% of trajectories. The embodiments include Franka, Piper, ARX X5, UR5e, Flexiv, and handheld single- or dual-gripper configurations. Tasks span precision assembly, deformable-object handling, pouring, wiping, opening, transport, semantic manipulation, and bimanual coordination. Every participating gripper finger carries the team’s vision-based tactile sensor. An open subset, OpenNeoData, releases more than 5,000 hours and 200,000 trajectories covering 257 task types. It is packaged in LeRobot format, divided by embodiment, and totals roughly 9.07 terabytes. Its RGB and gel-camera streams are aligned at 30 frames per second. Access is gated through Hugging Face, but the data itself is released under a noncommercial Creative Commons license. Data engineering receives more attention than usual. The pipeline checks coordinate frames, action units, absolute end-effector poses, video keyframe intervals, native tactile resolution, baseline validity, stream completeness, and media corruption. The paper notes that many of these failures do not necessarily make training loss look abnormal: a coordinate or action-convention mistake can produce a smoothly converging but unusable robot policy. Despite the “foundation model” terminology, the reported downstream numbers do not come from deploying one universal policy on every task. Each real task is adapted using a few hundred demonstrations, and each simulation task receives its own policy trained from one hundred demonstrations. The foundation-model claim therefore applies primarily to the pretrained initialization and shared tactile representation, not to zero-shot task generality in the evaluation. ALTER: Offline Improvement from Failures and Recoveries. The second half of N₀-VTLA addresses a different weakness of imitation learning. Clean demonstrations show how a task should proceed, but they rarely contain the states that dominate deployment: a towel slips, a box flap folds the wrong way, an object falls from the gripper, or a human intervenes midway through an autonomous attempt. ALTER stands for Advantage Labeling from Trajectory Events and Relative Progress. It uses a fixed deployment corpus containing clean teleoperation, autonomous policy rollouts, human-in-the-loop corrections, and recovery trajectories that deliberately begin from selected error states. Optimization itself requires no further environment interaction, although collecting the deployment corpus obviously required prior robot operation. The first step is constructing task progress supervision. A language model proposes a hierarchy consisting of the overall task, ordered stages, and low-level steps. A human reviews that template once. Individual demonstration boundaries are then localized using tactile contact changes, end-effector motion, gripper state, and visual event cues. Importantly, the tactile signals help create supervision; the project page says these signals are not necessarily inputs to the progress model itself. Stages are not assigned equal portions of the progress scale. Instead, each stage’s weight is proportional to its average duration in the clean demonstrations. A long manipulation phase therefore occupies more of the progress interval than a brief transition. This is more faithful to demonstrated temporal structure than uniformly spacing stage boundaries, although it also bakes the demonstrations’ timing conventions into the learned notion of progress. Clean demonstrations provide dense ordered pairs: one observation should score ahead of another by a known progress difference. Imperfect trajectories add sparse preference pairs around events. The frame after an object drop should rank below the frame before it; an observation after a successful human correction should rank above the state that triggered the intervention. A task-specific pairwise progress model learns from both types. Once trained, the progress model evaluates every retained trajectory in two ways. Comparing the current observation with the episode start estimates global task phase. Comparing it with the observation one action chunk later estimates local improvement or regression. Samples are compared only with other samples assigned to the same stage, preventing “late in the task” from automatically meaning “good action.” The top 30% of local changes within each stage are labeled positive, and the rest negative. Those labels become text. The policy sees either or appended to the task prompt while training on the entire deployment corpus. At inference, it is always prompted with the positive condition. The architecture and flow-matching action loss remain unchanged. This is offline RL in the broad sense of advantage-conditioned policy extraction. It does not use Bellman backups, an explicit Q-function, or a conventional actor-critic update. Operationally, it is conditional behavior learning where a progress model partitions stored actions into relatively productive and unproductive distributions. Whether one calls that offline RL or advantage-labeled imitation is partly taxonomic, but the mechanism is straightforward and potentially easy to transplant into other generative policies. The deployment corpora are substantial. Towel Folding uses 351 clean demonstrations, 286 autonomous rollouts, and 180 human-in-the-loop episodes. Bag Packing uses 213, 243, and 238. Cardboard Box Folding uses 550, 587, and 397. The human-in-the-loop totals include dozens of staged-recovery episodes for each task. What the Experiments Actually Establish. On nine real-robot NeoReal tasks, N₀-VTLA is compared with internally reproduced ACT and π₀.₅ policies. It achieves 47.2% average success, versus 29.4% for π₀.₅. Mean progress rises from 42.3 to 56.8 points. ACT completes none of the tasks and averages 10.2 progress points. N₀-VTLA has higher success than the strongest of those two baselines on every task and a higher progress score on eight of nine. The behavioral examples are more informative than the mean. Socket Plugging reaches 85% success versus 60% for π₀.₅. Board Insertion reaches 25%, while ACT and π₀.₅ both remain at zero. In reported rollouts, the visual baseline commits to insertion once the pose looks aligned and continues pressing when the plug or board meets an edge. N₀-VTLA lifts, realigns, and tries again after detecting unexpected contact. Bottle Standing tests a different contact regime. The object is a compliant empty bottle. π₀.₅ closes too far and lifts the bottle rather than stabilizing it against the surface. N₀-VTLA makes smaller gripper-aperture adjustments and reaches 30% success where both real-robot baselines score zero. The sensor does not directly measure force, but gel deformation provides a signal correlated with contact loading. The simulation campaign combines eight UniVTAC tasks with twelve NeoSim tasks. N₀-VTLA averages 83.1% on UniVTAC, compared with 67.1% for InternVLA-A1, the strongest external baseline there. On the harder NeoSim suite, it obtains 50.8% against 45.8% for π₀.₅. Weighted across all twenty tasks, that becomes the headline 63.8% versus 44.0%. That aggregate should not be read as universal dominance. N₀-VTLA is not best on every simulated task. π₀.₅ slightly leads on Insert Screw and Stack Plates; StarVLA-α leads on Grasp Chip; and Xiaomi-Robotics-0 obtains 71% on Unstack Cup, where N₀-VTLA reaches only 10%. Dual-arm performance remains a major bottleneck: N₀-VTLA averages 73.8% on the four single-arm NeoSim tasks but only 39.4% on the eight dual-arm tasks. One architectural detail may be relevant here, although the paper does not test this hypothesis directly. Current tactile tokens preserve separate sensor views by concatenating them, but the Stage 1 future-contact target is averaged across active tactile views. For bimanual tasks, averaging can discard which finger or arm should experience a particular future contact. A view-aware or embodiment-aware future target may preserve the asymmetry needed for handovers, coordinated insertions, and mutual support. The representation probes support the claim that (z) contains predictive contact information. Across 378 held-out queries, it retrieves its matching future-touch target 92.3% of the time from pools of roughly 32 candidates, compared with 3.2% chance. Using only the current tactile encoding reaches 57%. With 128 candidates, the predictive latent maintains 81% while the current encoding falls to 40%. Perturbation tests show that swapping tactile input initially changes the latent much more than swapping RGB and language. After end-to-end training, that ratio decreases, suggesting that the final latent mixes tactile evidence with task context rather than remaining a purely tactile code. A counterfactual action probe further shows that disabling touch changes the sampled trajectory at firm contact and grasp, while leaving it almost unchanged in free space. The latter is encouraging, though it is illustrated on a small number of qualitative episodes rather than a broad causal intervention study. ALTER raises success to 95% for Towel Folding, 80% for Bag Packing, and 75% for Cardboard Box Folding. Applying the same method to π₀.₅ reaches 90%, 75%, and 60%, so the tactile-pretrained initialization retains a consistent advantage after offline improvement. The final N₀-VTLA policies complete 43, 15, and 30 successful executions per hour on the three tasks. Where N₀-VTLA Sits Among Tactile Policies. ForceVLA represents the reactive force-sensing branch of this design space. It fuses real-time six-axis force-torque measurements into action decoding through a force-aware mixture of experts. Its core question is how current measured force should regulate motion. N₀-VTLA instead uses high-resolution optical touch and asks what contact change should accompany the coming action chunk. OmniVTLA focuses on tactile representation alignment across vision-based and force-based sensors. It trains a semantically aligned tactile encoder using the ObjTac dataset, which contains 135,000 vision-language-touch samples from 56 objects. Its emphasis is making tactile features compatible with semantic VLA representations; N₀-VTLA’s emphasis is temporal prediction and the location at which tactile features condition action. Tactile-VLA takes another route, combining tactile-aware reasoning with hybrid position-force control and using a pretrained VLM’s physical knowledge to generalize from a small number of demonstrations. HapticVLA goes in the opposite hardware direction: it uses tactile information during training, then distills a tactile token that can be predicted from vision and robot state so no tactile sensor is required at deployment. N₀-VTLA retains measured touch at inference because its aim is to respond to contact that vision cannot reliably infer. FTP-1 is perhaps the strongest comparison on tactile pretraining scale and sensor diversity. It aggregates about 3,000 hours from 26 sources and 21 sensor types, using heterogeneous encoders to produce morphology-aware tokens. N₀-VTLA is trained against a much larger overall corpus, but primarily around the team’s own vision-based tactile hardware. FTP-1 more directly targets cross-sensor generalization; N₀-VTLA more directly connects touch to a language-conditioned VLA and predictive action generation. Finally, the sibling N₀-TWAM report shows the more ambitious endpoint of this trajectory. N₀-TWAM jointly predicts future video, tactile state, and actions inside a world-action model. Relative to that system, N₀-VTLA is the lighter-weight option: it predicts a compact contact latent instead of decoding an entire sensory future. The trade-off is efficiency and architectural compatibility versus richer action-conditioned world modeling. What You Can Reuse—and What Is Still Missing. The public repository includes pretrained base weights, a task-level post-training toolkit, data conversion scripts, normalization utilities, real-robot and simulator servers, and example checkpoints for two simulation tasks. The base checkpoint is not directly deployable; it must be fine-tuned on a target task. The team explicitly does not release the large-scale pretraining pipeline. Adapting the model requires more than adding tactile files to an existing dataset. Episodes must be converted into the canonical 32-dimensional layout, camera and tactile slots must remain physically consistent, all streams must be synchronized, and normalization statistics must be recomputed for the new robot and action convention. A plain π₀.₅ checkpoint is not sufficient because it lacks the pretrained tactile predictor and projections. The baseline convention is also part of the model contract. Every episode must start contact-free, and deployment must reset the stored tactile baseline before the first observation. Drift, gel replacement, lighting changes inside the sensor, or accidentally capturing a baseline while touching an object will change every subsequent tactile difference image. There are a couple of documentation mismatches worth checking before reproduction. The paper formalizes (z) as ten latent tokens, while the repository summary describes five. The paper also describes an unmasked 32-dimensional base objective with unused dimensions trained toward zero, while the post-training documentation says inactive single-arm dimensions are masked. These differences may reflect a distinction between large-scale research training and the released adaptation path, but users should follow the exact checkpoint configuration and code rather than assuming the PDF and repository are byte-for-byte descriptions of the same implementation. Reproducing the reported foundation model remains difficult. The paper says pretraining used multi-node cluster-scale training but omits accelerator types, device counts, memory requirements, throughput, and detailed compute cost. Combined with the unavailable pretraining pipeline and the size of NeoData, the release is best viewed as a strong post-training starting point rather than a completely reproducible foundation-model recipe. The Main Open Questions. The first question is whether future-touch prediction remains useful with shorter receding-horizon execution. Requiring a complete fifty-step chunk is a strong deployment constraint. A streaming or overlapping-chunk policy could combine N₀-VTLA’s anticipation with genuinely rapid tactile feedback, but would require training the gripper schedule and action representation for that execution contract. The second question is whether the predictor should be action-conditioned. N₀-VTLA predicts the contact future associated with the current task context; it does not score alternative actions by their tactile consequences. Conditioning tactile prediction on candidate actions—or jointly denoising action and touch—would move it closer to model-predictive control and to the N₀-TWAM formulation. Third, sensor diversity remains unresolved. Baseline differencing and a frozen visual encoder are practical engineering choices, but they do not by themselves establish transfer from optical gel sensors to taxels, magnetic skins, six-axis force-torque sensors, or dexterous hands. The authors’ own future-work discussion points toward broader tactile hardware coverage. Fourth, the evaluation still leaves considerable headroom. The real-robot mean is 47.2%, so the system fails more than half its rollouts before ALTER. Dual-arm simulation remains below 40%. ALTER is evaluated on only three long-horizon tasks and relies on task-specific stage structures plus hundreds of deployment episodes. The report also presents point success rates under task-specific trial budgets without statistical uncertainty intervals, making small differences harder to interpret than the large headline margins. Finally, the real-world comparison is mostly against ACT and the no-tactile π₀.₅ backbone. That cleanly isolates the tactile pathway, but it does not establish superiority over ForceVLA, OmniVTLA, FTP-1, or other recent tactile systems under matched hardware and data. The paper is strongest as evidence that its tactile branch improves its own VLA family, not as a definitive ranking of every tactile architecture. Bottom Line. N₀-VTLA makes a persuasive case that touch should not be treated as a low-resolution camera bolted onto a VLA prefix. Its central move is to turn upcoming contact into a learned latent target, align that latent with an existing action expert in stages, and preserve the gains through task adaptation and offline deployment learning. The result is not yet a tactile world model, a universal cross-sensor policy, or a high-frequency tactile controller. It is a chunk-level VLA that learns a compact expectation of what contact should happen next—and uses that expectation to retry insertions, regulate grasping, and learn from failures that demonstrations leave out. For roboticists, the broader lesson may be more important than the particular architecture: modalities with different temporal statistics should not automatically share the same tokenization, placement, training schedule, or control loop. N₀-VTLA treats touch as an anticipated event in the action process. Its results suggest that this is a productive place to build from.