Latest / Embodied AI 101 / Two Clocks for Touch: How T-Rex Gives Dexterous VLAs a Reflex Arc
Transcript
- Two Clocks for Touch: How T-Rex Gives Dexterous VLAs a Reflex Arc. T-Rex: Tactile-Reactive Dexterous Manipulation • Dantong Niu, Zhuoyang Liu, Zekai Wang et al. • UC Berkeley, NVIDIA, Stanford, Panasonic, La Sapienza University, and ItalAI • arXiv preprint • 2026. Dexterity begins where vision becomes ambiguous. Most robot-manipulation demos look like visual problems right up until the decisive moment. The hand approaches an egg, a key, a page, or a nested plastic cup. Vision determines where to go and how to shape the grasp. Then contact happens, and the relevant state becomes partially hidden. The robot needs to know whether the egg is about to crack, whether the key is binding, whether one sheet or five have caught under a fingertip, and whether the upper cup has actually separated from the stack. At that point, another high-resolution RGB frame is often less useful than a small change in shear, force, or local deformation. More importantly, that change may need to alter the command already being executed. A policy that merely includes touch among its input tokens, but only consults those tokens when generating the next long action chunk, is tactile-aware without necessarily being tactile-reactive. T-Rex attacks that distinction at the level of the whole system. It does not simply append tactile features to a VLA. It co-designs a contact-rich dataset, a temporal tactile representation, a variable-rate transformer architecture, a flow-matching interface between slow planning and fast correction, a latency-aware training recipe, and a real-time deployment stack. Its core proposition is that semantic visuomotor planning and contact regulation should share task context but should not be forced to run on the same clock. That proposition is the most important part of the paper. T-Rex’s tactile expert matters, but so does the boundary around it: what information is cached, what is refreshed, how often it runs, which part of action generation it owns, and what data teaches it to make useful corrections. First, a clarification about the dataset release. The paper and current release materials use two different dataset scales, and downstream discussions should preserve the distinction. The full corpus reported in the June 15, 2026 arXiv paper contains 100 hours and 7,755 episodes, collected across 502 feasible object–motor-primitive combinations. The project’s public repository, however, says that the currently open LeRobot release contains approximately 50 hours and more than 5,400 trajectories. The Hugging Face viewer lists 5.47 million frame rows and episode indices extending to roughly 5.46 thousand, consistent with that public-subset figure. So, as of August 27, 2026, “50 hours and about 5,500 episodes” accurately describes the open subset, while “100 hours and 7,755 episodes” describes the complete corpus used and documented by the authors. Some language on the paper and project page loosely calls the entire 100-hour dataset open-source, but the repository is more explicit that only about half is presently public. This does not diminish the public release. Fifty synchronized hours from two 22-degree-of-freedom hands, ten fingertip sensors, three camera views, bimanual state and action, and natural-language annotations is substantial. But the distinction affects exact reproducibility: the released mid-trained checkpoint has seen more robot data than an outside group can currently download. The dataset is a curriculum over contact dynamics. The T-Rex team made a useful decision about the unit of data collection. Rather than spend its entire budget recording a small number of impressive long-horizon tasks, it organized robot play around object–motor-primitive pairs. The taxonomy begins with 207 household objects and 22 primitives. Their Cartesian product contains 4,554 possible pairs, but many are physically meaningless: one does not pour a rigid block or twist an object with no movable interface. After manually filtering infeasible combinations, the authors retain 502 pairs and collect about 16 demonstrations per pair on average. Episodes have a median duration of 29.8 seconds, with the middle half falling between 21.0 and 41.1 seconds. Collection took place over ten weeks. This is more than an efficiency trick. It encodes a hypothesis about how contact competence should be learned. Long tasks are compositions of lower-level interaction regimes: press until a latch releases, slide while maintaining normal force, twist while preventing slip, pinch without deformation, rub until two thin layers separate, stabilize with one hand while the other applies torque. If mid-training covers those regimes across many geometries and materials, downstream task demonstrations can focus on sequencing and task-specific conventions rather than teaching contact from scratch. The combinatorial design also gives the dataset a cleaner kind of diversity than simply accumulating unrelated episodes. Every retained pair asks how a reusable motor verb changes when applied to another noun. The policy sees that “press” means something different on a pillbox, a sponge, and a deformable tube, but that those interactions still share structure. This is a promising middle ground between narrow demonstrations and undirected play. Each episode contains three 640-by-360 RGB streams at 30 hertz: one head view and two wrist views. It also includes bimanual arm and hand state, wrist poses, actions, a language instruction, and tactile measurements from all ten fingertips. The released dataset contains the sensors’ raw grayscale images, estimated deformation maps, and estimated six-axis wrenches. The policy itself primarily consumes force histories, current force, and the derived deformation maps rather than repeatedly processing the raw tactile imagery. The collection protocol deliberately varies six tabletop backgrounds, randomizes object pose, and adds up to five distractors sampled from a pool exceeding 210 items. Episodes with unstable tactile streams, corrupted measurements, abnormal motion, or excessive joint velocities are filtered. Language instructions are generated from a commercial VLM given several head-camera frames plus the recorded object and primitive labels, and are subsequently checked by humans. One strength here is that the teleoperator and learned policy use essentially the same downstream control path. Manus gloves supply finger targets, VIVE trackers supply wrist poses, and the commands are retargeted through the robot’s kinematics and controller. This reduces the mismatch that occurs when demonstrations are recorded in one action space but deployed in another. The complete teleoperation stack, including synchronization, collision-aware arm control, recording utilities, and hardware mounts, is included in the repository. A 58-DoF robot with a 62-dimensional action language. The experimental platform is a fixed-base Dexmate Vega-1 with two seven-degree-of-freedom arms and two fully actuated, 22-degree-of-freedom Sharpa Wave hands. That gives 58 actuated arm-and-hand degrees of freedom. The policy nevertheless predicts a 62-dimensional action. This apparent discrepancy comes from representation rather than additional joints. For each arm, T-Rex predicts three-dimensional end-effector translation plus a six-dimensional continuous rotation representation. The two nine-dimensional arm commands contribute 18 values, and the two hands contribute 44 absolute joint targets, producing 62 dimensions in total. A differential inverse-kinematics layer converts the end-effector commands into arm-joint targets. Control is layered. A 300-hertz low-level loop tracks the targets emitted by the learned policy. The public repository describes the expensive visuomotor update as running at roughly 5 hertz and tactile refinement at roughly 20 hertz. The paper’s algorithm expresses the same relationship structurally: one slow plan is accompanied by four tactile-conditioned updates inside a 16-action chunk. This is worth emphasizing because “high-frequency tactile control” can mean very different things. T-Rex is not replacing a kilohertz force controller with a transformer. Touch updates a learned command trajectory more frequently than the vision-language stack does, while conventional low-level control continues to run at 300 hertz. Its fast stream is high-frequency relative to foundation-model inference, not necessarily relative to classical tactile servoing. Three experts, two clocks, one action trajectory. T-Rex is built around a Mixture-of-Transformer-Experts backbone containing three functionally specialized components. The latent expert processes vision and language and predicts future visual representations. The action expert produces the coarse visuomotor plan. The tactile expert revises that plan using current contact information. The latent and action experts are each listed at 1.41 billion parameters, using a Qwen3-VL-2B-family backbone, while the tactile expert is a smaller—but still substantial—620-million-parameter transformer with a reduced feed-forward width. The action and tactile experts both operate on 16-step, 62-dimensional action chunks. “Mixture” here should not evoke a conventional sparse MoE in which a learned router sends arbitrary tokens to interchangeable experts. The jobs are fixed by design. Vision and language go through the slow path; live touch goes through the fast path. Specialization is imposed by modality, objective, and invocation schedule. The action generator uses conditional flow matching. In plain language, training starts with a demonstrated action chunk and a randomly corrupted version of it. The network learns the direction that moves the corrupted sample toward a valid demonstrated action. At inference, it begins from random noise and follows the learned vector field over a small number of numerical integration steps until an executable trajectory emerges. The novel move is to divide this denoising trajectory between the two clocks. Cascaded flow matching. The slow action expert owns the first six of ten integration steps. It takes the noisy action sample from the beginning of the flow down to an intermediate point, corresponding to a split value of 0.4. At that boundary, the action is no longer pure noise, but it is also not yet the final command. It is a partially resolved plan containing the broad structure supplied by vision, language, robot state, and the pretrained visuomotor prior. T-Rex then caches both this intermediate action state and the transformer’s visual-language key-value context. The tactile expert receives a clone of that cached context, fresh tactile observations, and the partially denoised action. It executes the final four integration steps, turning the coarse proposal into an executable chunk. During the chunk, this terminal segment is recomputed at offsets zero, four, eight, and twelve. At the first offset, the slow and fast passes run back-to-back. At the later offsets, the expensive visual tower and action expert are skipped. The system retains the same coarse intent but completes the final section of action generation again using the latest force and deformation measurements. It is reasonable to call this a residual correction, but that description is only intuitive. The tactile expert does not simply add a small vector from a separate controller to a frozen action. It completes the lower portion of the same flow trajectory. This gives the two experts a common action distribution and common training target while reserving the final, fine-grained part of generation for current contact state. The split also regulates authority. If too little denoising is delegated to the slow expert, the tactile model must recover task-level structure from an underdeveloped proposal. If too much is delegated to the slow expert, touch has too little generative capacity left to change the result. The paper’s sweep finds that an intermediate split works best, consistent with the idea that tactile feedback should refine intent rather than reconstruct it. “Asynchronous” describes the policy cadence and cache lifecycle more than simultaneous GPU execution. The released server uses a single ZMQ request path and an explicit lock: a tactile request waits if a slow pass is still committing a new cache. This avoids races and stale half-written state, but it means the experts are serialized computationally even though they update the controller at different logical rates. There is also unavoidable staleness. The tactile expert may be reacting to contact at offset twelve while attending to visual context created at the chunk boundary. During training, T-Rex therefore shifts tactile and visual frame indices by delays sampled from the same set of offsets used at deployment. This is a small but important example of systems-aware augmentation: the model is trained on the timing imperfections it will actually encounter, rather than idealized perfectly synchronized input. Turning tactile history into a vocabulary of contact. A single force vector says what is happening now, but many useful contact states are defined by change. Incipient slip, a latch releasing, a cup separating, or a fingertip rolling over an edge can produce similar instantaneous magnitudes yet very different temporal signatures. For each fingertip, T-Rex takes a 16-frame history of six-axis wrench measurements—roughly half a second at the logged 30-hertz cadence. A shared one-dimensional convolutional encoder downsamples the sequence, pools it into a 256-dimensional embedding, and maps that embedding to the nearest entry in a 64-item codebook. Finger-identity embeddings let the same temporal encoder serve every digit without treating the thumb and little finger as interchangeable. The codebook is updated by an exponential moving average, with underused entries periodically reinitialized from current activations. Because most frames may contain little or no contact, reconstruction error is weighted by force magnitude so that the autoencoder cannot minimize its loss by becoming exceptionally good at representing “nothing is touching.” The output is one discrete temporal-force token per fingertip. Discretization is intended to suppress drift and turn noisy force traces into recurring contact motifs. It also creates a bottleneck: one code from a vocabulary of 64 cannot preserve every detail of a 16-frame, six-dimensional trajectory. T-Rex compensates by directly projecting the current, unquantized force vector into the policy. The discrete pathway supplies temporal pattern; the continuous bypass supplies immediate magnitude and direction. In parallel, a truncated ResNet-18-style encoder processes each current deformation map. It retains the first three residual stages, reprojects intermediate features to 128 channels, and is pretrained as part of a self-supervised tactile autoencoder before being frozen for policy learning. The deformation pathway describes contact geometry—edges, local pressure patterns, shear, and likely slip cues—while the force pathway describes global wrench dynamics. Their features are concatenated into the tactile token sequence consumed by the fast expert. This dual representation is one of the paper’s stronger design choices. A wrench is compact and easy to model but collapses spatial structure. A deformation map preserves geometry but may not expose net load or torque cleanly. T-Rex does not force either representation to stand in for the other. Acquiring touch during mid-training. T-Rex’s training recipe is as significant as its runtime architecture because the first stage contains no tactile data at all. The latent and action experts begin with large-scale human egocentric pretraining in the EgoScale lineage. T-Rex reports 22,889 hours of first-person video with wrist and retargeted hand-action supervision. The preceding EgoScale paper reported 20,854 hours, so T-Rex appears to use an expanded or differently counted corpus, although the new paper does not explain the numerical difference. EgoScale had already shown that human-action prediction improves with data scale and that the resulting motor priors transfer to a 22-degree-of-freedom robot hand. The second stage is tactile-grounded robot mid-training on the 100-hour T-Rex corpus. Here the action expert adapts from human egocentric observations and retargeted motion to the robot’s multiview cameras and executable action space. The tactile expert is trained to complete the fast portion of flow matching from live tactile context. An auxiliary future-visual-prediction objective encourages the shared representation to retain information about where the interaction is going, rather than letting tactile correction become a task-agnostic reflex. Finally, the model is post-trained with approximately 100 demonstrations for each benchmark skill. The 12 headline tasks are therefore not zero-shot evaluations. The authors do report zero-shot transfer on simpler primitive-level interactions, but the complex benchmark numbers reflect task-specific adaptation after human pretraining and robot tactile mid-training. Conceptually, this is an encouraging result for multimodal foundation models. It suggests that every useful sensor does not need to be present during the largest and most expensive pretraining phase. Broad semantics and kinematic priors can be learned from scalable human video, while a smaller, carefully structured robot corpus later teaches the model what its body and sensors mean during contact. What the benchmark actually measures. The evaluation includes 12 real-world tasks: page turning, fragile egg transfer, bimanual wiping, toothpaste application, separating nested cups, tactile Mahjong sorting, opening a lock, operating a pillbox, an acid–base manipulation sequence, extracting one card from a sleeve, dealing a playing card, and installing a lightbulb. Together they cover delicate force regulation, deformation-sensitive interaction, texture or geometry recognition through touch, bimanual stabilization, insertion, sliding, and prolonged force–motion coordination. Every method is evaluated for 16 trials per task with randomized object positions and orientations. However, the reported percentages require careful interpretation. Tasks use additive or progress-based rubrics that award credit for completed stages. The paper calls the resulting metric “success rate,” but a score of 65 does not necessarily mean that 65 percent of episodes achieved complete binary success. It is better understood as a macro-average normalized completion score over trials and tasks. On that metric, T-Rex scores 65, compared with 35 for EgoScale, 17 for π0.5, 15 for Tactile-VLA, 6 for Reactive Diffusion Policy, 6 for π0.5 augmented with tactile force, and 3 for ViTacFormer. T-Rex ranks first on all 12 tasks. Its strongest scores include 96 on page flipping, 78 on cup separation, 76 on acid–base manipulation, and 75 on egg transfer. The hardest task remains lightbulb installation, where T-Rex reaches 35. The headline advantage is therefore 30 percentage points over the strongest baseline, not merely a vague “30 percent improvement.” Relative to EgoScale’s 35-point score, the increase is much larger than 30 percent. The comparison supports the full T-Rex recipe, but it should not be read as a clean architecture tournament. ViTacFormer and RDP are trained separately per task from roughly 100 demonstrations, while T-Rex benefits from human pretraining and a large tactile mid-training stage. EgoScale, which shares the large-scale egocentric ancestry, is by far the strongest baseline. This is evidence that scale and transfer matter at least as much as the final tactile module. There are also adaptation caveats. Tactile-VLA originally consumes GelSight imagery, but the T-Rex reproduction replaces that input with ten six-axis force vectors. RDP’s original strengths were demonstrated in a different hardware and task setting. The authors keep robot setup, action space, demonstrations, and evaluation protocol consistent, which is valuable, but adapting every baseline to a fully actuated bimanual hand inevitably changes what each method is being asked to do. The ablations tell a more precise story. Removing touch drops the six-task ablation average from 65 to 42, the largest single decline. Deformation alone scores 54. Current force plus deformation, without the temporal VQ-VAE, reaches 58. Temporal and current force without deformation reaches 59. The complete spatial-temporal representation reaches 65. These results support both the temporal encoding and the complementarity of force and local contact geometry. Replacing asynchronous refinement with a synchronous design reduces the score from 65 to 60. That is meaningful and consistent across the selected tasks, but it also puts the architectural contribution in perspective. Most of the overall gain does not come from asynchronous execution alone. Touch, pretraining, mid-training, and their interaction contribute more. The training-stage ablation makes that interaction clear. Training from scratch averages 18. Tactile robot mid-training without human pretraining reaches 34. Human pretraining without the tactile mid-training stage reaches 45. Combining both reaches 65. Human data supplies broad semantic and motion competence; tactile robot data grounds that competence in executable contact dynamics. Neither substitutes for the other. The authors also compare primitive-organized mid-training with an equal-sized collection built around 11 specific tasks and vary downstream demonstration counts from 10 to 200. Primitive-based tactile mid-training produces stronger transfer and particularly helps in the low-data post-training regime. The evidence therefore favors the paper’s full-stack thesis: useful tactile behavior is not produced by a sensor token alone, but by the combination of the right data distribution, temporal representation, training stage, and control schedule. How T-Rex relates to earlier tactile policies. Reactive Diffusion Policy is the closest architectural ancestor. RDP also argues that action chunking creates an open-loop interval and divides control into a slow trajectory model plus a fast tactile-conditioned component. Its experiments showed rapid recovery from contact disturbances across three task-specific contact-rich settings. T-Rex’s advance is not inventing slow–fast tactile control, but embedding that idea inside a pretrained VLA-style model, coupling the experts through one divided flow trajectory, and scaling the embodiment to two 22-degree-of-freedom hands. ViTacFormer tackles a different part of the problem. It learns cross-attention between vision and touch and predicts future tactile signals as an auxiliary objective, driving an ACT-style bimanual policy. It demonstrated impressive task-specific dexterity, including an 11-stage sequence, on an earlier Sharpa Wave platform. T-Rex retains the idea that future prediction helps organize tactile representations, but adds within-chunk tactile recomputation and large-scale pretraining rather than relying mainly on task-specific imitation. EgoScale is both a baseline and the conceptual first stage of T-Rex. It establishes that egocentric human motion can serve as an embodiment-tolerant prior. T-Rex asks what must happen next: how do we convert a model that has watched vast quantities of human manipulation into one that understands the contact dynamics and sensor timing of a particular robot?. Finally, the π0.5-plus-tactile result is instructive. Concatenating one step of force measurements to a pretrained policy’s state reduces its score from 17 to 6. That does not show that touch hurts pretrained VLAs. It shows that an unfamiliar sensor can act as structured noise if the model lacks temporal encoding, modality grounding, and an explicit control role for that information. What researchers can reuse today. The release is unusually complete at the engineering level. It includes post-training and inference code, a separate branch with pretraining and mid-training scripts, pretrained and mid-trained checkpoints, a LeRobot-compatible public dataset, and the teleoperation and deployment stack. The inference server exposes explicit slow, fast, and combined request modes, and the robot client implements the four-update cadence described in the paper. There are still reproduction boundaries. The main branch is intended for task-specific post-training and deployment from the released mid-trained checkpoint. The complete human and robot mid-training corpora are not all public, and the reported training configuration uses 24 NVIDIA H100 GPUs. Reproducing the exact foundation model from raw data is therefore a different proposition from adapting the provided checkpoint. For a group using the same or a closely related hand and tactile stack, the obvious route is to start from the mid-trained checkpoint and collect task-specific episodes in either the repository’s JSON format or LeRobot v3.0. For a new tactile sensor, simply replacing the input dimensionality would miss the paper’s central lesson. A better adaptation would preserve three interfaces:. Learn a temporal representation from the new sensor’s recent history. Give the latest uncompressed reading a low-latency bypass. train the fast expert under the same visual–tactile staleness encountered during deployment. For a substantially different embodiment, the human-pretrained model may be a more appropriate starting point than the Sharpa-specific mid-trained checkpoint. The action expert would need to be re-grounded in the new robot’s command space, and the tactile expert would need new primitive-level interaction data. T-Rex does not yet demonstrate cross-robot transfer of the tactile component, so this is a research direction rather than a documented plug-and-play capability. The force-only ablation is encouraging for teams without dense fingertip deformation maps. Temporal plus instantaneous wrench input reaches 59 versus 65 for the complete model. That suggests most of the framework can be explored with force-sensitive fingertips or wrist sensing, while spatial tactile imagery supplies a useful final increment. What remains unproven. T-Rex is demonstrated on one robot, one hand design, and one family of fingertip sensors. The temporal codebook may resist drift within that setup, but the paper does not establish calibration-free transfer between sensor units, tactile technologies, or laboratories. There is also no palm sensing, leaving large areas of the hand effectively blind during enveloping grasps. The authors identify both calibration drift and sparse coverage as major hardware limitations. The fast expert is lightweight only in comparison with the larger visuomotor experts. At 620 million parameters, it still assumes substantial GPU infrastructure. The paper does not provide a detailed end-to-end latency table, jitter distribution, or response curve relating contact onset to changed motor command. The repository’s nominal 20-hertz fast rate is promising, but a dedicated perturbation benchmark would make the “reactive” claim easier to separate from general improvements in contact-aware imitation. The 12-task evaluation randomizes initial object pose but does not appear to systematically introduce external perturbations during execution. RDP, by contrast, explicitly evaluated disturbances before and after contact. T-Rex’s benchmark primarily shows that frequent tactile conditioning helps complete difficult interactions, not yet how gracefully the system rejects unexpected impulses, object motion, or sensor dropout. The metric also blends complete successes with partial progress, and the paper reports point estimates from 16 trials without confidence intervals. The 30-point margin over EgoScale is large enough that the main ranking is unlikely to be a scoring artifact, but uncertainty would be valuable for the harder tasks, particularly the 35-point lightbulb result. Finally, this remains behavior cloning. The documented failures—misaligned insertion, dropped keys, inaccurate placement, unintended multi-finger contact, excessive toothpaste force, and nonuniform card extraction—show that fast touch does not eliminate action-distribution shift or provide automatic recovery from states absent in demonstrations. Reinforcement learning, online correction, or model-predictive interaction could complement the current post-training stage, especially for tightly toleranced long-horizon tasks. The larger lesson. T-Rex’s lasting contribution may be less about one tactile encoder than about how foundation policies should be partitioned. The physical world does not evolve on a single transformer clock. Language-level intent changes slowly. Visual geometry changes at an intermediate rate. Contact forces can become decisive almost immediately. A unified model need not mean a uniform execution schedule. Three ideas stand out: broad priors can be learned without every eventual sensor; new modalities can be acquired during a dedicated embodiment-grounding stage; and fast experts should be assigned a precise part of action generation rather than asked to compete with the full policy from scratch. T-Rex is not yet a universal tactile foundation model. It is embodiment-specific, computationally heavy, behavior-cloned, and only partially reproducible from the public data. But it offers a credible blueprint for moving beyond tactile awareness toward tactile reaction. Instead of treating touch as one more token, it treats touch as a reason to reconsider when the robot is allowed to change its mind.