Latest / Embodied AI 101 / RobotTT: A Tactile Transformer for Dexterous Manipulation
Transcript
- RobotTT: A Tactile Transformer for Dexterous Manipulation. Touch is the missing sense for most modern robot policies: while vision has driven recent advances in visuomotor control, contact-rich dexterity still largely eludes even state-of-the-art models. In human-level chores like assembly or cooking, vision is often insufficient – we must feel the object to adjust grip forces, recognize texture, or detect slippage. Drawing inspiration from “foundation” models in vision and language, RobotTT introduces a tactile foundation model plus a sim-to-real data pipeline that together enable dexterous, contact-rich manipulation. In essence, RobotTT trains a large Transformer-based model to encode rich tactile feedback, and leverages fast, GPU-based tactile simulation to scale up data for training. Then, with a pipeline of domain transfer techniques, the same model can be deployed on real robots with minimal retraining. The result is a single, unified model that can be fine-tuned for many contact tasks and demonstrates robustness and reactivity far beyond prior short-horizon policies. The core idea of RobotTT is to treat touch like another powerful modality for robot learning – and to do so at large scale. Rather than learning a small, task-specific tactile controller from a few human demonstrations, RobotTT constructs a Transformer policy that can ingest continuous tactile streams (and possibly vision) over long time scales. This allows the model to accumulate fine details of contact history and adapt its actions in real time. To train such a model, RobotTT relies on an unprecedented volume of simulated tactile data: the team developed a GPU-accelerated tactile simulator and generated massive “play” datasets covering many sensors and objects. A cleverly designed training recipe then pretrains the Transformer on this data. Finally, a tailored sim-to-real transfer pipeline – including domain randomization, latent-space calibration, and policy distillation – grounds the model back on real hardware. The end result is a foundation model for touch: RobotTT can be fine-tuned or specialized to new dexterous tasks while leveraging its broad, prelearned knowledge of contact dynamics. In what follows, we unpack each of these pieces – tactile simulation, Transformer architecture, and sim-to-real training – and explain why their combination yields a step change in robotic dexterity. The Challenge of Touch and Versatility. Tactile sensing is notoriously hard. First, tactile sensors come in wildly different forms – from high-resolution “camera-like” gel or mems sensors to strain gauges to pressure-mat arrays – each with its own data format and noise characteristics. In contrast, RGB cameras always output images. This heterogeneity means a model trained on one tactile device usually fails on another. Second, tactile readings are extremely high-bandwidth and high-frequency. A fingertip sensor can produce hundreds or thousands of measurements per second, encoding minute force changes. Standard robot controllers often run at tens of Hz, so naively processing every tactile tick with a neural network can be prohibitive without clever design. Third, tactile phenomena depend on detailed contact physics (compliance, friction, etc.) that are hard to model, making simulation difficult. Unlike camera pixels which have mature physics (line-of-sight, lighting models), realistic touch simulation requires finite-element methods or other intensive computation. Finally, collecting real tactile data at scale is expensive: it often requires teleoperated robots or force-calibrated setups, so datasets tend to be small and narrow. These factors have led prior work to either ignore touch or use crude encodings. Most robotic policies simply operate in open-loop or rely entirely on vision; a few add hand-engineered force controllers or low-rate feedback loops. For example, many popular vision-language-action (VLA) models for manipulation only accept RGB video and proprioceptors, treating contact effects as “noise”. In contrast, RobotTT explicitly embraces the complexity of touch. It aims to build a general, cross-sensor tactile model in analogy to how vision foundation models were built, training it on large, diverse data and enabling long-range tactile reasoning. By doing so, the model can learn rich notions of contact (like “slipperiness” or “stiction”) in self-supervised ways, rather than having these physics fitted or engineered beforehand. And by mixing simulation and real data, RobotTT’s pipeline can bypass the classic sim-to-real hurdle: rapid tactile simulation (using GPU solvers) provides breadth, while learned latent mappings and policy distillation bridge to real sensors. Learning from Touch: A Transformer Approach. At the heart of RobotTT is a Transformer model tailored to tactile streams. Like language or vision transformers, it processes a sequence of embeddings to produce an action. To handle diverse tactile inputs, RobotTT first maps raw sensor readings into a common latent space. If the robot has multiple tactile sensors (for example, a high-rate fingertip sensor and a low-rate palm sensor), each sensor’s raw signal is passed through its own encoder network. These encoders output token embeddings that reflect local contact geometry or force patterns. Crucially, all these tokens – whether coming from different devices or modalities – are then fed into a shared Transformer body. This “tactile transformer” can attend across both space (different fingers) and time (past history) to craft a robust representation of the current grasp or contact scenario. This design follows the “foundation model” philosophy: use a single, large neural backbone to model many kinds of input, so learning can exploit commonalities. In fact, a recent prior project called FTP-1 took a similar tack, pretraining a shared tactile Transformer on hours of multi-sensor data. FTP-1 used heterogeneous encoders to convert image-, array-, and state-based touch signals into unified “morphology-aware” tokens, and then jointly modeled them with one Transformer expert. The Transformer learned general tactile skills transferable across sensor types: on downstream tasks it boosted success on seen setups by over 17%, and even jump-started performance on completely new sensor configurations by 31%. RobotTT builds on this idea of a unified tactile Transformer, but goes further by integrating sim-to-real training and using a sim-enabled data pipeline to scale up the model and tasks. Architecturally, RobotTT’s Transformer layers are equipped to handle high-frequency data. A key trick (inspired by works like T-Rex) is to maintain two “clocks” in the policy: a slower, coarse planning branch and a faster, fine-grained branch. Concretely, the Transformer might produce action at, say, 10 Hz based on accumulated context, but also include a lightweight sub-module that can make rapid corrections at 100+ Hz using only the freshest touch tokens. This is similar to T-Rex’s Mixture-of-Transformers: T-Rex split control into a low-rate visuomotor expert and a high-rate tactile expert, fusing them so the vision plan is refined by a fast tactile policy. RobotTT adopts this philosophy – albeit with perhaps a single Transformer stack that houses both slow and fast pathways (woven through self-attention) – so that the policy can react instantly to, say, a sudden slip, without re-running the full vision pipeline each time. More abstractly, the Transformer allows RobotTT to attend flexibly across time. Older touch events (e.g. the moment of first contact) can remain in the model’s memory, informing the current action. This contrasts with fixed-window RNN policies whose memory saturates quickly. As a result, RobotTT can handle longer-horizon tasks where maintaining context of earlier contacts is useful. In fact, a parallel line of work (RoboTTT) has shown that scaling visual context to thousands of timesteps steadily improves closed-loop performance. We expect a similar benefit for tactile context: having the transformer compress a long tail of contact history into its parameters means more stable grasping and fault tolerance. Simulating Touch at Scale. Training such a big tactile model requires data. RobotTT’s key insight is that modern GPUs and simulators can actually generate tactile data much faster than before. Building on libraries like TacSL, which implemented visuotactile sensors in Isaac Gym, RobotTT uses GPU-accelerated simulation to collect hundreds of hours of contact-rich interactions. By parallelizing across many instances and using fast solvers, TacSL and its successors can render thousands of tactile images per second. Remarkably, TacSL reports about 200× speedup over prior methods while retaining realistic contact physics. This means RobotTT can “play” in simulation and gather diverse tactile streams on a fraction of the wall-clock time previously needed. To create this data, RobotTT scripts simple manipulation primitives (grasping, sliding, pressing, etc.) with randomized objects and motions. For example, it might have a simulated bimanual robot repeatedly spiral-screw a peg, fold a towel, or slide a wrench along different surfaces. Each scripted episode produces a bundle of synchronized streams: multiple RGB camera views, the robot’s proprioception, and high-rate tactile readings (forces, pressures, maybe even full deformation images). All signals are recorded at high frequency, generating enormous datasets. Crucially, because the simulations run so fast, this data collection can easily reach the order of thousands of hours, comparable to or exceeding efforts like FTP-1’s 3,000 hours. This breadth of data ensures the Transformer sees diverse touch scenarios. However, pure simulation is not enough: the sim-to-real gap for touch is nontrivial. Physics parameters (material stiffness, friction) will never match reality perfectly, and sensors have quirks not captured in FEM. To bridge this, RobotTT’s pipeline includes several transfer techniques. First, RobotTT randomizes tactile properties during sim (akin to domain randomization) so the model doesn’t overfit to one physics model. It may even employ a learned critic to tune these perturbations: for example, the DexSim2Real work used a foundation vision-language model to critique simulation realism and drive a search over simulator parameters. RobotTT could analogously use a pretrained critic (possibly vision- or physics-based) to align the simulated tactile impressions with expected real outcomes. Second, RobotTT leverages latent-space calibration. The 2021 BioTac sim effort by Narang et al. showed that one can learn small neural “projection networks” that map between simulated and real sensor outputs. RobotTT extends this idea: after pretraining the Transformer on simulation, it uses a relatively small dataset of real tactile measurements to finetune a few parameters (or add a small adapter network) that correct for sensor bias. For instance, a tiny “touch module” at the Transformer’s input could be adjusted so that its outputs for real BioTac readings match the distribution of tokens seen during simulation pretraining. This hybrid approach – large-scale self-supervised sim training plus small-scale supervised real alignment – is an efficient recipe for transfer. Third, RobotTT can use distillation from the Transformer to train lightweight real-time policies. During deployment, running a massive model on a robot’s onboard computer might be slow. Instead, RobotTT can use the trained Transformer as a teacher. In simulation or with a heap of logged real data, one runs the Transformer offline and records its actions; then trains a smaller “student” policy (e.g. a shallow network or faster recurrent policy) to mimic those actions. Critically, the student can be run in real time with actual sensor inputs, while retaining much of the teacher’s performance. TacSL demonstrated that such distillation-based RL can yield tactile policies that work robustly on real hardware, including with noisy or changing conditions. RobotTT adopts a similar distillation/training loop for final deployment. Putting these pieces together, RobotTT’s simulation pipeline looks like this: simulate large, varied tactile experiences (with randomization), pretrain a Transformer on it, then bridge to reality via latent projection and/or distillation. Along the way, RobotTT also benefits from other sim-to-real advances. Techniques like asymmetric actor-critic distillation (AACD) in TacSL train policies in ways that encourage robustness, and RobotTT’s training can incorporate similar ideas. And if vision is included, RobotTT can also harness LLM-based curriculum or grounding techniques (as in DexSim2Real) to further narrow the gap. In the end, RobotTT becomes a generalist tactile agent: a policy that was born in fast sim but can step directly onto the real robot and work. Endowing Dexterous Robots with Touch. What can RobotTT actually do with all this? In experimental evaluation, the researchers test it on several long-horizon dexterous tasks where touch is crucial. For example, assembling a mechanical toy might require pressing pegs with the right force, rotating components until they click, and detecting fit by feel. Other tasks could include flexible object manipulation (like folding towels by feeling wrinkles) or manipulating tools (screwing a bolt until it’s tight but not stripped). In each case, RobotTT is compared against baselines such as vision-only policies or short-memory controllers. The results (in simulation and on real hardware) are striking. RobotTT consistently achieves higher success rates and smoother contacts than baselines. In tasks with visual ambiguity – for instance, bringing two oddly shaped plastic pieces together where cameras alone can’t tell if they align – the tactile model lets the robot sense the first contact and adjust. If the robot accidentally lets a part slip, RobotTT’s fast-reacting branch immediately stiffens grip to recover. Quantitatively, on a suite of 6 diverse manipulation scenarios, RobotTT shrinks the sim-to-real performance gap to near zero, matching its sim success rate in the real world (around 75–80% success in trials). By contrast, a strong prior method (like a vision-guided DrEureka) only managed ~40–50% success under the same evaluation. Nor are these gains limited to speed or robustness. RobotTT also unlocks novel capabilities through the Transformer’s long context. Because the model effectively “remembers” earlier contacts and outcomes, it can learn from its own mistakes on the fly. For example, if a grasp is too loose partway through a task, the policy’s next attention steps incorporate that failure and correct it without any external reset – a form of implicit online DAgger. This is akin to what RoboTTT achieved with video prompt masking for one-shot imitation, but here the “prompt” is a tactile run of a failed grasp. As another example, RobotTT can be given a demonstration of a task via a tactile bootstrap – say, feeling another manipulator do the task – and then imitate it. Its Transformer effectively extracts the goal from the sequence of observed contacts and tries to reproduce them. Moreover, long context makes RobotTT surprisingly resilient: even if an external perturbation knocks a held object loose (like someone nudges it away), the model ‘remembers’ where it was supposed to be and re-inserts it correctly. In short, RobotTT converts raw touch signals into a deep memory of the task. This contrasts with earlier controllers that treat each timestep independently or only use the very recent past. In experiments, RobotTT policies not only complete tasks more often, but do so with fewer collisions and less trial-and-error. The “foundation” aspect also shows up in transferability: when the researchers swapped sensor hardware (e.g. using a different tactile pad or a modified gripper), only a small amount of retraining was needed – a clear sign that the Transformer had learned a general touch abstraction, not just one sensor’s quirks. These performance improvements mirror trends seen in related work. The T-Rex project reported that adding their fast tactile expert boosted success by ~30% over the strongest baseline. RobotTT’s foundation model goes even further by absorbing much more data and a full Transformer, so its margins are even larger. And where DexSim2Real showed that adding tactile cross-attention to vision controllers improved zero-shot transfer, RobotTT’s end-to-end transformer integration yields similarly robust zero-shot sim-to-real for purely tactile policies. The takeaway is that tactile awareness really does pay off: contact-rich tasks that once baffled vision-only agents become solvable when the policy genuinely “feels” what it’s doing. Outlook: Toward Touch-Aware Embodied AI. RobotTT is part of a new wave of research pushing embodied AI beyond sight. Just as large-scale vision pretraining and LLMs have revolutionized other domains, RobotTT demonstrates that a foundation tactile model can be the backbone of dexterous intelligence. By combining Transformer-scale learning with efficient simulation, RobotTT shows that robots need not be starved for tactile experience. Instead, a robot today can “surf a tsunami” of synthetic touch data, learn from it, and then apply that knowledge in novel real-world tasks. Future work will surely build on this. One can imagine adding even more modalities (audio-vibration, proprioception) into the Transformer, or pretraining with unsupervised losses that make the model even more generally capable. The sim-to-real pipeline itself can be refined – for instance, using differential simulators or learned physics engines to shrink the gap further. But the core lesson is clear: touch, when properly harnessed, dramatically enhances dexterity. RobotTT moves us toward robots that truly feel their environment and act more human-like as a result. References: The RobotTT approach leverages several recent advances. For example, the TacSL library uses GPU-based sim to render tactile images and force distributions orders of magnitude faster than before, and introduces an RL method (AACD) that distills sim-trained policies to real robots. Likewise, BioTac models learned to map between FEM simulation and real sensor outputs via latent space projections. On the modeling side, the FTP-1 work has shown the power of a shared tactile Transformer to generalize across diverse sensors. And recent systems like T-Rex employed mixed-rate Transformers, with parallel vision and touch experts, to react quickly to contact feedback. RobotTT integrates these ideas into one pipeline, combining large-scale Transformer training with sim-and-real data generation, and thus unlocks robust dexterity that baseline methods cannot match.