Latest / Embodied AI 101 / Teaching Isaac Sim to Feel: A Real-Time Digital Twin for Capacitive Touch

Transcript

Teaching Isaac Sim to Feel: A Real-Time Digital Twin for Capacitive Touch · Embodied AI 101

• Teaching Isaac Sim to Feel: A Real-Time Digital Twin for Capacitive Touch. A Capacitive Tactile Sensor Digital Twin for Real-Time Synthetic Data Generation and Sim-to-Real Transfer in NVIDIA Isaac Sim • Berith Atemoztli De la Cruz Sánchez and Jean-Philippe Roberge • Control and Robotics Laboratory, École de technologie supérieure, Montréal • Applied Sciences, 16(15), 7708 • 2026. The missing sensor in the synthetic-data stack. Vision tells a robot where contact is likely to happen. Touch tells it what actually happened. That distinction becomes decisive as soon as manipulation moves beyond clean pick-and-place. During grasping, insertion, regrasping, cable handling, or manipulation in clutter, the relevant state is often hidden behind the robot’s own fingers. Camera calibration errors become contact errors. Small pose offsets change the pressure distribution. A grasp that looks geometrically valid may be supported by only one edge. And phenomena such as incipient slip, local compliance, or contact migration may be almost invisible externally. Yet tactile sensing remains poorly represented in mainstream robot-learning simulators. Rendering RGB images is now routine. Depth, segmentation, optical flow, LiDAR, and even event cameras have increasingly mature simulation pipelines. Tactile sensors are harder because their signal is mediated by a deformable body. Before the simulator can generate a measurement, it must reproduce contact, deformation, material response, and the sensor’s transduction mechanism. The useful signal is not merely the collision geometry or total contact force; it is the spatial pattern created inside a compliant sensing pad. The work by Berith Atemoztli De la Cruz Sánchez and Jean-Philippe Roberge attacks this gap by introducing a digital twin of a capacitive tactile sensor inside NVIDIA Isaac Sim. The released implementation targets the Robotiq TSF-85 fingertips mounted on a modified 2F-85 gripper. It includes a deformable sensing model, a learned mapping from deformation to tactile output, real-time visualization, and logging interfaces for synthetic-data generation. Robotiq and the authors’ CoRo laboratory released the corresponding Isaac Sim extension publicly in 2026. The most interesting aspect is not simply that a soft fingertip appears inside Isaac Sim. The central contribution is an engineering compromise that makes useful tactile synthesis possible at interactive rates: simulate enough mechanics to recover the contact deformation, then use a neural model—trained against real sensor data—to convert that deformation into the output distribution expected from the physical capacitive sensor. That makes this a hybrid digital twin. It is neither a fully analytical capacitor model nor a purely generative image model. It combines deformable-body physics, system identification, learned transduction, and deployment-matched software interfaces. Why capacitive touch is difficult to simulate. A capacitive tactile sensor typically contains conductive electrodes separated by a deformable dielectric. When an object presses the sensor, the dielectric compresses and changes the geometry of the local capacitive elements. The resulting capacitance changes are sampled across the array and converted into a tactile map. A naïve simulation might therefore start from the parallel-plate capacitor relationship: capacitance increases as electrode separation decreases. But a real fingertip is not a collection of independent one-dimensional springs. Compression at one point creates lateral strain elsewhere. The elastomer can bulge, shear, and redistribute load across neighboring taxels. The object’s curvature, edge orientation, indentation depth, contact angle, and material all affect the resulting pattern. Electrode geometry, cross-talk, nonlinear electronics, manufacturing tolerances, and signal normalization further separate the measured map from the idealized mechanical state. At the physics end of the spectrum, one could model the elastomer with a high-resolution hyperelastic finite-element model, reproduce the internal electrode geometry, and solve the electrical field for every deformation. That may be appropriate for sensor design, but it is far too expensive for a robot-learning loop that needs thousands or millions of contact observations. At the opposite end, one could paint a pressure blob wherever the collision engine reports contact. That is fast, but it ignores precisely the spatial structure that makes tactile data useful. A contact patch generated by a cylinder should not look like one generated by a thin edge, even when their total normal forces are identical. The paper’s approach occupies the middle ground. The simulator is responsible for producing a physically meaningful deformation field. A neural network is then responsible for reproducing the sensor-specific mapping from that deformation to the measured tactile map. In other words, mechanics and transduction are separated. This decomposition is valuable because these two subproblems have different requirements. Contact mechanics should generalize across object geometries and robot motions. Sensor transduction, by contrast, is specific to the fingertip construction, electronics, calibration, and output convention. A new mechanical scene should not require retraining the complete simulator, while a new tactile sensor should not require rewriting the contact engine. The research lineage: from offline FEA to a real-time twin. The 2026 digital twin builds on an earlier study from the same authors, titled A hybrid elastic-hyperelastic approach for simulating soft tactile sensors. That work developed two complementary deformation pipelines. The high-fidelity branch used Abaqus and hyperelastic material models. The authors fitted elastomer behavior using formulations such as Ogden and Mooney–Rivlin models, evaluating candidate fits for neoprene and polyurethane. The fast branch used the deformable-body machinery available in NVIDIA’s simulation stack, approximating the sensor with a simpler elastic model. In both cases, convolutional networks translated simulated deformation into synthetic tactile maps. The earlier system also recognized that not every contact requires the expensive branch. A broad, smooth contact patch can often be approximated well by a simpler model. A sharp edge, compound curvature, or fragmented patch may demand hyperelastic simulation. The authors therefore proposed selecting the modeling pipeline according to the geometric complexity of the contacting object and contact region. This is an important systems insight: tactile simulation should be adaptive, rather than enforcing one fidelity level for every frame. That study was grounded in 53,400 real tactile maps used to train, validate, and test the learned components. On twelve unseen objects, the reported best cases reached approximately 97 percent Structural Similarity Index Measure, or SSIM, for the hyperelastic pipeline and 90 percent for the elastic pipeline. Across a broader set of 2,000 tests, however, the reported averages were lower—about 0.72 for the Abaqus pipeline and 0.63 for the fast Isaac pipeline. Those two sets of numbers should be read together. The best-case results show that the representation can reproduce highly realistic contact maps. The aggregate results show that generalization across geometry and contact conditions remains nontrivial. The faster elastic model sacrifices some fidelity, but its computational profile makes integration into an interactive simulator feasible. The 2026 work effectively operationalizes that fast path. Instead of treating tactile synthesis as an offline experiment, it packages the deformation-to-signal pipeline as an Isaac Sim sensor. The public implementation contains a USD asset with a deformable sensing mesh, an ONNX version of the tactile prediction network, a Kit extension, interactive controls, and headless execution support. A separate companion dataset released around ICCRT 2025 contains 46,200 real and synthetic tactile samples collected with a Robotiq 2F-85 setup and generated in NVIDIA’s simulation environment. This is distinct from the 53,400-map corpus used in the hybrid-model study, and it provides another bridge between physical measurements and simulated contact patterns. Inside the Isaac Sim tactile pipeline. Conceptually, the runtime pipeline has four stages. First, Isaac Sim resolves contact between the manipulated object and the soft fingertip. The tactile sensor USD contains a deformable sensing mesh rather than a purely rigid collision surface. As the object presses into the pad, the deformable-body solver updates the mesh state. Second, the extension compares the current mesh with its undeformed reference configuration. This produces a deformation representation associated with the sensing surface. The released logger can preserve the per-node deformation as well as the fuller mesh state, which is useful for separating mechanical-model errors from neural-transduction errors. Third, the deformation representation is passed through the trained convolutional model. In the public implementation, that model is distributed in ONNX format and executed through ONNX Runtime. The output is the synthetic tactile map corresponding to the layout and conventions of the physical sensor. Finally, the map can be visualized as a heatmap, exposed to downstream code, and written to disk alongside the underlying deformation. The extension performs this cycle at the simulation refresh rate rather than launching a separate offline finite-element solve. This design gives researchers access to several different levels of supervision:. The simulator’s privileged contact state. The deformable mesh and per-node displacements. The predicted sensor map. The robot, object, and scene state associated with the contact. That separation is unusually useful. Suppose a real object produces the wrong tactile map after transfer. If the simulated deformation is already wrong, the problem lies in contact geometry, material parameters, friction, or collision handling. If the deformation is plausible but the map is wrong, the learned sensor model or calibration is the more likely culprit. A monolithic black-box generator would make that distinction much harder. The implementation supports both an interactive GUI workflow and a headless Python workflow. In the GUI, a user identifies the sensor prim, chooses the output location, enables real-time map visualization, and runs the scene. In headless mode, the sensor roots, logging options, and recording window are controlled through Isaac Sim’s settings system. Recording can be enabled only during the meaningful part of an episode—for example, from the beginning of gripper closure through the hold and release phases—while deformation calculation and inference remain active in the background. The released assets include a modified Robotiq 2F-85 gripper with TSF-85 fingertips and support for two sensors, corresponding naturally to the two gripper pads. An example scene approaches an object, descends to a grasp pose, and closes the fingers while recording the two tactile signatures. The repository also demonstrates integration with cuRobo for the arm motion surrounding the contact. As of August 4, 2026, the repository documents Isaac Sim 5.1.0 as the developed and tested version, with Isaac Sim 6.0 support still in progress. The listed reference system uses Ubuntu 22.04 and an RTX A2000 laptop GPU with 8 gigabytes of VRAM. That does not constitute a comprehensive performance benchmark, but it suggests that interactive operation is not restricted to top-end datacenter hardware. One small but relevant implementation detail is that the launch announcement describes HDF5 export, whereas the current repository documents CSV outputs for deformation, tactile maps, and complete mesh state. That likely reflects an evolving implementation or different paper and repository versions. Anyone building a data pipeline should therefore treat the checked-out repository version—not the announcement—as the authoritative I/O specification. What “real-time” and “sim-to-real” mean here. Real-time is an overloaded term in robotics. In this work, it primarily means that tactile maps are produced online at the simulation update rate. It does not mean that every physical channel of the TSF-85 is reproduced at its native bandwidth, nor does it establish a hard real-time deadline under every scene configuration. The physical fingertips expose richer and potentially much faster signals than a spatial pressure map alone. Robotiq’s user documentation, for example, lists kilohertz-rate dynamic and inertial channels. The public digital-twin extension is centered on the deformation-derived tactile map. It should therefore be viewed as a model of the spatial contact modality, not yet as a complete electrical and temporal emulation of every signal available from the hardware. The sim-to-real claim also has several layers. The first is geometric transfer: does a simulated edge, cylinder, or irregular surface produce a spatial pattern resembling the corresponding real contact?. The second is sensor-domain transfer: are the map’s gain, spatial spread, nonlinearities, and inter-taxel relationships close enough that a model trained on synthetic maps can consume real maps without relearning its entire representation?. The third is policy transfer: can a controller trained in simulation use real tactile feedback to complete a physical manipulation task?. This work directly addresses the first two layers. The real-data-trained deformation-to-map network is especially important. Rather than asking a fully analytical sensor model to reproduce every imperfection of the physical fingertip, the learned transduction model embeds those characteristics statistically. It acts as a form of data-driven system identification. Policy transfer is more conditional. A synthetic tactile classifier or contact estimator may transfer even when some contact dynamics are imperfect. A lifting or insertion policy may depend critically on friction, shear, latency, and temporal effects that the current simulator does not reproduce. The digital twin is thus an enabling sensor model, not a guarantee that arbitrary contact-rich policies will transfer zero-shot. The durability of the physical sensor helps this strategy. Robotiq reports testing the TSF-85 through approximately 2.3 million cycles at maximum gripper force without significant variation in its tactile output. A stable physical transfer target is easier to model than one whose baseline and sensitivity change substantially with wear. Nevertheless, this result does not eliminate unit-to-unit calibration differences, temperature effects, contamination, mounting tolerances, or elastomer aging in other operating conditions. It is also worth being precise about the phrase digital twin. In stricter industrial definitions, a digital twin often implies continuing synchronization between a specific physical asset and its virtual counterpart. The released TSF-85 system is primarily a calibrated virtual sensor model and synthetic-data generator; the repository does not describe continuous bidirectional synchronization with an individual physical fingertip. In contemporary robotics usage, however, “digital twin” is often applied more broadly to a deployment-matched virtual counterpart that shares geometry, interfaces, and expected measurements with the real device. How this work differs from other tactile simulators. Most prominent tactile simulators have focused on optical or vision-based sensors. TACTO provides a fast, open simulator for high-resolution vision-based tactile devices. Its core problem is rendering the internal camera image produced when an object deforms an illuminated elastomer. That makes it well suited to DIGIT-, GelSight-, and related optical sensors, where the sensor output itself is an image. Taxim tackles a similar modality through an example-based model for GelSight. Rather than reproducing the full optical process every time, it learns or reconstructs the relationship between contact geometry and the characteristic tactile image, emphasizing both speed and visual realism. TacSL pushes vision-based tactile simulation toward massively accelerated robot learning. It integrates GPU-based tactile rendering and contact-force extraction into Isaac Gym and reports more than a 200-fold speedup over the previous state of the art. TacSL also introduces asymmetric actor–critic distillation, allowing a privileged simulation policy to teach a policy that operates on observations available from real tactile sensors. The earlier BioTac sim-to-real work by Suresh and colleagues takes another route. It combines physics-based simulation with learned projections that align simulated and real tactile observations in a common latent space. Instead of requiring the raw signals to match perfectly, the downstream model is trained to operate in a representation where the domains are closer. The TSF-85 digital twin occupies a different point in this design space. Its output is not an internal camera image. It models a capacitive sensor whose map emerges from dielectric deformation and electrical transduction. The learned component is placed directly between simulated mechanics and sensor output. And the implementation is packaged around an industry-oriented fingertip mounted on a commercial parallel gripper. That positioning has practical advantages. A capacitive tactile map is lower-dimensional than a photorealistic GelSight image, reducing rendering and policy-encoding costs. It also exposes contact information in a form that may be easier to use for pressure centroid estimation, contact-area analysis, grasp symmetry, and edge detection. The trade-off is that it contains less explicit geometric detail than a high-resolution optical reconstruction and may not capture texture or shear as richly without additional channels. The comparison also exposes two distinct research goals. TACTO, Taxim, and TacSL ask how to simulate a broad class of vision-based tactile observations quickly. The TSF-85 work asks how to make a particular physical, capacitive sensor available inside an industrially relevant simulation stack. One emphasizes sensor-class generality; the other emphasizes deployment correspondence. Reading the validation results carefully. SSIM is a reasonable starting metric because tactile maps are spatial fields. It rewards preservation of contact shape, local contrast, and structure rather than comparing every taxel independently. A high SSIM suggests that the synthetic and real maps place pressure in similar regions and represent comparable contact geometry. But SSIM is not a complete manipulation metric. Two maps can have similar spatial structure while disagreeing in absolute force calibration. Conversely, a small spatial translation caused by pose-registration error can reduce SSIM even when the maps encode equivalent physical contacts. SSIM also says little about temporal behavior, hysteresis, slip onset, or whether a downstream policy chooses the correct action. For contact-rich learning, at least four evaluation families are needed:. Map fidelity, including SSIM and pixel- or taxel-level error. Physical quantities, such as contact centroid, area, peak pressure, integrated load, and edge orientation. Temporal fidelity, including loading and unloading curves, latency, hysteresis, and stick–slip transitions. Task transfer, measured by real-world success when a model or policy is trained primarily on synthetic observations. The reported best-case SSIM values show that learned deformation-to-tactile conversion is viable. The lower aggregate averages show where work remains. In particular, the gap between the hyperelastic and fast elastic pipelines is evidence that mechanical fidelity still matters even when the final signal is generated by a neural network. A learned transducer cannot reliably repair deformation patterns that are physically wrong outside its training distribution. The most convincing future validation would therefore combine held-out geometry with held-out dynamics. Training and testing on randomly separated frames from the same indentation trajectory risks leakage: adjacent frames are highly correlated. A stronger protocol would hold out complete objects, contact orientations, indentation rates, material properties, and sensor units, then evaluate both map quality and a downstream task on the real hardware. The most important limitation: static friction. The public repository is unusually direct about its central physics limitation. Isaac Sim’s deformable contact model, as used by the extension, represents kinetic friction but does not faithfully reproduce the static-friction “stick” phase between the soft fingertip and a rigid object. As a result, the extension is best suited to squeeze experiments and other tasks dominated by normal contact. Tasks that lift an object through fingertip friction require a workaround in the supplied examples. The maintainers state that they are investigating a hybrid co-simulation approach to recover the missing behavior. This matters because static friction is not a peripheral detail in tactile manipulation. It governs whether a grasp holds, when an object begins to slide, and how tangential load redistributes across the contact patch. A policy may learn the wrong correspondence between normal pressure and grasp security if the simulated object is artificially attached, constrained, or supported. For shape recognition from a stationary squeeze, this may be acceptable. For slip prediction, in-hand reorientation, peg insertion with lateral contact, or tactile servoing along a surface, it is a major limitation. The second limitation is temporal material behavior. Real elastomers are often viscoelastic. Their response depends on loading rate and history, and they exhibit relaxation and hysteresis. A frame-wise elastic deformation model plus CNN can reproduce spatial patterns without necessarily reproducing those dynamics. The third is scale. The extension currently supports up to two sensors per environment and has been demonstrated as an interactive Isaac Sim component. That is ideal for a parallel gripper, but it is not yet equivalent to a massively parallel tactile-skin simulator running thousands of environments. Deformable-body simulation and ONNX inference are both GPU-bound, so scaling will depend on mesh resolution, scene complexity, physics frequency, and batching strategy. Finally, the learned transduction model inherits the support of its training data. Contacts involving unseen indentation depths, unusual shear, very different object compliance, or sensor construction tolerances can push the model outside that support. Synthetic-data scale does not remove the need for targeted real calibration. How roboticists can use and adapt the system. The safest initial applications are those aligned with the model’s strengths: contact localization, pressure-centroid estimation, object-shape classification from a squeeze, grasp centering, contact-area monitoring, and snapshot-based grasp-quality prediction. These tasks primarily require realistic normal-pressure distributions and do not depend heavily on long-duration stick–slip dynamics. For a new project, the first step should be calibration rather than policy learning. Reproduce a small set of physical indentations with known objects, poses, and gripper commands. Replay equivalent contacts in simulation and compare deformation, tactile maps, and derived quantities. Adjust material stiffness, contact offsets, pad geometry, gripper calibration, and signal normalization before generating a large dataset. The second step is deliberate randomization. Useful parameters include elastomer stiffness, damping, friction, object pose, indentation depth, sensor gain, baseline offset, spatial blur, noise, and latency. The goal is not to randomize everything arbitrarily; it is to cover plausible uncertainty in the real system. Because the extension logs both mesh state and predicted output, randomization can also be divided cleanly between mechanical uncertainty and sensor-electronics uncertainty. The third step is to preserve the deployment observation boundary. A real policy cannot access simulator contact forces, exact object pose, or mesh nodes. Those signals can train a privileged critic, label synthetic examples, or supervise an encoder, but the deployed actor should consume only the tactile map and other measurements available on the physical robot. TacSL’s asymmetric learning setup offers a useful conceptual template for this separation. The fourth step is to mix synthetic and real data strategically. A modest real calibration set can anchor gain, noise, and temporal statistics, while simulation supplies geometric coverage and rare contact configurations. For representation learning, contrastive or latent-alignment objectives may be more robust than requiring perfect raw-map correspondence. The learned latent-projection approach used for BioTac is particularly relevant if multiple TSF-85 units exhibit systematic calibration differences. Adapting the framework to a different capacitive sensor should require three major substitutions: a new deformable USD asset matching the pad geometry, a paired dataset connecting deformation states to physical sensor outputs, and a retrained transduction network exported to the runtime format. The surrounding architecture—per-frame mesh extraction, inference, visualization, and logging—can remain largely unchanged. That modularity may ultimately be the paper’s most reusable contribution. Closing perspective. This work does not solve tactile simulation in general. It does something more concrete and immediately useful: it turns a specific, physical capacitive fingertip into a first-class sensor inside Isaac Sim. The key technical idea is the separation of mechanics from transduction. A deformable-body solver produces a contact-dependent mesh state. A network grounded in real tactile measurements turns that state into the output expected from the hardware. Packaging the result as both an interactive and headless Isaac Sim extension transforms an offline modeling study into a synthetic-data tool that roboticists can place inside existing manipulation workflows. The current boundary is equally clear. The system is strongest for normal-contact maps and squeeze-based tasks. Static friction, shear, viscoelastic dynamics, complete high-rate signal emulation, and massive parallel scaling remain open problems. Researchers should therefore treat the twin as a calibrated tactile observation model—not as proof that every policy trained with it will transfer. Still, this is an important shift. Tactile simulation is moving away from bespoke figures in individual papers and toward reusable sensor assets with deployment-matched outputs. Once touch becomes as accessible in simulation as RGB or depth, contact-rich learning can become a routine part of the robotics data stack rather than a specialized experiment.