Latest / Embodied AI 101 / ViserDex: Visual Sim-to-Real for Robust Dexterous In-Hand Reorientation
Transcript
- ViserDex: Visual Sim-to-Real for Robust Dexterous In-Hand Reorientation. Dexterous in-hand manipulation remains a frontier problem in robotics: controlling a multi-fingered hand to reorient and manipulate unfamiliar objects quickly and reliably. Recent breakthroughs have used deep reinforcement learning (DRL) to teach robots amazing tricks – for example, OpenAI’s hand that solves a Rubik’s cube in simulation – but most succeed only on simple, synthetic objects. Real-world items have complex 3D shapes, rich textures and reflexive properties, and a moving hand inevitably occludes much of the object during fast manipulation. As Bhardwaj et al. (2026) note, existing dexterous systems often rely on multiple cameras, depth sensors, or elaborate tactile arrays to close the perception gap. These add cost and complexity. The ultimate goal is to achieve monocular RGB dexterous control: a single camera on the robot’s palm, perceiving the object and feeding a controller so the hand can reorient that object to any desired pose. That is precisely the aim of ViserDex – a sim-to-real framework introduced by researchers at ETH Zurich’s Robotic Systems Lab – which couples state-of-the-art visual representations with a carefully trained policy to achieve zero-shot transfer on a real Allegro hand. The key insight in ViserDex is to use a novel graphics technique called 3D Gaussian Splatting (3DGS) to render extremely photorealistic simulations cheaply, and then to perform scene randomization directly in that Gaussian representation before rendering. In practice, each object is reconstructed as thousands of 3D “splats” or Gaussian blobs that encode color and lighting information. By perturbing groups of those Gaussians in a physically consistent way (for example, changing the colors or spherical-harmonic lighting coefficients of all Gaussians in a local region), the authors can emulate a vast variety of lighting and material conditions without full ray-tracing. This “pre-rasterization” augmentation generates imagery that closely matches the variations of the real world. Training a vision system on this rich data lets the robot predict object pose from a real camera image even under extreme lighting – all without ever picking up hardware data during training. The control policy itself is learned via RL with the help of a teacher-student distillation pipeline. A teacher policy in simulation has full access to the object’s ground-truth state, while a distilled student uses the visual pose estimates and proprioception only. To make learning more efficient, they further employ a performance-based curriculum that gradually increases task difficulty based on how well the policy is doing. The combination works remarkably well. Using just a single RTX 4090 GPU for most of the work (and a modest dual-GPU setup for the hardest tasks), the team trained both perception and control models on consumer hardware. At test time on a real Allegro hand with a wrist-mounted RGB camera, ViserDex reliably reoriented all five test objects to arbitrary goal poses – accomplishing on average over 25 consecutive successes even under adversarial, colorful lighting. This is, to the authors’ knowledge, the first demonstration of robust dexterous in-hand reorientation under such severe visual perturbations. In nominal lighting the system averaged 37–38 reorientations in a row before dropping each object, far surpassing previous monocular baselines (e.g. DeXtreme’s vision-only system managed only ~28 successes on the same cube object). Crucially, all of this was done without any special sensors or pre-training on real images – it was zero-shot sim-to-real from a single camera. In the sections below we dive into the ingredients of ViserDex. We first review the challenge of visual dexterity and why accurate simulation is so hard. Then we unpack 3D Gaussian Splatting and explain how ViserDex exploits it with domain randomization. Next we describe the learning pipeline: the RL teacher-teacher distillation and curriculum, and the monocular vision-based pose estimator. We conclude with the experimental results on hardware, and discuss why this advance matters for the future of dexterous robotics. The Challenge: Vision-Only Dexterity. Manipulating an object within a robot hand is deceptively demanding. The hand has many joints (the Allegro V5 has 16 degrees of freedom), and the object can tumble unpredictably as forces are applied. To reorient an object to a goal pose, the controller needs to know where the object is at all times – but if the only sensor is a camera, the object will often be partially hidden by fingers. As Bhardwaj et al. explain, this “perception–robustness bottleneck” limits most RGB-based in-hand systems. In practice, prior work often leans on richer sensing: for example multi-camera rigs or depth sensors surround the hand to get object pose, or full tactile gloves give fine contact info. These help perception, but at the cost of extra hardware and calibration. The ideal is to avoid them: use just one camera, and make the vision pipeline as robust as possible. A second challenge is that vision-based sim-to-real usually needs extremely photorealistic simulation, which is computationally expensive for RL. In a dexterous task, the hand may spend days of real time thousands of simulation steps. Traditional mesh-based rendering can’t easily provide the necessary diversity (lighting, textures, backgrounds) at such high speed; simple synthetic images only go so far. Even sophisticated large-scale synthetic data tools like Synthetica require ray-tracing or heavy compute. ViserDex circumvents this by a clever twist: it uses a renderer based on 3D Gaussian splats. In this representation, each 3D Gaussian blob carries color and reflectance (via spherical harmonics). Rendering becomes a matter of projecting all these Gaussians onto the image, allowing real-time photorealism that can even exceed classic rasterization in quality. Because the scene is explicit and compact, ViserDex can efficiently manipulate things like lighting properties at the Gaussian-level. In short, to train an RGB-only dexterous controller without giant clusters, we need (1) a simulation that looks very realistic and varied, and (2) a control learning approach that tolerates remaining gaps. ViserDex addresses (1) with 3DGS-based domain randomization, and (2) with a teacher-student RL method and careful curriculum. This allows training complex dexterous behaviors on a single desktop GPU, something unheard-of with prior mesh-rendering pipelines. 3D Gaussian Splatting for Realistic Domain Randomization. Gaussian splatting is a recent technique (circa 2023) for neural radiance fields that has turned heads in graphics and vision communities. Instead of storing a dense mesh or NeRF network, one records millions of small 3D “splats” – anisotropic Gaussians – each encoding a local patch of color and lighting. Kevin Kwok’s 3DGS viewer demos (for example on GitHub) show that spectacular photorealistic scenes (outdoors, objects, etc.) can be rendered at interactive rates. NVIDIA even introduced a Vulkan sample to accelerate it, noting that Gaussian splatting “enables real-time rendering of photorealistic scenes learned from small sets of images”. The key appeal is that once you build the Gaussian model, rendering just composites these splats without expensive ray tracing, making it far lighter weight than full NeRFs for new viewpoints. ViserDex brings this tech into the loop of a physics simulator. Rather than having the hand manipulate a standard mesh model of the object, we first convert each object into a 3DGS representation. Concretely, using tools like Instant-NGP or other pipelines, the team reconstructs each object’s appearance as thousands of Gaussians fit to images of the object. Now the simulator can move the object freely and then render what the camera sees via these Gaussians. Since the real system uses a fixed camera on the hand, ViserDex simulates this by applying the inverse transform: the object’s Gaussians are moved so that the static camera sees the correct relative pose. Why is this useful? Two reasons. First, the 3DGS renderer is incredibly fast and photorealistic. The Allegro hand and object can move quickly, but the camera view remains crisp and realistic throughout. Second, because the scene is by design a collection of Gaussians with attributes (color and spherical harmonic lighting), ViserDex can perturb those attributes in sophisticated ways before rasterizing. This is their notion of pre-rasterization augmentation. By clustering Gaussians spatially or by material, and then adding noise or scale changes to their color and spherical-harmonic coefficients, the authors simulate realistic lighting and material variation. It’s more powerful than simply jittering pixels or tacking on random backgrounds: the changes are consistent across the 3D model. For example, they group Gaussians into clusters by location (to mimic dirt or scratches on one patch of the object) and by base color (to mimic a material property changing uniformly). Then they apply additive or scaling noise to each cluster’s spherical harmonic coefficients. A Spatial Cluster perturbation might darken a local patch of the duck or glue on a colored smear; a Color Cluster change might shift the albedo of all “blue” splats on a toy’s surface; a global shift might emulate a change in ambient lighting. Each augmentation has only partial probability, so on each rendered image only some clusters are altered. They also add a small independent Gaussian noise group to every splat to simulate sensor noise and fine surface variation. Crucially, all these alterations affect the pre-render 3D model, so they produce natural shading effects rather than flat overlays. The upshot is that data rendered by ViserDex is both high-fidelity and diverse. In tests on real images, the pose estimator trained on ViserDex data significantly outperformed one trained with conventional mesh renders. Under normal lab lighting, ViserDex’s model achieved about 65% accuracy (within 10mm/10° error) on held-out poses, versus only 53% for a standard renderer and 56% for a usual domain-randomized mesh pipeline. Under “adversarial” lighting (multicolored moving lights), the gap was larger: 56% accuracy for ViserDex vs ~47% for the best baseline. A critical control was a “Naïve Gaussian” case where they turned on the 3DGS renderer but not the cluster augmentations – that scored only 36.5% under strong lighting. This shows that the 3DGS by itself isn’t enough; it’s the structured domain randomization in the Gaussian space that brings robustness. Standard 2D augmentation (color jitter, blur, etc.) simply cannot replicate the lighting effects of real shadows or reflections that these Gaussian perturbations capture. In sum, 3D Gaussian Splatting gives ViserDex a fast, flexible simulator: the object looks real and the team has direct handles (color, SH coefficients) to randomize it. This bridges the biggest sim-to-real gap – “How does the hand’s RGB view in sim ever match the unpredictable reality?” – without ever using real camera images in training. It’s photorealism-plus domain randomization all in one. Learning a Robust Policy: Curriculum and Distillation. With the perception pipeline in place, the next piece is how to learn the manipulation skill itself. ViserDex decomposes the problem: first train a teacher policy in simulation with privileged information, then distill it into a deployable student that only sees noisy sensor data. This decoupling avoids extremely sample-inefficient end-to-end learning with partial observations. The teacher policy is a standard DRL agent with full state feedback. The phase is implemented in NVIDIA’s Isaac Lab simulator, which runs the Allegro hand and the Gaussian object in physics. At each step the teacher sees full proprioception (joint angles) plus privileged exteroceptive info (the object’s exact position/orientation in the hand frame) and even fingertip contact forces. Its job is a goal-conditioned alignment task: it gets dense reward for reducing the rotation error to a target orientation, a bonus for reaching the goal, and small penalties on e.g. total torque. If the object is dropped or times out, the episode ends and resets. When the object hits the target orientation (within 0.4 radians), a new random target is sampled within the same episode, so the hand keeps reorienting continuously. Altogether this constructs a rich multi-task RL problem, akin to prior dexterous cube-reorientation work. They train this teacher via PPO. Critically, they found it useful to employ a performance-based curriculum to bootstrap learning. The curriculum has three parts that turn on gradually as the agent improves. First, the penalty terms (on object distance, velocity, etc.) start weak and grow stronger – so early on the agent can focus on any strategy to reach the goal, and only later is encouraged to do it smoothly. Second, they gradually introduce random action latency to simulate the real hand’s delays (how the robot’s control commands can lag). Third, they shorten the allowable time window to reach each new goal, forcing the policy to become more efficient. Each schedule is keyed to the agent’s moving average of consecutive successes: as the policy improves, each difficulty knob clicks up. This automated curriculum dramatically speeds up learning. In fact, an ablation in ViserDex shows that removing the curriculum (or even just the regularization schedule) causes learning to collapse to near zero success. With all components, the teacher converged quickly on each object. Because this is a high-dimensional problem, one might expect massive compute. However, ViserDex was explicitly designed for efficiency. The whole training was done on consumer-grade hardware. For many objects the teacher was trained on a single NVIDIA RTX 4090 (24GB), with no cluster needed. Even the most complex object (with finicky contacts) took about 90 hours on a modest two-GPU machine to complete. After that, they ran the student distillation phase for just 16 hours on one 4090. Contrast this with earlier dexterous projects that used supercomputer-scale resources: ViserDex’s runtime demonstrates that a single desktop can yield these advanced skills, thanks to efficient use of GPU physics (Isaac) and the 3DGS visual loop. Once the teacher policy is ready, the next task is to distill it into a student policy that will run in the real world with only sensor data. Here ViserDex uses a two-pronged approach. First, they corrupt the simulator’s object-pose observation (what would go to the student in real life) with realistic noise: random time delays, periodic dropouts, jitter, and calibration bias. This mimics the fact that the real pose estimator may be wrong for brief moments. Second, the student policy itself is given a recurrent architecture with a belief encoder-decoder. In practice, at each timestep the student takes noisy proprioception (joint readings) and the noisy exteroceptive pose estimate, and feeds these through a gated recurrent network to update a latent state. A decoding head tries to reconstruct the true privileged state from that latent, enforcing that the network learn an internal model of the object and contact dynamics. Simultaneously, a control head outputs actions that the student should take; these are trained with a behavior-cloning loss to match the teacher’s actions on the same states. During distillation, the student thus learns two things together: how to predict the object’s true orientation from partial info, and how to act. The recurrent encoder attracts all relevant history (so if a moment’s image is occluded, it can infer what happened). In tests, this design proved crucial. For example, if they replace the ViserDex pose estimator with a slower, off-the-shelf pose tracker called FoundationPose (which runs only at ~4 Hz), the performance plummets – reaching basically zero successful reorientations on average. The reasons are clear: the house-trained estimator in ViserDex runs at ~18 Hz, delivering fresh object poses, whereas FoundationPose is too slow and frequently loses track when the object suddenly flips or is occluded. The student’s recurrent filtering is designed for this high updates regime; with a lagging pose-sensor, it fails disastrously. In contrast, ViserDex’s student maintained robustness: the replay buffer noise perturbation test shows the recurrent belief decoder filtering out spikes that would otherwise confuse the controller. In summary, the control side of ViserDex is a classic teacher-student RL pipeline, augmented by a performance curriculum that reduces manual tuning and improves sample efficiency. By cleanly separating a privileged teacher (in simulation) from a robust student (in deployment), the authors avoid having to solve vision and control in one shot, which would be very data-hungry. Instead, the student only has to mimic a good teacher, smoothing over any sensor noise with its internal filter. And importantly, all this learning happened on a single consumer GPU, in hours to days rather than months. Monocular Pose Estimation via 3DGS Training. A linchpin of ViserDex is the vision stack: turning a single RGB image from the hand camera into an estimate of the object’s 6D pose. The pose feeds into the student policy as its “exteroceptive” input. To achieve this, ViserDex trains a neural keypoint-based pose estimator entirely on synthetic images generated from the 3DGS pipeline (no real images needed). Here’s how it works. First, for each object, they use their Gaussian splat scene in Isaac Lab to render many views of the object while it is held by the simulated hand. The rendering uses Isaac’s RGB pipeline and can incorporate the pre-raster augmentation described above. Since the simulator knows the exact object pose for each rendered frame, each image is paired with a ground-truth 3D pose label. To avoid bias from the hand shape, they first segment out just the object. (In practice, on the real robot they will deploy a separate mask network.) Next, they use a ResNet-34 CNN (pretrained on ImageNet) as a backbone, feeding it the cropped RGB object image. The network is trained to regress nine keypoints: the eight corners of the object’s 3D model plus the center. From those 2D keypoints in the image, the 6D pose is computed via a standard Perspective-n-Point (PnP) algorithm. (They train this end-to-end with data augmentation, so the network learns to predict keypoints under varied appearances.). Key implementation details make this robust. They operate on 120×120 crop windows upscaled to 224×224 for the ResNet, letting the network focus only on the object. They also leverage the very latest in segmentation: for deployment on the real robot, they fine-tune Meta’s “Segment Anything Model 2” (SAM2) for each object, so that in real time the camera feed is first masked to isolate the toy. This ensures the pose net always sees a clean object image. Finally, they use the exact same 3DGS-augmented imagery for policy training and for pose training, closing the loop between perception and control. The results speak for themselves. In testing on real images, the pose network trained with ViserDex’s 3DGS data vastly outstripped conventional baselines. Under nominal (white) lighting, ViserDex attained an average ADD error of ~10.2 mm (and 65.4% of poses within 10 mm/10° of true). By comparison, a standard high-performance renderer with typical 2D image augmentations only got ~53% accuracy, and even a version with heavy domain-randomization of mesh attributes scored ~55%. Under severe lighting shifts, the gap widened: ViserDex’s accuracy only fell to 56.3%, whereas the best baseline fell to ~47%. Most eye-opening, a naive use of Gaussian rendering without the smart perturbations gave just 36.5% under those conditions. This validates that the carefully structured 3D randomization – not just “better graphics” – is what yields robustness. On the robot, the pose estimator runs at roughly 15–20 Hz (limited by segmentation and inference time) with negligible delay. This high throughput is critical. As noted, an off-the-shelf 6D pose tracker that ran at only 4 Hz caused the system to fail almost immediately. In contrast, the custom net trained on ViserDex data keeps up with the fast motions and finger occlusions. In deployment trials where the authors simulated momentary pose dropouts (by injecting random noise), the recurrent student policy handled them gracefully because its belief module had learned to filter out sudden spikes. In effect, ViserDex has turned monocular vision from a liability into a strength. The system learns to “see through” occlusions and odd lighting by training on exactly those phenomena in simulation. The fact that both control and perception use Gaussian-splat-based rendering means there is no inconsistency: the policy knows exactly the kind of images its vision model sees during task execution. Experimental Setup and Results. ViserDex’s claims rest on real-world experiments. The team tested their system on a 16-DoF Allegro five-fingered robotic hand mounted on a fixed base, with a single RGB camera attached to the wrist pointing at the palm. They used five everyday test objects: a colored cube, a rubber duck, a textured toy model, a transparent shampoo bottle, and a glossy table-tennis ball (“globe”). These span simple to complex shapes and materials. The goal in each run was arbitrary: the object’s orientation should be rotated to a randomly chosen target orientation, achieved inches above the palm (the hand keeps it aloft). Importantly, they evaluated under both nominal lighting (a white overhead light) and adversarial lighting. The adversarial setup used multi-colored flashing lights aimed at the hand, causing dramatic color shifts and shadows in the camera view. This stresses the vision system. As metrics, they use “consecutive successes” (CS): how many goals in a row the hand reaches before it drops the object. This follows prior work conventions. Five runs per object and lighting were done. Pose estimation performance first (Table II and III in the paper). We already noted the overall accuracy gains. On individual objects, ViserDex’s poses were especially better on complex textures. For example, the 3D-printed toy had ~56% accuracy (vs ~32% for standard rendering) in normal light. Under harsh lights, ViserDex averaged 10–15 mm XYZ error on all objects, whereas the baselines were much noisier. In ablations, removing any one augmentation group (e.g. no spatial clusters) caused the estimator’s accuracy to drop several percent. All groups together gave the best mean accuracy by far. Next, policy performance in physics (sim-to-real). In nominal lighting, ViserDex achieved an astonishing average of 37.6 consecutive successes across the objects. That means most trials could reorient an object more than thirty times in a row before failing – and one object (the Globe) exceeded 200 in a single run. By contrast, the only prior vision-only dexterous-inhand work on hardware is DeXtreme (2023), which got 27.8 CS on a cube reorientation task. ViserDex beat that baseline on the shared cube: 35.4 vs 27.8. On other shapes, especially smooth symmetric ones (like the Globe or Duck), the policy was extremely reliable. On tricky items like the slick-labeled bottle, performance dipped (still ~14 CS on average), likely because the thin plastic label introduced friction effects not modeled exactly in sim. Notably, no object was specially tuned: “the same reward weights and hyperparameters were used across objects and no per-object changes were made”. This suggests the skill generalized well to variations. Under adversarial lighting, performance did drop (as expected) but remained robust: about 25 consecutive successes on average. Again, the globe and duck (round, symmetric) remained easy, but the colored lighting creates occasional false pose readings. Even so, this is the first report we know of continuous in-hand policy running for dozens of steps under extreme color flashes. In short, ViserDex “really worked” on the real hand. The authors also tested how each component contributed. Replacing their pose estimator with FoundationPose (a recent general 6D pose system) at runtime made the hand almost always drop immediately, confirming the need for high-rate, occlusion-friendly vision. Using a “standard telescoping curriculum” vs their performance one slowed convergence to the point of failure. And, simply training an RL policy “end-to-end” with fixed randomization (no 3DGS) never approached these numbers in their trials (the paper does not report a full baseline rollout, but the inferior pose accuracy implies a failed sim2real). Finally, a few noteworthy numbers: Because ViserDex was engineered for efficiency, training is not out of reach. The teacher took on the order of tens of hours per object on a single GPU (faster for simple shapes, up to ~90h for the hardest). The student distillation then ran in a few more hours. In sum, all of training finished on a desktop rig – compare this to prior works which often needed GPU farms. Once trained, the real robot executes in real time at about 15fps (camera+segmentation+pose+control) and can sustain operations indefinitely. Impact and Takeaways. ViserDex demonstrates a new paradigm for vision-based dexterity: explicitly simulate the visual world as a manipulable 3D representation, randomize that representation, and then train controllers purely in sim. The successful zero-shot transfer should encourage further use of 3DGS in robotics. The authors point out that Gaussian splatting fully breaks the traditional trade-off between realism and speed. Here it let them generate photorealistic images at high speed (millions of pixels per batch) and also afford the geometric control needed for domain randomization. It's a clever fusion of graphics and control. For practitioners, there are concrete lessons. First, a strong vision system with in-the-loop domain randomization can remove reliance on expensive sensors. A well-trained CNN keypoint detector, fed by realistic simulator images, handled the alleatory lighting and the brutal finger occlusions that stop naïve methods. Second, curriculum learning paid off. Without slowly ramping up penalties and delays, the agent couldn’t learn at all, showing that RL on such complex tasks benefits immensely from guided difficulty schedules. Third, the recurrent distillation approach turned teaching a policy on privileged data into a robust closed-loop agent – a recipe that will likely apply to many other sim-to-real tasks. Looking ahead, ViserDex focuses on static objects and one-handed reorientation. Extensions could include dynamic object sets (changing shapes mid-run), two-handed tool use, or longer-horizon plans. The integration of Gaussian splatting could also be applied beyond pose: one can imagine training perception pipelines for whole-arm grasping, navigation in cluttered scenes, or any scenario where photorealism and diversity matter. On the control side, the fact that this all ran on commodity GPUs suggests that research labs and even startups might adopt similar pipelines more easily than before. In closing, ViserDex is a compelling step toward truly vision-driven manipulation. It shows that with the right simulation tools, the line between lab and reality can blur: the same camera image fed to the policy in hardware was, in all practical senses, drawn “from the same distribution” as the images used in training – because they were literally rendered from physics in virtual space. As robotists seek to deploy dexterous hands in homes and factories, approaches like ViserDex hint at the possibility that we won’t need armies of cameras or lasers to make it work – just clever graphics, smart learning algorithms, and the willingness to lean heavily on synthetic data. Sources: The above summary is based on ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation by Bhardwaj et al. (ETH Zurich, 2026), along with related technical references and background on 3D Gaussian Splatting. The results discussed (pose accuracy, success rates, etc.) are drawn from the paper’s experiments and ablations. The NVIDIA Isaac Lab simulator was used for data generation. All claims about robustness, performance, and compute requirements are explicitly documented in the referenced work.