Latest / Embodied AI 101 / SimToolReal: Procedural Tool Generation and a Universal Objective for Zero-Shot Tool Manipulation
Transcript
- SimToolReal: Procedural Tool Generation and a Universal Objective for Zero-Shot Tool Manipulation. Robots are still not very good at using everyday tools – even simple tasks like picking up a spatula or swinging a hammer can stump them. Tools tend to be thin, odd-shaped, and require dexterous in-hand rotations or forceful motions. As the authors of SimToolReal put it, tool use “requires grasping thin objects, in-hand rotations to functional configurations, and maintaining stable grasps during interactions”. Collecting real-world demonstrations for such tasks (for example by teleoperating a robot) is tedious and expensive, so a natural idea is to train in simulation and transfer to the real world (sim-to-real RL). But most prior work takes a narrow approach: build a detailed simulation of one particular tool and task, hand-craft a reward function for that task, train a policy (or collect demonstrations), and hope the policy transfers. In practice this means every new object or tool category requires fresh modeling effort. SimToolReal (sometimes stylized “SimToolReal” in their project materials) tackles this head-on by flipping the paradigm. Instead of one-off simulators, it uses procedural generation to create a huge variety of random “tool-like” objects in simulation. Instead of one fixed task, it uses a single universal objective: move the object to a specified goal pose. The key insight is that by training a policy on many random shapes all with this same goal-reaching objective, the agent can learn fundamentally object-centric manipulation skills that generalize to new tools and new tasks without additional training. Amazingly, a single learned policy can zero-shot handle dozens of real tool-use tasks – drawing with pens, swinging hammers, flipping pancakes, using screwdrivers, and more – despite having never seen those specific objects during training. In effect, the approach produces a kind of “foundation skill” for tool manipulation. In this episode we’ll dive into how SimToolReal works: how they generate tools, how the RL is set up, and how they pull off real-world transfer. We’ll also see the new DexToolBench benchmark they introduce and what experiments they ran. By the end you'll understand how one can train a truly general robot tool-use policy in simulation and then deploy it on a physical robot without task-specific tuning. The Hardness of Tool Use. Let’s begin by appreciating why tool use is tough. A human using a tool can adapt fluidly to the tool’s shape: we know how to pick up a brush differently from a fork, we grip a hammer one way and a screwdriver another, and we apply different forces. But robots don’t yet have that universal know-how. If you give a robot a new thin object on a table, it might fail to even grasp it securely. If it does grasp it, swiveling it around to the right orientation often requires delicate wrist and finger control. Classification and perception are another hurdle: identifying “this is a pencil I need to use for writing” vs “this is a hammer I need to swing” adds more complexity. SimToolReal summarizes these challenges well: “tool manipulation represents a challenging class of dexterity, requiring grasping thin objects from flat surfaces, in-hand object rotations, and forceful interactions”. Moreover, collecting real data for these behaviors is hard (you’d need to teleoperate the robot with a real hammer and get it to swing a nail, over and over). That’s why they turn to sim-to-real RL: train in a simulator and deploy on the real robot. Most prior sim-to-real approaches to manipulation have one thing in common: they treat each task as separate. For example, to train a robot to use a particular screwdriver, you’d model that screwdriver’s geometry, define exactly what “use the screwdriver” means in that context, tune a reward for it (say, putting a screw in a hole), and train a policy. The result is often brittle and specialized: change the scene slightly or give a different-length screwdriver, and the policy fails. SimToolReal asks: What if we don’t specialize at all? What if we train one big policy on lots of random tools and tasks, all mixed together, with one simple goal?. Procedural Tool Generation and a Universal Goal. The heart of SimToolReal is to teach one policy everything about basic tool manipulation by simulation. The approach is:. Procedurally generate random “tool-like” objects in simulation. Instead of loading a few CAD models, they algorithmically create objects with different shapes and sizes. For example, they might generate a thin cylinder for a handle, attach a rectangular block as a head, or combine random primitives into new shapes. The exact method of generation isn’t spelled out in the abstract, but the phrase “Procedurally generate a large variety of tool-like object primitives in simulation” suggests they use code to randomly vary dimensions, add and remove parts, etc. The result is a zoo of synthetic tools: think sticks, rods, flat bars, etc. – anything that might resemble an everyday tool in geometry. By doing this, they create effectively infinite training data of different handle lengths, head sizes, curvatures, and so on. A single universal training objective: goal-pose reaching. With these random tools, the robot’s task is always the same kind of goal: move the tool from wherever it starts to some goal pose in space. In training, each episode picks a random tool and a random target pose for that tool. The policy receives as input the current pose of the tool and the goal pose (green mesh in their videos) and has to output robot actions to drive the tool towards the goal. Importantly, “without any object or task-specific training” means there is no special reward for different tools – the reward function is universal. It likely just encourages minimizing the distance (and orientation difference) between the tool’s current pose and the goal pose. So whether the tool is a skinny rod or a flat spatula, the objective is identical: reach the goal. This is what they call a “universal goal-reaching objective.”. Because the goal is pose-reaching, the policy is effectively goal-conditioned. During training, each step it sees (tool_pose, goal_pose) and receives feedback like “you are closer/further from the goal.” This setup forces the agent to figure out the right grasp and manipulation strategy for each new shape on the fly. For example, if the goal pose is far above the table, it must first pick up the tool; if the goal requires rotation, it may need to twist it in-hand. The beauty is: these skills emerge purely from the data – there’s no separate module for “grasp detection” or “rotate-in-hand” hard-coded. Training details. The SimToolReal implementation uses NVIDIA’s Isaac Gym simulator (for fast parallel training on GPU) and the with PPO or SAPG algorithms. The agent presumably controls the robot arm (a KUKA IIWA with a dexterous “Sharpa” hand, see below) and gets dense rewards based on how close the tool is to the target pose. They randomize which tool primitive and target pose each episode, so the policy never sees the same instance twice. Over time, the policy learns a versatile strategy: no matter what tool shape you hand it, it will attempt to align that tool with the goal pose. In other words, it has learned the general affordance of the tool – to move this object in 3D space. This is in stark contrast to prior “one-tool-one-policy” methods. Here there is one policy that handles all the tools. As the authors say: “We train a single RL policy to manipulate each [tool] to random goal poses”. The strategy pays off: the policy actually performs “general dexterous tool manipulation at test-time without any object or task-specific training”. In practice that means, at deployment time you can show the robot a brand-new tool (even one it has never seen) and a desired pose for that tool, and it attempts to achieve it. From Simulation to Reality: Execution Pipeline. Training in simulation is one thing; making it work on a real robot is another challenge. SimToolReal addresses sim-to-real transfer in a pragmatic way. First, because the policy is object-centric, it assumes access to the precise pose of the tool during execution. In simulation this is trivial (the sim engine knows the object pose). On the real robot, they need a perception pipeline that can track the tool’s pose in real time. SimToolReal uses a combination of vision and pose estimation: a ZED stereo camera looks at the workspace, and they run Meta’s Segment Anything Model (SAM) to segment out the tool, followed by NVIDIA’s new “FoundationPose” model to estimate the 6-DoF pose of that segmented object. In short: SAM + FoundationPose — a state-of-the-art system (CVPR’24 highlight) for “tracking novel objects”. This runs as a separate ROS node, continuously publishing the tool’s current pose to the robot. The hardware itself is a KUKA IIWA arm with a dexterous hand (called the “Sharpa” hand) as the end-effector. The ZED camera is mounted to view the table surface. Given a target pose, the perception system feeds the tool’s observed pose into the same policy loop that was used in sim. In other words, at run time the input to the policy is (current_tool_pose, goal_pose), and the output is desired robot joint positions (or velocities) to apply. Because the policy was trained in a simulated IsaacGym environment, they use a “simulated robot” node to interpret those actions and a “fake perception” node to feed in the real object pose back into the policy loop. To actually control the real arm, they replace the simulated nodes with an IIWA “Robot Node” that sends commands and the SAM/FoundationPose Perception Node to provide object pose estimates. Though the paper doesn’t emphasize it explicitly, this approach essentially side-steps much of the visual sim-to-real gap: the policy does not need to interpret raw images; it just sees object poses. Any difference between the simulated and real object shape is handled by the vision system, which matches the real object to a known mesh (they do have 3D models of the test tools for the tracker). Domain randomization of visuals or textures is thus not needed since we aren’t training from pixels. The main real-world challenges are accurate calibration and dynamics (e.g. friction differences), but the paper suggests the policy was robust enough to handle those with minimal fuss. The use of a high-performance tracker (FoundationPose) and careful calibration likely accounts for its smooth transfer. DexToolBench: A Zero-Shot Tool-Use Benchmark. To measure how well their generalist policy performs, the authors created DexToolBench, an evaluation suite of everyday tool tasks. Importantly, the key is to test zero-shot: the policy must succeed without having seen the exact objects or trajectories in training. The benchmark consists of 6 tool categories, each with 2 object variants, and 2 tasks per object — totaling 12 objects and 24 tasks. The categories (and tasks) include:. Hammer: Two types (claw hammer, mallet), tasks are to swing the hammer down (as if hammering) or swing to the side. Marker (Pen/Sharpie): Two markers (Sharpie and a Staples marker), tasks are draw a smiley face or write the letter “C” on a surface. Eraser: Two erasers (one flat, one with handle), tasks are wipe off a smiley face or wipe off a letter “C”. Brush: Two brushes (“blue brush” and “red brush”), tasks are sweep the brush forward or sweep it to the right. Spatula: Two spatulas (flat and spoon-style), tasks are serve a plate (like scooping motion) or flip something over (like a pancake). Screwdriver: Two screwdrivers (long and short), tasks are spin it clockwise (vertical) or spin it (horizontal screw labored). These tasks cover a wide range of motions: drawing, erasing, sweeping, hammering, flipping – all requiring different grasps and coordination. Crucially, none of these exact objects or tasks appeared in the procedural training set. This evaluates genuine generalization. During evaluation, the robot is given a human demonstration video or an example trajectory. The system extracts a sequence of desired goal poses from that demonstration. For instance, to benchmark “draw smiley,” a human might show the brush drawing a smiley on paper; the system then uses the tool’s pose at keyframes as the goals. The policy then tries to reach each of those goal poses in sequence. In the demo interface (built with NVIDIA’s Viser UI), the current target goal is shown as a green mesh around the tool, which advances once that goal is achieved. All trials are zero-shot: “we never train on the target object or task in simulation”. The entire DexToolBench (including the 3D models, trajectories, etc.) is released as part of their code, so other researchers can test their policies. They even provide a handy web-based demo for quick testing. The authors plan to open-source all training and evaluation code alongside DexToolBench. Results: One Policy to Rule Them All. How did this crazy idea of “train on random shapes, test on real hammers” work in practice? Surprisingly well. According to the paper, the SimToolReal policy outperforms naive baselines by a large margin. For example, a simple “fixed-grasp” baseline (where the robot always grabs the tool at a fixed mounting point without adjusting) or a kinematic retargeting approach (naively copying motions) were significantly worse. SimToolReal’s general policy beat those baselines by about 37% on their metrics, which is a huge lead. Even more striking, the generalist matched the performance of many specialized experts. That is, if you had trained individual RL policies each on a particular object and task (with a tuned simulation and reward), those specialists didn’t outperform SimToolReal by much. In the authors’ words: “we demonstrate that SimToolReal matches the performance of specialist RL policies trained on specific target objects and tasks”. In practical terms, one policy learned on random shapes did just as well on a brand-new hammer-swing task as a policy trained specifically for that hammer. Across 120 real-world rollouts on the 24 tasks (6 categories × 2 objects × 2 tasks), SimToolReal showed strong zero-shot success. For example, in real experiments the robot could swing a real claw-hammer down onto a target plane, draw a smile with a Sharpie, wipe it off with an eraser, and even spin an unfamiliar screwdriver in place – all without any new adaptation. The only training ever done was on synthetic “primitives,” yet it handled these everyday tools fluidly. The reported numbers paint a clear picture: the unified policy achieves robust success across categories. One can infer some qualitative behaviors: the policy must learn, for instance, to adjust its grasp on different tools (maybe even reposition fingers mid-task), to lift them off the table, and to apply the right motions. Watching the demo videos (available on the project page) shows the robot deftly adapting. In some failure modes, the policy still recovers: if a tool slips, it tries new grasps. The authors note that the policy exhibits “strong recovery behavior from failures”, meaning it often re-grasps or re-attempts until the goal is met (or the trial times out). Why SimToolReal Matters. SimToolReal represents a significant shift in how we think about training manipulation skills. Instead of engineering one-off solutions, it shows that diverse simulation data and one general objective can yield broadly capable policies. This is akin to how recent work in other domains tries to build “foundation models” by training on massive diverse data with a single task (e.g. language modeling for text, or ImageNet for vision). Here, the “foundation model” is a robot policy trained on procedurally generated tools. There are a few key takeaways for practitioners:. Domain Randomization ≠ Always Necessary: They didn’t mention extensive randomization of visual or physics parameters. By feeding the policy precise tool poses and relying on a strong perception system (SAM + FoundationPose), they largely sidestep the toughest sim-to-real issues. The main learning – how to grasp and move an object – stays nearly unchanged between sim and reality when given the right pose information. Goal-Conditioned RL Pays Off: Using a universal goal-reaching objective removes the need for custom rewards. In a sense, any behavior becomes possible if you set the goal appropriately. For tool use, most tasks can be framed as “move tool to this desired pose” (e.g. the end of the smiley-face drawing, the angulation of a hammer, the orientation of a flipped pancake). This sidesteps reward engineering for each scenario. A side benefit is that the policy can track trajectories of goals (multiple goals in sequence) simply by feeding them sequentially. Procedural Generation is Powerful: Procedurally shooting for diversity in shapes is more scalable than manually collecting CAD models. It means you don’t have to model a specific hammer or brush to succeed on a new one. The training set (random primitives) might look nothing like a real toothbrush or spatula, but the policy only cares about the core physics of moving a rigid object. This is a compelling demonstration that in some robotic domains, “more random things” beats “the right specific things.”. Benchmarking Dexterous Tool Use: DexToolBench itself fills a gap. Before this, there wasn’t a standard suite for testing zero-shot tool tasks. With this, future policies can be measured on exactly the same tasks and real objects. It’s analogous to how RLBench, Meta-World, or other benchmarks standardized grasping/pushing tasks. Now there’s a standard for tool manipulation across categories. Comparison and Context. SimToolReal fits into a growing trend of trying to make robots more generalists. For example, Ren et al. (CoRL 2022) showed that language descriptions could speed up learning of new tools by reusing previously learned behaviors. They essentially told the robot “this looks like a hammer” or “this is a rock you can use for swinging,” using language priors. In contrast, SimToolReal doesn’t use language at all – it relies purely on physics-generalization. Both approaches aim for broader skill, but SimToolReal’s recipe is: training diversity + single objective. Another related idea is artifactual “retargeting” methods, which try to map human hand motions onto a robot or use one known policy to act with a new object. The paper compares against a baseline like this (they mention “kinematic retargeting” in their code). Those methods often fail when the morphologies differ too much. SimToolReal sidesteps retargeting – there are no hand demonstrations, just goal poses. There’s also a body of work on grasp networks or affordance learning for tools – e.g. learning that “this object can be used as a hammer if grasped at the bottom”. SimToolReal presumably learns some implicit affordances, but within the policy weights themselves. It would be interesting future work to inspect whether the network has, say, separate strategies for rod-shaped vs plate-shaped tools, or whether it’s a fully continuous interpolation. In terms of sim-to-real for manipulation, much previous work (e.g., Popov et al. 2017, Levine et al. 2016) focused on single, specific tasks (like insertion or door opening). SimToolReal shows it’s possible to train at a larger scale of variety. This moves us towards “foundation models” in robotics: if we keep increasing the variety (not just tools but any object interaction), maybe a robot could develop very general skills. Limitations and Future Directions. No method is perfect. SimToolReal still has some constraints. For one, it assumes you can measure the tool’s pose accurately. In a real household, you might not know the tool’s exact 3D geometry or have a good tracking model. Here they did have the 3D models of the test tools and a calibrated camera. Extending this to fully unknown, deformable, or transparent objects would be a stretch. Also, the current tasks are all about moving rigid objects with one grasp (no re-grasping from arbitrary angles, no elastic or granular manipulation). Another point is that while the policy is “general”, it is still limited by the simulation variety. If the real tool is too different from anything the system saw (for example, a very large or squishy tool), it might fail. The procedural generation defines the domain of what’s “familiar”. Future work could make the generator even richer (different materials, hinges, etc.), or use random distortions to cover even more cases. A rich direction is to combine this with vision-language interfaces. For example, one could imagine a robot that sees a novel object, asks “what tool is this used for?”, then sets a goal pose (derived from language or a demonstration) and uses a SimToolReal-like policy to execute. The authors already show how to take a video demonstration and extract goals. Pairing that with, say, a GPT-based task description (“please draw a smiley with that marker”) could make it interactive and even more flexible. Finally, it would be interesting to see if similar ideas work for other domains of manipulation beyond tools: say, cooking ingredients (cutting vegetables at a target location), or assembly tasks with parts. The combination of procedural generation and goal-conditioned RL could unlock a lot of generalization. Conclusion. SimToolReal is a striking example of how scale and diversity in simulation plus a clever unified objective can yield truly generalizable robot skills. By training on thousands of simulated “random tools” with a single goal-reaching task, the authors produced one policy that can zero-shot handle a dozen real-world tools and tasks it never saw in training. For researchers and engineers, the takeaway is clear: when it comes to learning, it may be better to randomize broadly than to optimize narrowly. In practical terms, SimToolReal and its DexToolBench give us both a working recipe and a testbed for general-purpose tool-use policies. We may soon see follow-up works that push these ideas further – perhaps with richer tool models, improved perception, or integration of other sensorimotor signals. For now, SimToolReal stands as an impressive step towards robots that learn the essence of “using a tool” once and apply it everywhere. Sources: The above is based on Kedia et al.’s recent paper SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation and its accompanying project descriptions. Technical details on perception and hardware come from the authors’ code documentation.