Latest / Embodied AI 101 / A Sphere Every Robot Hand Can Speak

Transcript

A Sphere Every Robot Hand Can Speak · Embodied AI 101

• A Sphere Every Robot Hand Can Speak. Cross-Embodiment Robot Manipulation via a Unified Hand Action Space • Luis Felipe Casas, Robert Teal, Keval Shah, Abhijit Tadepalli, Wanxin Jin, and Yu Xiang • Intelligent Robotics and Vision Lab, University of Texas at Dallas; Intelligent Robotics and Interactive Systems Lab, Arizona State University • arXiv cs.RO and the 4th Workshop on Dexterous Manipulation at RSS 2026 • 2026. Robot learning has become reasonably good at sharing representations across cameras, scenes, tasks, and language instructions. Sharing actions is another matter. A model can ingest images from two robots with very different camera placements and still map them into a common visual feature space. But if one robot’s action vector contains 16 Allegro joint targets, another contains 16 LEAP targets with different kinematics, and a third contains more than 20 Shadow Hand coordinates, those vectors do not merely have different dimensions. Their individual entries mean different things. That is the problem tackled by the Unified Hand Action Space, or UHAS. Rather than asking a neural network to discover correspondences between incompatible joint spaces, UHAS places an explicit geometric interface between the policy and the hardware. The policy deforms a virtual sphere attached to the hand. A hand-specific controller then turns that deformation into joint commands. The central claim is not that a sphere perfectly describes every dexterous movement. It is that a sphere can provide enough common structure to support useful policy sharing across multiple anthropomorphic hands. The paper evaluates that proposition on in-hand cube reorientation using four embodiments: the four-finger Allegro and LEAP Hands, the five-finger Shadow Hand, and a simulated five-finger MANO human hand. It reports multi-hand training, leave-one-hand-out zero-shot transfer, limited cross-finger-count transfer, rapid adaptation, and physical deployment on LEAP and Allegro hardware. The results make UHAS an interesting example of a broader design pattern: instead of scaling the policy until it learns around action heterogeneity, first redesign the action interface so that heterogeneous robots expose comparable semantics. The Action Vector Is the Embodiment Bottleneck. Most multi-robot policies deal with action heterogeneity in one of three ways. They can standardize all robots around a common end-effector command, use separate action heads for each embodiment, or learn a latent action space with embodiment-specific encoders and decoders. Each option has drawbacks. A Cartesian end-effector action works well for robot arms because different manipulators can often be commanded through a shared wrist pose or twist. It is far less obvious what the analogous command should be for a dexterous hand. Fingertip positions are attractive, but fingertips alone omit the posture of intermediate links and the palm. A fixed canonical joint vector requires manually deciding which joints correspond. A learned latent space can discover richer mappings, but it introduces another training problem and often depends on paired trajectories, retargeting, or large multi-embodiment datasets. UHAS takes a more structured route. Its common variable is neither a joint vector nor a learned code. It is a deformation of the volume into which a hand naturally closes. Lateral motion is represented by moving regions around the sphere, while opening and closing are represented by radial expansion and contraction. This gives the policy an action with an interpretable geometric meaning, even when the receiving hands implement that action with different joint axes, link lengths, and finger counts. It helps to think of UHAS as an operational space for the entire hand. Conventional operational-space control asks a robot arm to produce a motion or force at an end effector while a kinematic solver handles the joints. UHAS asks a collection of fingers to reshape a virtual enclosure while its Cascade Inverse Kinematics module handles the embodiment. The controlled object is not literally constrained to the sphere, and the sphere is not an object model. It is a palm-attached virtual control surface representing coordinated hand geometry. This distinction matters. The policy is still learning a dynamic contact strategy. It is simply learning that strategy in a space whose coordinates are intended to retain their meaning across hardware. Constructing the Canonical Sphere. UHAS begins with a hand URDF and a designated open-hand configuration. The preprocessing pipeline identifies the palm and fingertip frames, computes the palm center from the finger roots, and measures the mean distance from that center to the fingertips. It chooses the sphere radius so that a quarter-circle arc on the sphere is approximately equal to this mean palm-to-tip length. Put differently, a fully extended finger is conceptually wrapped across about 90 degrees of the sphere. The sphere center is placed one radius outward from the palm along the palm normal. Its positive vertical axis follows that normal, while another axis points toward the middle finger, producing a consistent orientation across hands. All lengths are then divided by the hand-specific sphere radius. A small hand and a large hand therefore operate on the same unit sphere, while the original scale is retained for converting actions back to physical coordinates. This procedure encodes several assumptions. The hand should have a recognizable palm, an open configuration, and fingers that converge into a roughly enclosing workspace. Those assumptions are reasonable for the four hands in the paper. They are less obviously valid for highly underactuated grippers, tendon-coupled mechanisms, hands with articulated palms, or non-anthropomorphic arrangements. The next step establishes dense correspondences between the sphere and the interior surfaces of the hand. Points are sampled on the sphere and projected onto nearby points on the palm and fingers. Each corresponding hand-surface point receives a spherical address consisting of an azimuth and polar angle. When the hand articulates, the point moves in three-dimensional space, but its spherical address remains attached to the corresponding link. That stable address is the bridge between geometry and motion. A deformation specified at a particular region of the sphere can be transferred to whichever finger surface occupies that region on a given embodiment. It also gives UHAS something more informative than a sparse fingertip correspondence: intermediate finger surfaces and palm-facing regions can participate in the mapping. This construction is closely related to the authors’ earlier RobotFingerPrint work. RobotFingerPrint mapped gripper interior surfaces into a shared spherical coordinate system and used those coordinates to represent object-centric grasps across grippers. UHAS makes a significant conceptual extension: rather than using the sphere only to describe a completed grasp, it treats changes to the sphere as continuous control actions. Turning Sphere Deformations into a Compact Action. A dense displacement vector for every sphere point would be far too large for reinforcement learning. UHAS therefore restricts its deformation field to two components. The first is an azimuthal displacement. This moves a region laterally around the palm normal and corresponds roughly to finger abduction, adduction, or other side-to-side motion. The second is a radial displacement. Moving a region inward asks the corresponding hand surface to close around the sphere; moving it outward asks the surface to open. UHAS does not directly predict a separate polar-angle displacement. Instead, different radial values along the polar direction describe how the finger should conform to the virtual surface. The deformation is controlled through a sparse set of “driving planes.” Each plane passes through the sphere center and is initially aligned with a finger. Rotating a plane specifies the lateral action for that finger region. Each plane also contains a few driving vectors located at fixed polar angles, and the lengths of those vectors specify radial deformation. The full sphere deformation is reconstructed by interpolation between planes and vectors. The final configuration uses five driving planes and two radial vectors per plane. That produces 15 continuous action dimensions: five lateral controls plus ten radial controls. The two radial vectors are positioned at polar angles of 60 and 120 degrees, giving the policy separate influence over more proximal and more distal portions of each finger region. The radial values can range from negative two to positive two in normalized sphere units. Four-finger and five-finger hands still need the same tensor shape. For a four-finger hand, the method inserts an extra plane at the ring-finger position. The duplicated plane’s actions are averaged when the deformation is applied. The observations use an analogous duplication. This is a pragmatic rather than anatomically principled correspondence, but it allows one network to retain independent fifth-finger control when running on a five-finger embodiment. The paper’s ablations suggest that the cube task does not make heavy use of that fifth independent plane. Four-plane and five-plane variants perform similarly on Shadow and MANO, while policies with only three planes fail to learn reliably. The authors explicitly caution that the apparent dispensability of the fifth plane is probably specific to cube reorientation and its reward design. The number of radial vectors presents a similar capacity trade-off. Three vectors obtain slightly better final simulation metrics, but two train faster and reach nearly the same performance. One vector lacks flexibility, while four add action complexity without a reliable gain. The chosen 15-dimensional interface is therefore not a minimal representation, but an empirically useful operating point. Cascade Inverse Kinematics: The Embodiment Adapter. The policy never directly commands the hand joints. Its sphere deformation is passed to Cascade Inverse Kinematics, or CIK, which produces an embodiment-specific joint-position target. CIK first classifies every hand joint according to its dominant geometric effect. During preprocessing, each joint is swept across its range while forward kinematics tracks the corresponding fingertip in spherical coordinates. A joint that primarily changes fingertip azimuth is classified as lateral. A joint that primarily changes radial distance or polar position is classified as encompassing. The classification is performed once for each hand from its URDF. Lateral joints are handled through precomputed lookup tables. The pipeline samples each lateral joint, resolves the rest of the finger against the undeformed sphere, and records the resulting fingertip azimuth. At runtime, the desired plane rotation specifies a target azimuth, and the lookup table returns the corresponding lateral joint value. This avoids running a numerical optimizer for abduction or side-to-side motion. Once lateral joints are set, encompassing joints are solved sequentially from the finger root toward the fingertip. At each joint, CIK uses the sphere targets associated with that link and its descendants to select the one-dimensional joint rotation that best conforms the downstream finger surface to the deformation. Each joint is visited once in a forward cascade, and different fingers can be processed independently. This decomposition is deliberately lightweight. The authors report that CIK can run at roughly 150 hertz and was not the bottleneck in their physical system. Communication with the LEAP Hand and AprilTag pose estimation were slower. There is also a revealing task-specific detail. UHAS permits radial deformation to move through the sphere center into negative radius. Points with negative radius are discarded before CIK. If an encompassing joint is left with no reachable target points, it is commanded fully closed. This non-smooth rule lets the policy request an aggressive finger closure during fast cube motions, and the authors found it helpful for reorientation. It also illustrates that UHAS is not a purely neutral coordinate transform: its implementation contains priors shaped by the intended manipulation regime. CIK’s hard-coded semantics are both the method’s strength and its likely boundary. It gives the learning system a strong inductive bias and avoids paired cross-hand data. But joints with strongly coupled effects must still be assigned to one dominant class, and the solver assumes that individual finger chains can be treated largely independently. Complex tendon coupling, rolling joints, articulated palms, and contact-driven compliance may not fit this decomposition cleanly. A Unified Policy Also Needs Unified Proprioception. A common action space is not enough if the observation still contains embodiment-specific joint coordinates. UHAS therefore replaces raw hand joints with geometric proprioception. The preprocessing pipeline samples points along every finger chain. At runtime, forward kinematics provides their positions, while Jacobians and joint velocities provide linear and angular velocities. These values are expressed in the canonical sphere frame and divided by the hand-specific radius. The final policy uses only two points per finger: one around the finger midpoint and one at the fingertip. Four-finger hands again duplicate the ring-finger slot to match the five-finger observation layout. The task observation still includes information about the cube and target orientation. What changes is the representation of the hand itself. Instead of receiving arbitrary encoder ticks whose meaning varies by robot, the policy receives the current geometry of comparable locations along each finger. This is an essential part of the paper. It would be easy to attribute all transfer to the sphere action, but UHAS is really a paired observation-and-action interface. The network sees hand state in normalized geometric coordinates and emits commands in normalized geometric coordinates. The embodiment-specific joint space exists below both sides of the policy. The authors train state-based actor-critic policies with PPO using RSL-RL and a four-layer multilayer perceptron. Training runs in Isaac Lab with 1,000 parallel environments on a single NVIDIA A5000 GPU. The reward largely follows Isaac Lab’s cube-reposing environment, with additional penalties intended to prevent policies from overusing lateral joints or otherwise exploiting one embodiment’s special kinematics. Domain randomization covers object scale, mass and friction; robot mass, friction and actuator properties; palm inclination; and the placement of sphere-driving controls. The point is not only conventional sim-to-real robustness. Randomization also prevents a shared policy from depending too heavily on incidental dynamics of one simulated hand—for example, always expecting the cube to settle into a particular palm region. Despite the paper’s discussion of future robot foundation models, this is not yet a vision-language-action system. It is a compact, privileged-state reinforcement-learning experiment designed to isolate the action-space question. That narrower setup is a virtue for understanding the representation, but it should constrain how broadly we interpret the results. What the Experiments Actually Show. In simulation, each episode asks the hand to reach ten successive target cube orientations. A target must be reached within 30 seconds. If the cube falls, the environment resets and evaluation continues with the remaining targets. The paper reports both individual success rate and the average number of consecutive successful reorientations before the first drop. The in-distribution task is close to saturated for every controller. Single-hand UHAS policies reach success rates between 99.1 and 99.8 percent. Direct joint-control baselines range from 98.0 to 99.6 percent. UHAS usually has a small advantage in consecutive reorientations, but the absolute task-performance difference is not dramatic. The stronger result is that one multi-hand policy reaches between 98.7 and 99.5 percent across all four embodiments, essentially matching policies trained specifically for each hand. Leave-one-hand-out zero-shot transfer is more informative. A policy trained on the other three hands achieves 95.3 percent success and 7.7 consecutive reorientations on Allegro, 95.5 and 7.7 on LEAP, 85.7 and 4.4 on Shadow, and 98.1 and 8.9 on MANO. The Shadow result exposes a significant residual embodiment gap, but these policies are still performing a dynamic contact task on a hand excluded from training. “Zero-shot” here should be read precisely. It means that the target hand was held out while three other embodiments were available during training. It does not mean that a policy trained on one arbitrary hand will automatically transfer to any other hand. The appendix demonstrates why that distinction matters. One-to-many transfer from a single source is highly asymmetric. An Allegro-only policy scores just 8.7 percent on Shadow and 17.5 percent on MANO. A MANO-only policy reaches roughly 31 to 36 percent on the three robot hands. LEAP is the strongest single source, transferring at 95.3 percent to Allegro, 65.8 to Shadow, and 87.0 to MANO. The authors associate this with LEAP’s larger workspace and range of motion: a flexible source embodiment exposes the policy to behaviors that remain feasible on more constrained targets, while the reverse direction encourages overfitting to narrow capabilities. Crossing the four-finger versus five-finger boundary also degrades performance. Training on Shadow and MANO transfers at 66.2 percent to Allegro and 80.8 to LEAP. Training on Allegro and LEAP transfers at 83.2 percent to Shadow and 95.0 to MANO. These are meaningful results, but average consecutive reorientations drop much more sharply than individual success, indicating that long-horizon stability remains sensitive to morphology. The domain-randomization ablation is similarly nuanced. It substantially improves leave-one-out transfer to Shadow and MANO and gives a marginal gain on LEAP, but Allegro performance actually falls slightly. The most defensible conclusion is therefore not that randomization is uniformly beneficial, but that it particularly helps on targets whose geometry or dynamics are farther from the training distribution. One reported adaptation result needs clarification. The main text says a MANO-only policy is fine-tuned for 500 iterations on each target, compared with roughly 4,500 iterations from scratch. Yet the “zero-shot” entries in that adaptation table are identical to the three-source leave-one-out results, while the appendix reports far lower numbers for genuine MANO-only transfer. The 500-iteration improvement may still be valid, but version one of the paper leaves the initialization behind that table ambiguous. That inconsistency does not undermine the central multi-hand and leave-one-out findings, but it does mean the rapid-finetuning claim should be independently reproduced before being treated as settled. The Real-World Results Are Promising—and Modest. The physical setup tracks a tagged cube using one or two Intel RealSense cameras. The cube carries 24 AprilTags, four per face, and pose estimates are fused only when at least three tags are visible. For deployment on LEAP, the policy runs at 20 hertz because serial communication, rather than CIK, limits the control loop. On the physical LEAP Hand, the joint-control baseline averages 0.6 consecutive reorientations over ten trials. Zero-shot UHAS averages 0.9, the multi-hand policy 1.1, and the LEAP-specific UHAS policy 2.0. On Allegro, the zero-shot policy averages 0.8, while both the multi-hand and Allegro-specific policies average 2.1. Variance is high: some trials fail immediately, while one Allegro multi-hand trial reaches eight reorientations. This is credible evidence that the geometric controller survives contact with real hardware. It is not yet robust, sustained dexterity. The most accurate description is a proof-of-concept sim-to-real deployment in which UHAS generally beats direct joint control and occasionally produces impressive runs, while the mean performance remains only a few successful rotations. The engineering details explain why. The team performs system identification for the LEAP actuators, randomizes velocity limits, limits physical joint ranges to control overshoot, and uses an asymmetric actor-critic so the deployed actor depends mainly on more reliable positional signals. They also report missed serial reads and writes, gradual loosening of finger assemblies, deformation at the thumb root, and the need for custom reinforced 3D-printed parts. These details are not peripheral. They show that a unified high-level action representation does not unify actuator dynamics, sensing latency, mechanical compliance, or build quality. UHAS moves the policy above some embodiment differences, but the physical control stack still requires embodiment-specific work. Where UHAS Sits Among Cross-Embodiment Approaches. UHAS is most clearly distinguished by what it chooses to canonicalize. RobotFingerPrint and D(R,O) Grasp canonicalize hand–object relationships for grasp synthesis. RobotFingerPrint assigns dense spherical coordinates to gripper surfaces and predicts corresponding coordinates on objects. D(R,O) Grasp predicts a matrix of spatial relationships between a hand in its grasp pose and an object, then recovers kinematically valid grasps. Both are designed primarily around static grasp configurations rather than closed-loop sequential control. UHAS turns the canonical geometry itself into a time-varying action. One Hand to Rule Them All takes almost the opposite route. It defines a canonical, human-inspired URDF with a fixed 22-degree-of-freedom indexing scheme. Hands with fewer joints receive inactive slots, while an 82-parameter morphology representation describes their structure. This canonicalizes the mechanism: policies act in a standardized joint layout and can be conditioned on hand morphology. UHAS canonicalizes the desired geometric effect and lets CIK recover whatever native joint pattern produces it. That contrast is useful. Canonical joints preserve more direct control and may represent hand-specific maneuvers that cannot be reduced to sphere deformation. Canonical geometry is less tied to exact kinematic topology and may require less manual joint correspondence. Neither choice dominates universally; they encode different bets about which aspects of dexterity should remain shared. Latent Action Diffusion and XL-VLA learn the common action space rather than specifying it geometrically. Latent Action Diffusion uses retargeted, paired end-effector poses to train embodiment-specific encoders and decoders around a 16-dimensional latent. It then trains a diffusion policy in that shared space. XL-VLA learns a cross-hand latent that can be inserted into a vision-language-action architecture and decoded to several physical hand types. These learned spaces can potentially capture richer synergies than radial and lateral sphere deformation, but they require alignment data and introduce per-embodiment learned decoders. UHAS’s advantage is that its correspondence comes from the URDF and surface geometry. No paired demonstrations are needed merely to define the action space, and each action dimension remains interpretable. Its disadvantage is that the expressiveness ceiling is chosen by the designer rather than learned from data. At the broadest scale, CrossFormer and UniAct address action heterogeneity across entirely different robot categories. CrossFormer shares a transformer backbone while retaining action heads appropriate to each embodiment class, absorbing variation through large and diverse datasets. UniAct learns tokenized universal actions intended to capture atomic behavior across different physical controllers. UHAS is much narrower, but it offers a far stronger hand-specific inductive bias and demonstrates transfer with a lightweight MLP rather than a foundation-scale model. The likely long-term answer is not to choose between these levels. A generalist vision-language policy could predict UHAS trajectories for dexterous hands, Cartesian wrist actions for arms, and another structured action type for locomotion. Learned latent models could also operate on top of UHAS, using geometry as an alignment prior rather than forcing a latent space to discover all correspondences from scratch. What the Paper Gets Right. The strongest idea in UHAS is the separation of manipulation semantics from hardware realization. The policy decides how the hand’s enclosing geometry should change. CIK decides how a particular hand should realize that change. This is cleaner than asking one network to simultaneously learn cube dynamics, finger coordination, and arbitrary joint-space translation. The paper also recognizes that action unification and observation unification are inseparable. Using a 15-dimensional sphere action while feeding the policy raw Shadow or Allegro joints would leave the embodiment embedded in the input. Geometric finger points on both sides create a more coherent interface. Most importantly, the experiments expose rather than conceal the limits of transfer. Multi-source training is much stronger than arbitrary one-to-one transfer. Flexible source hands generalize better than constrained ones. Four-to-five-finger transfer is possible but weaker. Multi-hand policies can become conservative because they favor motions feasible on every embodiment. These are useful findings for dataset composition and morphology randomization, even beyond the particular sphere representation. Where the Abstraction May Break. The evaluation task is unusually compatible with the representation. A cube held near the center of an upward-facing palm is compact, convex, and naturally manipulated by repeatedly altering an enclosing finger geometry. UHAS still has to solve difficult contact dynamics, but the task aligns with the sphere prior. Less sphere-like skills may expose missing action dimensions. Examples include fingertip gaiting across a large object, manipulating a long tool, sliding an object along the palm, switching between precision and power grasps, using one finger as a hook, or exploiting articulated palm motion. An ellipsoid, superquadric, object-conditioned implicit surface, or collection of local surface charts may offer a more general control geometry. The tested hands also occupy a relatively narrow morphological family. They have four or five fingers, recognizable palm-centered layouts, and mostly independent serial finger chains. The results do not establish zero-shot control of three-finger hands, underactuated hands, tendon-coupled hands, soft hands, variable-stiffness devices, or grippers whose useful workspace does not resemble an enclosing sphere. CIK is kinematic and position-based. It does not directly represent contact force, tactile state, compliance, or torque distribution. Two hands may reach similar geometric configurations while producing very different contact mechanics. This is particularly relevant when policies transfer from a rigid simulated hand to flexible, backlash-prone physical hardware. Finally, the paper lacks an apples-to-apples comparison with alternative cross-hand representations on the same sequential task. The direct joint baseline establishes that UHAS is competitive for specialist control, but it does not tell us whether sphere deformation is better than a canonical joint layout, learned action autoencoder, fingertip operational space, or morphology-conditioned policy. A dedicated benchmark with shared tasks, held-out morphologies, and equal training data would make the contribution easier to position quantitatively. How a Robotics Team Could Use It. The public simulation repository contains the URDF-processing pipeline, Isaac Lab environments, pretrained models, evaluation scripts, and instructions for adding a new hand. It targets Isaac Sim 4.5 and Isaac Lab 2.2.1 and is released under Apache 2.0. For a new embodiment, a practical workflow would be:. Prepare a trustworthy kinematic model. Define an outward-facing palm frame, correctly oriented fingertip frames, realistic limits, and a fully open reference pose. The repository’s onboarding guide still requires visual checking and occasional cleanup of URDFs exported from other simulators. Generate and inspect the sphere mapping. Visualize surface correspondences, automatic joint classifications, lookup tables, and random deformations. A plausible-looking open hand is not enough; extreme deformations should remain kinematically coherent. Evaluate the pretrained multi-hand policy zero-shot, but expect adaptation. The leave-one-out results justify trying zero-shot control, while the real-world and one-to-one results argue against treating it as the default deployment mode. Randomize capabilities, not only physics. Vary joint ranges, actuator gains, control delays, palm orientation, and sphere-control placement. This can reduce reliance on motions unique to the source hands. Add a learned residual where geometry is insufficient. A small embodiment-conditioned residual over CIK targets could correct systematic mapping errors without forcing the main policy back into raw joint space. There are several especially promising extensions. Tactile taxels could be assigned spherical addresses, creating a shared contact observation space. A vision-language or diffusion policy could predict chunks of sphere deformations rather than joint trajectories. Arm-and-hand systems could combine a common wrist-pose action with UHAS for the fingers. The policy could also receive an explicit morphology embedding or capability mask, allowing it to exploit a target hand’s extra motion without sacrificing the shared action language. A more ambitious direction would make the control surface object-adaptive. The sphere is currently derived from the hand. Conditioning its shape on estimated object geometry could preserve the shared coordinate chart while better representing elongated, concave, or articulated objects. The key challenge would be maintaining temporal consistency as the object moves and contacts change. The Takeaway. UHAS does not solve universal dexterity. It does something more concrete: it shows that a carefully chosen geometric bottleneck can let one modest policy control multiple high-degree-of-freedom hands, transfer meaningfully to a held-out embodiment, and survive real-world deployment. Its most important contribution may not be the specific sphere. It is the proposition that cross-embodiment learning should begin by designing a common language of physical intent. For dexterous hands, that language may be the deformation of an enclosing surface: spread here, contract there, close this region, move that finger laterally. Joint commands become a hardware-specific translation layer. The simulation evidence is strong enough to make that proposition worth pursuing. The physical results and morphology gaps show how much remains unresolved. But if robot learning is going to aggregate data across an expanding ecosystem of dexterous hands, explicit geometric action interfaces like UHAS are likely to be an important part of the stack—not necessarily instead of learned latent actions, but as the structure that makes those latents easier to learn, interpret, and deploy.