Latest / Embodied AI 101 / HUG: Turning Everyday Human Grasps Into a Robot-Ready Prior

Transcript

HUG: Turning Everyday Human Grasps Into a Robot-Ready Prior · Embodied AI 101

• HUG: Turning Everyday Human Grasps Into a Robot-Ready Prior. Human Universal Grasping • Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, and Lerrel Pinto • New York University, Tsinghua University, and University of Michigan • arXiv preprint • 2026. The central bet: use the hands that already work. One of the deepest asymmetries in robot learning is that humans generate enormous amounts of physically grounded interaction data, while robots generate very little. We open drawers, pick up bottles, move dishes, carry bags, and handle awkward objects continuously. Yet dexterous robot learning still tends to build its grasping datasets through simulation, analytical optimization, or carefully instrumented teleoperation. Human Universal Grasping, or HUG, asks whether we can reverse that relationship. Instead of making humans operate robot hands, can we model the grasps humans naturally perform, place those grasps in a canonical hand representation, and translate the result to whatever robot hand is available at deployment?. The paper’s answer is a three-part system. First, it collects a large egocentric dataset called 1M-HUGs, containing roughly one million image–grasp pairs derived from 6,707 human grasp recordings across 41 buildings. Second, it trains a flow-matching model that takes an RGB-D observation and a user-selected object point and generates a human grasp represented with the MANO hand model. Third, it retargets that predicted human grasp to different anthropomorphic robot hands without retraining the grasp generator for each embodiment. The authors evaluate the resulting system on two robot hands, multiple stereo cameras, and both controlled tabletop and uncontrolled household settings. The headline result is a 66.7% success rate on a difficult 30-object tabletop benchmark, versus 43.7% for Dex1B and 32.7% for a Contact-Anchored Policy baseline. HUG also reaches 62.0% in an unseen home using a different camera, arm, and hand. Those numbers are notable, but the more important contribution is conceptual: HUG treats a canonical human hand as an interface between scalable human data and heterogeneous robot hardware. This is not yet universal manipulation. It is a static grasp generator followed by a minimal open-loop execution stack. But it offers a compelling answer to one of dexterous manipulation’s central questions: what representation lets human interaction data scale independently of the robot that will eventually use it?. Why dexterous grasp data is still a bottleneck. Parallel-jaw grasping benefited early from a convenient factorization. A grasp could often be represented as a rigid gripper pose, labels could be generated analytically or through robot trial and error, and execution remained relatively low-dimensional. Multi-fingered hands break much of that simplicity. Their action spaces combine wrist placement, orientation, finger articulation, approach trajectory, contact sequence, force, and embodiment-specific kinematics. Simulation has consequently become the dominant scaling strategy. DexGraspNet, for example, generated 1.32 million simulator-validated Shadow Hand grasps over more than 5,000 objects using differentiable force-closure optimization. Dex1B pushed this approach dramatically further, using generative models and simulation validation to construct one billion demonstrations for grasping and articulation. Such datasets can be generated much faster than physical robot experience, but their geometry, contacts, dynamics, and hand morphology are all products of a synthetic pipeline. Robot teleoperation provides the opposite trade-off. It produces grounded trajectories on the target hardware, but collecting broad coverage for every hand is expensive. Even when human motion is retargeted, operators must usually work within the robot’s sensing and control setup. The collected experience remains coupled to a particular morphology, workspace, and task distribution. Human video looks like an attractive escape route, but conventional internet or egocentric video generally lacks metric depth, calibrated camera motion, and reliable articulated hand pose. Web2Grasp reconstructs functional hand–object interactions from web images and then uses simulation to improve physical feasibility. That approach extracts useful semantic and functional information, but it must reason through substantial ambiguity in object scale, occluded hand pose, and single-image 3D reconstruction. HUG’s claim is that recent smart glasses change the data economics. A headset such as Project Aria Gen 2 provides synchronized cameras, metric stereo depth, camera poses, and tracked hand landmarks. The wearer can interact normally rather than drive a robot. The resulting grasp can then be expressed in a standardized human-hand space and reused across embodiments. There is also a subtler prior embedded in this choice. Analytical grasp synthesis asks for any physically valid grasp. HUG asks for grasps that people actually choose. Human grasps reflect far more than force closure: object semantics, likely mass distribution, handles, fragile regions, geometric scale, and the practical limits of approaching an object from a human viewpoint. HUG does not explicitly annotate all those properties, but they are partially encoded in the empirical distribution of human behavior. 1M-HUGs: one grasp, hundreds of viewpoints. The dataset collection protocol is clever because it separates the final grasp from the observations used to predict it. For each recording, a wearer stands in front of a stationary object and moves their head around it for approximately 15 to 30 seconds while keeping their hands out of view. They then reach in with the right hand and hold a stable grasp without lifting the object. Aria’s estimated camera trajectory is used to transform that final grasp backward into each preceding camera frame. One physical grasp therefore becomes hundreds of training examples pairing an object-only RGB-D observation with the same grasp expressed in that frame’s coordinate system. This is the dataset’s main scaling mechanism. HUG is not based on one million independently performed grasps. It is based on 6,707 physical grasp recordings, expanded across viewpoints into roughly one million RGB frames and an additional set of synchronized grayscale frames. The paper reports 27.8 hours of recordings, approximately 1,500 distinct object identities, and data collected in 41 buildings covering environments such as kitchens and bedrooms. After processing, the RGB and stereo-left grayscale streams together provide roughly two million model entries. That distinction matters. The behavioral diversity is on the scale of thousands of grasps, while the perceptual diversity is on the scale of millions of image–grasp transformations. HUG’s scaling experiment therefore combines several effects: more physical grasps, more objects and buildings, more viewpoints of each object, and more variation in illumination and camera geometry. The paper convincingly demonstrates that more frames improve performance, but it does not isolate how much comes from each source of diversity. The annotation pipeline begins automatically. A vision-language model compares pre-grasp and post-grasp frames to identify the object being held and propose points on its body and appendages. SAM3 then propagates an object mask throughout the recording. Heuristics identify a stable terminal grasp using hand motion, hand–mask proximity, tracking quality, and visibility. Crucially, every recording is subsequently reviewed in a web application where an annotator can repair the mask, choose a different grasp frame, approve the example, or reject it. The final filtering is fairly strict. Frames are removed when depth confidence is poor, too few hand landmarks would project into view, the propagated grasp does not overlap the object, or a hand is visible in the input image. The inputs are cropped and resized to 224 by 224 pixels. The grasp is always represented in the observation camera’s coordinate frame, so the network learns camera-relative placement rather than an object-centric pose that would require explicit reconstruction or pose estimation. From 21 landmarks to a canonical articulated hand. Aria provides 21 hand landmarks, but HUG needs a complete articulated representation suitable for generation, simulation, and retargeting. The authors fit the MANO hand model to each tracked skeleton. They optimize 15 finger-joint rotations while keeping the metric wrist transform supplied by Aria. Fingertips receive extra weight because small proximal angular errors can produce large contact errors at the distal links. An anatomical prior penalizes implausible joint configurations such as hyperextension, and the reported average fingertip fitting error is below two millimeters. MANO separates hand shape from articulation. HUG discards most of the shape variation by fixing every example to one canonical right-hand shape—specifically, a shape based on the first author’s hand—and recomputing the dataset labels while preserving their joint angles. This normalization makes a given pose vector mean approximately the same thing across collectors and allows one simulated hand to be used for evaluation. It also removes a nuisance variable that the model might otherwise entangle with object scale. The cost is that HUG does not model handedness, individual finger proportions, or the relationship between hand size and grasp strategy. All training grasps are right-handed, and the canonical hand is at least as large as the hands used for the benchmark. That becomes important when the prediction is retargeted to a substantially smaller robot hand. The model: RGB semantics, metric geometry, and generative hand pose. At inference time, HUG receives three things: an image, a registered metric depth map, and a pixel selected on the target object. The pixel click resolves which object should be grasped in a potentially cluttered scene. It also means HUG is not an end-to-end language-grounded grasping system. Object selection must come from a human click or a separate detector, segmenter, or task planner. The depth image is back-projected into a point cloud, and the selected pixel becomes a 3D query point. HUG predicts a 99-dimensional grasp: three values for wrist translation, six for a continuous wrist-rotation representation, and 90 values representing six-dimensional rotations for the 15 MANO finger joints. Holding the MANO shape fixed means the model only needs to generate placement and articulation. The perception stack has two complementary branches. A frozen DINOv2-Base vision transformer converts the image into 256 semantic patch tokens. Around the 3D query point, HUG crops the point cloud to a 30-centimeter-radius ball, samples 4,096 points, and encodes them with a trainable PointNeXt U-Net. The crop is important: spreading a fixed number of point tokens over the full camera frustum would leave too little local resolution around the chosen object. The two branches are fused through point painting. Each point-cloud region is projected back into the image, the corresponding DINO feature is sampled, and that feature is concatenated with the geometric point token. Metric centroid positions and the query point are represented with Fourier features before a four-layer transformer builds the scene-conditioning tokens. Camera intrinsics participate only in back-projection and projection, not as learned features. In principle, this makes the perception model agnostic to the intrinsics of the stereo camera, provided that the depth and calibration are reliable. The paper trains on Aria but deploys directly with ZED and RealSense cameras. Why flow matching?. A single object usually supports multiple reasonable grasps. A bottle can be held around its body, near its neck, or sometimes by an appendage. Regressing all demonstrations toward one average configuration risks producing a hand that corresponds to none of them. HUG therefore treats grasp prediction as conditional generation. During flow-matching training, clean grasp states are progressively mixed with noise, and the network learns a velocity field that transports noisy states toward the distribution of human grasps compatible with the observation. At inference, HUG starts from a random state and follows the learned flow for 50 Euler integration steps to obtain one grasp sample. The grasp state is divided into three tokens—translation, wrist rotation, and finger articulation—before entering six diffusion-transformer-style blocks. This prevents the 90-dimensional finger representation from numerically overwhelming the much smaller translation and wrist components, and it gives the network a cleaner way to model interactions among placement, orientation, and preshape. The training objective combines the standard velocity-prediction loss with direct geometric supervision. The predicted clean state is passed through MANO, and the resulting 3D landmarks are compared with the demonstrated hand in camera coordinates. This auxiliary landmark loss is weighted heavily, especially near the clean end of the flow path where decoded geometry is meaningful. That geometric term turns out to be essential. Removing it reduces test simulation success from 73.0% to 32.7% and increases fingertip contact error from 14.6 to 35.7 millimeters. In other words, predicting a plausible vector in MANO parameter space is not sufficient. The model needs explicit supervision on where the fingertips actually land in metric 3D space. The complete network has about 207 million parameters, of which approximately 121 million are trainable because DINOv2 and MANO are frozen. Training runs for 100,000 steps with an effective batch size of 128. The authors report roughly ten hours on two RTX 5090 GPUs, including periodic MuJoCo validation. MANO as an embodiment interface. HUG’s most consequential architectural choice may be that it predicts neither an Ability Hand configuration nor a WUJI configuration. It predicts a human hand. For tabletop experiments, the MANO grasp is retargeted to a six-degree-of-freedom Ability Hand using AnyTeleop. For the in-the-wild experiments, it is mapped to a 20-degree-of-freedom WUJI hand using WUJI’s retargeting system. The authors add a fixed alignment correction for the smaller Ability Hand but none for WUJI, whose scale is closer to the canonical MANO hand. Each robot starts from a hand-specific pre-grasp offset, interpolates toward the target, force-closes its fingers, and lifts. This is where the phrase zero robot data needs careful interpretation. The HUG neural network is trained without robot trajectories, robot grasp labels, or per-hand fine-tuning. But deployment is not free of embodiment engineering. It still requires a kinematic retargeter, coordinate alignment, a pre-grasp definition, joint-level closing behavior, and an arm trajectory. If the target hand cannot realize a useful analogue of the MANO pose, retargeting can fail. So the scientific claim is stronger and more precise when stated this way: HUG decouples learned grasp perception from embodiment-specific training, while retaining a small hand-specific retargeting and execution layer. That is still a meaningful shift. Approaches such as AnyDexGrasp also factor grasp perception from embodiment, but they train a hand-specific grasp decision model through hundreds or thousands of real-world attempts. AnyDexGrasp reports strong performance across three hands, including cluttered scenes, but some real robot data remains necessary for each new morphology. HUG instead wagers that anthropomorphic similarity plus retargeting is sufficient for useful zero-shot deployment. HUG-Bench: making real objects simulatable. The paper also introduces HUG-Bench, a set of 90 objects organized across five geometric categories—cylindrical, spheroidal, prismatic, appendaged, and amorphous—and three size bins. Every geometry–size combination contains six objects: four for simulation validation and two for testing. The resulting test split contains 30 objects, all unseen during training. It includes conventional items such as bottles and balls, but also thin, articulated, irregular, and oversized objects such as nail clippers, headphones, a picnic basket, an easel, and a storage bin. Each real object is reconstructed as a metric-scale simulation asset. The authors capture five Aria views, use a modified Multi-view SAM3D pipeline, align the result against stereo depth and the headset’s semidense SLAM reconstruction, manually verify scale and pose, make the mesh watertight, and generate convex collision components. The pipeline exports both MuJoCo and PyBullet-compatible assets. Simulation uses a physics-ready MANO hand executing an open-loop pre-grasp, closure, and vertical lift. Success means the object remains off the support surface at the end. The authors also report fingertip contact error, measuring whether the thumb and at least one supporting fingertip approach the object surface. A useful diagnostic displaces recorded human grasps before execution. Moving a grasp by only two centimeters along each coordinate axis—about 3.5 centimeters in Euclidean distance—roughly halves success for medium and large objects and reduces small-object success below 20%. This confirms that the benchmark rewards metric placement rather than merely moving a plausible-looking hand somewhere near the target. What the experiments say. In simulation, full HUG reaches 71.5% success on the validation objects and 73.0% on the test objects. Replaying the captured human grasps provides an imperfect oracle of 90.3% and 94.0%, respectively. The oracle falls short of 100% because of hand-tracking errors, imperfect reconstructed assets, rigid approximations of deformable objects, and the fact that the simulator executes a recorded closed-loop human grasp as a fixed open-loop robot trajectory. The modality ablations are especially informative. RGB alone performs poorly, reaching only 29.7% test success with more than ten centimeters of fingertip contact error. Point clouds alone are much stronger at 70.7%, showing that metric geometry carries most of the basic grasping signal. But the full RGB-plus-point-cloud system reduces contact error to 14.6 millimeters and avoids semantic mistakes such as grasping the leafy crown of a pineapple or the bristles instead of the handle of a brush. Geometry gets the hand to an object; RGB helps decide which structure is useful to hold. Removing either the local point-cloud crop or point painting reduces test success to about 58%. The crop improves local spatial resolution, while point painting directly associates visual semantics with metric regions rather than asking a later transformer to discover all image–point correspondences from separate token sets. The data-scaling result is perhaps the paper’s strongest evidence for its larger thesis. As the RGB training subset grows from 25,000 frames to one million, test success rises from 33% to 73%, while fingertip contact error falls from 54.2 to 14.6 millimeters. Neither curve appears saturated. Because the subsets expand at the video level, the result suggests that additional human recordings, environments, objects, and viewpoints continue to transfer to unseen objects rather than merely fitting the benchmark. It does not, however, tell us whether collecting more distinct grasps would be more valuable than capturing more views per grasp. Real robots. The tabletop setup uses a ZED stereo camera, an xArm, and the Ability Hand. HUG and Dex1B each receive one observation, produce one grasp, and execute it open-loop without motion planning. CAP uses its published parallel-jaw setup. Across 300 trials per method, HUG succeeds on 66.7%, Dex1B on 43.7%, and CAP on 32.7%. HUG succeeds at least once on 28 of the 30 objects. The checkpoint was selected only through simulation validation and was deployed on the test objects without per-object tuning or retries. The largest gains appear on objects where morphology and contact distribution matter. HUG grasps the storage bin in all ten trials, versus zero for both baselines. It reaches nine of ten on the picnic basket, compared with one for Dex1B and zero for CAP. It also performs well on spray bottles, an easel, a rubber duck, and grapes. A multi-fingered enveloping grasp can cage irregular bodies and distribute load in ways that a single antipodal contact pair cannot. But this comparison should not be read as a clean model leaderboard. CAP uses a parallel-jaw gripper, so the result measures embodiment as well as grasp intelligence. HUG-Bench deliberately includes objects where dexterity ought to matter. Dex1B uses the same Ability Hand as HUG, making it the more informative learned dexterous baseline, although its open-loop trajectories occasionally collide severely with the table and are counted as failures. For the in-the-wild evaluation, the system changes nearly everything: Aria replaces the ZED camera, a mobile YOR platform with an AgileX arm replaces the xArm, the WUJI hand replaces Ability, and an unseen home replaces the lab. The authors execute all 300 trials in one morning, placing objects across bedrooms, living areas, an office, and a kitchen. HUG reaches 62.0%, only 4.7 percentage points below its tabletop result, and succeeds at least once on 29 of 30 objects. That small drop is persuasive evidence that the perception representation is not tightly coupled to one camera or scene configuration. It also suggests that the canonical human-hand prediction survives substantial embodiment changes, provided a workable retargeter exists. Where HUG fits among learning-from-human approaches. HUG belongs to a broader movement that treats human interaction as scalable robot supervision, but different projects transfer different pieces of human behavior. EgoZero also learns from Project Aria data with no robot demonstrations. It converts demonstrations into morphology-agnostic 3D point actions and trains closed-loop policies for specific manipulation tasks. With roughly 20 minutes of human data per task, it reports 70% success across seven tasks on a Franka with a gripper. EgoZero transfers full temporal behavior but remains task-specific; HUG learns a much broader object-level grasp prior but outputs only a terminal grasp. DexWild collects 9,290 dexterous demonstrations across 93 environments using a portable human capture system. It aligns human and robot observations and actions, then co-trains policies on human and robot demonstrations. This produces richer task policies and cross-embodiment transfer, but unlike HUG it still uses robot data to ground most of its learned behaviors. Web2Grasp extracts functional grasps from web images, reconstructs 3D hand–object interactions, retargets them to a robot hand, and uses simulation to enlarge and clean the dataset. Its strength is functional information: whether a pen is held for writing or a spray bottle near its trigger. HUG has better metric grounding and substantially broader real-world visual coverage, but it has no task or function input. It learns a distribution of generic natural grasps rather than a commanded grasp such as “hold this mug for pouring.”. Seen together, these systems suggest that human data is not one monolithic resource. It can supervise contact priors, terminal grasps, trajectories, object-centric actions, or entire policies. HUG’s specific contribution is to show that terminal human hand geometry alone can be a powerful, scalable prior for zero-shot dexterous grasp generation. Reading “universal” and “zero-shot” carefully. HUG demonstrates several meaningful kinds of zero-shot transfer at once: unseen objects, unseen rooms, new camera intrinsics, and a new robot hand. The model receives no robot-specific fine-tuning and no trials on the HUG-Bench test objects. But “universal” should not yet be read literally. The model covers right-handed, single-hand, indoor grasps of stationary objects. It does not model left hands, bimanual coordination, object motion during acquisition, task-specific functional intent, force, tactile response, or a temporally extended approach. Its 224-pixel input reduces accuracy on very small objects, while large and distant objects are underrepresented in egocentric training data. The fixed MANO shape also prevents the predictor from adapting its strategy to the actual robot hand’s span. The system’s dominant failures come from control rather than grasp imagination. During tabletop trials, 42 attempts hit the object and eight hit the surface before the hand settled; in the home, the corresponding counts were 57 and eight. Additional objects slipped or were dropped during the lift. The model predicts a static pose with no contact-force target, and the executor interpolates toward it without collision-aware motion planning or visual correction. The football and wipe dispenser expose a different limit: no prediction can overcome a hand that is physically too small to wrap the object. Retargeting a canonical human configuration does not make morphology irrelevant. It simply moves morphology from the learned perception model into the feasibility of the retargeting layer. Finally, the paper emphasizes diverse generation, but the robot experiments execute only one sampled grasp per trial. There is no learned grasp critic, simulation-based ranking stage, or quantitative diversity evaluation. The generative formulation creates an important capability that the deployed system barely exploits. How a robotics team could use HUG. The most natural use of HUG is not as a complete grasping stack, but as a proposal distribution inside one. A production-oriented system could sample several MANO grasps for a selected object, retarget each candidate to the available hand, reject joint-limit and self-collision failures, and score the survivors with a hand-aware success model. A motion planner could then optimize the approach trajectory around the table and neighboring objects. During closure, tactile sensing or motor current could regulate force, while RGB-D feedback could update the wrist pose if the object shifts. This would preserve HUG’s best property—the scalable human prior—while adding the embodiment and control information that human video alone cannot supply. HUG could also serve as a low-level utility beneath a language-conditioned system. A VLM or open-vocabulary segmenter would translate “pick up the saucepan by its handle” into a target mask or point. HUG would propose a contact-rich hand configuration, and an embodiment-specific controller would execute it. Adding task conditioning during training would be the next step toward distinguishing a generic pickup from a functional pre-manipulation grasp. For researchers collecting their own data, the released pipeline may be as valuable as the model. It supports Aria Gen 1 and Gen 2 processing, metric stereo depth, MANO fitting, and live Gen 2 streaming. The accompanying pipeline turns short egocentric scans into metric, simulation-ready object assets. These tools make the paper’s human-data and real-to-sim workflows easier to extend beyond the original benchmark. As of August 1, 2026, the public release includes pretrained weights, inference and visualization code, the 1M-HUGs data, HUG-Bench assets, , and. The dataset card lists a 91.8-gigabyte CC BY 4.0 release. The repository does not yet include HUG training code, the HUG-Bench simulation evaluator, the full dataset-construction interface, or live RealSense and ZED streaming support. Reproducing inference is therefore possible, but reproducing the complete training and evaluation pipeline is not yet turnkey. The larger lesson. The most important part of HUG is not the particular transformer configuration or even flow matching. It is the factorization:. collect interaction where data is abundant;. express it in a reusable canonical morphology;. learn perception and grasp generation in that space;. postpone robot-specific constraints until deployment. That factorization is strongest when the source and target morphologies share structure. MANO is not a truly embodiment-free representation; it is an anthropomorphic one. But modern humanoid hands are now close enough to human geometry that this bias becomes useful rather than purely limiting. HUG also demonstrates an unusually productive use of egocentric camera motion. The wearer’s head movement is not just passive video. It becomes a label multiplier: one grasp defines supervision across hundreds of calibrated viewpoints. That idea should generalize beyond grasping wherever an interaction endpoint can be propagated through a known camera trajectory. The next version of this paradigm almost writes itself: left and right hands, variable hand shape, bimanual grasps, temporal approach trajectories, explicit functional intent, multiple sampled proposals, embodiment-aware feasibility scoring, and closed-loop force and vision control. The paper’s scaling curve suggests that the human-data side may still have substantial headroom. HUG does not solve universal dexterous manipulation. What it does show is that the robot may not need to experience every object through its own body before it can grasp it. If human interaction can be captured metrically, represented canonically, and retargeted reliably, then human hands become not just inspiration for robot design, but a practical data engine for robot learning.