Latest / Embodied AI 101 / DO AS I DO: Making Human Video Executable

Transcript

DO AS I DO: Making Human Video Executable · Embodied AI 101

• DO AS I DO: Making Human Video Executable. Do as I Do: Dexterous Manipulation Data from Everyday Human Videos • Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad “Mahi” Shafiullah, and Jitendra Malik • UC Berkeley • arXiv • 2026. Attribution clarification: Chris Paxton is not a paper author. The matching primary paper was submitted on June 17, 2026. The missing information between watching and doing. Imagine downloading a video of someone manipulating a kitchen tool. You can see the wrist trajectory, some finger motion, and the object moving through the scene. Now imagine converting that clip into commands for a dexterous robot. The tempting interpretation is that this is mostly a pose-estimation problem: reconstruct the human hand, solve inverse kinematics for the robot, and replay. But that leaves the essential manipulation question unanswered. What makes the object follow the hand? Matching fingertip positions does not establish the forces, contact sequence, or stability needed to reproduce the interaction. Physics-based retargeting methods such as SPIDER were developed precisely to address that gap between a kinematic demonstration and an executable robot trajectory. DO AS I DO tackles the additional difficulty that the demonstration itself is uncertain. Its starting point is ordinary monocular RGB video, not a clean motion-capture trajectory. Its output is a reconstructed interaction that has been translated into robot motion through simulation-based optimization. The project combines a video reconstruction system, a dynamics-aware retargeter, and a deployment path onto robot hardware. My reading is that the important contribution is not simply removing specialized capture hardware. It is deciding where to trust the video and where to let robot physics change the answer. The video supplies evidence about the intended interaction. Reconstruction turns that evidence into a reference. Optimization then searches for a way the robot can approximately realize that reference. That distinction matters for the headline claim. A system that generates useful robot trajectories from video is not automatically a system that learns a general manipulation policy from video. The released software is organized around reconstructing, optimizing, and replaying demonstrations—not around training a new vision-language-action model. Think of it as infrastructure for constructing robot data, rather than a replacement for the policy-learning stack. A modular pipeline with inspectable interfaces. The first stage reconstructs the human hand and manipulated object over time. The second maps the resulting interaction onto a robot hand and optimizes it in a physics simulator. Deployment adds another transformation: the optimized hand motion must become a feasible trajectory for the complete arm-and-hand system. These stages are separate in the official repository, with explicit intermediate artifacts. The perception stack combines several pretrained systems. SAM 3 provides segmentation. SAM 3D reconstructs object geometry. MoGe supplies pointmaps and camera information. HaWoR reconstructs hand motion. BootsTAPIR provides image-space motion estimates, and GeoCalib estimates gravity direction. The new object-tracking procedure connects these components rather than replacing them with one end-to-end network. That modularity is useful scientifically. If the robot misses the tool, there are several distinct hypotheses: the tool mesh is wrong, its depth is wrong, the fingers were reconstructed incorrectly, the kinematic mapping is poor, or the optimizer found a fragile interaction. Saving the intermediate representations makes those hypotheses distinguishable. The retargeting output also deserves a precise interpretation. The repository produces a simulation scene, a kinematic reference, an optimized trajectory, tracking metrics, and the resolved configuration. Those are substantial ingredients for robot learning, but they are not automatically a complete visuomotor training dataset. You still need to decide how observations will be paired with actions and what distribution the eventual policy should handle. For an engineering team, I would treat the interfaces between stages as part of the method. A polished final robot video tells you that something worked. An inspectable hand estimate, object track, and physics rollout tell you what to improve when the next clip does not. Making a single-image generator remember. The reconstruction contribution starts with an interesting choice of foundation model. SAM 3D is a generative system for recovering object geometry, appearance, and layout from a single image. Its training targets natural scenes where clutter and occlusion are common, making it a plausible source of priors for objects partly hidden by a manipulating hand. But single-image reconstruction is not the same problem as tracking. Suppose a mug is partially visible in one frame and mostly covered in the next. Independently reconstructing each frame permits the system to explain those images with different mug geometries. For manipulation, we want a different assumption: one rigid object persists, while its pose changes. DO AS I DO implements that assumption during generation. It chooses an anchor shape and guides subsequent inference toward that shape, while also biasing pose toward the preceding estimate. Although the paper calls this guided diffusion, the implementation applies the guidance inside a flow-based sampling process. Each integration step blends the generated shape and pose components toward reference states at the corresponding noise level. The useful intuition is that this is stronger than smoothing a completed pose sequence afterward. The temporal and shape constraints influence how the model resolves the current image. Instead of first accepting an unconstrained explanation and then repairing its trajectory, the system encourages a consistent explanation while it is being generated. The pose prior cannot be equally strong everywhere. A strong preference for the preceding pose helps suppress jitter when an object barely moves, but can resist genuine rapid rotation. The tracking code therefore adjusts pose-guidance strength using an image-space rotational-motion estimate, reducing the pull toward the previous pose when the observed rotation is larger. That estimate comes from BootsTAPIR point tracks. The implementation samples points in the object region, follows them between consecutive frames, and fits a two-dimensional rigid transform using singular-value decomposition. This gives a lightweight motion cue without requiring the complete three-dimensional tracking problem to have already been solved. There is a sensible division of labor here. The generative model supplies object-level visual priors; the point tracker supplies local evidence about change. I would not interpret the latter as a complete estimate of three-dimensional angular velocity. Its role is to regulate how conservative the generative tracker should be. Finally, the system generates multiple pose candidates. It groups them using translation and rotation distances, then uses cluster support and rendered-mask agreement to select an estimate. The implementation includes both clustering and likelihood-based selection paths. My caution would be that agreement is not certainty. Several samples can share the same mistaken interpretation of a symmetric or occluded object. Candidate consensus is useful evidence for selection, but I would preserve disagreement statistics rather than discard them; they could become valuable signals for downstream filtering. Aligning two plausible reconstructions. Even a good hand reconstruction and a good object reconstruction need not occupy compatible coordinate systems. DO AS I DO uses the reconstructed hand as its near-metric reference. Its alignment code compares the hand’s depth in the hand reconstruction with its depth in the MoGe pointmap, derives a scale conversion, and applies that conversion to the pointmap’s hand-to-object offset. It then solves a one-parameter alignment problem that adjusts object translation toward the resulting target position. The important distinction is between using an estimate as the reference and having ground truth. The algorithm needs an anchor, and the hand provides one. That does not make the hand’s scale or depth exact. In practical terms, I would expect errors in that anchor to propagate into object placement rather than disappear during alignment. This is why a convincing overlay is not a sufficient contact test. Consider two points on nearly the same viewing ray: they can project close together while remaining separated in depth. For a grasping system, the difference between touching and almost touching can determine whether the object moves at all. The alignment procedure addresses this geometrically, but its inputs remain estimated quantities. Gravity alignment solves another part of the coordinate problem. The released script runs GeoCalib across frames and aggregates its gravity estimates, including confidence weighting and outlier rejection. GeoCalib itself estimates camera calibration and gravity direction from image evidence. Knowing which direction is down still does not locate the demonstration inside a robot workspace. Translation and heading remain to be resolved. The deployment interface explicitly exposes workspace placement controls and recommends changing pitch and roll only when the gravity estimate is significantly wrong. That is a useful reminder that reconstructing a plausible interaction and registering it to a physical cell are different operations. Letting physics rewrite the reference. The retargeter begins with a geometric mapping. Using Mink, the implementation constructs wrist and fingertip targets for the robot hand, with fingertip positions receiving higher priority than wrist position and orientation. This produces an initial robot trajectory around which the dynamics-aware search can operate. That first trajectory is a proposal, not the answer. The physics optimizer perturbs candidate control sequences, rolls them out, scores them, and updates the nominal controls toward better candidates. The implementation uses an annealed sampling procedure: exploration changes over optimization, and high-scoring samples receive greater weight in the update. MuJoCo Warp supplies the parallel simulation substrate. This is where the method can depart from literal human-motion copying. If matching the reconstructed fingertips exactly produces a poor grasp, the optimizer can accept a different robot configuration that better maintains the intended object motion. The desirable correspondence is therefore functional: a robot interaction that realizes the reference, not a robot silhouette that perfectly resembles the human hand. The dataset-specific configuration uses a three-second planning horizon, a half-second advancement interval, a five-millisecond simulation timestep, 1,024 sampled controls, and up to 32 optimization iterations. Object tracking receives substantial weight, alongside hand tracking, terminal reward, and penetration penalties. Those settings describe a significant offline search. At the full configured iteration budget, multiplying samples, iterations, and horizon steps gives about 19.7 million simulated steps per planning update, before additional perturbation replicas. That is my arithmetic from the configuration, not a reported wall-clock benchmark. Parallel simulation makes such workloads practical to investigate, but the planning interval should not be mistaken for a guaranteed real-time deadline. For adoption, I would keep three clocks separate: simulation integration, advancement through the reference, and hardware command streaming. Confusing them can turn an offline data-generation result into an unsupported claim about online robot control. Three repairs for noisy demonstrations. Give the optimizer a recoverable beginning. The first repair is warmup. Before following the demonstrated motion, the system adds an initialization interval during which the object is held in place while the hand can adjust. The released implementation uses weld constraints during this period and then releases them for normal simulation. It also includes open-hand initialization and reference interpolation to help the fingers approach the starting grasp. Consider a video that begins with a tool already held in the hand. A small reconstruction error might place the robot’s fingers outside contact. If ordinary dynamics start immediately, the tool falls before the optimizer has found a useful interaction. Increasing the fidelity of the later trajectory does not repair that initial failure. Warmup changes that optimization problem. It gives the solver time to establish a viable starting configuration before asking it to reproduce motion. My interpretation is that this separates two questions that are often accidentally coupled: how to initialize the interaction, and how to continue it. But there is a deployment consequence. Holding an object fixed is an optimization device. A complete autonomous system must still establish the corresponding physical initial condition; warmup is not itself an autonomous grasp-acquisition policy. Reject interactions that work only without disturbance. The second repair introduces random external disturbances during simulated rollouts. The implementation supports stochastic perturbations with persistence across timesteps, rather than only isolated instantaneous kicks. The intended effect is intuitive. Imagine two candidate motions with similar nominal tracking error. One encloses a handle; the other balances it precariously against a fingertip. Disturbance testing provides a way to prefer the interaction with more stability margin, even if the video does not reveal the original contact forces. I would describe this as robust trajectory optimization, not learned reflex control. Searching for an action sequence that survives modeled disturbances is different from learning a feedback policy that detects and corrects unexpected slip. The deployment path streams the resulting trajectory; it does not turn the disturbance-trained search into a new visual controller. That distinction is particularly important when deciding how to reuse the output. A robust optimized trajectory can be a better demonstration, a better initialization, or a better reference for later learning. It is not automatically a substitute for feedback. Reward the contact state, not just nearby geometry. The third repair concerns transitions between holding an object and resting it on a support surface. The simulator adds penalties when the actual contact state disagrees with the reference stage: missing hand–object contact during a held phase, or missing object–support contact during a resting phase. The implementation explicitly separates these contact-state penalties from continuous tracking terms. This addresses a weakness of smooth geometric objectives. Being close to the demonstrated pickup motion is not equivalent to picking the object up. Contact introduces a discrete change in how the object is supported, and a small positional discrepancy can leave the system on the wrong side of that change. The labels remain proxies derived from reconstructed interaction geometry, so I would not treat them as tactile ground truth. Nevertheless, they encode something useful that a distance loss alone does not: whether the robot is supposed to be physically responsible for the object at that moment. Taken together, these repairs target initialization, stability, and contact mode. That is a more informative description of the contribution than simply saying the retargeter has a better reward. Reading the evidence at the right level. Reconstruction is evaluated on 160 DexYCB and 12 HOI4D clips, using ground-truth hands to isolate object performance. Against FoundationPose, DexYCB F-10 improves from 0.89 to 0.93; HOI4D F-10 ties at 0.91. Across 150 in-the-wild clips, human judgments favor the new tracker 67% of the time. These are useful but different kinds of evidence. The controlled benchmark asks about object reconstruction with the hand uncertainty removed. The preference study asks which reconstruction better follows the visible motion. Neither question, by itself, establishes the complete system’s contact accuracy on arbitrary footage. For my own evaluation, I would supplement both with interaction-centered checks: separation at expected contact points, temporal contact continuity, and sensitivity to estimated object scale. Those checks would target the interface where a visually plausible reconstruction becomes a physically difficult reference. Retargeting uses 655 reconstructed references and 1,352 OakInk2 trajectories. Success requires mean object-position error below 0.1 meters and mean rotation error below 0.5 radians. The reported cumulative ablation is:. | Retargeting configuration | Reconstructed references | OakInk2 | |---|---:|---:| | Annealed-sampling baseline | 25% | 72% | | Add warmup | 66% | 77% | | Add perturbations | 67% | 79% | | Add transition reward | 71% | 81% |. Reported simulation retargeting success rates. The standout is warmup: it supplies 41 of the total 46 percentage-point improvement on reconstructed references. My reading is that this is strong evidence for treating initialization as a first-class problem, rather than assuming the first estimated frame is a usable simulator state. The table is cumulative, however. It does not measure each component independently under every combination. I would not infer that perturbations are generally unimportant merely because their incremental change after warmup is small. The success definition also needs to stay attached to the number. Half a radian is about 29 degrees, and ten centimeters is not a precision-manipulation tolerance. The metric implementation measures object trajectory agreement; it does not directly measure task semantics such as whether a peg was inserted or a tool achieved its intended effect. There is another layer of interpretation: on reconstructed data, the reference itself is estimated. Low error means the robot reproduced that estimate. To claim successful recovery of the original human interaction, I would want to evaluate reconstruction fidelity and retargeting fidelity separately, rather than allowing one success rate to stand in for both. What the hardware demonstrations establish. The project showcases ten real-world motions with varied object geometries and grasps. Its deployment software maps trajectories onto UR3e arms with Sharpa Wave hands; the Wave platform has 22 active degrees of freedom per hand. The deployment process is not just loading the hand trajectory and pressing play. A simulation-based pass places the interaction in the workspace, solves arm inverse kinematics, and allows inspection and tuning of collision avoidance before producing arm-and-finger joint trajectories. The hardware streamer sends commands at 50 Hz. Its speed setting resamples the trajectory in time rather than changing that streaming rate. This is task-level replay of an optimized motion, not evidence of a learned visuomotor policy adapting online to arbitrary scene changes. The paper’s hardware rollouts run at roughly half speed. For me, these demonstrations establish an important existence result: the reconstruction and retargeting errors are not inevitably fatal when the resulting motion reaches hardware. But I would not use a collection of demonstrations to estimate deployment reliability. That would require repeated trials, controlled initial-condition variation, and explicit intervention accounting. The paper also limits reconstruction to hands and rigid objects, rather than full scenes. That boundary matters for interpreting tool-use examples. A tool trajectory can resemble whisking without the representation fully describing the bowl, its contents, or all environmental contacts. My practical question would therefore be: which parts of the task are encoded in the tracked object motion, and which depend on scene structure that must be supplied elsewhere?. That is not an objection to trajectory generation. It is a guide to where additional modeling will be needed. Where it sits among related approaches. SPIDER is the closest methodological parent. Chaoyi Pan and colleagues use physics-based sampling to convert kinematic human demonstrations into dynamically feasible robot trajectories, including curriculum-style virtual contact guidance. The DO AS I DO retargeting repository explicitly builds on SPIDER. Its distinctive emphasis is making that general approach useful with the uncertainty produced by monocular reconstruction. DexImit offers a different decomposition of video-to-robot transfer. It reconstructs interactions, identifies and schedules subtasks, synthesizes grasps, and generates motion through planning. Its low-level generation uses force-closure-based grasp synthesis and keyframe-based motion planning. That provides a useful contrast between inserting more explicit manipulation structure and searching more directly around a reconstructed continuous reference. VideoManip targets a different endpoint again. It combines reconstructed trajectories with contact optimization and demonstration synthesis, then trains manipulation policies. Its project page explicitly identifies its hardware rollouts as executions of a closed-loop DP3 policy. That makes it especially relevant if the desired artifact is a learned visuomotor controller rather than an optimized replay trajectory. I would choose among these directions by asking what uncertainty I want each stage to absorb. Explicit planning can encode useful task structure. Physics optimization can repair embodiment and contact mismatch. Policy learning can address variation at execution time. These are potentially complementary components, not necessarily mutually exclusive answers to “learning from video.”. The comparison also argues against treating “RGB only” as the entire novelty. The more consequential questions are what assumptions remain, what representation is transferred, and what kind of execution is actually evaluated. The scaling bottleneck is usable video. The final collection contains 500 human-verified trajectories: 53% from internet videos, 31% from egocentric sources, and 16% from generated videos. The filtering study is equally informative. Of 2,000 ten-second 100DOH clips, 187 contain meaningful interaction and 83 pass the reconstruction quality screen; the authors’ optimistic estimate is 107 usable clips. That is roughly four to five percent for this particular audit, not a universal constant for internet video. More importantly, it is a candidate-quality result upstream of complete robot deployment. I would not substitute it for an end-to-end data-production yield. The engineering implication is that a large video corpus and a large manipulation-demonstration corpus are different assets. A hand appearing near an object is not enough. For reconstruction, I would prioritize continuous clips with visible interaction, stable object identity, useful framing, and enough evidence to infer the relevant motion. That suggests a staged compute budget. Cheap filtering should precede expensive generative reconstruction and physics search. But those filters should be evaluated for recall as well as rejection rate: an overly conservative visibility filter could systematically eliminate precisely the complex interactions we eventually want robots to learn. The cost metric I would track is not downloaded video hours. It is accepted, useful trajectories per unit of compute and human review. Ultimately, the more demanding denominator is downstream learning improvement per unit of data-production effort. Generated video does not remove that accounting problem. It offers a way to propose additional interactions, but I would still screen for persistent object geometry, coherent motion, and plausible contact timing. The physics stage should be understood as a model-based optimizer and filter—not a guarantee that every visually convincing generated clip can be repaired into a meaningful physical task. How I would bring this into a robotics stack. As of September 26, 2026, the official repository is available. Its reconstruction documentation specifies an NVIDIA GPU with at least 32 GB of memory, access to the SAM model weights, MANO assets, and multiple software environments. The dependency setup is a substantial part of reproduction, not incidental packaging. I would begin with the bundled whisking example, not a random internet clip. The retargeting viewer can display the human reconstruction, the kinematic robot reference, and the physics-optimized trajectory together. Establishing that known example gives you a concrete reference for coordinate conventions, assets, and expected intermediate outputs before introducing new perception failures. Next, I would build a small, deliberately varied evaluation set and categorize failures by stage. Include clips that begin with an existing grasp, clips containing pickup or placement, and clips with substantial object rotation. The goal is not an impressive montage; it is to discover whether the system’s main bottleneck in your domain is reconstruction, initialization, morphology, or dynamics. For another robot hand, I would start with the kinematic interface: fingertip sites, wrist frames, joint limits, and the quality of the initial reference. Then I would inspect contact geometry and actuator behavior before tuning the physics objective. The IK implementation makes those target correspondences explicit, which is a better starting point than treating the hand model as a drop-in mesh replacement. I would also pin and save the actual resolved configuration. The repository’s generic defaults differ from its DO AS I DO dataset override, including simulation timing and planning horizon. Reproducing the named method requires more care than loading whichever default file happens to be easiest to find. For hardware, follow a separate validation path. The deployment documentation provides dry-run and reduced-speed options, requires the manufacturer’s hand SDK, and calls for checking the robot’s home configuration and emergency-stop access. A trajectory that looks acceptable in an isolated hand simulation should not bypass full-cell inspection. Finally, I would test learning value explicitly. My next experiment would hold the policy architecture and evaluation tasks fixed, then compare training with and without the retargeted data. I would report transfer benefit alongside reconstruction acceptance, simulation success, and human review effort. Those measurements answer whether this pipeline is producing merely executable examples or genuinely useful experience for a learning system. The durable idea. The most reusable idea here is to treat reconstructed human motion as evidence about an interaction, not an exact specification of robot behavior. For perception, that means asking a generative model to maintain a persistent object explanation across time. For control, it means allowing the robot’s motion to change when geometry, contact, and dynamics demand it. The project’s reconstruction and simulation stages make those two kinds of reasoning explicit. My takeaway is not that video has made dexterous data free. It is that video can become a useful starting point for constructing robot experience—provided we budget for uncertainty, optimization, filtering, and validation. The remaining research question is then sharper than “Can robots learn by watching?” It is: Which parts of an observed interaction should become constraints, which should remain uncertain, and which should the robot be allowed to reinvent?. DO AS I DO gives a concrete, inspectable answer for an important slice of that problem.