Latest / Embodied AI 101 / HumanCLAW: The Model Sees the Chair but Loses the Body

Transcript

HumanCLAW: The Model Sees the Chair but Loses the Body · Embodied AI 101

• HumanCLAW: The Model Sees the Chair but Loses the Body. HumanCLAW: Can Vision-Language Models Act Through a Body? • Siyao Li et al. • Meta, Nanyang Technological University, University of Washington, Brown University, and Northwestern University • arXiv preprint • 2026. A robot walks toward a chair. The chair is visible. The path looks clear. The model says it is close enough, stops, turns around, and sits—onto empty floor. What failed?. In a conventional robotics evaluation, the answer is frustratingly ambiguous. Perhaps the vision-language model estimated the chair’s location incorrectly. Perhaps the task planner selected the wrong maneuver. Perhaps the locomotion controller drifted. Perhaps the body lost balance. Perhaps the sit controller was poorly calibrated. Episode-level success collapses all of these possibilities into a single bit. HumanCLAW is interesting because it tries to separate them. It places a frozen, off-the-shelf vision-language model in control of a simulated human body, but delegates the realization of local motions to a deliberately reliable generative motor layer. The body still has volume, inertia-like motion continuity, collisions, gravity, and contact with movable objects. What it largely does not have is the ability to fail because of torque control, tracking error, or loss of balance. The benchmark can therefore ask a narrower question: given an egocentric image and a task, does the VLM know what this body should do during the next half-second?. The headline answer is no—not reliably. HumanCLAW-Bench contains 1,218 episodes in 41 indoor scenes. The strongest evaluated model reaches only 16.8% full interaction success on the benchmark’s 597 sitting episodes. More important than the number is the shape of the failure: models can often recognize a target once it enters view, yet fail to bring their body to it, know when they have arrived, place themselves for contact, or notice that a limb is colliding with the environment. The Missing Middle of Embodied AI. There are two familiar ways to connect foundation models to action. The first is the hierarchical agent approach. A language or vision-language model issues symbolic commands such as “go to the kitchen,” “open the drawer,” or “pick up the cup.” A simulator script, navigation stack, or library of predefined skills handles execution. This arrangement is effective for studying planning and task decomposition, but it can hide the physical meaning of action. A command may complete atomically even though a real body would need to approach, orient, avoid collisions, establish contact, and recover between motions. The second approach is an end-to-end vision-language-action policy. Systems such as RT-2, Octo, and π₀ train on robot trajectories so that images and language map directly to robot actions. These systems produce executable control, but the resulting policy entangles web-scale semantic knowledge, embodiment-specific action data, and motor behavior. When an episode fails, it is difficult to determine whether the general-purpose model reasoned badly or whether the learned action distribution failed to cover the state. HumanCLAW targets the space between these regimes. Its actions are not abstract task completions, but neither are they joint torques. The VLM chooses parameterized, atomic whole-body skills such as walking a local displacement, turning through an angle, stepping sideways, climbing, or sitting to a specified height. Those decisions become continuous human motion and are then subjected to contact and gravity. This makes the work less a proposal for a complete humanoid policy than a diagnostic instrument. The authors call the capability under examination action intelligence: selecting, parameterizing, and sequencing physical actions while observing their consequences in a closed loop. Their core claim is that conventional spatial benchmarks test whether a model can describe spatial relationships, while HumanCLAW asks whether that understanding survives contact with a body. How HumanCLAW Turns a VLM into a Body Controller. HumanCLAW has three major components: a decision harness around the VLM, a skill-conditioned human-motion generator, and a “half-physics” simulator. The decision harness. Every 0.5 seconds, the frozen VLM receives the high-level task, the current egocentric image, a registered list of skills and their rules, and a compact textual history of recent states and decisions. Instead of asking the model to jump directly from image to action, the harness elicits three levels of reasoning. First, the model produces a structured visual state. It describes visible objects from left to right, estimates their distance, states whether the target is visible, and assesses nearby obstacles and affordances. This externalizes the model’s interpretation of the current frame. Second, it selects or maintains a mid-level objective. This is a commitment lasting several decision steps—for example, entering a doorway, continuing up a staircase, approaching a visible couch, or turning to place a seat behind the body. The objective can be inherited from the previous step or revised when new visual evidence arrives. Third, the model emits one atomic skill and its parameters in structured form. Sitting on a sofa is deliberately not exposed as a single semantic action. The VLM must compose the behavior: approach the sofa, reduce the remaining distance, rotate until its back is correctly placed, and only then request a sit at an appropriate height. This is an important design choice because it prevents a scripted function from quietly solving the central spatial problem. Before execution, a short-context verifier examines the proposed skill. The verifier is triggered by the action type and asks narrowly framed questions using the current observation: Is the forward lane actually clear? Has the target really been reached before stopping? Is there a stair underfoot? If the body turns before sitting, will the seat end up behind it? A rejected proposal is replaced by another action from the same pool. Actions whose safety cannot be inferred from the camera, such as stepping backward into an unseen region, receive no equivalent check. Architecturally, this resembles a planner-plus-critic loop, but both operate over a physical skill interface. The verifier is not merely cosmetic. As the ablations later show, it is one of the primary reasons the system works at all. A motion generator as a semantic actuator. A VLM can write. It cannot reasonably emit more than a hundred pose dimensions at 30 frames per second while preserving human motion continuity. HumanCLAW bridges that gap with a diffusion-transformer motion prior. The base motion model observes five recent poses and generates the next 15 frames, giving one 0.5-second chunk at 30 Hz. The body state has 219 dimensions, and the base network is a roughly 38-million-parameter, ten-layer DiT trained by flow matching on motion-capture data. Rather than compressing the motion history into a single vector, the five history poses and 15 future poses are represented as one token sequence, allowing future motion to attend directly to recent pose and velocity structure. The base prior is intentionally skill-agnostic. Semantic control is added through one ControlNet-style adapter per skill. The selected skill routes generation to the appropriate adapter, while its continuous parameters specify outcomes such as local pelvis displacement, heading change, lateral distance, stair geometry, or final sitting height. This is effectively a motion-level application binary interface. The VLM sees a stable, human-readable function call; the motor system sees a continuous condition tied to measurable geometry. A new primitive can be added by defining its schema, collecting or filtering suitable motion clips, and training another adapter without retraining the base prior or the other skills. The authors test whether this interface actually obeys its commands. Reported achieved-to-commanded ratios are about 0.97 for walking, 0.99 for turning, 1.00 for side-stepping, 0.99 for stepping backward, and 0.98 for sitting height. Stair motions exhibit lower but stable gains of roughly 0.74 to 0.79. Compared with MoMask prompted using the same numerical requests, HumanCLAW’s controller has much lower variance and responds more faithfully to magnitude. This does not prove perfect execution in every contact-rich state, but it supports the intended interpretation that local skill semantics are predictable. A kinematic body in a dynamic world. Pure animation playback would let a humanoid walk through a wall or climb unsupported through the air. Fully dynamic humanoid control would reintroduce balance, torque tracking, and falls. HumanCLAW chooses an intermediate mechanism. The environment runs in AI Habitat with Bullet rigid-body physics. Scene objects follow collision, gravity, friction, articulation, and displacement. The human body, however, is actuated using equivalent root and joint velocities derived from the generated pose sequence rather than simulated torques. The simulator runs at 120 Hz with compliant passive joints. Walls can block the body, feet can push furniture, and stairs must physically support vertical motion, but the humanoid is not expected to solve balance control. This “half-physics” design is the paper’s most consequential methodological decision. HumanCLAW does not remove embodiment; it removes one important source of embodiment failure. The benchmark asks whether the selected motion is appropriate, not whether a learned low-level controller can track it. What HumanCLAW-Bench Actually Measures. The benchmark uses the 41 validation houses from HSSD, a human-authored synthetic indoor-scene collection designed for realistic embodied navigation. It defines 1,218 episodes over chairs, beds, couches, potted plants, toilets, and televisions. Each episode supplies an initial body pose and a category-level target; reaching any instance of that category is acceptable. The dataset is divided into two subsets. The 621 chair, plant, and television episodes end after navigation. The 597 bed, couch, and toilet episodes continue to sitting. Thus, the reported 16.8% interaction score is measured over the sit subset, while finding and navigation are measured over all 1,218 episodes. Plausibly movable items are converted into dynamic rigid bodies, allowing careless motion to disturb the scene. Large fixtures and target furniture remain static. Difficulty is characterized along three geometric axes. Distance is the geodesic path length to the nearest target. Choice approximates route branchiness using major turns and rooms traversed. Obstacle measures clutter near the route. This is more informative than a single easy-or-hard label because a long open corridor, a short maze of doorways, and a cluttered room stress different behaviors. The success criteria are deliberately both objective and subjective:. FindSR requires the target to occupy at least 100 semantic pixels in the 512-by-512 egocentric view and for the model to acknowledge seeing it. NavSR requires the body to come within 20 centimeters of the target and actively stop. InteractSR requires a sit whose pelvis makes physical contact with the target, followed by appropriate completion behavior. Geometry-only variants remove the acknowledgement requirement. That distinction is central: an agent that reaches the chair but does not know it has arrived has not completed the benchmark’s intended cognitive task. HumanCLAW also reports collision frequency, the number and displacement of disturbed objects, motion jerk, average rollout length, and token consumption. These secondary metrics prevent inactivity from being mistaken for safety and successful but wildly oscillatory motion from being mistaken for coherent control. The Leaderboard: Seeing Is Not Arriving. The nine evaluated VLMs all use the same motion layer and benchmark harness. Their headline success funnel is:. | VLM backbone | FindSR | NavSR | InteractSR | |---|---:|---:|---:| | Gemini-3.1 | 64.9% | 42.4% | 16.8% | | Gemma-4-31B | 58.1% | 28.7% | 11.1% | | Gemini-2.5 | 58.5% | 21.6% | 3.5% | | GPT-5.5 | 55.1% | 13.9% | 3.4% | | Claude-4.8 | 32.6% | 8.6% | 1.5% | | Qwen3.6-27B | 51.0% | 20.9% | 0.2% | | Qwen3.6-35B-A3B | 44.6% | 5.8% | 0.0% | | Qwen3.5-27B | 37.8% | 13.5% | 0.0% | | InternVL3.5-38B | 46.8% | 0.8% | 0.0% |. These are the paper’s full-validation numbers; model names are reproduced as reported by the authors. The important pattern is the two-stage collapse. Models first lose a large fraction of episodes between seeing a target and stopping beside it. They lose another large fraction between navigation and successful physical interaction. Even Gemini-3.1 navigates to only about 65% of the targets it actively finds. Four models achieve no more than 0.2% interaction success. Open and closed models do not form a clean hierarchy. Gemma-4-31B is second overall and outperforms the evaluated GPT and Claude variants on all three success metrics. At the same time, task success and motion coherence are only weakly coupled. GPT-5.5 produces the lowest motion jerk despite low navigation success, while Gemini-2.5 achieves stronger task progress but frequently alternates direction and receives the worst jerk score. Cost is similarly non-monotonic. The model consuming the most input and output tokens per step is not the strongest. Low environmental disturbance can also be misleading: one Qwen variant disturbs fewer objects largely because its episodes terminate early and it completes no interactions. HumanCLAW’s broad metric set is useful precisely because “success,” “safety,” and “coherence” can each be gamed by different forms of failure. The Anatomy of Failure. Search fails before recognition. For the strongest models, the gap between objective target visibility and acknowledged visibility is relatively small. Gemini-3.1, for example, objectively brings a target into view in 69.9% of episodes and acknowledges it in 64.9%. Gemma-4-31B’s corresponding numbers are 67.7% and 58.1%. This supports a qualified version of the authors’ claim: once a target is genuinely visible, recognition is often successful. The more common problem is getting the camera into a state where the target can be seen. The automated failure taxonomy attributes 38% of find failures to ineffective exploration. Another 10% bring the body near a target without turning to view it, and 3% make progress before giving up and retreating. Genuine “target visible but not recognized” failures account for 23%. The qualitative examples make this distinction vivid. In one episode, an agent spends all 100 decisions searching outside a house for a chair. It keeps describing walls, paving, and grass correctly, but never applies the commonsense prior that a chair is more likely indoors. The issue is not object recognition; it is exploration policy and semantic search. Navigation fails at self-localization and termination. Across all nine model rollouts, there are 5,473 cases in which the model actively finds its target. Of those, 3,706—68%—still fail to navigate successfully. Nearly two-thirds of these navigation failures involve estimating the body’s own state relative to the target. The error appears in opposite forms. In 20% of failed navigation cases, the body reaches the target but continues issuing movement commands because it does not recognize arrival. In 14%, it remains jammed against geometry while repeatedly attempting locomotion. Another 30% stop too early, declaring that a visibly distant object is within reach. The model can describe a chair and still have no calibrated estimate of whether its pelvis is 10 centimeters or a meter away. This is why the subjective success criterion matters. If the evaluator counted any transient pass within 20 centimeters, it would credit a system that accidentally touched the target and walked away. HumanCLAW instead treats termination as part of competence: the agent must know that the state transition it wanted has occurred. Interaction exposes the missing body model. Of 726 episodes that successfully reach and stop near a sitting target, 513—71%—still fail the interaction. The authors attribute 81% of these failures to body placement rather than failure to issue a sit command. The dominant mode is sitting into empty space. In 58% of failed interaction attempts, the agent requests a sit but its pelvis never contacts any surface. Fourteen percent land on the floor or the wrong object. Another 9% briefly establish valid seat contact but continue acting and move themselves off again. Only 19% reach the seat and simply never request the sit. Collisions tell the same story. Legs and feet collide during roughly 28% to 45% of steps depending on the model, arms and hands during 20% to 35%, and the head much less often. These are exactly the body regions least observable from a forward-facing egocentric camera. The model may see the surrounding furniture, but it does not reliably infer the swept volume of the limbs it is moving through that furniture. The paper’s “ghost” metaphor is apt, provided it is read carefully. Current VLMs behave as though the camera is moving through the world, not as though the camera is attached to a body with a pelvis, feet, elbows, contact surfaces, and action-dependent occupancy. The Ablations Point to Structure, Not More Context. The harness ablations use Gemma-4-31B on a 100-episode mini-validation set. Their strongest result concerns the verifier. Removing it reduces navigation success from 27% to 2% and interaction from 18.9% to zero, while finding changes only modestly. Without the verifier, the agent often stops prematurely or fails to close the approach loop. History is necessary, but only in compact form. With no textual history, navigation falls to 11% and interaction to zero. Increasing history from ten decisions to 20, 50, or 100 does not improve navigation beyond roughly 26% to 28%, while per-step input grows from about 4,700 to nearly 13,000 tokens. Longer context preserves more observations but does not produce a better state estimate. Visual history behaves even worse. Supplying ten image frames drops navigation from 27% to 13% and interaction from 18.9% to 3.8%. More raw evidence appears to dilute attention rather than create a persistent map. This is an argument for structured memory—objects, rooms, revisitation state, body displacement, and uncertainty—not for indefinitely appending screenshots. Finally, removing the mid-level objective leaves finding and navigation roughly intact but reduces interaction to zero. Sitting requires a short routine whose individual steps may temporarily make the target less visible: approach, rotate, back into alignment, sit, and stop. Purely reactive half-second decisions cannot reliably preserve that commitment. Together, these results suggest a concrete agent-design principle: compress the past, preserve the current objective, and verify risky transitions using a short fresh context. How HumanCLAW Relates to Nearby Approaches. Traditional ObjectGoal navigation and vision-and-language navigation generally model the agent as a point, cylinder, or moving camera. Benchmarks such as ALFRED, VirtualHome, and PARTNR add richer task structure but still rely heavily on symbolic actions, simulator-defined skills, or scripted interactions. HumanCLAW’s contribution is not a more complex task description; it is making the route from decision to consequence pass through a continuous full body. End-to-end VLAs take the opposite position. RT-2 jointly fine-tunes a vision-language model on web tasks and robot trajectories, expressing robot actions as tokens. π₀ places a flow-matching action expert on top of a pretrained VLM and trains across multiple robot embodiments. Octo similarly pursues an open generalist robot policy. These approaches are designed to produce competent control, not to isolate zero-shot reasoning. HumanCLAW sacrifices deployability in order to obtain cleaner attribution. MolmoAct is conceptually closer. It explicitly separates depth-aware perception, editable mid-level spatial traces, and low-level actions. The critical difference is training: MolmoAct learns its action reasoning and control from robot data, whereas HumanCLAW keeps the VLM frozen and supplies action through an external motion tool. One asks how to train a better embodied policy; the other asks how much embodied competence is already latent in a general VLM. VideoGameBench offers another useful comparison. It also places frozen VLMs in long-horizon closed-loop environments using raw visual observations. But key presses are resolved by a game engine and do not require the agent to reason about a continuous body’s contact geometry. SpatialVLM and related spatial VQA work probe distance and relations directly, but usually without requiring each answer to produce an irreversible world transition. On the motor side, HumanCLAW inherits ideas from receding-horizon motion generators such as PRIMAL: pretrain a general motion prior, then attach controllable adapters for semantic or spatial behavior. Its distinctive move is to treat this motion model as an agent tool—a reusable physical implementation behind a stable language-call interface. A Critical Reading of the Claims. The strongest claim HumanCLAW supports is that current VLM-plus-harness systems cannot infer and maintain target-relative body state from egocentric RGB and textual history. That is a valuable and well-localized result. The broader phrase “VLMs lack embodied self-awareness” needs more qualification. HumanCLAW deliberately provides no proprioceptive state, explicit camera pose, body transform, contact sensor, or tactile observation. Collisions affect the next RGB image only indirectly. When a foot hits furniture outside the field of view, the model receives no direct event saying that motion was blocked. A camera-only agent failing to feel an invisible collision may reveal inadequate inference, inadequate observation, or both. The paper explicitly acknowledges this ambiguity. That observation suggests the most important missing experiment: compare RGB-only performance with controlled additions of pelvis displacement, joint state, body masks, depth, contact bits, and target-relative pose. If a compact contact-and-egomotion channel removes most failures, the central issue is multimodal state estimation rather than a deep inability to form a self-model. If failures persist, the stronger cognitive interpretation becomes more convincing. The decision–control boundary is also chosen rather than absolute. Giving the model a primitive delegates a substantial amount of contact-rich movement to the motor layer, while requiring the VLM to perform fine target alignment assigns work that a practical local controller might handle. A finer skill vocabulary would classify more errors as control failures; a coarser one would classify more behavior as successful high-level planning. The authors correctly list skill granularity as a limitation. The verifier presents a related issue. HumanCLAW is described as evaluating off-the-shelf VLM action intelligence, yet navigation almost disappears without substantial harness engineering. The actual object of evaluation is therefore the VLM embedded in a particular reasoning, memory, and verification protocol. That is not a flaw—real agents require harnesses—but leaderboard differences should not be interpreted as intrinsic model rankings independent of the interface. The automated root-cause analysis is reproducible rather than manually subjective, which is a strength. But its rules are ordered, with the first matching condition assigning the episode’s sole label. A trajectory that is both spatially confused and jammed may receive only one diagnosis depending on rule order. The reported percentages should therefore be understood as frequencies under a transparent operational taxonomy, not unique psychological explanations. The benchmark’s interaction scope is narrow. Only beds, couches, and toilets participate in the 597-episode sit subset. There is no reaching, grasping, opening, carrying, handover, tool use, bimanual coordination, or deformable-object interaction. Sitting is a clever first probe because it requires whole-body target-relative placement, but HumanCLAW-Bench is not yet a comprehensive embodied-action evaluation. Finally, HumanCLAW does not compare its frozen reasoners with policies trained directly on the benchmark’s action distribution. The authors’ philosophical bet is that generalizable action intelligence should arise primarily from reasoning rather than trajectory fitting, but they explicitly acknowledge that their experiments identify the location of current failures, not the best way to eliminate them. There is also a practical reproducibility caveat. As of August 2, 2026, the public repository contains the paper materials and a release plan, but states that the harness, motion weights, simulator, benchmark episodes, and evaluation code are still being prepared. Independent replication therefore remains pending. What Roboticists Should Build Next. The first priority should be a causal sensor-ablation study. HumanCLAW already logs objective geometry and model claims, making it well suited to isolating which state variables unlock competence. Start with RGB only, then add depth, egomotion, pelvis velocity, joint state, body-part contacts, and an explicit target-relative transform one channel at a time. The resulting curve would distinguish visual spatial reasoning from state observability. Second, replace long raw history with a persistent body-and-world memory. A useful representation need not be a complete metric map. It could track visited rooms, doorways, last-seen target bearing, estimated displacement since sighting, current locomotion status, recent contact events, and confidence that a mid-level objective remains valid. HumanCLAW’s history ablations strongly suggest that representation quality matters more than context length. Third, introduce a skill-level forward model. Before executing , , or , the agent should predict the next target-relative body pose, approximate swept volume, expected contacts, and whether the action will make the completion condition observable. This can remain above torque control. It is essentially model-predictive control over semantic skills rather than joint commands. Fourth, preserve the paper’s short-context verifier pattern. Long-horizon planning and local safety should not necessarily share a single context. A persistent planner can maintain task progress, while a fresh critic checks high-risk transitions such as stopping, entering a narrow passage, descending stairs, or committing to contact. The dramatic verifier ablation indicates that this architectural separation may matter more than simply scaling the backbone. Fifth, compare three training regimes under the same body, observations, and skill vocabulary: frozen VLM reasoning, a VLM trained specifically on body-state and action-outcome prediction, and an imitation or reinforcement-learning policy fitted to HumanCLAW trajectories. Such a comparison would directly test the paper’s reasoning-versus-action-data hypothesis instead of treating it as a philosophical prior. The benchmark should then expand interactions while preserving diagnostic separation. Reaching to touch a marked surface, leaning without collision, opening doors, pushing movable objects to targets, carrying objects through narrow spaces, and sitting with different body morphologies would test progressively richer self-models. Dynamic humans and changing scenes would further reveal whether the agent maintains a live state estimate or merely recognizes static visual configurations. For teams building real systems, HumanCLAW’s most reusable idea may be the stable skill interface rather than the human avatar itself. The same design can place a VLM above certified locomotion and manipulation controllers on a mobile manipulator or humanoid. The key is to log both physical success and the model’s belief about success. A robot that happens to complete an action but cannot identify the resulting state is not ready for reliable long-horizon autonomy. Final Take. HumanCLAW’s most important result is not the 16.8% score. It is the decomposition that makes the score interpretable. Today’s strongest VLMs can often name the target and reason fluently about the room. What they cannot reliably do is maintain a calibrated, action-conditioned estimate of where their own body is within that room. They stop too early, continue after arriving, walk while jammed, ignore limb collisions, and sit where they imagine the seat should be. HumanCLAW does not prove that reasoning alone will produce general robot intelligence, nor does it fully separate model weakness from missing proprioception. It does, however, expose a neglected middle layer between symbolic planning and motor control. That layer—body-relative state, consequence prediction, and calibrated termination—may be exactly where the next generation of embodied foundation models needs to learn how to stop behaving like a moving camera and start behaving like an agent with a body.