Latest / Embodied AI 101 / WOLF-VLA: Vision-Language-Action for Humanoid Walking

Transcript

WOLF-VLA: Vision-Language-Action for Humanoid Walking · Embodied AI 101

• WOLF-VLA: Vision-Language-Action for Humanoid Walking. Over the past few years, vision-language-action (VLA) models have surged as a new paradigm for robot control – essentially training large vision-language models to see and follow instructions with a robot’s body. In manipulation, for example, recent multi-modal transformers (like OpenVLA) have learned to pick and place objects or manipulate tools by processing camera images and text commands in one end-to-end model. But up to now, these VLA agents have mostly been used for arm movements and object handling. Extending them to whole-body humanoid locomotion – where balance, contact dynamics, and stability constraints dominate – has been largely unexplored. Humanoid walking and transitional motions are contact-rich and highly dynamic: they demand precise balance control (ZMP, center-of-mass stability, etc.), coordinated multi-joint motions, and safety against falls. Moreover, there has simply been a lack of large, realistic datasets of human or robot walking demonstrations paired with visual scenes and language instructions. WOLF-VLA, introduced by Boukheddimi et al., steps into this gap. It combines whole-body optimal control trajectory synthesis with vision-language learning to train an instruction-following locomotion policy. In short, WOLF-VLA generates feasible, optimized walking motions in simulation and uses them to teach a VLM how to walk and move a humanoid robot from natural-language directions. The key insight of WOLF-VLA is to leverage classical optimal-control tools to bootstrap the data-hungry language model. The authors use a full-body dynamic model of a humanoid and solve optimization problems to produce stable, physically realizable motions for a variety of tasks. Each optimized solution is a sequence of joint configurations (or velocities) that accomplishes a locomotion task – e.g. walking to a target, turning, squatting, etc. Crucially, these trajectories are dynamically feasible by construction (meaning they respect contact constraints and joint limits). Then each trajectory is paired with a (simulated) camera view of the world and a text instruction describing the task. For example, one datapoint could be a motion that moves the robot to a colored landmark with a phrase like “Walk to the green box on the ground”. The vision input is an egocentric image from the robot’s perspective, so it sees the box and its surroundings. The authors compile such examples across six families of locomotion tasks, with variations in object colors, placements, environments, and visual distractors. This yields a large, multi-modal dataset of humanoid walking: state trajectories, first-person images, and textual commands. Finally, they train a transformer-based VLA model to map (image, text) → (body actions). At run-time, the model takes a camera frame and an instruction like “Approach the red target and squat” and generates the full-body control for the robot. By fusing optimal-control data and deep learning, WOLF-VLA achieves robust instruction-following locomotion in simulation. The learned policy shows strong reasoning (it correctly interprets complex instructions) and is resilient to varying start positions and visual settings. The authors carry out ablations confirming that each modality – language, vision, dynamics – contributes to performance. They also promise to release the entire dataset and evaluation suite, creating “a reproducible dynamically consistent benchmark” for language-conditioned humanoid locomotion. In total, WOLF-VLA’s contribution is twofold: a method of generating and learning from data to combine VLA models with whole-body optimal control, and a first-of-its-kind open dataset/model for humanoid locomotion tasks. In what follows, we unpack why this matters and how the approach works. Why Vision-Language for Locomotion?. Humanoid locomotion has always been treated differently from manipulation. Traditional control methods rely on dynamics models (LIPM, centroidal) to plan foot placements or center-of-mass, and on low-level controllers (QP-based whole-body controllers or model predictive control) to track those motions. By contrast, language-driven policies are a newer idea in robotics, boosted by large-scale transformer models. An anatomy-of-VLA survey notes that VLA models “are driving a revolution in robotics, enabling machines to understand instructions and interact with the physical world”. These models use big vision-language backbones (often LLMs paired with convolutional encoders) and learn from huge multi-modal datasets (like Internet videos, teleoperated robot logs, etc.). With enough data, they can generalize to new instructions, objects, and environments. For manipulation tasks, they set new records in flexible, zero-shot control. However, as Boukheddimi et al. point out, the same paradigm has not been applied to complex bipedal walking. Typical locomotion tasks (walking, turning, squatting, climbing stairs) require precise whole-body coordination. There are frequent changes in contact (feet leave or land), high-order dynamics, and strict stability requirements. Offloading this to a purely learned model is risky if the training data or model is insufficient. Moreover, collecting real-world demonstrations of a humanoid performing a wide variety of walks and maneuvers with corresponding camera views and language descriptions would be extremely expensive and slow. These tasks also raise safety concerns – a learned model must not command the robot to fall or break itself. The WOLF-VLA team explicitly cites these challenges: “data scarcity, the absence of dynamically consistent demonstrations, and the difficulty of encoding optimality and safety in learning-based pipelines” have so far kept VLA models away from whole-body locomotion. In other words, standalone VLA training (like one might do for a robot arm) would struggle without huge, well-curated datasets that include all the nuances of humanoid balance. WOLF-VLA addresses this by using optimal control (OC) to produce safe, stable examples first, then using them to train the VLA model. This sidesteps the scarcity problem: instead of recording hours of human or animal walking, the paper synthesizes “dynamically feasible humanoid trajectories” in simulation. We’ll see in the next section how they do this at scale. Once you have such a dataset, the benefits of VLA become clear: the model can learn to reason about language instructions (“squat”), vision cues (see the green box), and action in one unified network. It effectively learns a high-level “map” from semantics to motion. For instance, given an instruction like “move the blue block to the red zone”, a VLA policy might decompose it into locomotion subgoals (walk to the blue block, squat, grab, walk to red zone, place). WOLF-VLA, focusing on locomotion, essentially learns interpreters that convert natural instructions into appropriate gait and posture changes. This aligns with how other VLA works have extended capabilities beyond raw teleoperation: OpenVLA, for example, trained on 970k robot episodes and produces generalist manipulation policies from text, and MotionVLA used a dual-stream model to generate realistic human-like motions from images and text. WOLF-VLA can be seen as a similar leap, but in the domain of whole-body balance and travel rather than arm movements or pedestrian animation. By closing the loop between high-level planning (language) and low-level control (walking dynamics), it opens up humanoid robots to more flexible, purposeful behavior. Creating a Multi-Task Humanoid Dataset. A core component of WOLF-VLA is the new humanoid locomotion dataset. The authors “construct a comprehensive dataset of dynamically feasible humanoid trajectories across six locomotion-related task families”. While the paper doesn’t list the tasks explicitly in the abstract, the phrase suggests each family corresponds to a type of motion or scenario (for example: walking to a target, turning in place, stepping over an obstacle, squatting and rising, climbing, etc.). They enrich these tasks with variability: each is parameterized by changes in the environment (such as obstacles or terrain), object colors and positions, and even visual distractors in the scene. This variety means that the model can’t simply memorize a single solution; it must interpret the scene and instructions. How do they generate this dataset? The key is using whole-body optimal control motion synthesis. In practice, this means solving motion-planning problems for the humanoid with physics. For example, to get a trajectory for “walk to the red ball 2 meters ahead,” the system might set up an optimal control problem with cost terms for reaching the ball position while minimizing energy or deviation from an upright posture, subject to full dynamics and contact constraints. Techniques like direct collocation or shooting methods can then compute feasible joint trajectories. This is an established idea in legged robotics: for instance, Ishihara et al. use hierarchical MPC to drive full-body motions for agile tasks, and Boukheddimi’s own prior work did constrained trajectory optimization on a closed-chain humanoid. By adopting such methods, WOLF-VLA ensures that every example in the dataset is physically valid. As the authors put it, this yields a “reproducible dynamically consistent benchmark for whole-body humanoid locomotion”. Each entry in the dataset comprises:. Joint trajectories (the action sequence): the optimized sequence of robot joint angles or velocities over time for a given task. This is the ground-truth action to be imitated. Ego-centric images: first-person camera views of the environment while performing the task. These give the contextual visual input (e.g. seeing the colored object or obstacles). Natural-language instruction: a textual command describing the task (e.g. “Walk to the green pillar and squat”). These instructions presumably correspond exactly to the task the trajectory accomplishes. Because the trajectories are generated by optimal control, the dataset covers only those movements that are truly feasible and safe. This is in contrast to many imitation datasets for manipulation where raw human teleop data may contain suboptimal or even invalid moves. It also means the dataset can be quite large: by randomizing task parameters (target location, object color, obstacle layout, textures), they can synthesize thousands to tens of thousands of variations. The paper abstract highlights that this is a large-scale dataset, though exact size isn’t given. By comparison, OpenVLA used nearly a million episodes for manipulation‍. WOLF-VLA’s scale may be smaller (since humanoid simulation is heavier), but is nonetheless much larger than any previously available humanoid dataset with language. Once collected, this multimodal data becomes the training ground for a VLA model. In essence, the model learns to regress or predict the joint actions from the fused inputs of image and text. The learning problem can be thought of as conditional sequence modelling: given the current camera frame and goal instruction, predict the next step of the trajectory. Because the data is generated in simulation, the authors can ensure synchronized observations and high-fidelity state information (exact joint angles, velocities, contact forces if needed) for supervision. This is crucial for stable learning: the model sees precise ground-truth motions rather than noisy human demos. The VLA Model and Training. With the dataset ready, the WOLF-VLA framework trains a vision-language-action model on it. Though the arXiv abstract doesn’t detail the architecture, by analogy to other VLA papers we can infer the general approach. Likely they start from a pre-trained multi-modal backbone (e.g. a vision-language model like CLIP/SigLIP combined with an LLM) and fine-tune it. For example, OpenVLA fused a SigLIP+DINOV2 vision encoder with a Llama 2 7B language model to output discrete action tokens. Similarly, MotionVLA used a Qwen 3.5–based model to lay out motion tokens in time. WOLF-VLA’s goal is analogous: process an RGB image (from the humanoid’s belly camera perhaps) and a text instruction, and output a sequence of actions for the legs and body. The output actions could take a few forms, depending on design. One option is discrete “action tokens” decoded into continuous controls (like desired joint velocities) – this is the approach in OpenVLA where the transformer’s vocabulary represents quantized action snippets. Another is simply having the model predict the next joint positions/regulators in a continuous form (with regression). In either case, the VLA learns to do “imitation” of the optimal trajectories, guided by language. For instance, if the instruction is “turn 90 degrees left”, the model learns the sequence of joint commands that accomplish a smooth pivot, because those commands appear in the training trajectory for a corresponding task. If the scene has two objects marked in blue and red, and the text says “go to the red one”, the model learns to select the direction corresponding to the red object in the image. At run-time, we can imagine WOLF-VLA’s policy loop like this: the humanoid robot has a goal instruction (for a high-level mission planner or human), and it continuously streams first-person camera frames. Every time step, the VLA model encodes the current frame and the instruction, and emits the next chunk of the locomotion command (this might be at, say, 10 Hz or so). A lower-level controller would then execute that command by applying torques or velocities to the joints at high frequency. Indeed, a similar decoupling is seen in WholeBodyVLA’s pipeline: at inference it ran the vision-language model at low frequency (~10 Hz) to output both arm and leg action commands, and then a dedicated locomotion policy acted at 50 Hz to execute them. WOLF-VLA could use a comparable setup: e.g. a PD controller or simple receding-horizon planner to track the pose outputs. The abstract doesn’t spell out the control stack, but it emphasizes that the trained VLA “generat[es] humanoid locomotion policies directly from natural-language instructions”, implying end-to-end mapping. To be clear, WOLF-VLA is not purely a reinforcement learning system. Rather, it uses supervised learning from the optimal-control generated data. This contrasts with many state-of-the-art locomotion controllers that rely on RL (e.g. PPO) to learn stable gaits. In fact, the VLA here essentially imitates solutions found by optimization. This has pros and cons: on the plus side, it avoids inefficient exploration and immediately inherits stable behaviors. On the minus side, it may be limited by the quality and diversity of the optimizer’s outputs. Yet given the cost of RL training on complex robots and the difficulty of hand-crafting rewards for language tasks, this imitation route is a sensible bootstrap. Results: Generalization and Robustness. Although the full paper is needed for quantitative details, the abstract reports that the VLA policy is effective and robust across tasks. It “exhibits strong reasoning and robustness to initial-condition variability, and competitive performance across several tasks and environment settings”. In practice, this means the trained policy can take a command-image input and successfully complete the motion in simulation at high success rate, even when starting from a slightly different pose or facing minor changes in the visuals (lighting, extra objects, etc.). This is plausible: the network has seen randomized scenes at training, so it learns to disregard irrelevant changes (like distractor objects) and focus on the instruction. The “competitive performance” likely means that on each of the 6 task families, the VLA model’s success rate or tracking accuracy is on par with what one would get by planning with the original optimal controller (or some baseline controller) – at least in the contexts they tested. Importantly, the authors performed ablation studies to quantify each input modality’s contribution. By systematically removing the text or image inputs (or perhaps adding noise), they saw how much performance dropped. The abstract indicates that each modality did matter (“demonstrates the impact of each modality on model performance”). For example, if you remove the language input and only give the image, the model would no longer follow instructions – it might wander or pick an arbitrary direction. If you remove vision and just train on text, the model might have a hard time relating instructions to actual positions in the environment (it would have no context of where the target is). The ablation confirms that both language and vision cues are needed for the full capabilities. (This echoes findings in other VLA works, where text and vision together provide a big advantage for generalization.). The abstract does not report specific numbers, but highlights that WOLF-VLA successfully bridges vision, language, and control in simulation. It even mentions that the resulting benchmark allows “scalable transfer of instruction-driven locomotion policies”. The idea is that by releasing the data and model, others can fine-tune or adapt it to new humanoid platforms or tasks. For instance, one could imagine taking the learned VLA policy and fine-tuning it with a small amount of real robot data, to move closer to real-world walking. The dataset itself – being “dynamically consistent” – can also be used to benchmark new approaches (for example, to test if a new RL-based humanoid policy can learn the same instruction tasks better or faster). In summary, preliminary results suggest that leveraging whole-body optimal control data is a viable route to training VLA policies for locomotion. The policy learned to handle the instructed maneuvers reliably in the simulator. While real-world testing is future work, the approach should in principle supply a strong prior control strategy – akin to how humans practice walking with various goals before hearing any command. Relation to Prior Work. WOLF-VLA sits at the crossroads of several research threads. On the locomotion side, it builds on a long history of optimal control and RL for bipedal walking. Early approaches used simplified models (like inverted pendulums and ZMP planners) to chart safe footsteps, and updated them with feedback control. More recent works have incorporated whole-body dynamics: for example, Ishihara et al. used a hierarchical model predictive control that plans full-body motions with a fast stance controller. Boukheddimi et al. (2023) optimized humanoid trajectories taking into account closed-loop kinematics in hybrid robots. Those works show that computing optimal humanoid motions is feasible and beneficial. WOLF-VLA leverages that by running similar optimizers offline. RL-based controllers are also prominent in legged robotics. For instance, Huang et al. recently proposed ECO, which frames energy efficiency as explicit constraints in an RL loop to get stable humanoid walking. Others have trained policy networks end-to-end for walking or gotchas like running, turning, etc. But these RL controllers normally require careful reward shaping for each task (and usually do not accept language commands directly). WOLF-VLA bypasses reward engineering by using language supervision, which can express goals naturally. In that sense, it is complementary to RL: RL might tune a locomotion policy on a single task, whereas WOLF-VLA trains a policy that switches tasks based on text. Future work could even combine them: one could imagine using WOLF-VLA’s trained policy as a warm start for an RL fine-tuning step on a real robot (or vice versa). On the multi-modal side, WOLF-VLA follows a clear lineage of VLA research. OpenVLA and VLM-assisted control projects have shown the power of large vision-language models in robotics. WholeBodyVLA specifically tackled a mixed locomotion-manipulation setting on a humanoid: their system used a pretrained latent action model (LAM) for manipulation and a specialized locomotion RL policy (LMO) for movement, combining them at runtime. WOLF-VLA can be viewed as a sibling in this family: it still uses a combined vision-language framework, but focuses the action space purely on locomotion. In WholeBodyVLA, the VLM output two streams (arm actions and locomotion commands), whereas in WOLF-VLA likely the model focuses entirely on the latter. Thus WOLF-VLA simplifies the action task but raises the challenge of full balancing and stepping. Another related line is integrating vision or language in locomotion jobs. Some work has explored vision-based foot placement or terrain adaptation (e.g. vision-guided quadruped steps), but seldom with language. A notable precedent is CobraNet by Deeplab group, which gave language instructions (like “go to the chair”) to a quadruped, but that system used a vision planner plus RL, not a single VLA model. WOLF-VLA is unique in training a joint model from scratch. The use of language also resembles recent humanoid projects that learned from human video demonstrations with language; for example, “EgoHumanoid” allowed learning locomotion from human egocentric video, though without language. WOLF-VLA’s novelty is tying language directly to walking in a trainable model. Finally, the open release of their work connects to the growing trend of shared robotics data. OpenVLA and other VLA datasets (like Bridge) have galvanized generalist manipulation research. Until now, there has been no analogous “Open Humanoid Locomotion” dataset with language. By promising to release the trajectories and simulation suite, WOLF-VLA could become a standard benchmark – in the spirit of OpenX-Embodiment, but for bipedal movement. This will help compare future approaches (e.g. new neural architectures, new RL methods) on a level playing field. Using and Adapting WOLF-VLA. For practitioners, WOLF-VLA offers several concrete assets:. Dataset and Simulation: Once released, researchers can download the whole-body humanoid trajectories, along with the environment models and scripts. This affords an immediate training playground for vision-and-language-conditioned policies. For example, one could fine-tune a different VLA architecture (like a smaller model or a different training strategy) on the same data to compare performance. Or, one could modify the tasks (e.g. adding new obstacle types) and re-run the optimal control optimizer to expand the dataset. Trained VLA Policy: The authors imply they will release model checkpoints. Other teams could use the pre-trained VLA policy as a starting point, even applying it on different humanoid simulations. For instance, a lab working on a new humanoid could fine-tune WOLF-VLA’s policy in their own physics engine for sim-to-real transfer. Because the policy is trained on diverse data, it may adapt more quickly than training from scratch. Benchmark Suite: The evaluation suite can standardize comparisons. If you invent a new method for language-conditioned walking (say a cross-modal transformer or a diffusion model for actions), you can test it on the same six tasks in simulation to see how it stacks up (just as the WOLF-VLA authors did). This makes research reproducible and speeds iteration. In applying WOLF-VLA, it’s important to remember its current scope: it was developed in simulation with an idealized humanoid (likely assumed perfect sensing and modeling). Real hardware adds noise, latency, and unmodeled dynamics (like flexibility in the structure or slip on the feet). So a direct step to robot requires careful sim2real work (e.g. domain randomization, fine-tuning with real perturbations). Nonetheless, the vision-language capability is a huge asset: once the robot can interpret “turn left” or “climb the stairs” from a human’s command in human language, it can be far easier to integrate with higher-level planning or user instructions. Broader Impact. WOLF-VLA points toward an exciting vision: programmable, language-directed humanoid agents. Imagine rescue robots that can be told “go to the red door” or elder-care robots that can follow “walk over and sit down gently”. Traditionally, programming such missions would require precise coordinate maps or button presses. Vision-language frameworks allow more natural communication. Moreover, by grounding instructions in rich perception data, these robots could navigate complex, unstructured spaces more fluidly. There are also technical lessons here. The combination of optimal control + learning is powerful. The paper shows that we no longer have to choose between rigorous control theory and modern deep learning – we can fuse them. The optimizers guarantee safety and feasibility, while the learning model grants flexibility. This approach might generalize beyond locomotion: any domain where dynamics are well-understood could use this workflow to generate synthetic data. For example, one could imagine ‘analogous VLA datasets’ for aerial drones (with flight dynamics) or for articulated industrial machinery. On the flip side, there are limits to consider. WOLF-VLA’s performance depends heavily on the quality of the optimizer. If the optimizer doesn’t explore certain creative or efficient maneuvers, the learned policy won’t either. Also, because the model is trained on a fixed distribution of environments (the six task families), it may not generalize to completely new tasks outside that scope without further training. Extensive interpretability and safety evaluations would be needed before letting such a system loose on a real robot around humans. Nonetheless, the authors’ plan to release the dataset and code will allow the community to iterate on these problems. WOLF-VLA establishes a much-needed stepping stone in humanoid robotics: it demonstrates that building vision-language agents for walking is possible and practical. By blending optimal control mastery with modern AI-scale learning, it opens the door to humanoids that can walk somewhere because you asked them to, not just because they learned a reward function. This fusion of modalities – seeing, thinking, and moving with two legs – is precisely the kind of embodied intelligence that can take robots into more open-ended real-world roles. Sources: The above analysis is based on the WOLF-VLA abstract and related vision-language-action and humanoid locomotion literature.