Latest / Embodied AI 101 / CLAP: Converting Vision-Language Models into Vision-Language-Action Models via Language-Prompted Actions
Transcript
- CLAP: Converting Vision-Language Models into Vision-Language-Action Models via Language-Prompted Actions. Integrating modern vision-language models (VLMs) into real robot control is a key challenge in embodied AI. A vision-language-action (VLA) model takes an image observation and a natural-language instruction as input and produces robot actions to execute the task. In current practice, building a VLA typically means starting with a pre-trained VLM (for example a large vision transformer fused with a language model) and extending it with a new action head or output layer, then fine-tuning on large robot demonstration datasets. This design yields an end-to-end model that maps image+text to continuous actions (e.g. changes in robot joint angles or gripper commands). The upside is that the backbone of the model already knows vision and language (leveraging massive pretraining), so it can generalize to new tasks. The downside is that one often needs huge amounts of robot data (tens of thousands of demonstrations) to train the action head, and the added decoder means extra parameters and complexity. For example, Google’s RT-2 (Robot Transformer) from 2023 was one of the first models to fuse vision-and-language with robot actions, but it required a full demonstrations dataset and a dedicated action decoder. The new CLAP approach (by OMRON’s SINIC-X team) takes a radically different tack: it converts any pre-trained VLM into a VLA using only language, with no architectural changes. In effect, CLAP treats actions as just another piece of language input. Instead of adding a numeric action head, CLAP prepends natural-language descriptions of the robot’s possible actions to the text prompt given to the VLM. By framing actions as words or phrases, the method lets the powerful image-text model do the heavy lifting. In practice, CLAP fine-tunes the VLM so that the correct action-descriptive text becomes associated with the image and instruction in the model’s latent space. In other words, the action selection problem is recast as a language understanding problem: the model learns to “understand” an appropriate action description from the image+instruction, just as it would understand a question-answer. This simple trick turns the VLM into a VLA without adding any new layers or heads. The implications are significant. Since CLAP takes advantage of the inherent vision-language encoder and text generator, it can run entirely in the existing VLM framework. No new output heads means zero architectural redesign and near-instant compatibility with any off-the-shelf vision-language model. By contrast, most prior VLAs had to include a bespoke action decoder (where we must decide how to represent actions by tokens or embedding) and train it from scratch. CLAP’s language-based interface also leverages the VLM’s pre-trained language understanding: the model already has learned semantics for words describing motion, objects, and relations, which it can now coopt to represent actions. In essence, each discrete action or control primitive is turned into a short phrase (e.g. “reach out and grasp the cup”, “move left”, etc.), and the VLM is trained to match images with those phrases in context. Despite its simplicity, CLAP achieves compelling results. The teammates evaluated CLAP on the LIBERO benchmark for lifelong robot learning – a suite of manipulation tasks where each task comes with visual observations (from workspace and wrist cameras) and a language description of the goal. Remarkably, using a 2-billion-parameter VLM backbone and less than 6 hours of fine-tuning, CLAP attained 90.8% success across the LIBERO tasks. (For comparison, other state-of-the-art VLA methods such as VLA-Adapter report around 99% success on specific LIBERO subsets, but after longer training [~8 hours] with extra model parameters.) This result suggests that CLAP’s language-prompting paradigm can harness the VLM’s knowledge very effectively, even with much less additional training. CLAP’s heavy reliance on language is reminiscent of how large language models use prompt engineering. Here the “prompt” includes not only the task instruction but also an appended action description. For example, if the task instruction is “put the red cube inside the bowl,” CLAP might train on prompts like “Action: grasp the red cube; Instruction: put the red cube inside the bowl.” The VLM’s job (during fine-tuning) is to associate the image of the scene with the correct action phrase. Since the VLM already aligns images with text, adding the right action text biases it to produce that action. Once trained, at test time CLAP can operate by scoring or generating action phrases given the current image and instruction. In this way CLAP sidesteps the need to explicitly output low-level control signals, while still accomplishing the manipulation. In effect, CLAP demonstrates that a VLM can do end-to-end control solely through its language interface. It leverages the same vision-language encoder for understanding both instructions and actions. This stands in contrast to orchestration-based approaches (the “tool-using” style of GPT-4V/Claude) where high-level models call external robot primitives (e.g. ). Instead, CLAP stays in the end-to-end regime: image+language to actions, which means very low inference latency and smooth continuous output since the model is directly deciding actions. In principle, CLAP’s prompts could also be adapted to any discrete-action domain; any task whose action space can be described in words is potentially addressable. Of course, language-prompting has limits. It presumes that all useful actions admit clear textual descriptions. Fine motor primitives at very fine granularity might be hard to verbalize. The current CLAP evaluation on LIBERO relies on rather discrete manipulation steps (“pick up object”, “move object to position”, etc.), which match well to phrases. It’s an open question how well CLAP would scale to tasks needing continuous, high-precision control described by continuous outputs (like torques or high-frequency trajectories). In practice, one might combine CLAP’s high-level planning with a low-level controller, or extend the phrasing to cover sequences of moves. Another consideration is that because CLAP injects actions as language into the prompt, it could inherit the brittleness of language-based models – for instance, small changes in wording might affect outcomes if not robustly learned. Yet the upside is striking. CLAP’s “plug-and-play” nature means we can reuse any next-generation VLM for control with minimal effort. As vision-language models keep improving on perception and grounding, CLAP theoretically hands those gains over to robotic skills automatically. GPUs and time required are tiny: just a few hours of fine-tuning (likely on standard robot datasets) sufficed here. By contrast, previous VLA training regimes often needed thoroughly curated simulations and heavy compute to learn from scratch. Moreover, the approach provides a clear interpretation: one can examine the learned phrasing to understand what “action” the model has latched onto, offering some transparency not present in purely numeric output heads. CLAP’s success also highlights a broader trend: language as a universal interface. Just as CLIP showed that vision-text alignment could enable zero-shot recognition, CLAP shows that vision-language alignment can enable nearly zero-shot action selection. The vision-language model already knows about objects and motion verbs; it just needed a little help associating them with robot affordances. This is reminiscent of how some recent works have used language to bootstrap control – for example, by instructing LLMs to predict state changes or to bridge modalities – but CLAP does so without any external LLM or symbolic planner, purely within the frozen vision-language backbone. In summary, CLAP offers a refreshingly simple recipe: to turn a VLM into a VLA, simply feed it action descriptions as text and fine-tune on a few demonstrations. The result is a robot policy that still “knows” language and vision in a single model, now extended to output actions via language. On the LIBERO benchmark, this yielded 90.8% success after only six hours of post-training, showing that even a modest VLM (2B parameters) can handle complex multi-step instructions once guided by language prompts. While more study is needed on generalization and edge-cases, CLAP opens the door to rapid, language-driven adaptation of powerful pre-trained models for embodied control. Sources: The CLAP concept was demonstrated by the OMRON SINIC-X group (CLAP: Converting Pretrained VLMs to VLAs via Natural-Language Actions). It builds on the notion of vision-language-action models (VLAs) that combine a VLM backbone with an action decoder. Typical VLAs (e.g. Google’s RT-2) require adding a new action head and fine-tuning on large robot datasets; by contrast, CLAP keeps the architecture unchanged and uses language prompts for control. Vision-language models have shown great prior knowledge to leverage (providing strong visual/language priors for new tasks), but normally this comes at the cost of massive demonstration data. CLAP’s innovation is to sidestep that cost. The reported 90.8% success on LIBERO tasks (about 25 simulated tabletop tasks with images and instructions) is notable; for context, contemporaneous work (VLA-Adapter) achieves ~99% on comparable LIBERO tasks with 8h training. This suggests that prompt-based action description is a surprisingly effective way to harvest a VLM’s capabilities for control.