Latest / Embodied AI 101 / Language Is Not Just a Prompt: Four Ways Robots Turn Words into Action

Transcript

Language Is Not Just a Prompt: Four Ways Robots Turn Words into Action · Embodied AI 101

• Language Is Not Just a Prompt: Four Ways Robots Turn Words into Action. Bridging language and action: A survey of language-conditioned robot manipulation • Xiangtong Yao, Hongkuan Zhou, Oier Mees, Yuan Meng, Ted Xiao, Yonatan Bisk, Jean Oh, Edward Johns, Mohit Shridhar, Dhruv Shah, Jesse Thomason, Kai Huang, Joyce Chai, Zhenshan Bing, and Alois Knoll • Technical University of Munich, Bosch, University of Stuttgart, UC Berkeley, Microsoft, Google DeepMind, Carnegie Mellon University, Imperial College London, Princeton University, USC, Sun Yat-sen University, University of Michigan, and Nanjing University • The International Journal of Robotics Research, OnlineFirst • 2026. The Paper in One Sentence. Tell a household robot, “Please tidy the desk,” and almost every unsolved problem in embodied AI appears at once. The robot must infer what counts as untidy, identify the relevant objects, understand which ones may be moved, choose appropriate destinations, sequence its operations, generate collision-free motions, regulate contact, detect failures, and decide when the desk is tidy enough. The sentence specifies intent, but it does not uniquely specify a reward function, symbolic plan, trajectory, controller, or stopping criterion. That gap is the subject of Bridging language and action. Its central contribution is not another chronological history of language grounding or another catalog divided into reinforcement learning, imitation learning, and foundation models. Instead, the authors organize the field according to the functional role language plays inside the manipulation loop: language can evaluate states, condition a policy, support planning and reasoning, or participate directly in a unified vision-language-action model. The survey then compares approaches along five engineering axes: action granularity, data and supervision, cost and latency, environment and evaluation, and task specification. That framing is useful because language-conditioned manipulation is no longer a single research problem. It is a family of architectures that place language at very different interfaces. In one system, language is compiled into a reward. In another, it is an embedding fed into a visuomotor policy. Elsewhere, it is an internal planning representation. In a modern VLA, language, images, proprioception, and actions may all be processed by one trainable network. Those systems may accept the same instruction while solving fundamentally different learning problems. The publication history reflects how rapidly this landscape has changed. The manuscript first appeared on arXiv on December 17, 2023, and reached version seven on June 22, 2026. The peer-reviewed article was first published online by The International Journal of Robotics Research on July 29, 2026. During that period, its taxonomy evolved from categories centered on reward shaping, policy learning, neuro-symbolic AI, and foundation models into the more durable, role-based organization used in the final article. Why Language Is a Systems Problem, Not an NLP Feature. Language is attractive as a robot interface because it is compositional, accessible, and remarkably information-dense. A user can describe objects, attributes, spatial relations, temporal order, preferences, prohibitions, and hypothetical conditions in one utterance. “Put the clean cups in the upper cabinet, but leave my blue mug on the counter” combines category recognition, state estimation, coreference, exclusion, spatial reasoning, and a user-specific preference. But language does not live in the robot’s control space. Manipulation ultimately requires geometry, timing, contact, force, and feedback. Words such as “carefully,” “slightly,” “securely,” or “near” are meaningful to a person because they are interpreted through shared physical and social experience. A robot must turn them into measurable constraints. Even seemingly concrete instructions leave questions unanswered: which coordinate frame defines “left,” how close counts as “next to,” whether “open the drawer” means ten centimeters or its mechanical limit, and whether speed or peak force determines “gently.”. Language-conditioned manipulation therefore contains at least three grounding problems. There is referential grounding, which associates words with entities and regions in the scene. There is task grounding, which turns an instruction into goals, constraints, subgoals, or rewards. And there is motor grounding, which connects those goals to actions that a particular embodiment can execute. These problems are coupled. A policy cannot grasp “the small red cup” if perception does not isolate the correct instance. A planner cannot reason about pouring if its skill library lacks a controllable pour primitive. A language model may produce a semantically plausible plan that is physically impossible because the gripper cannot reach the object or because the proposed grasp will not support the required force. The survey’s most important conceptual move is to ask not simply whether language is present, but where language exerts causal influence over behavior. The Taxonomy: Score, Steer, Think, or Act. A compact way to remember the paper’s taxonomy is that language can score, steer, think, or act. First, language can score states and trajectories. It may define a reward, cost, value map, success condition, or preference signal. The language does not directly command the motors; it tells an optimizer what outcomes should be desirable. Second, language can steer a learned policy. The policy receives the current observation and instruction and predicts an action. Language becomes part of the policy’s conditioning context, alongside images, depth, proprioception, or action history. Third, language can help the robot think. An LLM, semantic parser, knowledge graph, program generator, PDDL planner, or behavior tree decomposes a task, selects skills, checks preconditions, and revises the plan. Finally, language can be integrated into a model that directly acts. Unified VLA models adapt vision-language backbones so that robot actions are generated alongside—or in a representation analogous to—language tokens and other multimodal outputs. These are not mutually exclusive boxes. SayCan combines language-model planning with learned affordance values. VoxPoser uses language reasoning to construct spatial value maps. A VLA may generate a verbal subgoal before emitting continuous actions. A deployed robot may use all four roles at once. The taxonomy should therefore be read as a description of the dominant interface through which language influences control, not as a claim that every system belongs to exactly one category. Route One: Language as an Evaluator. The first route treats language as a specification of what should happen, rather than a direct prescription of how to move. In reinforcement learning, that often means translating an instruction into a reward signal. Earlier approaches learned associations among language, observations, demonstrations, and task progress. A model might learn that “push the red block toward the blue bowl” corresponds to decreasing one spatial distance while maintaining constraints on the other objects. In inverse reinforcement learning variants, demonstrations help recover the objective implied by the instruction. Foundation models have expanded this idea in two directions. LLMs can generate executable reward code, while VLMs can serve as learned critics that judge whether an image, video, or trajectory satisfies a language goal. The survey’s associated repository groups methods such as Text2Reward, Eureka, Video-Language Critic, ReWiND, and language-to-reward systems under this broad role. Text2Reward, for example, asks an LLM to synthesize dense reward functions from natural-language task descriptions and environment APIs. In its reported manipulation experiments, policies trained with generated rewards matched or exceeded expert-written rewards on many of the tested tasks. Eureka similarly treats an LLM as a reward-code designer that can iteratively improve its proposals using training statistics. A closely related strategy maps language into cost fields or value maps instead of a single scalar reward. VoxPoser is a representative example. Given RGB-D observations and an instruction, it combines LLM-generated programs with visual-language grounding to construct three-dimensional affordance and constraint maps. A motion planner then optimizes within those maps. Language might create attraction around a drawer handle, repulsion around obstacles, an orientation constraint on the end effector, and a target region corresponding to “half open.”. This separation is attractive when a team already has competent low-level controllers, trajectory optimization, model-predictive control, or reinforcement-learning infrastructure. The language component can specify the objective without having to relearn control from scratch. Rewards and value maps can also be easier to inspect than the latent conditioning of a monolithic policy. But an evaluator is only useful if it measures the right thing. A language-generated reward may contain exploitable shortcuts, omit a safety constraint, or assign high value to visually plausible but mechanically unstable states. A VLM critic may recognize that a mug appears inside a sink while missing that it was dropped hard enough to break. Dense rewards can accelerate learning while quietly changing the intended task. And if the same imperfect evaluator generates training labels and later reports success, evaluation becomes circular. For practitioners, this route is most compelling when the hard problem is objective engineering, while perception and control are comparatively mature. It is less compelling when success depends on contact states or hidden physical variables that a language or vision model cannot reliably observe. Route Two: Language as a Policy Condition. The second route feeds language directly into the policy. In words, the model learns: given the current observation, recent history, and instruction, produce the next action or action sequence. This is the most straightforward formulation of language-conditioned behavior, but it covers several different learning regimes. In language-conditioned reinforcement learning, one policy can represent a family of tasks. Language may identify the active goal, support hindsight relabeling, or provide a structure over related tasks. Experience gathered for one instruction can sometimes improve another because the visual and motor representations are shared. The difficulty remains the familiar robotics bottleneck: online exploration is expensive, sparse rewards are difficult, and a semantically meaningful instruction does not make unsafe physical exploration acceptable. Imitation learning avoids much of that exploration by pairing instructions with expert trajectories. CLIPort is a canonical example: an end-to-end imitation-learning system with separate semantic “what” and spatial “where” pathways, trained to solve multiple tabletop tasks from language-conditioned demonstrations. Its architecture illustrates an enduring principle—the representation that recognizes a requested object need not be identical to the representation that localizes a precise pick or place action. Other work trains on unstructured play, teleoperation, or retrospectively labeled trajectories. This is important because collecting free-form behavior is often easier than requesting a balanced set of demonstrations for every possible instruction. A trajectory in which a person opens a drawer, retrieves an object, and closes the drawer can be segmented and assigned several descriptions at different temporal scales. Foundation models can help generate or normalize those labels, although synthetic linguistic variety cannot replace missing physical behavior. Benchmarks such as CALVIN focus on the resulting long-horizon composition problem. A policy must execute sequences of language-specified skills rather than isolated pick-and-place episodes, exposing errors in memory, task completion detection, and transitions between skills. The survey also tracks RLBench, Meta-World, VIMAbench, LIBERO, and real-world datasets such as Open X-Embodiment and DROID, each covering different embodiments, sensors, action spaces, and horizons. Diffusion policies add another important ingredient. Manipulation is intrinsically multimodal: there may be several valid grasps, approach directions, or collision-free trajectories for the same instruction. A regression policy trained with a simple point estimate can average these alternatives into an action that belongs to none of them. Diffusion-based policies instead learn to generate samples from a structured action distribution, often predicting temporally coherent chunks rather than one command at a time. The survey places language-conditioned diffusion, PlayFusion, multimodal diffusion transformers, goal-conditioned score models, and generalist policies such as Octo in this broader policy-conditioning family. These methods can model diverse demonstrations and smooth action sequences, but iterative denoising and long action chunks introduce latency and reactivity trade-offs. A chunk that looked sensible when generated may become inappropriate after an object slips. The central weakness of direct conditioning is that successful behavior can conceal shallow grounding. A policy may associate particular words with familiar backgrounds, object locations, or demonstration styles. Paraphrase robustness does not imply robustness to new geometry; visual generalization does not imply understanding of negation; success on individual skills does not imply reliable composition. For this family, evaluation should independently vary language, objects, layouts, dynamics, and temporal structure. Otherwise, “language generalization” may only mean that a frozen text encoder maps two similar sentences to nearby embeddings. Route Three: Language as a Medium for Planning and Reasoning. The third route inserts an explicit cognitive layer between instruction and control. Before foundation models, this often meant semantic parsing and symbolic planning. A language command was converted into predicates, object relations, goal states, or a program. A task planner searched over actions with defined preconditions and effects, while a motion planner checked geometric feasibility. Knowledge graphs and scene graphs supplied relations that were difficult to infer from raw pixels alone. The appeal is auditability. A robot can expose a plan such as: locate the mug, verify that it is empty, grasp the handle, open the dishwasher, place the mug in the upper rack, and close the door. Preconditions and forbidden transitions can be checked before execution. Symbolic representations also support explicit negation and logical constraints more naturally than many end-to-end policies. The failure mode is the interface between symbols and reality. A logically valid plan is useless if “mug graspable” is produced by an unreliable detector, if a skill’s effects are poorly modeled, or if the planner assumes that every object labeled “drawer” opens in the same way. Hand-designed ontologies and skill libraries can become deployment bottlenecks. LLMs relax some of that brittleness by providing broad priors over task structure and common-sense sequencing. SayCan is an influential example. It combines a language model’s estimate that a skill is useful for an instruction with a learned value function estimating whether the robot can successfully execute that skill in its current state. The language model proposes what would make semantic sense; the affordance model grounds the proposal in the robot’s capabilities. Code as Policies moves from selecting skills to generating executable programs. Few-shot examples teach an LLM how natural-language commands correspond to robot APIs, perception calls, geometric computations, and control routines. Because the output is code, the system can compose loops, conditionals, calculations, and reusable functions rather than producing only a flat textual plan. ProgPrompt similarly uses program-like structures to generate situated task plans. The distinction between open-loop and closed-loop reasoning is crucial. An open-loop planner generates a complete plan once and hopes execution follows its assumptions. A closed-loop system repeatedly observes the environment, checks progress, summarizes failures, and replans. The survey groups modern approaches according to planning, prompt engineering, iterative reasoning, feedback, and combinations of LLMs with PDDL or behavior trees. VLMs make this reasoning more situated. Contrastive visual-language models can ground open-vocabulary objects and regions, while generative models can produce textual scene descriptions, answer questions about task state, or imagine goal images. A VLM-based success detector might answer whether a drawer is open; an image-editing model might transform the current scene into a visual subgoal; a controller can then attempt to realize that transformation. The main lesson is that language-model reasoning becomes useful only when connected to three disciplined interfaces: an updated state representation, an explicit inventory of executable capabilities, and a verification mechanism. Without them, an LLM produces descriptions of actions rather than robot actions. Fluent decomposition should never be mistaken for reachability, stability, or control authority. Route Four: Unified Vision-Language-Action Models. The fourth route tries to absorb the interfaces into a unified trainable model. RT-2 established the modern VLA framing by adapting vision-language models so that robot actions could be represented and predicted in a token-like form. The model was trained using both web-scale vision-language knowledge and robot trajectories, allowing semantic concepts acquired from Internet data to influence robotic control. OpenVLA made this direction more accessible by releasing an open-source VLA trained on diverse robot demonstrations. Its architecture adapts pretrained visual and language components to action prediction, demonstrating that a general-purpose policy can be built from openly available foundation-model components and cross-embodiment data rather than an entirely proprietary stack. More recent systems explore alternatives to simple discrete action tokenization. The survey distinguishes autoregressive prediction, direct regression, diffusion, and flow matching. Physical Intelligence’s π0, for example, uses a flow-based action model to generate continuous action trajectories, targeting the fine-grained multimodality that is awkward to represent with a small discrete vocabulary. The survey organizes current VLA optimization around several fronts. On the perception side, researchers are expanding data sources, improving three-dimensional grounding, and adding heterogeneous sensing. RGB is often sufficient for semantic recognition but not contact-rich manipulation. Depth, proprioception, force, and tactile signals reveal physical states that language and images alone leave ambiguous. The associated literature already includes tactile-language-action and force-aware VLA variants. On the reasoning side, the challenge is to preserve the broad knowledge of the pretrained VLM while learning robot control. Aggressive fine-tuning can improve action prediction while degrading open-world recognition and reasoning. Proposed remedies include modular adapters, memory mechanisms, explicit chains of subgoals, future prediction, and integration with world models. On the action side, the field is searching for representations that are expressive, efficient, and transferable. Discrete tokens integrate naturally with language-model training, but quantization can discard control precision. Continuous diffusion or flow models preserve multimodality but require specialized heads and inference procedures. Action chunks improve temporal coherence and reduce repeated decoding, but longer chunks can reduce feedback responsiveness. On the adaptation side, generalist pretraining is only the beginning. A deployable model must be fine-tuned efficiently for a new robot, camera configuration, gripper, environment, and task distribution. The survey tracks parameter-efficient adaptation, object-centric fine-tuning, reinforcement-learning post-training, and methods that preserve foundational knowledge while learning new motor behaviors. It is worth being precise about the word “unified.” Many VLAs still contain pretrained vision encoders, a language-model backbone, projection layers, robot-state embeddings, and a separate action decoder. They are unified in training objective and inference pipeline, not necessarily homogeneous in architecture. More importantly, semantic scale and motor scale are not equivalent. Web data can teach a model what a colander is and why someone might place pasta in it. It does not teach the friction coefficient of a particular plastic colander, the compliance of a gripper pad, or the torque required to lift it when wet. VLA progress depends on connecting broad semantic priors to sufficiently diverse physical experience. The Five Axes That Decide Whether a System Is Useful. The paper’s four-way taxonomy explains what language does. Its five comparative axes explain whether a method is practical. Action granularity is the first axis. High-level systems output skill names or subgoal sequences; low-level systems output poses, velocities, joint targets, or torques. High-level actions make planning and language grounding easier, but performance is bounded by the skill library. Low-level actions offer flexibility and end-to-end adaptation, but demand much more data and make safety harder to verify. Hierarchical systems are attractive precisely because language and control operate at naturally different time scales. Data and supervision form the second axis. Reward-based systems require interaction or a dynamics model. Behavioral cloning requires demonstrations. Planners need skill definitions, environment APIs, or symbolic annotations. VLAs need heterogeneous trajectories aligned with language and observations. Internet data provides semantic diversity, but robot data remains the scarce source of contact and control knowledge. Synthetic captions can multiply descriptions of known behavior; they cannot create physical behavior absent from the dataset. System cost and latency are the third axis. A cloud-hosted LLM can spend seconds constructing a task plan, while a local force controller may need to react within milliseconds. Treating both as one inference problem is often unnecessary. A practical design may use slow semantic reasoning asynchronously and a compact policy for high-frequency control. Model size, network reliability, memory footprint, action-decoding iterations, and replanning frequency should therefore be reported alongside task success. Environment and evaluation are the fourth axis. CALVIN, RLBench, Meta-World, VIMAbench, LIBERO, and real-world datasets differ substantially in embodiment, reset conditions, sensing, action representation, language diversity, and horizon. The survey’s benchmark tables make the fragmentation visible, but they also show why headline success rates are rarely comparable. A meaningful evaluation should test more than whether a model completes familiar episodes. It should measure sensitivity to paraphrases and ambiguous instructions; generalization across objects, layouts, and embodiments; recovery after perturbations; calibration of failure detection; and compliance with negative constraints such as “do not touch the knife.” Long-horizon evaluation should distinguish planning errors from perception, transition, and low-level control errors. The final axis is task specification. Language is not automatically superior to goal images, videos, demonstrations, or structured task graphs. Language communicates intent and abstraction well, but often underspecifies geometry. A goal image provides concrete visual structure but may not reveal which relations matter or which intermediate states are acceptable. A demonstration conveys motion style and contact strategy but may entangle task essentials with one operator’s embodiment. The strongest interfaces will likely be multimodal. A user might say, “Arrange these like the example, but keep the fragile bowl on top,” combining a visual target with a linguistic exception. Language is best understood as one complementary goal representation, not a universal replacement for every other form of task specification. The Central Debates Behind the Taxonomy. The survey does not present the field as converging on one winning architecture. Instead, it reveals several persistent tensions. The first is modularity versus end-to-end learning. Modular systems expose plans, maps, rewards, and constraints. Components can be replaced and failures localized. End-to-end policies can co-adapt their representations and avoid brittle hand-designed interfaces. But this is a spectrum rather than a binary choice. A VLA with a separate action head is already modular internally; a symbolic planner whose predicates come from opaque neural networks is not fully interpretable. The second is whether learned world models can provide the predictive ability needed for robust long-horizon manipulation. A world model could let the robot evaluate candidate actions before executing them, generate synthetic experience, and reason about delayed consequences. Yet small errors in contact, friction, deformable objects, or occluded state can compound over long imagined rollouts. A visually convincing predicted future is not necessarily a dynamically valid one. The paper’s surrounding discussion identifies this as a central unresolved debate rather than a settled route to generality. The third tension is scaling versus real-time control. Larger models absorb more semantic and visual knowledge, but manipulation is constrained by deadlines, energy, onboard compute, and communication reliability. The likely answer is not simply a smaller or larger monolith. It is architectural separation between deliberative reasoning, predictive planning, reactive visuomotor control, and hard real-time safety loops. What the Survey Gets Right—and What It Cannot Resolve. The role-based taxonomy is the paper’s strongest contribution. It remains useful even as particular model families change. A new foundation model can still be analyzed by asking whether it defines an objective, conditions a controller, performs deliberation, or generates actions. That is more durable than organizing a survey around fashionable architecture names. The framework also encourages hybrid design. The four categories are not competing research tribes. They are possible layers of one robot system. A language-conditioned critic can monitor a VLA; an LLM planner can call diffusion-based skills; a symbolic safety layer can reject an otherwise plausible neural plan. The unavoidable limitation is breadth. The survey spans reward learning, reinforcement learning, imitation learning, diffusion, neuro-symbolic planning, LLMs, VLMs, VLAs, world models, and heterogeneous benchmarks. These areas use incompatible assumptions and evaluation protocols. A comparison table can expose those differences, but it cannot turn them into controlled experimental evidence. And the field moves faster than conventional survey publication. This paper required seven arXiv versions between late 2023 and mid-2026, while its companion repository continues tracking new reward models, multimodal VLAs, world-action models, adaptation techniques, and benchmarks. It should be used as a conceptual map and living bibliography, not as a permanent leaderboard. The Research Agenda: From Fluent Commands to Dependable Behavior. The first priority is factorized generalization. Researchers should report linguistic, semantic, visual, spatial, dynamical, temporal, and embodiment generalization separately. A system that handles new nouns may fail on new spatial relations. A system that follows paraphrases may still fail when the same task requires a different grasp strategy. Second, robots need interactive grounding. Natural human instruction is ambiguous, incremental, and corrective. A dependable robot should ask which cup the user means, confirm dangerous interpretations, accept mid-execution corrections, and explain why a request is infeasible. Treating language as a one-shot prompt discards much of what makes language valuable for collaboration. Third, uncertainty must become a first-class control signal. Language models and VLMs should not merely output plans and success judgments; they should expose calibrated uncertainty that determines when the robot observes again, switches strategies, requests help, or stops. Runtime monitors should evaluate progress independently of the policy rather than trusting the same model that selected the action. Fourth, safety must combine semantic and physical constraints. A model needs to understand that a knife is hazardous, but it also needs verified collision limits, force thresholds, reachable sets, and emergency stopping. The survey explicitly identifies generalization and safety as central open problems. Neither can be solved by linguistic alignment alone. Finally, future systems must bridge semantic scale with physical diversity. Cross-embodiment datasets, human video, simulation, synthetic labels, and Internet knowledge all help, but their action representations do not align automatically. Progress will depend on learning what can be shared—task intent, object affordances, subgoals, state transitions—while keeping embodiment-specific control grounded in the robot’s actual kinematics and sensors. A Practical Architecture Suggested by the Survey. For many labs, the most promising near-term system is not a pure instance of any one category. A visual-language model can ground the instruction and produce object-centric scene representations. An LLM or neuro-symbolic planner can turn the request into a monitored skill graph. A VLA or diffusion policy can execute each contact-rich skill using local visual, tactile, and proprioceptive feedback. A separate language-conditioned critic can assess task progress. Hard-coded or formally verified safeguards can enforce forces, collisions, forbidden regions, and stop conditions. That design uses language in all four roles: as an evaluator, a policy condition, a reasoning representation, and part of a unified action model. It also keeps each role at an appropriate time scale. The question for a new project is therefore not, “Should we use an LLM, diffusion policy, or VLA?” It is: Where is task information being lost between the user’s intent and the robot’s motors, and what role should language play at that boundary?. That is the enduring value of Bridging language and action. It replaces a vague ambition—make robots understand language—with a much sharper systems question: where, exactly, should words enter the loop so that they change physical behavior in a useful, verifiable, and safe way?.