Latest / Embodied AI 101 / When Vision Cannot Know: MulPlanLM Lets Robot Planners Lift, Weigh, and Decide
Transcript
- When Vision Cannot Know: MulPlanLM Lets Robot Planners Lift, Weigh, and Decide. MulPlanLM: multimodal robotic task planning with vision-language models and physical feedback • Young-Chae Son, Dong-Han Lee, Soo-Chul Lim • Department of Mechanical, Robotics and Energy Engineering, Dongguk University • Intelligent Service Robotics, Volume 19, Article 90 • 2026. The problem is not always better visual reasoning. Imagine asking a robot to pick up an opaque container only if something remains inside. The camera can identify the container, estimate its pose, and perhaps read its label. But the pixels do not reveal whether it is empty. No amount of image prompting fixes that observability problem. The robot has to interact with the object—typically by lifting it—and use the resulting force measurement to infer the hidden state. That is the motivating case for MulPlanLM, published on August 19, 2026. The paper asks how an off-the-shelf vision-language or large language model can incorporate not only what the robot sees, but also what the robot physically discovers while acting. Its answer is a hierarchical, three-agent planning pipeline that turns a natural-language goal, detector-annotated images, and wrist force measurements into a structured scene description, a human-readable plan, and finally restricted Python robot-control code. The key word here is planning. MulPlanLM is not an end-to-end vision-language-action policy, a learned tactile representation, or a force controller. It does not predict joint torques or contact trajectories directly. Instead, it treats force-derived information as symbolic evidence available to a language-based task planner. Vision says which object is which and where it is. Force supplies facts such as approximate weight or whether a container appears empty. The planner then reasons over both. That distinction makes the paper more modest than a general-purpose physical foundation model, but also more practically legible. MulPlanLM shows a relatively lightweight way to add physical state information to an existing LLM-based robot stack without collecting a large multimodal training dataset or fine-tuning a new action model. The paper’s main contribution. MulPlanLM contributes three ideas that are best understood together. First, it makes non-visible physical properties first-class planning inputs. Force is not used merely as a low-level collision signal or compliance feedback. It helps answer semantic questions relevant to a task: Is this object heavier? Does this box still contain blocks? Does this container exceed a specified weight threshold?. Second, it divides responsibility among an Extractor, Planner, and Sub-planner. The Extractor converts heterogeneous observations into structured environmental information. The Planner produces an explicit natural-language procedure. The Sub-planner grounds that procedure in object coordinates and a small, validated action API. This creates inspectable interfaces between perception, deliberation, and execution rather than asking one model call to go directly from an image and instruction to arbitrary robot code. Third, the authors evaluate the framework on a real UR5e across 15 tasks, three model configurations, and three sensory conditions: vision plus force, vision only, and force only. This is important because the paper’s central claim is not that force is universally useful. It is that the two modalities cover different failure modes, and that their combination becomes decisive when task-relevant state is only partially observable. The novelty, then, lies more in system architecture and sensory orchestration than in a new neural architecture. MulPlanLM uses existing multimodal and language models, structured prompts, conventional detection and pose-estimation infrastructure, and bounded robot primitives. Its research question is whether this composition is enough to enable useful physical reasoning. How MulPlanLM works. The sensory and execution stack. The physical platform consists of a Universal Robots UR5e arm, its wrist force/torque sensor, an OnRobot 2FG7 parallel gripper, and two Intel RealSense cameras. A RealSense L515 supplies the scene image used by the Extractor, while a D435 supplies object positions for execution. An object detector overlays identifiers and bounding boxes on the image so that the language model can refer to detected instances consistently. This separation between semantic imagery and geometric position estimates is worth noting. The VLM does not have to infer executable metric coordinates from pixels. It reasons about objects using annotated images and structured identifiers, while the Sub-planner receives a separate list of object poses. That reduces the geometric burden on the foundation model and makes the eventual code more deterministic. Force is gathered after the robot grasps and lifts an object. The system samples the three translational force components at 100 hertz for one second and passes their average into the Extractor. Although the UR5e sensor is a six-degree-of-freedom force/torque sensor, the reported model input uses the three force components rather than the torque channels. The averaging operation suppresses high-frequency noise and gives the model a stable summary, but it also discards contact transients and temporal structure. The result is a deliberately heterogeneous pipeline:. Images remain images, augmented with detector output. Positions are maintained as geometric data for code generation. Force measurements are serialized into numerical text. The task goal and all intermediate representations are language or structured JSON. This is therefore late, symbolic multimodal fusion. MulPlanLM does not learn a shared vision-force embedding. Instead, it translates each source into a representation an off-the-shelf language model can consume. Agent one: the Extractor. The Extractor receives the user’s goal, the annotated RGB frame, the averaged force measurement when available, and a reusable set of interpretation guidelines. It produces two structured JSON sections: one describing visible objects and another describing the visible environment. Object records can contain an identifier, name, shape, size, distinguishing features, and an estimated or measured mass. Environment records capture elements such as labeled regions, paper markers, colors, text, and relative locations. Because the output is text rather than a model-specific latent state, the same interface can be handed to different Planner models. The prompt includes several safeguards against casual hallucination. It explains that the number above a detection is an object identifier. It tells the model that either image or mass data may be unavailable. Most importantly, it discourages associating a force measurement with an object unless the robot is known to be holding that object. This is a simple but meaningful attempt to preserve sensor provenance: a stale payload measurement should not become an invented property of whichever object is currently visible. The Extractor is also task-conditioned. It receives the goal rather than blindly transcribing every possible scene detail. That lets it focus on whether the current task depends on color, shape, label text, weight, or contents. In a larger deployment, this kind of task-conditioned perception could reduce prompt size and irrelevant context, although MulPlanLM does not study that efficiency question directly. What the Extractor does not provide is a persistent probabilistic belief state. Its output is a structured snapshot. It can process new feedback at designated points, but the framework does not maintain calibrated confidence over competing hypotheses or a durable object history. That omission becomes important in the longer conditional tasks. Agent two: the Planner. The Planner receives the task goal and the Extractor’s environmental analysis. Its prompt also describes the available high-level capabilities, offers examples of action combinations, and imposes procedural constraints. For example, the Planner is told that the robot can move, control the gripper, and request an environmental or mass check. It is reminded that mass can only be measured while an object is being held. It must divide the solution into numbered steps and lettered substeps, and it is encouraged to avoid comparisons that the task does not require. Its output is a natural-language plan rather than code. A plan might say: identify candidate blocks, approach the first block, grasp and lift it, request a mass measurement, return it, repeat for the remaining candidates, compare the recorded values, and hand over the lightest block. This intermediate representation serves several purposes. It gives the language model room to reason at the task level without simultaneously handling exact poses and Python syntax. It gives a human operator something readable to inspect. And it reduces error propagation by making the intended procedure explicit before it is translated into commands. Still, the Planner’s output is not a formally verified symbolic plan. Preconditions, postconditions, and object-state transitions remain embedded in natural language. The hierarchy improves modularity and debuggability, but it does not provide the guarantees of a classical planner or behavior-tree verifier. Agent three: the Sub-planner. The Sub-planner receives the task goal, the Extractor output, the natural-language plan, low-level execution constraints, and the object-position list obtained from the RGB-D pipeline. It generates Python using a tightly restricted API. The public project description lists operations for moving the tool center point to a target pose, reading the current tool pose, commanding the gripper, and checking a visual or environmental condition. Arbitrary Python and unrestricted robot commands are not permitted. Generated code is statically checked, and invalid formatting or unauthorized calls trigger regeneration, up to three attempts. Commands that pass the language-level check are still subject to controller-level workspace and safety constraints. This is a strong engineering choice. Language models are substantially easier to contain when they select and compose a small vocabulary of vetted capabilities rather than writing directly against a full robot SDK. The approach is analogous to giving an agent a narrow tool interface with typed arguments. It is also a limitation. The robot can perform sequences of predefined motions and sensing actions, but it cannot invent a new compliant manipulation strategy or synthesize a continuous force-control policy. The expressivity of the whole system is bounded by the hand-engineered primitives. Feedback, but not full closed-loop replanning. The system diagram contains a feedback loop: after execution, new camera and force observations can return to the Extractor. Generated code can explicitly request a condition check, allowing subsequent branches to depend on what the robot observes or measures. This makes MulPlanLM more than a purely open-loop code generator. It can include active sensing and conditional execution. But it should not be confused with continuous receding-horizon replanning. The authors themselves identify explicit state tracking, postcondition verification, and closed-loop replanning as future improvements. In the current framework, feedback is mainly consumed at planned checkpoints, and failures can arise when generated code does not update or propagate intermediate conditions correctly. The distinction matters. Code regeneration handles malformed programs before execution. A condition query can handle a known branch during execution. Neither mechanism necessarily recovers from an unexpected physical outcome that was absent from the original plan. Force measurement as an epistemic action. MulPlanLM’s most interesting conceptual move is to treat manipulation as a way of asking the environment a question. A robot that lifts a box is not always trying to relocate it. It may be performing an epistemic action: an action whose purpose is to reduce uncertainty. The subsequent force reading reveals something that was unavailable from passive observation. The robot can then choose its task action based on the new information. Consider the paper’s box-sorting example. Two sheets of paper are labeled to indicate “empty” and “remaining.” The robot must place a box on the appropriate sheet. Vision supplies the box pose and the semantics of the two labels. Lifting the box supplies evidence about whether it contains anything. The multimodal system succeeds because it can connect those facts. Vision alone knows where the alternatives are but not the box state. Force alone can suggest the state but cannot reliably interpret the visual labels. This is a useful pattern beyond weighing. A robot might squeeze an object to estimate compliance, pull on a drawer to determine whether it is locked, probe a surface to infer friction, or monitor motor current to detect obstruction. MulPlanLM’s broader lesson is that planners should be able to select sensing actions whose value lies in revealing hidden task state. Experimental design. The authors construct 15 real-world tasks, split evenly among vision-based, force-based, and genuinely multimodal tasks. Every task is evaluated with five repeated trials under each model and sensory condition, with success scored as a binary completion of a predefined mission. With three models and three input conditions, the design implies 675 robot trials. The three model configurations are:. GPT-4V for visual extraction paired with GPT-4.1 for planning and code generation. Gemini 2.5 Pro. Llama 4 Maverick. The prompts remain consistent across tasks. The comparison is therefore primarily an inference-time evaluation of different foundation models inside the same agent structure, not a comparison of separately trained robot policies. The vision group tests shape and size selection, visual matching, color similarity and difference, and semantic object knowledge—for example, choosing an appropriate ball for golf. The force group tests hidden quantities such as the number of metal balls in a container, whether contents remain in an opaque package, whether blocks have been removed, and which of two visually similar blocks is heavier. One task deliberately moves the target outside the useful camera view, making physical feedback necessary for the subsequent decision. The multimodal group combines visual identity with physical state. Tasks include placing containers according to labels on paper, visually finding blocks before weighing them, applying different weight rules depending on block color, and lifting multiple bolt-filled containers to identify the lighter one. These are not merely tasks where either modality would suffice; their decision rules explicitly join appearance or text with force-derived state. The sensory ablations need to be interpreted carefully. “Force only” does not mean that the robot operates without execution geometry. The Sub-planner still needs object positions to reach and grasp targets, and the authors describe this condition as combining positional and force cues. The ablation removes semantic image information from the reasoning pipeline rather than turning the robot into a completely blind force controller. The setup also treats object detection as reliable. There is no separate detector benchmark, and the scenes are controlled tabletop arrangements. Consequently, the experiment is mainly testing multimodal reasoning, conditional planning, and code execution—not open-world perception. What the results show. The headline overall results from the paper’s Figure 6 are as follows:. | Model configuration | Vision + force | Vision only | Force only | |---|---:|---:|---:| | GPT-4V + GPT-4.1 | 77.3% | 45.3% | 17.3% | | Gemini 2.5 Pro | 77.3% | 54.7% | 20.0% | | Llama 4 | 69.3% | 45.3% | 17.3% | | Average across models | 74.6% | 48.4% | 18.2% |. Thus, averaged across the three displayed model configurations, providing both modalities raises success by about 26 percentage points over vision only and 56 points over force only. GPT and Gemini tie on the reported multimodal aggregate, while Gemini is strongest in the vision-only condition. The authors caution that these model differences are specific to this prompting and robot setup and should not be interpreted as a general foundation-model leaderboard. The category-level averages make the mechanism clearer:. | Task category | Vision + force | Vision only | Force only | |---|---:|---:|---:| | Vision-based tasks | 80.0% | 78.7% | 9.3% | | Force-based tasks | 72.0% | 25.3% | 40.0% | | Multimodal tasks | 69.3% | 41.3% | 2.7% |. On vision-based tasks, force adds almost nothing on average. In fact, its effect varies by model: GPT improves from 76% to 84%, Gemini remains at 80%, and Llama drops from 80% to 76%. That is a healthy result because it shows that multimodality is not automatically beneficial. An irrelevant modality can be neutral or even mildly distracting. The situation reverses on force-based tasks. Vision-only performance averages 25.3%, because appearance cannot reveal hidden counts, payloads, or small weight differences. Force alone reaches 40%, showing that physical feedback contains useful task information. Combining force with vision reaches 72%, because the system can associate the measurement with the correct object and preserve the scene context needed to act. The strongest evidence for complementarity comes from the multimodal group. Vision plus force averages 69.3%, compared with 41.3% for vision alone and only 2.7% for force alone. Gemini reaches 80% in this category with both modalities, versus 52% with vision and 4% with force. GPT reaches 68%, versus 40% and 4%, while Llama reaches 60%, versus 32% and zero. Force-only performance collapses here because weight does not identify labels, colors, object categories, or target regions. Conversely, vision can sometimes exploit priors or accidentally choose the correct branch, explaining why vision-only performance remains above chance on some multimodal tasks. But only the combined condition has access to the information demanded by the task specification. Where the system fails. The framework is strongest on single-object judgments and simple thresholds. Performance degrades when the robot has to update state after several intermediate actions, maintain values for multiple objects, execute deep conditional branches, or compare a sequence of nearly identical measurements. The paper identifies the heavier-of-two-blocks task as particularly difficult because the weight gap is small. More broadly, the authors report that many failures arise in generated conditional logic: a measurement may be obtained correctly but not carried through to the right later branch. In other words, the bottleneck often shifts from sensing to procedural state management. That is an important finding. Adding a sensor can solve observability without solving memory. Once a robot has weighed three objects, the planner needs an explicit record of which measurement belongs to which object, which comparisons remain unresolved, and which postconditions have been achieved. Natural-language context can approximate that bookkeeping, but it is not a substitute for a structured task state. Execution time follows the same complexity gradient. The project page estimates roughly 15 seconds for vision-based pick-and-place tasks, 30 to 40 seconds for force-based tasks, and around a minute for multimodal tasks. These are video-based estimates rather than instrumented timing results, so the paper does not separate model inference, code generation, sensing, robot motion, and recovery latency. One reporting detail is also worth flagging for reproduction. The category-level model percentages on the project page do not exactly reconstruct the aggregate bars in Figure 6 under a straightforward equal-weight calculation, even though each category contains the same number of tasks and trials. For example, the listed GPT multimodal-condition category scores average to 74.7%, whereas Figure 6 reports 77.3%. This may be a project-page transcription or aggregation detail, but a reproducibility package should clarify the underlying trial counts. Where MulPlanLM sits relative to prior work. SayCan grounded language-model plans by scoring proposed skills with learned value functions, constraining semantic plausibility with embodiment-specific feasibility. MulPlanLM takes a different route: its low-level actions are already restricted, while physical sensing supplies missing state information. SayCan asks whether the robot can execute a skill; MulPlanLM often asks what the robot must physically measure before choosing one. Inner Monologue showed that language models could use scene descriptions, success signals, and human feedback to update embodied plans. MulPlanLM follows the same broad closed-loop philosophy but extends the feedback vocabulary with numerical force-derived evidence. Its limitation is that the update mechanism remains less systematic than a persistent state estimator or full replanning loop. The closest conceptual precedent may be Interactive Planning for Partially Observable Robotic Tasks, which explicitly has an LLM direct information-gathering actions, infer hidden state from observations, and continue planning. MulPlanLM instantiates that idea in a concrete vision-and-force manipulation setting, with lifting and weighing as information-gathering actions. At the representation level, PaLM-E learns to inject continuous sensor encodings directly into an embodied language model. MulPlanLM instead converts force measurements into structured text and uses commercial or open foundation models without multimodal retraining. That makes the system easier to assemble and swap across models, but its fusion is semantically shallow compared with a jointly trained sensor-language representation. The modular composition resembles Socratic Models, in which independently pretrained models communicate through language to acquire new multimodal capabilities. Meanwhile, VoxPoser represents the opposite end of the execution spectrum: it uses language and vision models to compose spatial value maps and generate dense, closed-loop trajectories. MulPlanLM sacrifices that geometric expressivity in favor of a small, inspectable action API. Finally, frameworks such as ReplanVLM place explicit error correction and task replanning at the center of the architecture. MulPlanLM validates code before execution and can query conditions during a plan, but robust postcondition monitoring and recovery remain future work. What the evaluation does not yet establish. The central ablation compares sensory inputs, not architecture. There is no reported comparison against a single-agent multimodal prompt, direct multimodal code generation, a two-stage planner, or a classical finite-state controller given the same sensor summaries. The experiments therefore support the value of complementary physical information inside MulPlanLM, but they do not isolate how much of the gain comes specifically from the three-agent hierarchy. Likewise, there is no ablation of force processing. The study does not compare raw samples with averaging, force vectors with a calibrated mass scalar, or three-axis force with full force/torque histories. Averaging one second of data is sensible for tabletop weighing, but it narrows “physical feedback” to approximately static payload inference. Contact timing, slip, compliance, friction, and dynamic interaction remain outside the demonstrated scope. The experiments use controlled object layouts, a known workspace, predefined motion primitives, and dependable object detection. Fifteen handcrafted tasks are enough to show the phenomenon but not to establish broad generalization. Five repetitions per cell also leave substantial uncertainty for a stochastic planning stack, particularly when the outcome is binary. The success metric combines several failure sources into one number. A failed trial might result from visual interpretation, force ambiguity, an incorrect high-level plan, bad object-measurement association, code generation, motion execution, or a condition that was not updated. The qualitative analysis identifies some of these, but a stage-wise error taxonomy would make the scientific conclusions sharper. Safety is similarly bounded rather than comprehensive. API whitelisting, static checks, retries, and workspace constraints are valuable defenses. They can prevent malformed code and out-of-bounds commands. They cannot guarantee that a semantically valid action is appropriate—for example, that the selected object is safe to lift or that a sequence will not destabilize a neighboring object. Most importantly, MulPlanLM still lacks an explicit world state with durable object identities, measurement histories, preconditions, and postconditions. The paper’s failures suggest that this—not broader language knowledge—may be the next major performance lever. How roboticists could build on it. A practical successor to MulPlanLM would preserve its modularity while making the interfaces less dependent on free-form language. First, place a structured state estimator between the Extractor and Planner. Each object should have a persistent identifier, pose, grasp state, measurement history, uncertainty, and verified properties. The LLM could propose updates, but deterministic code should own the state transition. Second, represent sensing actions explicitly. Rather than hiding a mass check inside natural-language steps, define actions such as , , and , each with known preconditions, expected observations, timeout behavior, and uncertainty estimates. The planner can then reason about information gathering as a real action class. Third, attach verifiers to every stage. Validate the Extractor’s JSON schema, compile the high-level plan into a behavior tree or temporal logic representation, type-check object references in generated code, and verify postconditions after each physical action. A failed postcondition should trigger local recovery or replanning, not simply allow the original program to continue. Fourth, improve physical summaries before giving them to an LLM. Gravity-compensated mass estimates, confidence intervals, grasp stability indicators, and comparisons such as “object A is probably heavier than object B” are likely more reliable than asking a general model to reason directly over noisy force components. Temporal force encoders could be added when the task depends on slip or contact dynamics. Finally, evaluate against simpler baselines. A compelling follow-up would compare the three-agent system with a single multimodal model call, a classical state machine, an LLM plus symbolic planner, and an end-to-end VLA. Difficulty should be varied systematically through weight gaps, occlusion, lighting, detector noise, branch depth, and the number of objects that must be remembered. The bottom line. MulPlanLM makes a straightforward but consequential argument: a robot planner should not be limited to properties visible in an image. Some task states can only be discovered through physical interaction, and those discoveries need to participate in high-level reasoning. The paper’s strongest evidence is the task-dependent pattern of its results. Force provides almost no average gain when vision already contains the answer. It provides a large gain when weight or hidden contents matter. And when the task requires both visual semantics and physical state, neither unimodal condition comes close to the combined system. MulPlanLM is not yet a general physically grounded agent. Its force representation is simple, its workspace is controlled, and its feedback loop lacks persistent state and comprehensive replanning. But it demonstrates a useful design principle for embodied AI: when the world is partially observable, the planner must be able not only to look and act, but to act in order to learn what is true.