Latest / Embodied AI 101 / When the Command Disappears: What Counts as Robot Intelligence?
Transcript
- When the Command Disappears: What Counts as Robot Intelligence?. Robot Independent Intelligence: When Language Vanishes in Vision-Language-Action Models • Kai Gui, Shun Gui, Yan Luximon • IEEE Robotics and Automation Letters • 2026. Research scope, October 3, 2026: I located a bibliographic record reproducing the paper’s abstract, but could not retrieve its full text through IEEE Xplore or find an accessible author manuscript. I therefore cannot verify its detailed algorithms, experimental tables, supplementary material, or code. This article distinguishes the paper’s reported claims from evidence in related primary research and from my own technical analysis. It is not a substitute for a full-text methodological review. The harder question is what to do. Imagine a robot facing a table with a mug, a plate, and a partly open drawer. With the instruction “put the mug in the drawer,” the immediate research problem is recognizable: identify the objects, interpret the spatial relationships, find a feasible grasp, and execute the requested transformation. Now remove the instruction. The scene has not become harder to perceive. The robot’s kinematics have not changed. Yet the decision problem is fundamentally different. Should the robot close the drawer, clear the table, move the mug, or leave everything alone? Several actions may be physically feasible, and several may look reasonable. None necessarily reflects what the user wants. This is the distinction worth keeping in mind throughout this episode: competence at executing a goal is not the same as competence at selecting one. A demonstration of successful manipulation can obscure that distinction. Watching a robot complete a sensible-looking task, we may credit it with understanding what needed doing. But the objective might already have been supplied through a prompt, a benchmark convention, a restricted skill library, or the arrangement of the scene. The central question is therefore not simply whether a robot can move without receiving a command. It is whether we can identify, evaluate, and constrain the process by which it decides that a particular intervention is appropriate. Contributions and experimental results: the evidence boundary. The accessible abstract frames the problem as acting without explicit instructions—not language features collapsing inside a trained network. It introduces semantic-level vision-language entropy to characterize task uncertainty; proposes separating 3D perception, linguistic interpretation, reasoning, action generation, and execution; and argues that this supports plausible autonomous task choice with less dependence on demonstrations and fine-tuning. The reported evaluation spans different entropy levels, compares against OpenVLA under conditions described as identical, and claims better performance and robustness. The reproduced abstract provides no numerical comparisons. That is enough to identify the research direction, but not enough to establish the strength of the evidence. In particular, I cannot verify the entropy estimator, task taxonomy, robot platform, perception pipeline, reasoning model, baseline adaptation, trial counts, success criteria, ablations, or effect sizes. These are not minor omissions for this particular topic. If a robot may select its own task, the definition of success becomes part of the scientific contribution. Likewise, an entropy measure cannot be interpreted without knowing which alternatives it assigns probabilities to. The remainder of this article therefore examines the technical questions that determine whether such a system represents a meaningful advance. Where I describe an implementation or experiment, it is a proposed way to operationalize the idea—not a reconstruction of an inaccessible section of the paper. One terminological caution also matters: the available description concerns independence from explicit task instructions. It should not be read as evidence of embodiment-independent control or transfer across robot morphologies. Entropy only becomes meaningful after defining the alternatives. To understand the proposed research direction, consider our own simplified task-selection problem. A robot observes a cup beside a sink. We enumerate three possible objectives: wash the cup, place it in a cupboard, or leave it where it is. An instruction might concentrate the robot’s belief on one objective. Without that instruction, several alternatives could remain plausible. An uncertainty measure over those alternatives would describe something different from uncertainty over the next motor command. The robot might know exactly how to grasp the cup while remaining unsure whether grasping it is appropriate. That separation is valuable. But implementing it requires decisions that a term like “semantic entropy” does not resolve by itself. First, what counts as a distinct task? “Put the cup away” and “store the cup in the cupboard” may express the same objective. Treating them as separate possibilities would make wording diversity look like decision uncertainty. Conversely, merging “move the cup somewhere safe” with “put the cup in the cupboard” could erase a meaningful distinction. Second, where does the candidate set come from? A system restricted to five familiar tasks faces a different inference problem from one allowed to invent arbitrary objectives. Apparent certainty may reflect a narrow task vocabulary rather than strong situational understanding. Third, how are candidate probabilities obtained? Human judgments, empirical task frequencies, repeated model samples, and model likelihoods would give different meanings to the resulting score. Without the estimator, we cannot tell whether a number represents ambiguity in the environment, variability in a language model, or uncertainty in a particular robot’s policy. For evaluation, I would distinguish three questions:. What is present? The robot may be uncertain about an object’s identity, state, or pose. What should happen? The scene may support several plausible objectives. Can the robot achieve it? The selected objective may exceed the robot’s capabilities. These uncertainties suggest different responses. A new camera view might resolve object identity. Asking the user might resolve intent. Neither necessarily fixes an unreachable grasp. There is also an identifiability limit. Suppose two users prefer different outcomes in an otherwise identical observable scene, and the robot has no information about their preferences. Additional reasoning cannot recover information that was never provided. It can select a likely default, but that is not equivalent to discovering the actual intention. For that reason, higher task uncertainty should not automatically imply more elaborate physical behavior. It may instead justify clarification, observation, or abstention. My main criterion for an entropy-based framework would therefore be behavioral: does its uncertainty estimate help the robot choose between acting, gathering information, and waiting? A score that only ranks scenes by apparent ambiguity is useful descriptively. A score that improves those decisions has a stronger claim to operational value. Turning the idea into an inspectable architecture. At an architectural level, the most useful interpretation is a system in which task selection can be inspected separately from motor execution. The following is how I would build a research prototype around that principle. The first interface would expose a structured scene estimate. Each relevant object would have an identity hypothesis, geometry, pose, state, timestamp, and uncertainty. Object relationships would be represented explicitly where they matter: inside, supported by, reachable from, obstructing, or associated with a particular workspace. This representation would need to preserve uncertainty rather than merely serialize the detector’s best guess. A beautifully formatted scene description is not a reliable state estimate if it silently converts weak evidence into categorical facts. The second interface would represent candidate objectives. Crucially, it would distinguish observed facts from assumptions about desirability. “The drawer is open” belongs in the first category. “The drawer should be closed” belongs in the second unless a standing instruction or operating rule establishes it. The third interface would connect objectives to robot capabilities. A task could be contextually appropriate but infeasible because the required tool is unavailable, the target is unreachable, or the action would violate an operating constraint. Feasibility should constrain selection before an unconstrained plan becomes an executable command. The fourth interface would produce a typed action specification. Rather than passing unrestricted prose directly to a controller, I would require explicit objects, target states, preconditions, termination conditions, and permitted motion parameters. Missing references or inconsistent units should produce a rejected plan, not an improvised interpretation downstream. Finally, an execution monitor would compare the predicted outcome with the observed result. If the mug slips or the drawer remains blocked, the system should invalidate affected assumptions. Otherwise, a modular planner can remain perfectly coherent while reasoning about a world that no longer exists. The benefit of this design is not that modularity guarantees intelligence. It is that failures become more attributable. We can ask whether the system misperceived the scene, selected an inappropriate objective, generated an infeasible plan, or failed during execution. There is a corresponding cost: information can disappear at every interface. A compact object representation may omit deformability, contact state, or visual details needed for manipulation. A symbolic action vocabulary may exclude useful behaviors. I would therefore treat the interfaces as experimentally testable bottlenecks, not assume that explicit representations are inherently sufficient. The key design question is where to expose structure without discarding the evidence needed by the next component. A closely related paper from the same authors. There is a separate, accessible study that helps contextualize this architectural direction: “Explicit task reasoning empowering robotic manipulation,” by Shun Gui, Kai Gui, and Yan Luximon, published in npj Artificial Intelligence on August 10, 2026. That work connects separated perception, language, reasoning, and action components through numerical representations, rather than learning a task-specific manipulation policy from demonstrations. Its manuscript describes representing detected objects through names, 3D bounding boxes, and coordinates, then using those descriptions to construct numerical action plans. Its abstract reports six experimental suites and a 91.67% success rate across 240 real-world manipulation trials. Those numbers belong to the npj paper, not to the RA-L paper under discussion. The conceptual connection is useful: structured interfaces offer a way to make intermediate decisions inspectable. But the distinction between the two research questions remains important. Demonstrating that numerical scene descriptions support instruction-conditioned planning does not establish that a robot can select appropriate objectives without instructions. Nor should shared authorship license transferring implementation details, evaluation results, or limitations between the papers. Without the RA-L full text, that would be an unsupported shortcut in the review itself. Why outperforming OpenVLA needs careful interpretation. OpenVLA is a relevant reference point because its primary paper presents an open-source vision-language-action policy trained on robot demonstrations, with support for adaptation to new settings. Its central interface connects visual observations and a language instruction to robot actions. However, a comparison against an instruction-conditioned policy can test several different things. It could test manipulation quality when both methods receive the same objective. It could test robustness to missing instructions. Or it could compare an entire task-selecting system against a policy that was not given an equivalent mechanism for choosing tasks. All three experiments can be informative. They do not support the same conclusion. The first comparison I would request holds the goal fixed. Every method receives an identical objective and equivalent observations. That isolates execution performance, although differences in state representations and controllers would still need accounting. The second comparison holds execution fixed. Different high-level modules propose goals, but the same downstream controller attempts them. That gives a clearer view of whether the contribution lies in selecting useful objectives. The third comparison combines the systems: use the proposed reasoning module to generate an instruction for OpenVLA. If this hybrid closes much of the performance gap, the result would suggest that goal specification is the main missing capability—not that learned action policies are inherently unsuitable. Observation parity is equally important. A reasoner receiving object identities, metric coordinates, and carefully defined relations has a different information interface from a policy receiving RGB images. Such a system-level comparison may be practically valuable, but it should not be interpreted as isolating reasoning ability. I would also inspect what “no language” means operationally. An empty user-command field is not necessarily an absence of task guidance. A system prompt might specify that the robot should tidy, assist, organize, or correct anomalies. A restricted task library might carry similar information implicitly. The proper question is not whether those priors exist. Some prior specification is unavoidable in a designed system. The question is whether they are disclosed and matched fairly. Finally, I would want performance broken down by ambiguity regime rather than compressed into a single average. A method could excel at choosing among familiar tasks while struggling with explicit instructions, or execute reliably once a goal is chosen while selecting inappropriate goals. Without those decompositions, a headline advantage cannot tell us which component improved—or which capability remains absent. The relevant lineage: planning, programs, and feedback. For context, several primary research efforts provide useful comparison points. These are conceptual neighbors, not claims about which baselines the inaccessible paper actually evaluated. SayCan connects language-based task planning with grounded robot capabilities. Its central mechanism combines what a language model considers useful with value functions estimating whether available skills can succeed. That makes it a natural reference for separating semantic task relevance from physical feasibility. The distinction here is that grounding a supplied instruction does not, by itself, resolve which objective to pursue when no instruction is supplied. Code as Policies demonstrates another form of explicit intermediate representation: language-model-generated programs that call perception and control functions. This makes it relevant to any argument that inspectable structures can connect high-level reasoning to embodied behavior. For our analysis, the important lesson is architectural. A system need not choose exclusively between an opaque action policy and unconstrained natural-language deliberation; programs can expose a structured interface between them. Inner Monologue brings environmental feedback into language-based planning, including information about scene changes and task outcomes. It is especially relevant to the execution-monitoring problem. Selecting an initially plausible task is not enough if the robot cannot revise its plan after failure or changed circumstances. Feedback is what lets the system distinguish the intended trajectory from the one that actually occurred. Hi Robot, formally “Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models,” separates high-level interpretation from lower-level execution through a hierarchical design. It provides a useful counterpoint to framing explicit reasoning and learned control as competing paradigms. A high-level model can interpret complex requests while a lower-level VLA handles concrete physical behaviors. Taken together, these approaches suggest a more precise novelty question than “does the robot reason?” We should ask what new variable the system is responsible for selecting. Does it decompose an already specified objective? Choose among feasible skills? Generate an executable program? Recover from failure? Or decide what outcome would be appropriate before any task has been assigned?. That last responsibility changes the evaluation problem substantially. Existing planning machinery may help implement it, but the evidence must address the appropriateness of objective selection, not merely the quality of the resulting motion. For me, this is the productive way to position the research direction: as an extension of grounded planning toward less explicitly specified objectives, rather than a blanket replacement for instruction-conditioned robot learning. Designing an experiment that really tests task choice. Here is the experimental program I would propose. It is not a description of the paper’s unavailable protocol. Start with a controlled family of scenes and a bounded set of possible objectives. For each scene, establish which tasks are physically feasible, which are permitted, and which would satisfy the user’s preferences. These sets should not be assumed identical. Then vary the information supplied to the robot while keeping the underlying scene fixed. One condition provides a complete instruction. Another provides an underspecified request. A third provides only a standing operating objective. A fourth removes the immediate command while retaining explicitly documented background rules. This intervention is cleaner than evaluating separate task collections under labels such as “low ambiguity” and “high ambiguity.” With matched scenes, we can more directly attribute behavioral changes to missing task information. Next, vary visual evidence while holding the instruction condition fixed. Introduce ambiguous object states, partial occlusion, unfamiliar arrangements, or conflicting contextual cues. This helps distinguish uncertainty about the world from uncertainty about what the user wants. Counterfactual pairs would be particularly valuable. Suppose the same object arrangement is appropriate to tidy in one context but should be preserved in another. The distinguishing evidence must be available to the robot. If it is not, the experiment should acknowledge that successful preference recovery is impossible rather than treating a wrong guess as a reasoning failure. Ground truth also needs care. Open-ended scenes may admit multiple acceptable outcomes. I would prefer an explicitly defined acceptable set, supplemented by preference rankings where appropriate, rather than one arbitrarily selected “correct” task. That does not mean every plausible action should receive credit. A robot allowed to choose its own goal could otherwise select the easiest transformation, repeatedly perform low-value actions, or exploit an evaluator’s broad notion of helpfulness. I would report at least four separate quantities:. Objective appropriateness: Was the selected task acceptable given the available evidence and operating rules?. Conditional execution success: Once an appropriate objective was selected, did the robot achieve it?. Overall useful completion: Did the full system select and complete an appropriate task?. Interaction and restraint quality: Did it ask, observe, or refrain when those responses were warranted?. That last category prevents the benchmark from rewarding activity for its own sake. For ablations, I would remove or replace one capability at a time: task priors, structured geometry, feasibility filtering, execution feedback, and the ability to ask questions. I would include an oracle-goal condition to estimate how much performance is lost during objective selection, and an oracle-state condition to expose perception bottlenecks. A simple baseline would also be essential. A task-frequency prior combined with a reliable controller might perform surprisingly well in a repetitive environment. Beating that baseline would provide stronger evidence than showing that a model accepts an empty instruction field. Finally, assessors should judge outcomes without being persuaded by the system’s explanation. A fluent account of why the robot acted is not itself evidence that the action was appropriate. When human evaluation is necessary, I would separate judgments of the chosen objective, physical outcome, and explanation quality. This experimental structure would make the claim considerably more falsifiable: we could locate both the value of autonomous task selection and the conditions under which it should yield to interaction. Robustness, interpretability, and data efficiency are separate claims. Three desirable properties often travel together in discussions of modular reasoning systems. I would evaluate them separately. For robustness, the first requirement is to specify the disturbance. Missing instructions, paraphrased instructions, changed object layouts, perception errors, and execution failures challenge different components. Improvement under one does not logically imply improvement under the others. My preferred robustness analysis would trace errors through the interfaces. If an object pose is perturbed, does the planner notice inconsistency, produce a different target, request a new observation, or confidently continue? The important outcome is not just whether the final task succeeds, but whether uncertainty is handled before it becomes physical error. For interpretability, explicit intermediate outputs are valuable because they expose hypotheses. But I would not equate an explanation with a faithful account of the decision process. The stronger test is intervention: change a represented fact, constraint, or assumed objective and determine whether the corresponding action changes appropriately. For example, if the plan says an object is unreachable, correcting the reachability information should alter the plan. If the explanation changes while the commanded action remains unaffected, the explanation may not identify the operative decision pathway. For data efficiency, the accounting should separate new task-specific data from prior training. RT-2 explicitly uses web-scale vision-language pretraining alongside robot data, while OpenVLA also depends on substantial prior robot training. Consequently, “no new manipulation demonstrations” is a scoped deployment claim, not evidence that a system operates without learned knowledge. A useful comparison would count demonstrations, perception annotations, manually authored skill definitions, prompt examples, calibration work, and integration effort. It would also report inference latency and computational requirements. My concern is not that engineering priors invalidate a method. Every practical robot system embodies prior choices. Rather, replacing demonstration collection with carefully designed interfaces may be an excellent trade—but it should be described as a trade, not as eliminating the need to encode knowledge. Separating these claims makes the work easier to use. A system might offer excellent inspectability without superior robustness, or reduce adaptation data while requiring more integration effort. Those distinctions matter to anyone deciding whether to reproduce or deploy it. What I would build first. For an applied research team, I would start with bounded initiative, not unrestricted goal invention. Define a small operating domain, a standing objective, a known skill set, and explicit limits on intervention. Within that domain, let the robot identify opportunities to act without receiving a fresh command for every step. The first milestone would be reliable objective selection with a fixed controller. That keeps motor-learning improvements from obscuring the high-level question. Once the selection mechanism is credible, a learned VLA, a classical planner, or a mixture of controllers could provide execution. I would make every proposed goal carry an evidence record: which observations motivated it, which standing rule authorized it, which assumptions remain uncertain, and what outcome would count as completion. This is a proposed engineering interface, not a demand for verbose natural-language reasoning at control frequency. The system would also need an action budget. A wrong but reversible rearrangement is different from an irreversible intervention. Task uncertainty alone cannot capture that difference. The decision to act should account for consequences and recoverability as well as confidence. Finally, clarification should be treated as a productive action rather than a failure of autonomy. A robot that asks one well-targeted question and then completes the correct task may be more useful than one that confidently guesses and executes perfectly. This leads to a less theatrical, more operational definition of independence: the robot reduces the need for continual supervision while preserving the user’s ability to specify priorities and boundaries. Under that definition, reasoning and learned control are complementary. The research opportunity is to improve the handoff between deciding what is worth doing and carrying it out—not necessarily to replace one family of methods with the other. Bottom line. The compelling issue here is the boundary between goal-conditioned competence and appropriate initiative. To cross that boundary convincingly, a system must do more than produce a sensible-looking action in the absence of a fresh command. Its task assumptions must be visible, its uncertainty must support useful decisions, and its evaluation must distinguish choosing an objective from executing it. For roboticists, that suggests a concrete agenda: test goal selection separately, compare matched information interfaces, include hybrid planner–VLA baselines, and reward clarification or restraint when the evidence does not justify action. The most important question is not “Can the robot do something without being told?” It is “Can it justify selecting this outcome, given what it knows and what it is allowed to change?”. Remaining research gap: I found the target paper’s bibliographic record and reproduced abstract, plus relevant primary research, but not the RA-L full text. Its numerical results, precise entropy formulation, implementation, and experimental validity therefore remain unverified here. Uploading the paper and supplement would enable a genuine section-by-section technical review rather than the source-limited analysis above.