Latest / Embodied AI 101 / Show-Harness Gives VLMs a Robot Keyboard
Transcript
- Show-Harness Gives VLMs a Robot Keyboard. Show-Harness: Just a VLM Agent Can Play Robots • Show Lab, National University of Singapore • September 9, 2026. The news: fewer learned components, not less robotics. Behind the AI Native Foundation headline is a substantive research release. Show-Harness includes real-robot demonstrations, public control software, model adapters, and demonstration data. Its repository supports Franka and AgileX Piper arms alongside simulation backends—enough infrastructure to investigate the claim, rather than merely watch a compelling video. The central result is that a pretrained vision-language model can select fine-grained robot actions without robot-specific fine-tuning or a separate learned action head. A second route adapts smaller VLMs through fine-tuning; that route is explicitly not zero-shot. The distinction matters. This is not evidence that every pretrained VLM can operate arbitrary hardware. It is evidence for a more useful hypothesis: before training another robot policy, ask whether the existing model has been given an action interface it can actually use. My read is that Show-Harness challenges where we put the learning—not whether robotics still needs a control stack. What the VLM actually controls. Imagine a gripper approaching a banana. Instead of predicting joint angles or calling a complete “pick up banana” skill, the model chooses something closer to a keyboard command: move left, move down, close the gripper. The released training data uses nine basic units: six translation directions, grasp, release, and completion. Each example pairs camera observations with the selected action. The action symbol does not itself specify a metric displacement. An embodiment-specific interpreter supplies the movement magnitude and translates the symbol into robot motion. The extended interface also supports incremental rotations. Underneath, conventional robotics remains. The Franka implementation uses impedance control; the Piper implementation streams joint targets. The transferable component is the model-facing vocabulary, not identical motor commands across machines. So there is still a policy: the VLM selecting actions. What disappears is the requirement for a separately learned neural motor interface. And the surrounding software does substantially more than parse action names. Plugins provide camera-view guidance, turn gripper height and contact state into textual feedback, maintain subtasks, carry recent action history, and recover from empty grasps. Other plugins can highlight an interaction point or defer a planning decision until the robot has revealed the necessary evidence. The loop is not obligatorily one model call per tiny movement, either. Selective action chunking allows short open-loop sequences during transport, while adaptive stepping changes movement granularity. That preserves frequent feedback where alignment matters without demanding equally expensive deliberation everywhere. This is the important architectural middle ground: more physically specific than choosing entire skills, but much more structured than asking a VLM to discover motor control from scratch. Read the denominator before the headline. The main cross-task experiment contains ten tasks: five objects placed into either a plate or a bowl, with ten trials per task. Teddy bears and chess pieces are excluded from the fine-tuning demonstrations. Across that suite, the authors report 89% success for zero-shot Gemini-3.1 Pro, 86% for fine-tuned Qwen3.5-2B, and 57% for the strongest compared baseline. Those are substantial differences within this evaluation. The zero-shot agent also completes all twenty trials in each of four environmental-shift conditions: background, lighting, viewpoint, and distractors. Separately, a simulation-trained small model succeeds in thirteen of twenty real-robot trials, versus zero for both compared trainable VLA baselines. There are important boundaries. The fine-tuned policy learned from both Franka and AgileX demonstrations; its cross-embodiment result is therefore not transfer to an entirely unseen robot. The VLA baselines were trained on continuous trajectories converted from the same demonstrations—a controlled low-data comparison, not a universal ranking of architectures. The “plain VLM” description also understates the harness. On the five plate-placement tasks, removing subtask planning drops success from 96% to 60%; removing failure recovery drops it to 72%. I would interpret these results as a strong argument for testing the complete system. I would not interpret them as evidence that action names alone solve manipulation, or that small laboratory trial counts establish deployment reliability. The practitioner context: abstraction still matters. A useful external perspective comes from Shmuel Berman and colleagues’ July 9, 2026 evaluation published by Anthropic. They tested language models across multiple robot bodies and control interfaces. Their central observation was that performance depended heavily on the interface: direct joint-level control generally struggled, while pretrained controllers and simple orientation aids enabled more meaningful behavior. That work predates Show-Harness and is not an independent replication. But it supplies relevant practitioner context: a foundation model’s apparent embodied competence changes dramatically depending on what the surrounding software asks it to predict. Anthropic’s experiments also found that increasing reasoning effort often brought little benefit. In some settings, additional deliberation appeared to interfere with reactive behavior. That is a useful warning against treating more inference-time thinking as an automatic robotics improvement. There is an equally important counterweight from the VLA side. Physical Intelligence’s original π0.5 release described generalization to unfamiliar homes through co-training on robot trajectories, semantic tasks, and other heterogeneous data. Learned action models are not necessarily restricted to memorizing motor patterns from one laboratory. The productive question is therefore not “symbolic interfaces or neural policies?” It is: which physical decisions should remain with the foundation model, and which should be delegated to a specialized controller?. Show-Harness makes one particular division of responsibility look more competitive. GUMI turns control into a data pipeline. The most immediately reusable part of the release may be GUMI, the GUI Manipulation Interface. Humans operate the robot through buttons or keyboard commands. Agents can use the same controls, and human interventions during autonomous rollouts can be recorded as corrections. Demonstration and inference share the same action representation. The public dataset contains 164 real-robot episodes with roughly eight thousand labeled samples, plus 230 simulated episodes. Its capture logs retain gripper state, end-effector pose, and image references, supporting alternative supervision targets rather than only the semantic-action labels. That is useful engineering. A demonstration is not merely a video that must later be translated into robot actions; it already contains an explicit action choice associated with the observation that preceded it. The released small-model adapters use LoRA fine-tuning with the vision tower frozen. Their training setup turns action selection into a compact language-output problem rather than adding a dedicated continuous-control head. The team describes adaptation taking only a few H200 GPU-hours. That is an incremental fine-tuning cost, not the cost of creating the underlying foundation model. A plausible workflow follows: use a frontier model or human operator to collect and correct behavior, adapt a smaller model, then continue gathering interventions through the same interface. The release provides components for that workflow; it does not establish that the entire process can run unattended. Speed helps explain why the second route matters. The project reports seconds per step for frontier models, versus approximately 12–33 Hz inference for adapted smaller models. Neither number, by itself, establishes end-to-end task throughput. Zero-shot learning is not zero-shot integration. The repository’s setup instructions require site configuration, camera identification, a calibrated safety floor, and a starting pose. The supplied calibration values are examples, not universally safe defaults. The model card exposes another revealing detail: direction conventions matter. Its fine-tuned models follow the Franka view convention, while deployment on the first-person AgileX setup requires swapping forward and backward actions. It also warns that using the wrong chat template can silently put inference inputs outside the training distribution. Those are not cosmetic implementation details. A robot can execute a perfectly valid symbol in the wrong direction. Likewise, a model can keep producing syntactically correct responses while operating under a subtly changed prompt contract. The browser interface carries operational cautions too. GUMI’s documentation says its servers are unauthenticated and should remain on trusted networks. It also warns that pausing cannot recall a command already sent; an emergency stop must remain reachable. A deterministic interpreter is valuable because its command mapping can be inspected. It is not a guarantee that interaction with the world will be harmless. The demonstrated scope also remains primarily single- and dual-arm manipulation with parallel-jaw grippers, rather than general humanoid or dexterous-hand control. For deployment, I would still ask for measured intervention rates, task completion times, recovery behavior, and failure severity—not just success percentages. What changes for robotics teams. For a research group, Show-Harness deserves consideration as a serious baseline before committing to another embodiment-specific training pipeline. The experiment I would want is straightforward: compare a semantic-action harness and a learned-action policy under matched observations, intervention rules, and wall-clock budgets. For industry, my near-term expectation is more modest than “general-purpose robots without training.” The promising applications are supervised manipulation, rapid task prototyping, and demonstration collection—places where reducing adaptation effort may matter more than maximizing continuous-motion performance. The next convincing result would combine unfamiliar hardware, longer task sequences, and transparent reporting of human assistance. That would test whether this interface advantage survives beyond the current evaluation. The news is not that robots no longer need policies. It is that interface design can determine how much policy learning a robot actually needs. Source note: The supplied X post could not be retrieved directly. This article uses the research team’s paper, project materials, repository, and model/data documentation, with separate practitioner context from Anthropic and Physical Intelligence.