Latest / Embodied AI 101 / Show-Harness: Can Small VLMs Replace Frontier Models in Robot Control?
Transcript
- Show-Harness: Can Small VLMs Replace Frontier Models in Robot Control?. Show-Harness: Just a VLM Agent Can Play Robots • Yanzhe Chen et al. • Show Lab, National University of Singapore • arXiv • 2026. Cold Open. Picture a robot looking at a banana beside a plate. The next useful decision might be embarrassingly simple: move left, move down, close the gripper. What would happen if we made those decisions the model’s entire output language?. That is the engineering question behind Show-Harness. Its authors give the robot something resembling a game controller: a small vocabulary of movements whose physical meanings are fixed in advance. A frontier vision-language model can use that interface directly; a compact model can learn to use it from demonstrations. Imagine the practical payoff. You demonstrate through the same buttons the policy will later press. You train an adapter, put the model beside the robot, and keep observing after each movement. The interesting question is what that interface buys you. Does it simply make outputs easier to parse, or does it change how much robot learning you need? And how much of the larger agent can fit inside that smaller controller?. The Idea. The compact controller consumes a scene view, a wrist-camera view, the task instruction, and a short record of recent movements. It returns one action symbol. The released basic policy chooses among six translation directions, closing the gripper, opening it, and declaring completion. Each iteration brings fresh camera observations back into the decision process. Those symbols go to a robot-specific interpreter. The model selects a movement; the interpreter translates it into the corresponding hardware command. In the released demonstrations, translation units share a two-centimeter step across Franka and AgileX. The dataset also fixes a common directional convention, reconciling the rigs’ different viewpoints. Think through a hypothetical grasp. The gripper is slightly behind the banana. The policy chooses a forward movement, examines the resulting wrist image, then decides whether to descend. Closing and lifting become further decisions in the same language. The useful abstraction is a small physical change with an observable consequence. A task can unfold through repeated decisions rather than requiring the model to specify the entire trajectory at once. There is an important lineage here. Code as Policies showed that language models could generate robot programs containing control calls, calculations, conditionals, and feedback loops. That work made programming interfaces a bridge between language reasoning and physical execution. Show-Harness explores a different placement of that bridge: the model repeatedly selects incremental actions from a compact alphabet. The frontier-model route adds a configurable harness around those decisions. Plugins organize camera information, describe proprioceptive state, maintain recent-action context, and support subtask planning. Situated planning can defer a decision until new evidence becomes available. Recovery logic can detect an empty grasp and initiate another attempt. These are explicit parts of the control system, implemented as separable modules. The compact-model route uses supervised action imitation. The authors adapt Qwen3.5-2B with LoRA while freezing the vision tower. Training examples pair observations and instructions with the demonstrated action symbol. The released model card provides adapters for several backbone sizes trained on the same real-robot corpus, making model scale an independently adjustable part of the recipe. A useful way to understand this design is to separate which way to move from how that movement is executed. The learning problem becomes selecting among meaningful alternatives. The robot implementation supplies the operational meaning of each alternative. That division gives the interface itself a substantial role in shaping what the policy must learn. Data collection uses that same division. GUMI, the GUI Manipulation Interface, exposes actions through browser controls and keyboard inputs. Humans or GUI-operating agents can demonstrate behavior, with each step recorded as an observation-action pair. The public real-robot corpus contains 164 episodes from Franka and AgileX. A separate simulation corpus combines demonstrations from ManiSkill and RoboLab. Both use the same basic action vocabulary and observation packaging. That is the elegant systems contribution: demonstration collection, policy training, and execution share one control language. The representation is doing useful work at every stage. The Result. The headline number is 86 percent success for the fine-tuned compact model across the main cross-task evaluation. The strongest compared baseline reaches 57 percent. The compact policy also finishes close to Show-Harness’s frontier-powered version on this test. The central experiment combines five objects with a plate or bowl, producing ten real-robot pick-and-place tasks. Each receives ten trials with randomized placements and a fixed action budget. A teddy bear and chess piece are held out from fine-tuning. For the trainable VLA comparisons, π0.5 and GR00T receive continuous end-effector trajectories converted from the same demonstrations used by Show-Harness. That holds the demonstration source constant while comparing different learning and control recipes. The engineering significance is the combination of a compact model and a deliberately structured output space. When comparing systems trained from a common demonstration source, the action representation becomes an important candidate explanation for the performance gap. Here, making the target easier to express is part of the learning strategy. There is also a simulation-to-real result: a policy trained only on simulated demonstrations succeeds on the real Franka, while both tested trainable VLA baselines fail in that evaluation. The released simulation adapter is a single policy trained across both simulators. This is a useful direction for practitioners because the interface creates an explicit place to standardize training and deployment. You can ask whether the camera interpretation, movement meanings, and observation format agree, instead of treating transfer as an entirely opaque property of the weights. The most revealing ablation examines what makes the action language effective. In the zero-shot system, arbitrary action labels work almost as well as meaningful names when the prompt explicitly explains their physical effects. Written conventions provide substantial grounding; familiar names provide a useful prior. That result makes interface specification a research variable. Choosing the vocabulary and documenting its meaning belong alongside backbone selection and fine-tuning strategy. The compute claim is concrete: Qwen3.5-2B fine-tuning takes under two hours on an H200, and the experiments serve local models on an RTX 5090. The workflow starts from a pretrained VLM and produces a LoRA adapter that the released serving stack can load. The Biggest Weakness. The compact controller is not a capability-equivalent replacement for the complete frontier agent on reasoning-heavy tasks. The paper tests searching for a hidden object and arranging letter blocks to spell “SHOW.” The standalone fine-tuned policy achieves 10 percent success, compared with 85 percent for the frontier system with situated planning. Supplying Gemini-generated subtask instructions substantially improves the compact policy. The implementation makes the distinction clear. The released compact runtime has no separate subgoal planner or stage machinery. It directly maps the current views, task text, and recent movements to an action. The frontier route includes explicit planning support around its controller. This matters because action imitation and complete-agent distillation are different promises. A controller can become very good at executing a specified objective without learning the process that selects the right intermediate objectives. Putting an identified object into an identified receptacle supplies much of the task structure in the instruction. Searching behind occluders asks the system to choose informative actions and revise what happens next. The latter demands a different kind of competence from choosing the next corrective movement. The comparison changes both the model and its surrounding machinery, so it does not establish a fundamental ceiling on small models. Its system-level implication is narrower and decisive: the evaluated standalone compact route does not reproduce the full frontier system’s reasoning performance. A convincing capability-preserving replacement would run those reasoning tasks with an entirely local stack, explicitly account for every planning call, and compare systems with matched planning support. That would reveal whether a small planner, additional demonstrations, or a different training objective closes the gap. Until then, “without sacrificing task capability” should refer to the demonstrated manipulation regime. The paper’s own reasoning experiments do not support extending that phrase to the complete agent. Verdict. Show-Harness deserves attention from teams building language-conditioned manipulation with modest demonstration budgets. Its strongest contribution is a concrete way to make the prediction problem easier while retaining a closed perception-action loop. For an imitation-learning researcher, the interesting object is the action interface. Treat it as something to design and evaluate alongside the neural architecture. Ask which distinctions the model should learn, which conventions can be fixed explicitly, and which details belong in the robot implementation. Those choices determine what your demonstrations actually teach. The comparison with SmolVLA is useful. SmolVLA pursues affordable robotics through a compact vision-language backbone, a learned flow-matching action expert, and asynchronous execution of action chunks. Its experiments include simulation and real robots, and its deployment design targets consumer hardware. Show-Harness contributes a complementary design point: ordinary VLM adaptation toward a small, explicit control vocabulary. These approaches suggest a productive research question: which output representation makes the best use of the data and feedback available for your task? Model size is one part of that question. The shape of the action prediction problem is another. The release also supplies substantial engineering material. The repository includes GUMI collectors, robot runners, plugin implementations, and the training pipeline. Published adapters and demonstration datasets provide concrete starting points for experimentation. That makes the project valuable beyond its reported success rate. A team can investigate the interface, change the learning recipe, and study how the same demonstrations support different controller designs. The practical takeaway is direct: make physical decisions easy to express, easy to demonstrate, and easy to interpret. Show-Harness turns that principle into a working manipulation system. For researchers deciding where to spend their next unit of complexity, the interface deserves a place beside the model. Outro. Watch whether this shared action language becomes a reusable training interface across more laboratories and robot platforms. The enduring question is where to put complexity. You can put it into a larger model, a richer action decoder, a planning system, or the contract connecting perception to motion. Show-Harness makes that last choice unusually tangible. A robot policy’s output is more than a tensor. It is an agreement about what the machine will do next. Designing that agreement can change the entire learning problem. That is the idea worth carrying into your next experiment.