Latest / Embodied AI 101 / HexaShell: Can Debugging Code Generalize RGB Robot Manipulation?
Transcript
- HexaShell: Can Debugging Code Generalize RGB Robot Manipulation?. HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents • Hongcheng Gao, Alex Zhou et al. • HexaFuture • arXiv preprint • 2026. Cold Open. A robot pushes a T-shaped block onto its target, then “corrects” it out of place. The culprit is a camera-area calculation. HexaShell’s optimizer repairs that calculation, adds a regression test, and the next attempt succeeds. The durable result is a software change that the next episode can reuse. The foundation model stays fixed; the robot’s working toolkit improves. A failed manipulation has become something closer to a failed software test. We usually ask what a policy has learned about the world. This experiment asks where that learning should live. In weights? In the prompt? Or in a function that measures an object more reliably every time it runs?. If experience can become executable, the question shifts: what should the robot repair, and how far does the repair travel? Today on Embodied AI 101: robot learning with a code diff. The Idea. HexaShell operates a simulated dual-arm ARX X5 in RoboDojo’s Isaac Sim environment. It uses Codex with GPT-6-Astra at medium reasoning effort for both the acting agent and the tool-repair agent. During execution, the actor examines the scene and chooses commands. Between development attempts, the repair agent changes the software available to it. Those roles share a deliberately small robot-facing interface, the command. Commands move or orient a gripper, change its opening, request observations, or finish an episode. Below that interface, inverse kinematics and timed joint trajectories convert targets into motion. The language model chooses what to do; conventional control infrastructure carries it out. RGB describes the visual input, not the entire observation. The system also receives camera calibration and the robot’s own state, including joints and gripper poses. It sees one head camera and two wrist cameras. Its tool API excludes simulator object poses and task-scoring code. That is the precise meaning of controlling from observable information here. Now comes the useful division of labor. The vision-language model selects a target in an image. Geometry turns that selection into a location. A calibrated pixel defines a ray, not a unique point. Intersect that ray with a measured plane, or match the point across two views and triangulate. If the wrist needs to move for a better view, that motion is an action and uses the episode’s movement budget. Even the table height is measured. The released helper tries candidate heights, projects one camera’s image into another, and chooses the height where their appearance agrees. It retains that measurement within the episode and exposes diagnostics about the agreement between views. This is classical multiview vision packaged as a callable tool: a numerical procedure that supplies metric structure to the agent. Tools are Python modules paired with short interface descriptions. The actor sees how to call them; the implementation runs behind the API. Each task enables a specific collection, alongside shared utilities. Development proceeds through attempts, logged failures, code edits, tests, and retries. Tools are developed on ten layouts per task, then frozen for evaluation. The RGB tools start from earlier depth-based versions; the model’s weights stay fixed. This sits in a clear research lineage. Jacky Liang and colleagues’ Code as Policies established that language models could compose robot API calls, numerical computation, and feedback loops into executable policies. It made code an expressive action representation rather than merely a way to print a plan. URAI, introduced by overlapping HexaFuture authors in Make Code as Policy Great Again, already separated an agent that builds tools from one that selects and executes them. Validated revisions persisted across episodes. HexaShell brings that division together with a minimal command interface, RGB geometry, and evaluation across the full manipulation suite. The Result. The headline is 39.7% success on RoboDojo’s category-balanced metric. VPP2, the strongest learned policy in the paper’s October 14, 2026 leaderboard snapshot, scores 32.3%. Those are the numbers behind the authors’ state-of-the-art claim. Understand the denominator. RoboDojo divides its 42 simulation tasks into generalization, memory, precision, long-horizon execution, and open-vocabulary tasks. The overall metric gives each category equal weight, with generalization combining standard and cluttered scenes. The headline describes balanced coverage across capabilities, rather than the fraction of all recorded episodes that succeeded. Evaluation uses ten new layouts per task, one attempt per layout, with frozen tools and no optimizer. RoboDojo’s checker judges success. The prompt contains the instruction, a task recipe, and development notes. The actor still responds to fresh observations; its software stays fixed. The supporting comparison I find most useful is the depth experiment. On matched development layouts, the geometric RGB version outperforms the depth-network version overall. Pushing the T-shaped object and playing the xylophone improve substantially. This comparison focuses attention on how the system obtains coordinates for the controller. For a roboticist, that is an actionable result. A learned depth estimator and calibrated multiview geometry solve related problems through different assumptions. Here, exploiting the known camera arrangement and actively obtaining views pays off. The engineering lesson is to invest in explicit measurements alongside a capable model, rather than treating every spatial decision as another language-model prediction. The task results also show why the abstraction is interesting. HexaShell completes language-conditioned stacking, remembered button-press sequences, and tic-tac-toe layouts. Those episodes require symbolic choices to become concrete manipulations: which object, which order, and which target matters now. Read the result as evidence for a particular allocation of work. Semantic interpretation remains flexible. Numerical operations become executable and repeatable. The policy is the combination: a model making decisions through a library that has already absorbed experience. That is the architectural story behind the aggregate score, and it connects directly to URAI’s separation of tool construction from feedback-driven execution. The Biggest Weakness. The biggest weakness is that the headline is not a matched test of task generalization. HexaShell develops tools for every evaluated task. RoboDojo’s published training split deliberately excludes its Open-category tasks, because that category is intended to test whether policies can recombine existing skills for unseen task specifications. Those are different learning conditions. Consider language-conditioned block stacking. A policy encountering that task without task-specific training is being tested on transfer. An agent that has already spent development episodes repairing tools for that task is being tested on how well its adapted solution handles another layout. Both experiments are useful. They answer different questions. Freezing the tools before evaluation creates a meaningful boundary around the new layouts. It does not make the task itself unseen. Moving learned knowledge into code changes its representation, not its status as learned knowledge. Persistent experience is exactly what makes tool evolution compelling. A comparison has to count software adaptation as adaptation. This matters directly to the score: Open is an equally weighted category and one of HexaShell’s strongest. Its contribution cannot automatically be read as evidence of the same transfer capability that the benchmark’s original training split was designed to measure. No weight updates is not synonymous with zero-shot evaluation. Here, the optimizer can change mechanisms used to interpret observations and execute actions. Those changes belong on the learning side of the comparison. The decisive experiment is straightforward: build the tool library on a declared training partition, freeze it, and evaluate unseen task specifications under the same rules as competing policies. Alternatively, give every method a matched task-adaptation phase and label the comparison accordingly. That would distinguish transferable tools from efficient specialization without changing the central method. Verdict. Robot-agent builders should care because HexaShell makes the interface itself a place where capability can accumulate. The productive question is not only which model to call. It is which recurring operation should become a tested tool that the model can invoke again. That reframes agent development around reusable computations. Perception and control researchers should recognize a familiar advantage in the architecture. Object selection and geometric estimation are separated. That separation gives you a concrete place to investigate a bad command: the chosen image feature, the recovered location, or the motion generated from it. Each software component becomes something you can test separately. The released table-height implementation is a useful artifact to inspect. It exposes measurement diagnostics and can run on saved observations. You can examine how image agreement changes with the hypothesized plane without rerunning an entire manipulation episode. This is the kind of local, inspectable computation that makes an agent’s behavior easier to study. For robot-learning researchers, the conceptual payoff is the location of memory. Code as Policies made programs a policy representation. Persistent tool revision makes a program library a record of experience. A successful repair changes the computation available on future episodes, not just the explanation attached to the last failure. My take: HexaShell is a strong demonstration of coding agents as robotics software engineers. Its most useful lesson is to combine flexible semantic decisions with explicit measurement and reusable execution. Treat the library as part of the policy you are building. Give it clear inputs, observable outputs, and a development history that can be inspected. The practical prize is a controller whose experience produces better tools for the next decision. Outro. The next development worth watching is a library that grows in useful structure: better measurements, clearer motion interfaces, and reusable recovery procedures. That is a research direction, not simply a larger collection of generated scripts. Return to the opening: the failed manipulation becomes a regression test and a changed function. That is the promise to carry forward. Experience can become an executable asset. A robot can carry something from one episode into the next that another engineer can read, inspect, and improve. Sometimes the most important thing a robot learns is not another response. It is a better function. This has been Embodied AI 101.