Latest / Embodied AI 101 / NEO-VLA: Why Visual Readout Depth Matters for Robot Manipulation
Transcript
- NEO-VLA: Why Visual Readout Depth Matters for Robot Manipulation. From Pixels to Actions – Towards Native Vision-Language-Action Policies • Weichen Fan, Haiwen Diao et al. • NTU S-Lab, SenseTime Research, Ropedia • arXiv • 2026. Cold Open. Your robot knows which object is the tennis ball. It knows which object is the can. Now it has to balance the first on the second, releasing the ball onto a surface smaller than the ball itself. That is one of NEO-VLA’s physical manipulation tests, and it makes the paper’s architectural question concrete. Where should a controller get the visual information for that final placement? From the completed output of an image encoder? Or from intermediate representations, while the model is still processing the scene? Think about the difference between identifying the target and deciding exactly when to open the gripper. NEO-VLA turns that question into an experiment: change which backbone layers an action expert can read, then measure what happens to control. The story is about the interface between seeing and acting—and giving researchers a way to investigate that interface directly. The Idea. At the center is NEO-2B, a forty-layer native vision-language backbone. The two-billion-parameter label describes the backbone, with a narrower action expert attached. The visual front end is a two-stage convolutional patch embedder, with an overall stride of thirty-two. The architectural simplification is removing a separately pretrained vision tower: visual computation happens inside the shared backbone. The foundation comes from the earlier NEO paper, From Pixels to Words. Its transformer processes image patches and text within one stack, using spatial-and-temporal positional information and modality-aware attention. Patches within an image can attend bidirectionally; text retains causal attention. This gives the shared model connectivity suited to images while preserving its language-model structure. Large-scale image-text learning and multimodal instruction tuning establish the representations that robot learning subsequently adapts. The policy’s input prefix adds previous and current camera views, camera intrinsics and rotations, the task instruction, robot state, and a structured description of the embodiment and action frame. Missing robot components are masked. One interface can therefore represent single-arm, bimanual, and arm-and-body configurations. This is a practical design choice for learning across heterogeneous robot datasets. On the action side, NEO-VLA follows the flow-matching family established by Physical Intelligence’s π0. The expert learns how to transform noisy action trajectories into demonstrated trajectories, conditioned on observations. This generates continuous motor commands rather than spelling actions out as discrete language tokens. The action-generation machinery has a clear predecessor; NEO-VLA’s distinctive investigation concerns the visual representations supplying it. The expert predicts thirty-step chunks with slots for two arms and a body. It reads cached keys and values from selected backbone layers, reusing the observation computation during sampling. The designer chooses whether those connections expose early processing, later representations, or a mixture across depth. That choice becomes an explicit policy-design variable. The robot pretraining corpus contains 2.23 million episodes and approximately 9,900 hours, mixing real recordings with simulation. Its nine dataset families include Open X-Embodiment, DROID, AgiBot-World, RoboCOIN, Galaxea Open-World, RoboMIND, RH20T, Bimanual YAM, and simulated InternData-A1. The processed representation includes embodiment-specific normalization and validity masks, allowing usable action targets to survive when another field is unreliable. Sampling also differs from raw dataset volume. The simulated source is downweighted, so its large number of stored timesteps does not translate directly into dominance during training. This distinction matters when interpreting a headline episode count: the learning mixture determines what the model repeatedly encounters. Training first freezes the backbone while fitting the action expert and conditioning modules. It then unfreezes the model and combines robot learning with multimodal understanding examples. A subsequent stage adapts using in-house action data. The progression separates learning an action readout from jointly changing the representations it reads, while continuing to exercise the model’s vision-language capabilities. Now the central ablation: hold expert capacity fixed and change its readout locations. A particularly clean comparison retains the full backbone in both cases, contrasting a depth-spanning map with a later-layers-only map. Early access helps. With the backbone frozen, early-layer readouts are strongest; after full fine-tuning, access distributed across depth works best. The Result. The contrast to remember is 69.4 percent versus 10.7 percent. Those are RoboTwin 2.0 clean-to-randomized success rates for NEO-VLA with multimodal co-training and with action-only fine-tuning. Retaining understanding supervision radically changes the outcome. RoboTwin is a simulated bimanual manipulation benchmark. Its randomized environments vary clutter, backgrounds, lighting, table height, and language. Clean-to-randomized evaluation trains on clean demonstrations and tests under environmental randomization. It asks a specific operational question: can a policy continue performing a familiar manipulation skill after the conditions surrounding that skill change?. That evaluation direction is useful because it separates the training environment from the robustness test. Imagine collecting demonstrations on a tidy work surface, then changing the lighting and introducing irrelevant objects around the targets. The desired motor behavior may remain familiar, while the observation no longer matches the demonstrations. RoboTwin turns variations of that situation into a repeatable evaluation protocol. The accompanying understanding tests strengthen the co-training result. Multimodal co-training keeps question-answering performance close to its starting level. The action-only variant stops producing valid text. Together, these findings make preserved multimodal competence look like a working component of the policy, rather than merely a credential inherited at initialization. The backbone comparison adds another piece. The authors compare NEO against Qwen3-VL and Gemma-3 using the same action-expert design and robot-training corpus. NEO leads with frozen backbones and after full fine-tuning. Routing Qwen’s expert directly to intermediate vision-tower features does not recover that advantage. The result favors considering representation learning and action readout together. The intervention study makes the layer findings more concrete. Replacing an attachment’s cached representations with those from another observation changes predicted translations most at early layers. Matched-magnitude noise has little effect. The controller is sensitive to the information carried by those connections, rather than simply to the size of a disturbance. Standard LIBERO confirms strong in-distribution manipulation. On LIBERO-Plus, NEO-VLA leads the authors’ aggregate comparison and performs particularly well under camera changes. Real-robot tests cover ball placement, tape-and-ball sequencing, and can reorientation: fifty demonstrations and twenty trials per task, split between familiar and shifted setups. The reported success counts match or exceed π0.5 across these tests. The Biggest Weakness. The biggest weakness is incomplete end-to-end reproducibility. That directly qualifies the description of “fully open code and data.” Downloading a capable policy and reconstructing the experiment that produced it are different levels of access. As of November 10, 2026, the Hugging Face dataset card says the release contains processed trajectories, frame references, and metadata, but no camera observations. Users must obtain those from upstream publishers and match episode identities, frame order, and timestamps. The repository does not provide an automatic conversion or linking tool for that reconstruction. The model card identifies another missing artifact in the same reproduction chain: the original NEOpp-2B-SFT starting checkpoint is not released. The available standalone stage-two backbone has already undergone joint vision-language and action training. Substituting that artifact changes the starting point of an experiment. These details matter particularly for this paper’s argument. Its experiments ask what information a pretrained native backbone already contains and how robot learning should access it. If an outside team reconstructs a different image stream or begins with an already action-trained backbone, a changed outcome becomes harder to diagnose. Did the proposed interface fail to transfer, or did the reproduction begin from a different experiment?. Open weights still enable meaningful evaluation and adaptation. The higher bar is a versioned path from the documented starting backbone and source observations to the reported checkpoints. Until that path is complete, the accurate description is an open implementation with released policies and processed trajectory metadata, rather than a self-contained foundation-model training release. The distinction follows directly from the published artifact documentation. Closing that gap would let other labs change one ingredient and know what they actually kept fixed. That is the kind of accessibility that turns a useful result into a reliable experimental foundation. Verdict. VLA backbone designers should pay attention. NEO-VLA gives readout depth a place alongside backbone size, action horizon, and training mixture in the policy-design discussion. The most interesting question is where action-relevant information becomes available—and how to expose it to the controller. For representation-learning researchers, this is a productive meeting point with robotics. A feature can support recognition, language grounding, or precise motion prediction, and those uses invite different experiments. Here, changing the action expert’s access provides a concrete way to investigate the relationship between internal representations and behavior. For teams building on existing models, the release provides useful entry points. There are action-pretrained and jointly trained checkpoints, along with policies adapted for LIBERO and RoboTwin. The model packages include checkpoint-specific normalization records; the pretraining releases also include native training state. Those artifacts support evaluation, continued training, and adaptation from an established policy. The deployment story is similarly concrete. The paper serves its Pika robot from a single RTX 3090 with 24 gigabytes of memory—a workstation-scale inference setup. The deployment interface sends camera observations, proprioception, and an instruction to an HTTP policy service. It returns action chunks in physical units, with the robot’s own controller handling execution. This separates model serving from robot-specific drivers in a way an engineering team can integrate into an existing stack. The experiment I would prioritize around this work is representation access: keep the task and action machinery stable, then investigate which internal computations are useful to control. That framing connects architectural research to something the robot actually does. It also gives smaller research groups a focused question to pursue using released policies. The takeaway: NEO-VLA makes visual readout depth an explicit, testable part of robot policy design. Outro. Return to the ball and the can. Recognizing both objects gets the robot to the right place. The final release asks a more precise question about what the model knows at the moment it acts. Watch next for independent experiments that vary visual processing and action readout under a shared training recipe. That would turn this architectural direction into a particularly useful body of evidence. For now, carry one question into your next VLA experiment: when the gripper opens, which representation is supplying the information that makes the placement succeed?.