Latest / Embodied AI 101 / VL-GRiP3 and the Case for Keeping VLMs Away from the Motor Loop

Transcript

VL-GRiP3 and the Case for Keeping VLMs Away from the Motor Loop · Embodied AI 101

• VL-GRiP3 and the Case for Keeping VLMs Away from the Motor Loop. VL-GRiP3: A hierarchical pipeline leveraging vision-language models for autonomous robotic 3D grasping • Mirco Polonara, Xingyu Yang, Luca Carbonari, and Xuping Zhang • Polytechnic University of Marche and Aarhus University • Robotics and Computer-Integrated Manufacturing, Volume 100 • 2026. At first glance, VL-GRiP3 sounds like another attempt to turn a vision-language model into a robot policy. It is almost the opposite. The paper’s central argument is that a vision-language model should interpret what a person wants, identify the relevant objects, and organize a task—but should not be trusted to invent precise six-degree-of-freedom robot motion directly. Metric perception and physical execution are instead delegated to explicit geometric modules: RGB-D reconstruction, CAD registration, a learned 3D grasp planner, and a conventional robot-control library. That distinction makes VL-GRiP3 interesting. Rather than asking whether a sufficiently large vision-language-action model can absorb the whole manipulation stack, the authors ask a more industrially grounded question: where in a robot system does generative intelligence actually help, and where is it an unnecessary liability?. The result is a hierarchical system that accepts language, segments the requested part, localizes a destination, reconstructs hidden geometry using a CAD model, generates a six-degree-of-freedom grasp, emits a symbolic action sequence, and executes it on a robot. The authors report a 94.67 percent grasp-and-place success rate over 150 randomized trials. Although the journal volume is dated August 2026, the article was accepted on January 18 and made available online on January 27, 2026. The paper in one sentence. VL-GRiP3 places a compact VLM at the semantic boundary of an industrial manipulation system, while preserving explicit geometry, constrained action primitives, and conventional robot control below that boundary. That is a much more consequential design choice than it may initially appear. End-to-end VLAs generally try to learn a direct mapping from images and instructions to robot actions. Depending on the system, those actions might be Cartesian deltas, joint commands, discretized action tokens, or trajectories generated through a diffusion policy. The advantage is conceptual simplicity: one model receives observations and emits behavior. The disadvantages are familiar—large demonstration requirements, difficult debugging, uncertain out-of-distribution behavior, and poor visibility into why a particular motion was selected. VL-GRiP3 instead breaks manipulation into inspectable stages. A PaliGemma backbone handles object and target perception and produces a high-level action script. OverlapPredator aligns a known CAD model with the partial object cloud. M2T2 generates geometry-aware grasp poses. Finally, a small library of robot primitives carries out the generated script. The work therefore sits somewhere between a classical industrial perception pipeline and a modern embodied-AI system. It retains the engineered interfaces of the former while using a VLM to make those interfaces accessible through language. Why this problem is harder than “pick up the red part”. The motivating setting is high-mix, low-volume manufacturing, particularly small and medium-sized enterprises. These environments are awkward for both conventional automation and current general-purpose robot learning. Traditional industrial cells are reliable when the workpiece, fixture, camera pose, and motion sequence remain fixed. But changing the part often means updating perception models, waypoints, grasp parameters, and robot programs. That engineering cost can dominate the economics of small-batch automation. At the other extreme, a generalist VLA promises rapid retasking through natural-language commands, but introduces uncertainty into the portion of the stack where millimeters and degrees matter. A plausible-looking action is not sufficient when a rigid gripper must clear surrounding geometry, contact an irregular metal component, and place it without collision. Random bin picking adds another difficulty. An RGB-D camera only observes visible surfaces. The rear and underside of an object are absent from the measured point cloud, and neighboring parts may hide precisely the features that determine whether a grasp is feasible. A grasp network operating directly on that partial observation can choose a pose that appears valid from the camera but intersects unseen geometry. VL-GRiP3 exploits an important industrial asymmetry: although the live scene may be uncertain, the parts are usually not epistemically unknown. Engineered components often have CAD models. Instead of learning to hallucinate the missing surface, the system estimates how the known CAD geometry is situated in the scene and uses that aligned model to fill in the unobserved portion. This is not open-world object manipulation. It is a structured compromise: natural interaction and variable object poses, but a known catalog of manufactured parts. For many factories, that is not a weakness. It is the actual operating regime. Walking through the hierarchy. The easiest way to understand VL-GRiP3 is to follow a command through the system. Consider an instruction such as “move TAC2 into the red target.”. Language becomes a set of constrained subproblems. Commands can arrive as text or as speech transcribed by Whisper. The instruction is then decomposed into separate prompts for three functions:. segment the object to manipulate;. detect the destination region;. generate the high-level robot action sequence. The paper describes these as specialized roles of a single PaliGemma backbone: detector, segmentor, and action planner. In practical terms, “single backbone” means reuse of the same model family and representation rather than one indivisible forward pass. The released code exposes separate parameter-efficient checkpoints for segmentation and action generation, invokes the model sequentially, and unloads modules to release GPU memory. There is also a deterministic layer before the VLM. The reference runner uses regular expressions to extract a manipulation verb, an object phrase, an optional target color, and directional shifts. It recognizes verbs such as “move,” “place,” “put,” “position,” and “shift,” then converts the original command into terse prompts such as “segment tac2,” “detect red target,” and “move in red target.”. This is a revealing implementation detail. The language interface is more flexible than a conventional teach pendant, but the current system does not hand an unrestricted natural-language goal to an autonomous reasoning agent. A parser first forces the request into the ontology understood by the rest of the cell. That constrained semantic interface is arguably appropriate for manufacturing, even if it makes the phrase “free-form language” somewhat generous. PaliGemma emits structured perception tokens. PaliGemma combines a SigLIP visual encoder with a Gemma language decoder. The paper uses its ability to represent spatial outputs as tokens rather than attaching an entirely separate segmentation architecture. Bounding boxes are represented by four location tokens. Each token selects one of 1,024 normalized coordinate bins, which are rescaled to the image width and height. Segmentation masks are represented through tokens from a 128-entry vector-quantized codebook and decoded into a binary mask using a lightweight VQ-VAE. The action role emits an ordered sequence of symbolic manipulation primitives. This gives the same basic architecture three output vocabularies:. spatial tokens for target localization;. mask tokens for object segmentation;. text-like tokens for robot actions. The segmentation mask is combined with the depth image to lift the selected pixels into 3D. For the destination, the center of the detected bounding box and its median depth produce a target position in the camera frame. Camera-to-robot calibration then maps these quantities into the robot coordinate system. The division between object and destination perception is sensible. The object needs a detailed mask because its geometry will drive registration and grasping. The destination, at least in the demonstrated pick-and-place tasks, can be represented much more simply by a box center and depth estimate. That also exposes the system’s task assumptions. It is designed for placing a part into a relatively permissive target region, not for mating two components under tight orientation and tolerance constraints. A bounding-box centroid is an adequate goal for a colored tray. It is not, by itself, an assembly pose estimator. The partial cloud is matched to CAD. After segmentation, the system has an object-specific point cloud—but only for surfaces visible to the RGB-D camera. OverlapPredator registers the associated CAD cloud to that partial observation. OverlapPredator is well matched to this problem because it was designed for point clouds with limited overlap. It produces a learned descriptor for every point, along with estimates of whether the point lies in an overlapping region and whether it is likely to admit a reliable correspondence. VL-GRiP3 combines those two confidence estimates, retains the most promising points, and establishes mutual nearest-neighbor matches in descriptor space. A weighted rigid-alignment procedure then estimates the CAD model’s rotation and translation. Once transformed into the observed pose, the CAD points are merged with the measured scene points. The important point is that the system does not truly “see” through occlusion. It replaces missing measurements with a registered prior. If the part identity and registration are correct, that prior is exceptionally informative. If registration is wrong, the robot acquires a complete but confidently incorrect model of the object. This makes registration confidence a safety-critical interface. A production version should not simply pass every registration onward. It should reject solutions with weak overlap, large residuals, implausible transforms, or disagreement with visible depth boundaries. Nevertheless, the idea is powerful. Multi-camera systems reduce occlusion by acquiring more observations, but introduce additional calibration, synchronization, mounting, and maintenance. VL-GRiP3 buys a complete geometric representation from one RGB-D view by using information that an industrial digital thread often already contains. M2T2 turns completed geometry into a grasp. The augmented object cloud is then passed to M2T2. A PointNet++ encoder extracts multiscale geometric features. A masked-transformer decoder predicts likely contact regions, and an action decoder estimates the parameters needed to form six-degree-of-freedom grasp and placement poses, including approach direction, contact direction, and gripper width. VL-GRiP3 adapts the public M2T2 implementation to consume the object-specific cloud produced by segmentation and CAD registration. In other words, M2T2 does not have to discover which points belong to the commanded object or reconstruct the scene itself. Those ambiguities have already been resolved upstream. Each candidate has an objectness score and a maximum contact confidence. The implementation rejects candidates unless both exceed 0.60, ranks the survivors, retains the three best for visualization, and sends the highest-ranked pose to the action layer. This is where the hierarchy earns its name. The VLM decides which object matters. Segmentation restricts the metric observation to that object. Registration inserts the missing geometry. M2T2 reasons over contacts and gripper configuration. The robot controller receives one selected pose rather than a distribution of language-conditioned action tokens. Each module progressively narrows the problem and changes representation:. language becomes semantic labels; semantic labels become pixels; pixels and depth become points; partial points become a completed object; completed geometry becomes grasp poses; and a selected pose becomes a parameterized robot primitive. The action model emits a program, not motor commands. The final PaliGemma role generates a semicolon-separated action script. A representative sequence is:.. These symbols map to methods in a conventional robot library. “Grasp” loads the pose selected by M2T2. “Target” loads the destination estimated by the detector. The remaining commands invoke predefined motions and gripper operations. Execution uses Universal Robots’ RTDE interface and a Robotiq gripper. This is a much safer use of generative output than directly predicting Cartesian motion. The action vocabulary is small, and unrecognized symbols can be rejected. The VLM may rearrange available skills, but it cannot silently invent a new torque command. There is an open question about how much a learned action planner contributes in the demonstrated task. For a simple pick-and-place operation, the valid program is nearly fixed. In the released runner, action inference even uses a fixed example image rather than the live scene image, suggesting that the action head primarily learns command syntax and task templates. A hand-written finite-state machine could likely execute the demonstrated behavior just as well. The action model becomes more valuable if the skill library grows to include inspection, regrasping, tool use, conditional recovery, or different placement strategies. In the current task, it should be understood as a proof that structured language generation can dispatch robot skills—not yet as evidence of rich visual task planning. What the experiments establish. The paper reports that its fine-tuned VLM segmentation modules perform comparably to YOLOv8n on the task dataset. That is meaningful because token-generating VLMs are often assumed to be too spatially imprecise for manipulation. In this constrained domain, PaliGemma appears able to serve as both the semantic interface and the object mask generator without sacrificing the downstream task. The end-to-end system achieves 94.67 percent success over 150 randomized grasp-and-place trials using a UR3e cell and one RGB-D camera. That percentage corresponds to 142 successful trials and eight failures, assuming conventional rounding. A rough 95 percent Wilson interval runs from about 89.8 to 97.3 percent. So this is strong evidence that the system works repeatedly in its tested cell, but not yet enough to characterize rare failures at industrial production scale. The distinction between trial reliability and generalization is important. One hundred and fifty randomized trials measure repeatability over variations represented in the experiment. They do not establish open-world object generalization, broad linguistic competence, or robustness across cameras, grippers, lighting conditions, and factories. The released runner is explicitly configured around three named CAD assets—TAC1, TAC2, and TAC3—and asks the operator to identify one of those names in the command. That is consistent with a known-SKU manufacturing cell. It also means the result should not be interpreted as grasping arbitrary unseen industrial components. The authors additionally report that the modular, CAD-augmented pipeline is more reliable and controllable than evaluated end-to-end VLA baselines. The broader conclusion is plausible: explicit pose generation and a constrained controller should be easier to inspect than an unconstrained learned motor policy. But the comparison is best read as evidence for this task regime, not as a universal ranking of modular systems over VLAs. End-to-end policies are solving a broader adaptation problem, while VL-GRiP3 is allowed to exploit object-specific CAD and engineered motion primitives. Where VL-GRiP3 sits in the manipulation landscape. The paper usefully spans three normally separate research traditions. The first is classical and learning-based grasp perception. Analytic planners search CAD models for force-closure contacts. GQ-CNN and GG-CNN learn grasp quality from depth observations. GraspNet-style systems predict full six-degree-of-freedom grasps, while QT-Opt demonstrates how large-scale real-world reinforcement learning can acquire closed-loop grasping behavior. These approaches concentrate on the contact problem but generally assume that some external system has already selected the relevant object and task. The second tradition is language-conditioned task planning. Frameworks such as Instruct2Act and Dexbotic treat a language or vision-language model as a planner over conventional perception and robot skills. Their advantage is modularity, but many demonstrations use simple top-down picks, 2D points, or coarse object poses. The third is end-to-end VLA learning. The VL-GRiP3 paper places systems such as CogACT and π0 in this category and contrasts their unified action generation with the modular framework approach. It also points to the scale of the RT data ecosystem as evidence that generalist policies demand demonstration resources beyond what most SMEs can collect. VL-GRiP3’s contribution is not a new foundation model, registration algorithm, or grasp architecture. PaliGemma, OverlapPredator, and M2T2 are existing components. The contribution is their division of labor and the industrial assumptions connecting them. That is worth emphasizing because systems papers are sometimes undervalued when their components already exist. In robotics, however, the interface between representations often determines whether a method can leave the benchmark. VL-GRiP3 defines an unusually clean interface between semantics and geometry: the VLM chooses and segments; the geometric stack measures and plans; the controller executes. The deeper idea: CAD as an embodied world model. The most interesting conceptual element may be the role played by CAD. In contemporary embodied AI, “world model” often means a learned latent predictor that imagines future observations or dynamics. VL-GRiP3 uses a much older and more exact form of world knowledge: an engineered model of the object itself. The CAD file says what geometry must exist even when the sensor cannot observe it. Registration binds that prior to the current scene. The resulting cloud is therefore neither pure measurement nor learned completion. It is a hybrid state estimate composed of observed surfaces and model-derived hidden surfaces. For manufactured objects, this may be more useful than a generic neural shape-completion model. A learned completion network asks what shape is statistically plausible. CAD tells the robot what shape this specific component was designed to have. That suggests a broader direction for industrial embodied AI. Foundation models need not replace the factory’s existing digital artifacts. They can act as interfaces into them. CAD, bills of materials, process plans, tolerance specifications, tool databases, and robot skill libraries can all serve as structured priors beneath a language-enabled orchestration layer. In this interpretation, the VLM is not the world model. It is the router that selects the appropriate pieces of an already rich industrial world model. The important limitations. The first limitation is the system’s dependence on correct object identity and CAD availability. This is appropriate for known parts, but the pipeline cannot simply accept an unfamiliar object and retain the same geometric completion strategy. New parts require a CAD asset, registration support, segmentation data, and configuration of the object catalog. Second, CAD augmentation transfers uncertainty rather than removing it. Occlusion disappears from the final point cloud, but only because the pipeline assumes the registration is correct. The current architecture would benefit from propagating registration uncertainty into grasp scoring rather than converting a single rigid transform into apparently certain geometry. Third, the manipulation loop appears largely open-loop after the initial image. There is no central role for continuous visual servoing, tactile feedback, slip detection, or post-grasp verification. A small registration or calibration error can therefore persist through approach, closure, lift, and placement. For irregular metal parts, even a simple wrist force-torque check or gripper-width verification could substantially improve failure detection. Fourth, the action space is more restricted than the language-facing presentation suggests. The released parser accepts a finite set of verbs and target patterns. Commands must identify one of three TAC objects for CAD selection. The action decoder ultimately emits symbols from a small primitive library. This is still useful natural-language programming, but it is not unrestricted language grounding. Fifth, “single VLM backbone” should not be confused with a single jointly optimized perception-action model. The repository uses separate PEFT checkpoints and sequential inference. That preserves modularity but introduces loading and orchestration overhead. It also raises the question of whether sharing the architecture yields a practical advantage over using a dedicated segmenter and a smaller language model for action syntax. Sixth, the public implementation is not entirely unattended. It contains operator prompts before registration, M2T2 inference, and physical execution, including an option to inspect the grasp visualization and repeat inference. These are sensible research safeguards, but they mean that the released code represents a supervised experimental workflow rather than a fully autonomous production daemon. Finally, reproducibility is partial. The repository includes the integration code, sample outputs, and incorporated versions of OverlapPredator and M2T2. However, users are instructed to create their own dataset and train both PaliGemma and registration checkpoints. The main script also contains machine-specific absolute paths. The tested software stack uses Python 3.10, PyTorch 2.3.1, CUDA 12.1, and an RTX 4090 with 24 gigabytes of memory. The code is valuable, but it is not a one-command reproduction of the reported experiment. How I would adapt the system. For a known-part bin-picking application, I would preserve the paper’s central hierarchy but harden every interface. The VLM should output a schema-constrained task object rather than unconstrained text. That object would contain the SKU identifier, destination identifier, task type, and any requested offsets. A deterministic validator would reject unknown parts, ambiguous destinations, and unsupported operations before perception begins. The registration stage should expose confidence and residual diagnostics. If multiple CAD poses are plausible because of symmetry, the system should preserve that pose set and score grasps across the alternatives. It should not collapse a symmetric part into an arbitrary orientation unless the downstream task is invariant to that symmetry. Grasp selection should include robot reachability and collision checking before a candidate enters the action layer. The current contact confidence indicates whether M2T2 likes the object-gripper relationship, but not necessarily whether the arm can reach the pose without striking the bin, camera mount, table, or neighboring objects. I would also close the loop at three points: reobserve before final descent, verify gripper closure after contact, and inspect the scene after lift. These checks can remain conventional and deterministic. They do not require a learned policy. For deployment with many SKUs, keeping every adapter resident may be impractical. A sensible architecture would share one general segmentation model, retrieve the CAD asset and registration configuration from the recognized SKU, and cache frequently used models. If part appearance varies dramatically, lightweight per-family adapters could be loaded dynamically. For open-world objects, the CAD completion stage would have to change. Options include multi-view active perception, category-level pose estimation, learned shape completion, or grasp planning directly on partial geometry. But removing CAD also removes one of VL-GRiP3’s strongest sources of reliability. At that point, one should expect lower certainty or spend more time gathering observations. Most importantly, I would resist expanding the VLM’s authority merely because it is available. If a finite-state machine can safely express a task, let it do so. Use the VLM where semantic variation is real—interpreting human intent, resolving references, selecting skills, or choosing among process alternatives. Use geometry and control where physical constraints dominate. The verdict. VL-GRiP3 is not a general robot foundation model, and that is precisely why it is valuable. It shows a credible path for incorporating VLMs into industrial manipulation without asking them to solve metric geometry and motor control through generative imitation. The VLM provides the flexible language-and-vision interface. CAD supplies object-level prior knowledge. Registration turns that prior into a scene-specific model. M2T2 generates contact-aware six-degree-of-freedom grasps. A constrained skill library executes the result. The reported 94.67 percent success rate suggests that this division of labor can work well for a known-part, single-camera manipulation cell. The evaluation does not establish broad generalization, and the released implementation remains more constrained and operator-gated than the word “autonomous” implies. But the paper’s architectural lesson is stronger than any single benchmark number. For industrial embodied AI, the winning system may not be the one that learns the most. It may be the one that knows exactly which responsibilities should never have been handed to a generative model in the first place.