Latest / Embodied AI 101 / Four Keypoints, One Careful Pour: Connecting Language to Precise Robot Motion

Transcript

Four Keypoints, One Careful Pour: Connecting Language to Precise Robot Motion · Embodied AI 101

• Four Keypoints, One Careful Pour: Connecting Language to Precise Robot Motion. Bridging VLM and KMP: Enabling Fine-Grained Robotic Manipulation via Semantic Keypoints Representation • Junjie Zhu et al. • Zhejiang University; Ningbo University of Technology; Robotics Research Center of Yuyao City • IEEE Robotics and Automation Letters listing: 2026; arXiv manuscript analyzed: 2025. Source note — October 6, 2026: The October 2026 journal listing is corroborated by library metadata, but I could not retrieve the journal text through the supplied DOI. The accessible full manuscript is arXiv:2503.02748v1, submitted March 4, 2025. Methods and results below refer to that version; I do not assume that it is identical to the final journal article. The discussion separates reported findings from engineering interpretation and proposed follow-up experiments. The failure between understanding and execution. Imagine a robot holding a mug, with several other mugs on a table. A person asks it to pour water into the blue one beside the box. Suppose the robot understands the sentence perfectly. It identifies the correct blue mug, selects a pouring skill, and produces a smooth trajectory. Yet the pour still fails: the held mug’s rim clips the receiving mug, or the water passes just outside its opening. What failed was neither necessarily language understanding nor trajectory smoothness. The failure occurred between them. “Pour into that mug” did not become a sufficiently precise description of the spatial relationship the robot needed to establish. Now imagine the opposite failure. The robot has an excellent pouring trajectory, demonstrated carefully and reproduced accurately. Move the receiving mug, rotate its handle, or replace it with a differently proportioned instance, and the trajectory is no longer appropriate. The motion model knows a movement, but the system has not supplied the right conditions for adapting it. These examples motivate the question running through this episode: what information should cross the boundary between semantic reasoning and motion generation?. An object identifier is too little. A complete world model may be unnecessary. A desired gripper pose is useful, but someone still has to derive it from the actual interaction. My reading of this work is that the most interesting design choice is an intermediate representation: a small set of meaningful geometric landmarks, interpreted through task constraints and connected to a demonstrated motion prior. The research question is not simply whether a large model can command a classical controller. It is whether the interface preserves the details that determine physical success. For a roboticist, this is also a debugging question. If the final pour fails, can we distinguish the wrong target from the wrong geometric relationship, and both from a poorly adapted trajectory? A modular representation gives us places to inspect those possibilities. The central lesson I would take into implementation is therefore narrower than “combine foundation models with movement primitives.” It is: make the interface explicit enough that a correct semantic decision can become a testable geometric condition. What VL-MP actually connects. The verified implementation is called VL-MP. GPT-4o and GroundingDINO handle dialogue and target retrieval; an HRNet-based bridge estimates 3D landmarks at the mug’s rim, mouth center, base center, and handle. Demonstration-derived interaction constraints determine the terminal pose, with two-degree orientation sampling and selection near the starting pose. Video-derived pose trajectories supply task-local probabilistic motion models. Local Feature Enhanced KMP, or LFE-KMP, resamples endpoint neighborhoods and assigns those points very small covariance before refitting and fusing local predictions. That compact description contains three different kinds of computation. Understanding their separation is more useful than treating the pipeline as one large policy. The first is semantic selection. Which object does the person mean, and what operation is being requested? GroundingDINO provides language-conditioned object detection: category names and referring expressions can specify what to find. This is a useful connection between language and visual candidates, but it is not itself a metric description of a pouring interaction. Detecting the right object and locating its functionally relevant geometry remain different problems. For example, consider two uncertainties that sound similar in conversation. “Which cup?” is uncertainty about the intended target. “Where exactly is this cup’s opening?” is uncertainty about its geometry. Asking the user a clarifying question can resolve the first while doing nothing for the second. Conversely, improving depth estimation cannot tell us which of two equally plausible cups the user intended. I would preserve that distinction in the software interface. The target-selection module should return an identity and evidence for that identity. The geometric module should return measurements, frames, and uncertainty. Passing one generic confidence score between them would conceal too much. The second computation is task specification. This is where semantic intent becomes geometric conditions rather than merely another function name. There is an important historical qualification here. Code as Policies was already more expressive than a sequence of canned skill calls. Its language-generated programs could process perception, perform spatial calculations through numerical libraries, and express feedback or waypoint-based policies. Consequently, the meaningful comparison is not “old systems selected functions, whereas this system understands geometry.” It is how geometry is represented, measured, and connected to the motion backend in a particular implementation. The third computation is motion adaptation. Once we have a meaningful start condition and goal relationship, how should the robot get between them while retaining useful structure from a demonstration?. That is where movement primitives enter. Importantly, a good answer to this third question does not rescue a bad answer to the second. An accurately tracked trajectory can still implement the wrong interaction. Why semantic keypoints are an effective interface. The conceptual precedent for using a handful of landmarks is particularly clear in kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation. kPAM represents objects using semantic 3D keypoints and formulates manipulation objectives and constraints over them. The attraction is category-level consistency: corresponding landmarks can retain their functional meaning across objects whose detailed shapes differ. An instance-specific CAD model is not the only route to specifying useful manipulation geometry. For our illustrative pouring problem, think about what each landmark could contribute. The mouth center provides a reference for the receiving opening. A base-to-mouth direction provides an approximate longitudinal axis. A handle supplies an asymmetric feature that can help distinguish orientations otherwise ambiguous for a nearly rotationally symmetric cup. A rim landmark identifies a location directly relevant to the interaction. The useful abstraction is not “four points reconstruct the mug.” They do not. It is “these points may retain the particular geometric relationships needed by this skill.”. That distinction determines where the representation will work. A sparse landmark set might be sufficient to align a pouring rim, while remaining insufficient to establish clearance between every surface on both objects. It might describe a rigid vessel well while saying little about liquid motion. I would treat representation sufficiency as a task-specific hypothesis, not a property conferred by the word semantic. The perception architecture matters for a related reason. HRNet’s original contribution was to maintain high-resolution representations while repeatedly exchanging information across parallel resolution streams. That design targets spatially accurate keypoint localization rather than only image-level recognition. It explains why a dedicated landmark estimator is a sensible component when the downstream computation is sensitive to where a feature lies, not just what object is present. But selecting an appropriate backbone is only the beginning of a measurement pipeline. In an implementation audit, I would ask whether the same physical landmark is predicted consistently across viewpoints, partial occlusions, and changes in object proportions. A visually plausible point is not necessarily a stable task coordinate. Consider a rim landmark that drifts around a circular opening as the camera moves. Every prediction might lie on the rim, yet a controller interpreting it as a persistent pouring-side landmark would receive inconsistent information. The annotation convention and temporal correspondence matter as much as the detector’s average pixel error. K-VIL: Keypoints-Based Visual Imitation Learning provides another relevant foundation. It learns object-centric task representations from human demonstration videos, including keypoint-based geometric constraints, associated local frames, and movement primitives. Its execution machinery uses a keypoint-based admittance controller. This establishes a broader context in which landmarks are not merely perception outputs: they are variables through which demonstrated interactions can be represented and controlled. For a VL-MP-style implementation, my preferred interface would therefore resemble a typed geometric record. It would distinguish landmark identity, coordinate frame, estimated location, observation reliability, and task role. That is more informative than a bare pose vector—and much easier to interrogate when a successful-looking plan becomes an unsuccessful physical interaction. From geometric goals to a distribution of motions. To understand the backend, start with the original Kernelized Movement Primitives work by Yanlong Huang, Leonel Rozo, João Silvério, and Darwin Caldwell. KMP learns from a reference distribution rather than only fitting a single mean curve. The reference describes expected outputs and their covariance at different inputs, often time or movement phase. Kernel-based prediction then reproduces and adapts those distributions without requiring an explicitly chosen finite set of basis functions. Desired via-points or endpoints can be incorporated with associated covariance. Crucially, the original work also includes local-coordinate representations for task-dependent extrapolation; local frames are not, by themselves, a new invention of VL-MP. For a professional audience, the useful mental model is a motion prior with structured flexibility. Some parts of a demonstrated movement may vary substantially without affecting success. Other parts may need to be reproduced closely. A probabilistic representation provides a way to express that distinction, although interpreting the resulting covariance requires care. Now consider task parameterization. Sylvain Calinon’s tutorial on task-parameterized movement learning describes representing demonstrations in multiple coordinate systems associated with the task. Changing those frames changes the circumstances in which the learned movement is retrieved. Gaussian representations allow information from different frames to be transformed and combined, rather than expressing everything permanently in the original world coordinates. An intuitive example is a transfer motion between two objects. Near departure, the movement should depend strongly on the source arrangement. Near arrival, it should depend strongly on the destination. If the destination moves, we generally do not want to translate the entire original trajectory rigidly. Nor do we necessarily want to interpolate between two endpoint poses while discarding the demonstrated approach. I find it useful to picture each task frame as making a proposal about the movement. The source-relative model proposes how to leave. The destination-relative model proposes how to approach. Their compatibility is what matters. Combining Gaussian proposals is not the same as taking an unweighted average of curves. Their uncertainty shapes their influence, including which directions are tightly constrained. A narrow distribution can dominate a particular geometric requirement while leaving other directions comparatively flexible. That is the mechanism that makes local probabilistic representations useful—and also makes incorrect covariance choices consequential. Orientation introduces another complication. Rotations cannot safely be treated as arbitrary unconstrained coordinates. Calinon’s Gaussians on Riemannian Manifolds explains how local tangent-space representations, mappings back to the manifold, and transport between tangent spaces support probabilistic learning with geometric quantities such as orientations. The practical objective is to preserve the geometry of the variable while still benefiting from familiar statistical operations. For pouring, I would not regard that as mathematical decoration. The difference between a useful tilt and an inappropriate orientation is part of the task itself. Still, none of these ingredients should be mistaken for a complete motion-planning certificate. A distribution over end-effector poses answers a different question from whether a particular robot can traverse those poses within its joint, collision, and actuation constraints. My architectural preference is to keep those responsibilities separate: use the primitive to propose a task-shaped motion, then validate and execute that proposal through embodiment-aware machinery. Why an endpoint neighborhood matters more than an endpoint. Here is a simple thought experiment. Draw two points on a page and demand that a curve pass through both. That requirement leaves enormous freedom. The curve can approach the destination from above or below, turn sharply just before arrival, or flatten an important arc. Now add several nearby points at the beginning and end. Those neighborhoods begin to constrain departure and arrival behavior, not merely position. This is the useful intuition behind local feature enhancement. A related technical disclosure, CN118682751A, describes a task-parameterized KMP construction that samples local portions of reference trajectories, assigns the resulting desired points very small covariance, merges them into an expanded reference, and recomputes predictions. It also describes transforming local results and combining their Gaussian distributions. This separate disclosure helps explain the implementation pattern, but should not be treated as evidence that every disclosed parameter setting was used in the paper’s experiments. My interpretation is that these local samples act as additional motion requirements. Instead of saying only “end here,” the representation says something closer to “arrive through this kind of local movement.”. That can be useful when the final approach contains functional structure. For an illustrative pour, position and tilt may need to change together. For an insertion, the approach direction may matter. For lifting an object out of a constrained arrangement, the departure behavior may matter more than the middle of the transport. There is also a trade-off. More strongly weighted local samples preserve more of the original behavior, but reduce freedom to accommodate a genuinely different situation. If the new arrangement requires a different approach, aggressively preserving the demonstrated neighborhood could be exactly the wrong bias. The covariance interpretation deserves particular attention. In Uncertainty-Aware Imitation Learning using Kernelized Movement Primitives, Silvério and colleagues distinguish notions of variance associated with demonstrated variability from uncertainty associated with missing training evidence. Their work uses KMP to reason about variability, correlations, and uncertainty together. It is a reminder that “small variance” does not have one universal meaning across a robot-learning pipeline. In particular, assigning tiny covariance to a desired point is an optimization instruction. It does not prove that the point was measured accurately. I would therefore maintain two separate concepts: how important a geometric condition is, and how confidently its location is known. A safety-critical condition inferred from an unreliable observation should trigger better sensing or a pause—not automatically become an extremely confident numerical anchor. What the experiments show—and what they do not. The pouring success criterion is transferring half a cup into the selected mug without spillage. The preprint reports ten trials per condition:. | Method | Single target | Ambiguous targets | |---|---:|---:| | VLM+Prim | 5/10 | 2/10 | | Diffusion Policy | 4/10 | 0/10 | | VL-MP | 9/10 | 8/10 |. These are the reported results, not independently reproduced measurements. The trajectory experiment uses six GShape demonstrations and reports:. | Method | Smoothness score ↓ | Shape similarity ↑ | |---|---:|---:| | KMP | 0.4127 | 64.95% | | TP-GMM | 0.0911 | 84.89% | | Proposed method, labeled “SemKMP” | 0.0663 | 92.76% |. The similarity measure averages alignment between corresponding trajectory-segment directions. The physical results are encouraging, but the trial count matters. Under an independent Bernoulli-trial assumption, my calculation of a 95% Wilson interval for eight successes in ten trials is approximately 49% to 94%. That is not an author-reported interval; it illustrates the uncertainty associated with estimating a success probability from a small sample. The Wilson construction is described in the NIST statistical handbook. The assumption is also consequential. Repeated trials using closely related objects, arrangements, and calibration conditions may not provide the same evidence as independent samples from a broad deployment distribution. I would want the evaluation unit specified carefully: repeated execution, new arrangement, new object instance, or new task. The Diffusion Policy comparison needs similar restraint. The original Diffusion Policy represents a visually conditioned action distribution using denoising diffusion and executes predicted action sequences through receding-horizon control. It is not inherently a dialogue-based target-disambiguation system. A poor result on an ambiguous-instruction benchmark therefore cannot, by itself, establish that diffusion is an inferior representation for the underlying motor skill. The follow-up I would request is straightforward: give every motion backend the same correctly selected target and equivalent geometric information. Then compare execution. Separately, compare target selection while holding the execution backend fixed. The handwriting result is useful as a diagnostic, but I would not promote a direction-matching score into a general geometric guarantee. As a counterexample, translating a curve preserves all its segment directions while potentially placing it in entirely the wrong location. A trajectory metric can reward a desirable property without establishing endpoint accuracy, collision clearance, or task success. Likewise, I would interpret the reported smoothness number as a benchmark-specific score until its implementation, sampling convention, and physical interpretation are reproduced. The most defensible empirical takeaway is thus about the tested system and conditions: the proposed combination merits closer investigation. It is not yet a universal ranking of motion-learning families, nor a calibrated claim of industrial positioning accuracy. The closest comparisons are not simply end-to-end VLAs. For conceptual comparison, ReKep is especially important. ReKep represents manipulation requirements as Python functions mapping 3D keypoints to numerical costs. Vision and vision-language components generate these relational constraints, and hierarchical optimization produces end-effector actions. Its project also demonstrates closed-loop replanning using tracked keypoints, including recovery when objects are perturbed during a multi-stage interaction. This is a distinctly geometric use of foundation models, not merely language selecting a canned action. The distinction I would emphasize is where the motion structure comes from. A constraint-optimization approach can search for behavior satisfying the specified relationships. A demonstration-centered approach adds a stronger prior about what that behavior should look like. Neither is automatically preferable. The important question is whether the task is better specified by its outcomes and constraints, or also by the detailed manner in which it should be performed. A wiping pattern, dispensing motion, or carefully demonstrated departure maneuver may benefit from a trajectory prior. A new arrangement requiring a substantially different route may instead benefit from greater planning freedom. VoxPoser offers another interface choice: language models and perception construct composable 3D value maps used for motion generation. It illustrates that a semantic-to-geometric connection need not take the form of sparse landmarks. Spatial fields and landmark constraints provide different ways to expose task structure to a numerical backend. My practical comparison would be representational rather than ideological. Which interface expresses the relevant interaction with the least ambiguity? Which can be measured reliably? Which makes failure diagnosable?. K-VIL is also a useful reminder that object-centric constraints and demonstrated motion structure already form a productive combination. Its contribution includes extracting such representations from very few human demonstrations rather than requiring every constraint to be manually specified. Taken together, these works suggest a useful research agenda: investigate how constraints, motion priors, and online feedback should be combined. The interesting boundary is not “classical robotics versus foundation models.” It is the boundary between what should be inferred semantically, what should be measured geometrically, and what should be enforced numerically. How I would adapt and test the idea. I would begin with an implementation that is deliberately less ambitious than the complete language-driven system. First, I would establish an oracle-geometry baseline. Supply correct object identities, calibrated landmarks, and task frames manually or from a trusted measurement process. Then test motion adaptation under controlled changes in start and destination arrangements. If the backend fails with correct inputs, introducing a VLM will make diagnosis harder rather than improve the experiment. At this stage, I would vary translation, rotation, and object proportions separately. “Generalization” is too broad a label to explain a failure. A model that handles moved targets might still fail when a target rotates. A representation robust to scale changes might remain sensitive to which side of an object is visible. Second, I would audit the landmark schema before investing heavily in detector training. Every point needs a stable definition. I would ask annotators to identify it across unfamiliar instances and viewpoints, then examine disagreement. If the intended landmark is ambiguous to informed humans, the resulting training target is unlikely to become reliable merely through a larger network. The evaluation should include task-space sensitivity. A small image error can be harmless for one landmark and consequential for another. Rather than reporting only average keypoint accuracy, I would perturb each landmark and measure the resulting change in the proposed interaction pose. Third, I would validate the object-to-gripper relationship explicitly. A trajectory describing the held object must be converted into a robot command through the actual grasp transform. I would test that conversion independently, including changes in grasp placement. Otherwise, errors in grasp geometry can be misdiagnosed as failures of perception or trajectory generalization. Fourth, I would perform a factorial ablation. Keep the demonstrations, selected target, and low-level execution machinery fixed. Vary the geometric interface and the local trajectory enhancement independently. The useful comparison is not only full system against unrelated baselines; it is whether each proposed mechanism changes the failure modes it is supposed to address. I would also sweep the strength and extent of local enhancement. How many neighboring points are preserved? How strongly are they weighted? At what displacement or rotation does preservation become counterproductive? These tests would expose the operating region of the prior rather than compressing it into one success number. Fifth, I would add execution monitoring without assuming that replanning means restarting the whole trajectory. If the receiving vessel moves, or the held object slips, the system needs a rule for updating geometry, preserving progress, and deciding whether continuing remains sensible. This is a separate design problem from generating a good initial path. Finally, I would insist on embodiment-aware validation before execution: reachable poses, joint limits, collisions involving the held object and robot links, and an appropriate timing profile. In my proposed architecture, the movement primitive is a candidate generator, not the final authority to move the hardware. On reproducibility, I did not locate an author-verified public VL-MP implementation or dedicated project release during this research. That is a search limitation, not proof that none exists. The public kPAM implementation and Diffusion Policy repository are useful adjacent resources for constructing comparison systems, but neither substitutes for reproducing VL-MP. Before calling an implementation a reproduction, I would seek the landmark training protocol, exact trajectory preprocessing, covariance settings, prompt templates, baseline training details, and evaluation splits. The takeaway: preserve meaning all the way to motion. The enduring idea here is not that a VLM should replace a controller, or that movement primitives should replace learned policies. It is that the interface between understanding and action deserves its own design, supervision, and evaluation. A useful interface should retain object identity without discarding interaction geometry. It should communicate a goal without pretending that the goal specifies the whole movement. And it should let a motion prior preserve useful demonstrated structure without turning that structure into an unquestionable requirement. For a lab considering this direction, I would frame the opportunity quite concretely: choose one interaction where the relevant geometry can be expressed through stable landmarks, where demonstrations contain valuable approach or departure behavior, and where failures can be decomposed experimentally. Build the geometric and motion interfaces first. Add language-driven selection once those interfaces are measurable. The accessible preprint supplies a promising system-level result, but final journal revisions and a complete reproducibility package remain unverified in this assessment. My bottom line is nevertheless clear: the valuable research problem is not merely connecting language to motion. It is ensuring that the information needed for physical success survives that connection.