Latest / Embodied AI 101 / Three Ways to Buy Back Robot Data: Augment the Demonstration, Import the Semantics, Benchmark the Physics

Transcript

Three Ways to Buy Back Robot Data: Augment the Demonstration, Import the Semantics, Benchmark the Physics · Embodied AI 101

• Three Ways to Buy Back Robot Data: Augment the Demonstration, Import the Semantics, Benchmark the Physics. Data-Efficient and Contact-Rich Manipulation Through Diffusion Augmentation and Vision-Language Models • Daniel Seita • University of Southern California • AAAI-26 New Faculty Highlights • 2026. Bibliographic note: The supplied DOI, , appears to be mismatched. The official AAAI proceedings record lists the paper as Data-Efficient and Contact-Rich Manipulation Through Diffusion Augmentation and Vision-Language Models, with DOI. This episode uses that record and the underlying technical papers as the primary sources. Robotics has a data problem, but contact-rich manipulation has something more specific: a data geometry problem. If a vision classifier sees a translated image of a cat, the label remains “cat.” If a robot policy sees a translated image of a gripper, however, the correct action may also have to translate. Once the gripper touches an object, the permissible transformation becomes even more constrained. With two arms holding the same object, the labels for both arms are coupled. A visually plausible augmentation can therefore be physically impossible, kinematically infeasible, or simply paired with the wrong control target. This is the problem at the center of Daniel Seita’s AAAI New Faculty Highlights presentation. The paper itself is a one-page overview rather than a conventional standalone algorithm paper. It connects three recent projects: D-CODA, which generates constrained synthetic demonstrations for bimanual imitation learning; IMPACT, which uses vision-language models to decide which incidental contacts are acceptable during motion planning; and ManipBench, which tests whether vision-language models actually understand the low-level physical consequences of robot actions. The presentation also points toward ROPA, a subsequent extension of the augmentation idea to third-person RGB-D observations. Taken together, these projects articulate a useful research thesis: data efficiency does not have to mean squeezing a few more percentage points out of a smaller behavioral-cloning dataset. It can also mean generating valid counterfactual experience, importing semantic knowledge learned outside robotics, and replacing expensive robot evaluations with carefully validated proxy tests. The connecting thread is contact. D-CODA detects when contact changes the valid augmentation manifold. IMPACT asks which contacts should be allowed. ManipBench asks whether foundation models can predict what contact will do. A Research Program, Not One Monolithic System. It is important not to read this AAAI paper as if D-CODA, IMPACT, and ManipBench form a single deployed stack. They do not. They address distinct points in a robot-learning pipeline and use different robots, assumptions, representations, and evaluation protocols. D-CODA is an offline data-generation method for visual imitation learning. Its main concern is maintaining consistency between synthetic camera observations and bimanual joint-space actions. IMPACT is primarily a motion-planning framework, not an imitation-learning method. It imports zero-shot semantic priors from a vision-language model, converts them into a spatial cost representation, and uses a structured planner to select contact-rich paths. ManipBench is an evaluation framework. It turns manipulation decisions into multiple-choice visual questions so that many vision-language models can be screened without running thousands of full robot trajectories. That separation is a strength. Rather than asking one large model to perceive the scene, infer material properties, predict contact dynamics, plan a trajectory, and directly command the motors, the work assigns different jobs to different tools. Generative vision models synthesize observations. Kinematics and optimization reject infeasible labels. Vision-language models provide semantic judgments. Classical graph search produces a plan. Benchmarks measure whether the imported priors are useful. This is a distinctly hybrid view of embodied AI, and it is arguably more realistic than the idea that scale alone will make an end-to-end policy understand every physical edge case. Why Contact Changes the Economics of Robot Data. A standard imitation-learning dataset records observations and expert actions. Its coverage is concentrated around successful trajectories, because that is what the demonstrator was trying to produce. At deployment time, small policy errors push the robot away from that distribution. The robot then encounters wrist-camera views, arm configurations, and object arrangements for which the demonstrator supplied no recovery action. The standard answers are familiar. Collect more demonstrations. Use DAgger and repeatedly query an expert in states visited by the learned policy. Generate trajectories in simulation. Add visual augmentations. Train on large multi-robot datasets. Contact-rich tasks make each of these harder. Resetting a deformable object may be slow. Accurate contact simulation may require unavailable material, friction, and compliance parameters. A failed bimanual action can wedge an object between the arms or trigger force limits. And ordinary image augmentation is not enough, because changing the apparent camera pose generally changes the correct action. The situation becomes especially delicate after contact. Before contact, the two end effectors may have considerable independent freedom. Once both grippers support the same tray, drawer, ball, rope, or bag, the object imposes a relationship between them. The space of valid states effectively contracts onto a contact manifold. Randomly perturbing each arm can tear the synthetic grasp apart even if the two generated images look individually convincing. Seita’s broader research statement frames this as a need to move beyond standard physical demonstrations using three complementary strategies: synthetic augmentation that preserves action correctness, pretrained models that provide zero-shot semantic reasoning, and automated data-collection systems based on simulation, self-supervision, or dedicated physical infrastructure. Deformable objects and dense clutter are used as stress tests precisely because they expose the weaknesses of naive data scaling. D-CODA: Counterfactual Demonstrations That Respect Contact. D-CODA stands for Diffusion for Coordinated Dual-Arm Data Augmentation. It addresses eye-in-hand bimanual imitation learning: both arms carry wrist cameras, and a policy maps the two camera views to target joint positions for the two arms. The method begins with an ordinary demonstration dataset. Each trajectory contains synchronized left and right wrist images and corresponding joint-space actions. D-CODA then learns to answer a counterfactual question:. If the wrist cameras had been at slightly different poses at this moment in the demonstration, what would both cameras have seen, and what joint-space action should the robot have executed?. The desired output is not merely another rendering of the same scene. It is a new training state paired with a label that can guide the policy back toward a successful trajectory. Joint Novel-View Synthesis. D-CODA trains a conditional latent diffusion model on pairs of observations sampled from the original trajectories. The source consists of left and right wrist images at one set of poses. The targets are wrist images at another set of poses from the same trajectory. Relative camera transformations are supplied as conditioning information. Images are encoded into a VQ-GAN latent space, and a U-Net-style denoiser jointly predicts the noise added to the two target views. Relative camera transforms, along with their inverses, are injected into attention layers. Joint generation matters because the synthesized left and right images should agree about object identity, pose, orientation, color, and scene state. Running two completely independent view generators would make it easier for the arms to receive mutually inconsistent observations. This is already more structured than generic visual augmentation. But the central contribution is how D-CODA samples the transformations and derives the new actions. Free-Space and Contact-Rich States Need Different Symmetries. D-CODA uses SAM2 to segment and track the grippers over a demonstration. When depth is available, the method combines segmentation masks with depth filtering to detect gripper-object contact. Without depth, it uses an image-based heuristic related to whether the gripper remains fully visible in the wrist view. For contactless states, the two camera poses can be perturbed independently within configured translation and rotation ranges. That is a reasonable approximation while the arms are approaching an object. For contact-rich states, D-CODA applies an identical perturbation to both arms. The intuition is that if both grippers are coupled through the manipulated object, moving them together is more likely to preserve the relative grasp geometry. A global optimizer searches for perturbations satisfying minimum movement, table-clearance, inter-end-effector-clearance, and inverse-kinematics constraints. The implementation uses dual annealing for this search and a Levenberg–Marquardt inverse-kinematics solver for feasibility checks. Once a camera perturbation is accepted, the corresponding end-effector pose is transformed and converted into target joint positions. Invalid inverse-kinematics solutions are rejected. Valid synthetic states replace original states at intervals within the trajectories, while the unmodified states remain in the dataset. That last detail is important. The policy sees both the expert manifold and nearby counterfactual states. The synthetic samples teach recovery; the original samples continue to teach task completion. Conceptually, D-CODA resembles an offline, generative approximation to DAgger: it creates states that might arise from execution error and attaches corrective labels without returning to the environment or querying the demonstrator. What the Experiments Show. D-CODA was evaluated on five simulated bimanual tasks derived from PerAct2 and RLBench: coordinated ball lifting, tray lifting, box pushing, simultaneous button pressing, and rope straightening. The downstream policy was ACT, trained on 100 original demonstrations plus the generated samples. The camera-ready paper reports 2,250 simulation trials and 300 physical trials across three real-world tasks. With two wrist cameras, D-CODA achieved success rates of 73.3% on Lift Ball, 44.0% on Lift Tray, 56.0% on Push Box, 53.3% on Dual Push Buttons, and 30.7% on Straighten Rope. ACT without augmentation obtained 56.0%, 37.3%, 36.0%, 46.7%, and 18.7%, respectively. D-CODA outperformed the tested baselines on four of the five tasks. The exception was box pushing, where VISTA’s augmented overhead view made the target region easier to observe than wrist cameras alone. The “more expert data” baseline is particularly interesting. Adding another 100 demonstrations did not consistently outperform the original 100-demonstration ACT policy. That does not prove synthetic data is generally better than human data. It does show that which states are added can matter as much as how many trajectories are added. D-CODA deliberately adds nearby recovery states rather than more samples from approximately the same successful distribution. On the physical UR5 setup, D-CODA with ACT completed 17 of 20 ball-lifting trials, 14 of 20 drawer-lifting trials, and all 20 block-pushing trials. ACT without augmentation obtained 15, 7, and 15 successes. The drawer result is the clearest improvement: the success rate doubled from 35% to 70%. An additional experiment with the π₀-FAST vision-language-action model showed a large improvement on ball lifting but no meaningful improvement on the difficult drawer task, suggesting that the augmentation is policy-compatible but cannot compensate for every weakness in the downstream model. The constrained sampler also mattered. On simulated ball lifting, full D-CODA achieved 73.3%, compared with 57.3% when contact-state perturbations were sampled without constrained optimization. Replacing the original latent encoder with one trained on a less task-relevant image distribution reduced performance to 53.3%, reinforcing that generated image quality and geometric validity both affect the final policy. The Deeper Contribution. The most useful way to think about D-CODA is as constrained counterfactual data generation. The diffusion model supplies appearance. Kinematics supplies action labels. Contact detection decides which transformations are legal. Optimization rejects samples that violate the robot’s embodiment. This decomposition is more important than the particular choice of ACT or latent diffusion architecture. It offers a general recipe for robotics augmentation: identify the task’s valid symmetries, determine how labels transform under those symmetries, and narrow the augmentation space whenever physical constraints remove degrees of freedom. D-CODA is nevertheless only geometrically contact-aware. It does not synthesize force, tactile, torque, compliance, or object-internal state. Applying the same rigid perturbation to both arms is sensible for a ball or tray, but less obviously valid for a deformable bag, cloth, or cable under tension. Its camera poses must also remain reasonably close to the training distribution, and diffusion artifacts can corrupt the observation-label relationship. The paper explicitly limits D-CODA to wrist views. ROPA, the follow-on system referenced by the AAAI overview, tackles the third-person case. It fine-tunes Stable Diffusion to synthesize new robot poses and uses a modified ControlNet to generate geometrically corresponding depth, while skeleton representations and constrained kinematics provide pose and action supervision. ROPA was evaluated on five simulated and three physical tasks, suggesting that the contact-preserving augmentation idea can extend beyond eye-in-hand sensing, though at the cost of a more elaborate rendering pipeline. IMPACT: Turning Collision Into a Graded Decision. Classical motion planners generally treat collision as binary. A configuration is valid or invalid. That is appropriate when an obstacle is a human, a glass vessel, or a fixed machine enclosure. It is unnecessarily conservative when the obstacle is a pillow, foam block, empty cardboard box, or lightweight toy that can be brushed aside safely. IMPACT—Intelligent Motion Planning with Acceptable Contact Trajectories—asks a more nuanced question: if collision-free reaching is impossible or inefficient, which objects can the robot contact, from which directions, and with what likely consequences?. The system is deliberately modular. A vision-language model supplies semantic priors about contact tolerance. A geometric procedure predicts the consequences of pushes. A contact-aware A* planner searches over robot and object configurations. From VLM Judgments to a Spatial Cost Field. IMPACT starts with multi-view RGB-D observations of a cluttered tabletop. SAM2 segments and annotates the objects, and GPT-4o receives an annotated image plus a task-agnostic prompt describing general principles of contact tolerance. The model assigns each object an integer cost from zero to ten. High values mean contact is less acceptable. A soft toy might receive a low cost; a wine glass receives a high one. The target object is assigned a negative cost to attract the planner. GPT-4o is used zero-shot, so IMPACT requires no task-specific training set of acceptable and unacceptable contacts. A scalar semantic cost is not enough. Pushing a box toward open table space is different from pushing the same box toward a glass. IMPACT therefore builds an anisotropic cost map in which contact cost depends on direction. The system first constructs a three-dimensional voxel grid containing the VLM-derived object costs, then projects it into a top-down map. Objects with costs of five or below are treated as potentially pushable; those at six or above are treated as high-cost obstacles. At boundary points on low-cost objects, IMPACT estimates the inward push direction from the surface normal. It samples nearby push distances and angles and classifies the resulting hypothetical outcomes: no further collision, collision with another low-cost object, collision with a high-cost object, or collision with the target. These outcomes receive categorical safety values, weighted by how likely the sampled push is relative to the nominal surface-normal direction. The resulting directional score is blended with the original VLM cost. This division of labor is well chosen. The VLM answers, “What kind of object is this, and how worried should I be about touching it?” The geometric model answers, “What is near it, and where will it probably go if pushed from here?”. Planning Over a Changing World. IMPACT’s A* planner searches over three primitives: move, rotate, and push. Its state includes not only the end-effector pose but also the accumulated displacement of low-cost objects. A push therefore branches into a new world state in which the contacted object has moved. Push displacement is approximated using the angle between the commanded motion and the object’s surface normal. The planner’s cost includes travel and rotation costs, the anisotropic contact cost, gripper overlap with costly regions, explicit collision penalties, and proximity to high-cost objects. This is a simplified planar interaction model, not a general contact simulator. But it captures an essential property that ordinary cost maps miss: an object that was initially blocking the path may no longer occupy the same location after an intentional push. The resulting trajectory is converted back into three-dimensional robot motion. Crucially, the VLM never directly outputs waypoints or motor commands. It supplies a semantic prior within a planner whose behavior can be inspected and constrained. Experimental Results. The simulation study used 20 PyBullet scenes containing YCB objects alongside fragile meshes generated with TRELLIS. The current paper version reports 3,200 simulated trials and 200 physical trials. A trajectory counted as successful only if it reached the target, kept its accumulated semantic contact cost below ten, limited total contact duration, and displaced human-designated unsafe objects by less than ten centimeters. IMPACT reached the target in 78% of simulation trials and achieved a composite success rate of 73.75%. The corresponding A* planner without directional push analysis reached 58.25% success. A collision-free A* variant achieved only 15.75%, while LAPP—a learned language-conditioned collision-planning baseline—achieved 25%. VLM costs also improved RRT and RRT*, but their success rates remained below that of contact-aware A*. Assigning zero cost to every object reduced A* success from 73.75% to 65%, showing that simply allowing all collisions was not sufficient. The semantic prior helped the planner distinguish useful contact from reckless contact. Similar ablations reduced the success of RRT and RRT*. A study with 25 participants compared videos of trajectories without revealing the underlying method. Across the simulation comparisons, trajectories using VLM-generated costs were preferred most often. The same participants later evaluated physical trajectories. On a Franka Panda setup, IMPACT achieved 61% success, compared with 49% for LAPP on scenes containing objects seen during LAPP’s fine-tuning and 40% on scenes with unseen objects. IMPACT itself remained zero-shot. The failures reveal the method’s boundaries. The robot can become caught on an obstacle, rotate its gripper into an object, or receive VLM costs that do not align with human preferences. The physical model is planar and approximate, while execution occurs in a three-dimensional world with friction, compliance, shape uncertainty, and arm-link collisions. Still, IMPACT demonstrates a compelling architectural principle: use foundation models for semantics, not for unverified mechanics. Let the VLM identify fragility and broad affordances. Let geometry, uncertainty models, and constrained planning handle physical consequences. ManipBench: Test the Model Before Making It an Agent. The third project changes the meaning of data efficiency. ManipBench does not reduce the number of demonstrations needed by a policy. It reduces the number of expensive robot rollouts needed to discover that a vision-language model lacks a required physical capability. ManipBench contains 12,617 multiple-choice questions and evaluates 33 vision-language models drawn from ten model families. Its questions span pick-and-place, articulated-object interaction, deformable manipulation, and dynamic tasks. The benchmark combines questions generated from public robot datasets, an in-house fabric-manipulation setup, and simulation environments. Rather than asking a model to emit a continuous trajectory, the benchmark presents candidate contact points, placement regions, motion keypoints, or action arrows. A model may have to select where to grasp an object, which tile should contain the object after a successful placement, where to contact a drawer and in which direction to push it, or how to strike a ball toward a target. The fabric portion is especially revealing because it separates several capabilities that are often collapsed into a single success metric. Questions test fabric-state recognition, task ordering, spatial reasoning, keypoint correspondence, temporal action understanding, action length, inverse dynamics, fabric–rigid-body interaction, fabric–fabric interaction, and counterfactual outcomes. Simulation questions include carrot placement, drawer closing, rope straightening, sweeping, and ball shooting. The evaluation found substantial variation across models and task categories, along with a persistent gap from human performance. More importantly, the authors report a strong correlation between model trends on ManipBench and performance when those models were used in real manipulation experiments. That correlation provides some evidence that the multiple-choice abstraction measures capabilities relevant to downstream action rather than only generic visual question answering. ManipBench nevertheless remains a proxy. Candidate actions are provided, so the model solves recognition and selection rather than unconstrained action generation. Most questions do not expose compounding control errors, latency, active perception, force regulation, or closed-loop recovery. A model can select the right contact point from four annotations and still fail to localize that point accurately enough for execution. The right interpretation is therefore not “high ManipBench score implies robot competence.” It is “low relevant-subset performance is a warning not to build a robot system around that model without additional adaptation.” As a screening tool, this can save considerable hardware time. The Common Pattern: Spend Structure Instead of Data. D-CODA, IMPACT, and ManipBench operate at different stages, but they share a consistent philosophy. D-CODA buys demonstration coverage with geometric and kinematic structure. It does not generate arbitrary scenes; it perturbs around demonstrations while trying to remain on a valid contact manifold. IMPACT buys contact labels with pretrained semantic knowledge. It does not ask humans to annotate every object arrangement, but it also does not trust the VLM to predict detailed dynamics. Semantic costs are filtered through geometry and search. ManipBench buys evaluation scale with action abstractions. It replaces many full rollouts with questions about the decisions that would determine those rollouts, and then checks whether benchmark trends transfer to physical tasks. This gives us three distinct currencies of data efficiency:. Experience efficiency: extract more corrective training states from each demonstration. Supervision efficiency: reuse semantic knowledge instead of collecting contact-tolerance labels for every scene. Evaluation efficiency: test candidate models cheaply before committing to physical experiments. The work also suggests a precise definition of useful robotics augmentation. Useful synthetic data must preserve not just visual realism but the full relationship among observation, embodiment, action, and contact. A photorealistic image with the wrong action label is worse than no augmentation because it teaches a physically incorrect conditional distribution. Likewise, useful foundation-model reasoning must be attached to an interface whose assumptions are inspectable. IMPACT’s object costs can be visualized, overridden, or combined with hard safety constraints. This is preferable to hiding semantic judgments inside an end-to-end action generator when failure costs are high. What Practitioners Can Borrow. For a team working on bimanual imitation learning, the most transferable lesson from D-CODA is to treat augmentation as a constrained robotics problem rather than an image-processing problem. Log calibrated camera poses, end-effector poses, joint configurations, gripper state, and preferably depth or force signals. Divide trajectories into phases with different constraint structures. Generate small perturbations first, reject samples with uncertain synthesis or invalid kinematics, and measure whether the policy actually learns recovery rather than merely becoming invariant to the augmented images. For deformable tasks, identical rigid perturbations will probably be insufficient. The same framework could instead enforce distance, tension, topology, or learned latent-state constraints. Tactile and force observations should be transformed or regenerated alongside vision. Otherwise the policy may receive a synthetic camera image indicating one contact state and a real sensor vector indicating another. For cluttered planning, IMPACT’s semantic–geometric separation is a strong starting point. A production version should retain hard exclusion zones for humans and truly hazardous objects, use the VLM only to initialize uncertain soft costs, and replan after every meaningful contact. The hand-designed push approximation could be replaced by an object-centric learned dynamics model that outputs both predicted displacement and uncertainty. Contact should be attempted only when the worst credible outcome remains acceptable. For foundation-model selection, a ManipBench-style evaluation should be customized to the downstream system. If the intended application is cable routing, generic pick-and-place accuracy is not enough. Construct questions around topology, occlusion, tension, contact sequencing, and recovery. Then run a small, controlled physical study to verify that the proxy ranking survives embodiment and perception noise. What Remains Unsolved. The largest open issue is that contact-rich data cannot be made fully cheap merely by generating more RGB. D-CODA expands the visual and kinematic neighborhood of demonstrated trajectories, but it does not discover fundamentally new interaction strategies or unseen mechanics. Training its diffusion model still requires task-relevant trajectories, and its best results come from training directly on the target-task data. Its few-shot generalization experiment works less well than training on 100 target demonstrations, indicating that generative augmentation densifies experience more readily than it transfers contact knowledge across tasks. IMPACT imports semantic knowledge, but “acceptable” contact is contextual and subjective. Whether a bottle can be pushed depends on whether it is open, full, valuable, contaminated, supporting another object, or close to the table edge. A single zero-to-ten cost cannot capture all of that. The anisotropic map recovers some context through local geometry, but richer systems will need object state, relational semantics, uncertainty, and user-specific preferences. ManipBench measures isolated decisions more readily than closed-loop competence. Future benchmarks will need to connect question answering to calibrated spatial outputs, temporal prediction, uncertainty, and intervention risk. They must also guard against benchmark contamination as public robot datasets become part of foundation-model training corpora. Seita’s stated future direction follows naturally from these limitations: multimodal augmentation that keeps vision, touch, force, and language mutually consistent; cross-embodiment transfer; object-centric representations for predicting contact outcomes; and systems that decide when to grasp, push, brace, or use a different part of the hand or arm. Closing Take. The central lesson of this AAAI highlight is not that diffusion models, VLMs, or multiple-choice benchmarks individually solve contact-rich manipulation. It is that scarce robot data becomes more valuable when we explicitly represent the structure surrounding it. D-CODA preserves the geometry of coordinated contact while generating recovery states. IMPACT combines semantic priors with directional physical reasoning rather than demanding collision avoidance everywhere. ManipBench asks whether those priors survive contact with low-level manipulation questions before we trust them on hardware. The broader message is pragmatic: do not generate more data indiscriminately. Generate the states the policy is missing, preserve the labels that physics requires, import only the knowledge foundation models are qualified to provide, and benchmark that knowledge against the decisions the robot must actually make.