Latest / Embodied AI 101 / Meaning, Motion, and Two Hands: A Task Graph That Actually Plans

Transcript

Meaning, Motion, and Two Hands: A Task Graph That Actually Plans · Embodied AI 101

• Meaning, Motion, and Two Hands: A Task Graph That Actually Plans. Semantic–Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning • Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, and Georgia Chalvatzaki • PEARL Lab and IKIDA at TU Darmstadt, Hessian.AI, Robotics Institute Germany, and the Center for Cognitive Science • IEEE Robotics and Automation Letters • 2026. The premise of this paper is straightforward: a robot cannot robustly reproduce a bimanual task if its representation captures only the action sequence or only the geometry. It needs both. It must understand that a bottle is being lifted in order to pour into a bowl, but it must also track where that bottle, bowl, and the two hands are moving. More importantly, it must preserve enough of this structure to decide what to do next when different demonstrations perform the task in different orders. Franziska Herbert and colleagues turn that premise into a compact, object-centric task model. They represent human demonstrations as temporal scene graphs, encode those graphs with a message-passing neural network, and decode them into future actions, target objects, and object trajectories. They then transfer the learned representation to a physical bimanual robot, where its predictions are combined with a library of Probabilistic Movement Primitives, or ProMPs, for online action selection. A quick bibliographic note: the PEARL Lab announced the paper’s RA-L acceptance on June 15, 2026, and the supplied DOI now resolves to IEEE Xplore. The open manuscript analyzed here is arXiv version 2, dated May 30, 2026. Why bimanual imitation is a representation problem. Many bimanual manipulation tasks are not fixed trajectories. Consider clearing several objects into a container. One demonstrator may begin with the object nearest the right hand; another may use the left hand first. The two hands can change actions asynchronously, and demonstrations may contain different object sets. Even when the high-level outcome is identical, the action ordering, hand assignment, and interaction geometry can differ. A sequence model can learn that “lift” often follows “approach,” but it can struggle when the identity and location of the relevant object changes. A geometry-only model can predict that a hand is moving toward a bottle, but without semantic task structure it may not know whether that motion is part of lifting, pouring, placing, or simply repositioning. The paper’s claim is that these two forms of information should be modeled together rather than fused only at the end. This distinction becomes especially important in bimanual tasks. The left and right hands do not necessarily share action boundaries. One hand may continue holding a bowl while the other finishes stirring, places the whisk, and moves toward another object. A representation based on a single global action token therefore throws away useful structure. Herbert and colleagues instead maintain separate action-object histories for each hand while embedding both hands and all objects in the same relational graph. The work should not be interpreted as end-to-end imitation from pixels to torques. The robot is assumed to possess an action vocabulary and a corresponding library of executable movement primitives. Human demonstrations must be segmented into categorical actions with a target object for each hand. The learned model is primarily responsible for representing task progression, forecasting object motion, and selecting which existing robot skill to execute next. The central idea. The paper’s central object is a semantic–geometric task graph. At each point in a demonstration, the graph describes:. what entities are present,. where they have moved recently,. how they relate semantically and spatially,. and which overall task is being performed. The graph encoder receives no action labels. Action history enters only in the decoder. This separation is intended to prevent the perceptual representation from simply memorizing action sequences and to make the encoder reusable when moving from human to robot demonstrations. On the robot, the authors freeze this encoder and fine-tune only the decoder using ten to fifteen robot demonstrations per task. That is the main architectural contribution, but the system-level contribution is just as important. The model does not stop at offline recognition or forecasting. Its semantic predictions propose action-object pairs, its geometric predictions test whether those proposals are motion-consistent, and a conventional skill library executes the result. The paper is therefore positioned between symbolic task planning and end-to-end visuomotor learning. Constructing the semantic–geometric graph. Each graph contains one node for every tracked object and one node for each hand. Rather than create a separate copy of every node at every time step, the authors attach a temporal feature history to a single node. This substantially reduces the size of the graph compared with an explicit space-time construction. A node contains a one-hot object identity and a history of its three-dimensional centroid. The history consists of ten sampled slots, with the experiments using a stride of ten video frames. The representation therefore captures motion over a wider temporal window without forcing the message-passing network to operate on a much larger number of nodes. The graph is fully connected and bidirectional. Every directed object pair has an edge whose features encode one or more semantic relations. These include static relations such as one object being to the right of another, as well as dynamic relations such as one object approaching another. Contact, proximity, and directional relationships are computed heuristically from tracked object positions using a relation taxonomy inherited from work on enriched semantic event chains. Because multiple relations can be true simultaneously, edges use multi-hot rather than one-hot features. Finally, the graph has a global feature containing the task identity. This gives message passing access to context that does not belong to any single object or relation. That last detail qualifies the paper’s use of “task-agnostic.” The encoder is decoupled from action labels, but it is not completely task-independent: it receives a one-hot task ID. A more precise description would be action-label-decoupled scene representation. The experiments demonstrate reuse across embodiments and joint training over known tasks, not zero-shot inference of a previously unseen task. The required perceptual interface is also more structured than raw video. A third-person RGB-D system must provide object and hand centroids, object identities, and bounding boxes. Human training data additionally requires action segmentation and action-object labels. The semantic edges are inexpensive because they are generated automatically, but the pipeline still depends on reliable tracking, identity assignment, and task annotation. The graph encoder. Node, edge, and global features are first projected into a shared 64-dimensional latent space. Rotary position embeddings are applied along the temporal dimension, using the actual frame identifiers to preserve relative timing. The message-passing neural network then performs three rounds of structured information exchange. Within each round, it updates edges first, followed by nodes and the global embedding. An edge can therefore combine its previous representation with information from its sender, receiver, and the whole graph. Nodes subsequently aggregate their incident edge information, and the global state summarizes the evolving scene. Unlike the older Dreher graph-network baseline, the weights are not shared across message-passing iterations. Each iteration can learn a different stage of relational reasoning. Residual connections allow the network to refine rather than replace its previous representation. Message passing is interleaved with temporal self-attention over the feature histories, giving the model both an object-relation operator and an explicit mechanism for reasoning across time. The resulting representation contains node embeddings, edge embeddings, and a global graph embedding. This is important because the downstream outputs operate at different granularities. Action progression is decoded primarily from the global state, while object motion is decoded from object-level node embeddings. The complete proposed model has approximately 449,000 parameters. That is deliberately small. The closest encoder baselines range from roughly 313,000 to 462,000 parameters, while the non-graph Transformer contains about 418,000. The comparison is therefore primarily about inductive bias rather than gross model scale. Forecasting at two different clocks. A subtle but valuable part of the method is its distinction between frame-level continuation and semantic task progression. Suppose the right hand is currently stirring. The action at the next video frame will almost certainly still be “stir.” Predicting that next frame accurately says little about whether the model understands the task. The more informative question is what action will begin after stirring ends. The decoder therefore produces both an immediate next action-object prediction and a future semantic action-object prediction corresponding to the next high-level primitive after the current one terminates. The first is predicted from the current action-object pair and the temporally averaged global embedding. A second classifier uses that result and the same scene context to predict the later semantic transition. For each hand, the model also retains the twenty most recent semantic action-object pairs. These histories, the immediate prediction, and the future semantic prediction are embedded into Transformer queries. Action start times receive rotary temporal encoding, which matters because the two hands can switch actions at different moments. Two Transformer decoders then process these queries. The action-object decoder attends to the full temporal sequence of global graph embeddings and outputs framewise action and target-object predictions over a ten-step horizon. The motion decoder attends to node embeddings and predicts a three-dimensional trajectory for every object over the same horizon. Training combines weighted cross-entropy losses for the action and object classifiers with mean-squared error for object trajectories. The motion loss is multiplied by 1,000 to balance its numerical scale against the classification objectives. The auxiliary immediate and semantic-future classifiers help supervise task transitions, while the horizon heads support continuous online forecasting. As in action-chunking policies, consecutive model calls produce overlapping forecast windows. These are reconciled through a temporal ensemble with exponentially decaying weights. The intent is to suppress prediction jitter and produce a smoother rolling estimate of future actions, objects, and motions. From predictions to online action planning. At deployment, the model is not asked to generate robot joint trajectories directly. Each high-level action has an associated ProMP learned from teleoperated robot demonstrations. These movement primitives contain full six-dimensional task-space trajectories and gripper commands, and they can be conditioned on the current pose of the target object. After a primitive finishes, the model produces distributions over possible next action-object pairs. A symbolic precondition mask removes clearly invalid combinations, such as grasping an object that is already held. The planner retains the top four remaining candidates. For each candidate, the model rolls out a short predicted object trajectory conditioned on that action-object pair. The corresponding ProMP is also conditioned on the current state. The planner then computes a covariance-normalized distance between the neural motion prediction and the ProMP’s mean trajectory. In intuitive terms, it asks whether the object motion expected by the task model resembles motion that the proposed robot primitive can plausibly produce. The final candidate score multiplies semantic probability by geometric consistency. A proposal can therefore be rejected either because the action-object pair is unlikely or because its expected motion conflicts with the robot’s learned skill. A final workspace feasibility check runs before execution, and the planner is queried again if the selected primitive cannot be executed. Left- and right-arm actions are scored separately to accommodate asynchronous boundaries. This is a pragmatic hybrid architecture. Neural prediction supplies task context and ranking, while masks, feasibility checks, and ProMPs provide explicit execution structure. Consequently, robot success should be attributed to the complete system, not solely to the MPNN. Experimental design. The human experiments use two data sources. The first is the KIT Bimanual Actions Dataset, or KIT Bimacs, containing five cooking tasks and four workshop tasks. Six subjects perform ten takes per task. The dataset supplies RGB-D recordings, hand-action labels, object bounding boxes, three-dimensional positions, and semantic object relations. The authors add action-object annotations, remove background detections, smooth and standardize trajectories, and augment the data through mirroring and temporal resampling. The authors also record four tasks in their own environment: cooking, wiping, clearing objects from a table, and taking objects out of a box. Four subjects perform each task ten times. Cooking and wiping overlap with KIT Bimacs, while clear-up and take-out add higher object variability. Counting the two new tasks alongside the nine KIT tasks gives eleven distinct tasks. Evaluation uses leave-one-subject-out cross-validation and averages four random seeds. Models are trained separately on individual tasks, jointly over groups of tasks, and on a mixed dataset combining cooking and wiping demonstrations from the two environments. Metrics include action and object classification accuracy for the immediate, semantic-future, and ten-step predictions, plus root-mean-square error for object motion. The baselines are well chosen for isolating representation components. The Dreher-style graph uses semantic edges but no geometry. The Lagamtzis-style relational GCN uses object identities and geometry but omits learned edge and global representations. A conventional Transformer sees sequential object identities and geometry but no semantic graph. A decoder-only model removes the encoder completely. All share the proposed prediction decoder, making the comparison largely a test of the scene encoder. One caveat is the stopping protocol. The authors observed that the MPNN converged faster and evaluated every model at the same epoch. This is defensible as a fixed-compute comparison, but it may understate the eventual accuracy of slower-converging baselines. Results on human demonstrations. The aggregate result is not that the graph wins by the same amount everywhere. Rather, the graph’s advantage increases as action ordering and object participation become less predictable. For fixed or nearly fixed tasks such as cooking, the Transformer and even the decoder-only model can often match the MPNN on action classification. A strong action history is sufficient to memorize much of the sequence. The paper also observes that ten-step framewise action accuracy is an imperfect measure of task understanding: because primitives persist for many frames, a model can score well by repeating the current action. The semantic-future prediction is more discriminative because it requires anticipating the next primitive boundary. The differences become clearer in wiping, cereal preparation, clear-up, and take-out, where object choices or action order vary. Here the decoder-only model deteriorates, and the proposed MPNN generally produces the strongest object forecasts. In the clear-up task, for example, the model must use the evolving scene to determine which objects remain and which hand can safely approach them; action history alone does not uniquely determine the answer. Motion forecasting is the most consistent win. Across all displayed tasks and multi-task settings, the MPNN has the lowest trajectory error. The semantic-only Dreher graph performs especially poorly because it has no direct geometric history, while geometry-only and sequence encoders cannot exploit explicit relational changes as effectively. The qualitative cooking example is informative about remaining failure modes. Predicted object trajectories closely follow the ground truth, and the correct actions appear in approximately the right places. Nevertheless, short approach and retreat actions are assigned unrealistic durations, and the right-hand prediction flickers between placing and stirring near the end. The representation understands the broad task while still struggling with precise temporal boundaries. Multi-task and mixed-environment training do not substantially degrade the proposed model. Mixing KIT and locally recorded cooking and wiping demonstrations often improves results, presumably because the additional data outweighs the environment shift. This is useful evidence of robustness, although it remains generalization across known tasks and object vocabularies rather than zero-shot transfer to new tasks. Human-to-robot transfer. The transfer experiment is one of the paper’s strongest components. The authors compare three regimes: training only on the small robot dataset, applying the human-trained representation directly to robot data, and fine-tuning only the decoder after human pretraining. For cooking, robot-only training achieves immediate and semantic-future action accuracies of roughly 74.6 and 65.9 percent, with a motion error of 5.47 centimeters. Human pretraining without robot fine-tuning raises the two action metrics to about 92.8 and 92.5 percent and reduces motion error to 3.04 centimeters. Decoder-only fine-tuning reaches approximately 94.5 and 96.4 percent, with motion error falling to 2.12 centimeters. The clear-up results tell a similar story. Robot-only training is already strong on immediate action prediction but poor at choosing future objects: its future and horizon object accuracies are only 28.3 and 20.5 percent. Human pretraining raises those values to 45.1 and 48.7 percent. Fine-tuning increases horizon object accuracy to 55.7 percent and lowers motion error from 3.52 to 2.35 centimeters. Not every individual metric improves—the future-object score dips slightly after fine-tuning—but the overall task-progression and motion picture becomes substantially stronger. The robot evaluation uses ten trials per method and task. Success is defined as the proportion of completed subtasks, rather than merely whether a single final condition is reached. Cooking uses varied object placements but a fixed action-object order. Clear-up varies the types of objects that must be removed. On cooking, the proposed MPNN, Transformer, decoder-only model, and semantic-only Dreher graph all obtain a perfect reported completion rate. Only the Lagamtzis baseline falls noticeably short, at 72.5 percent. The MPNN nevertheless produces the lowest motion error at 4.69 centimeters. This result is appropriately modest: when task order is fixed, a sophisticated relational representation is not required for success. Clear-up is the consequential result. The MPNN reaches a completion rate of 1.0, compared with 0.799 for Dreher, 0.739 for decoder-only, 0.732 for the Transformer, and 0.332 for Lagamtzis. Its motion error is 4.40 centimeters, again the lowest. Its planner infeasibility rate is about 10.8 percent, versus approximately 33 to 66 percent for the learned baselines. The authors report that the sequence models sometimes propose approaching objects on the wrong side of the table. A fine-tuned Florence-2-Base model achieves zero completion on both tasks. It tends to repeat the currently executing action and produces extremely poor position estimates. This is evidence that a generic vision-language model is a poor drop-in replacement under this data and interface budget. It is not, however, evidence that graph models broadly dominate modern VLAs: Florence-2 is not a robotics action model, and predicting continuous three-dimensional motion from a tiny supervised robot dataset is not its native use case. The evaluation also allows repeated recovery. A trial fails only after the same planning request is rejected ten consecutive times. This is sensible for an online planner, but it means the reported completion rates include the benefits of re-querying, symbolic masks, physical feasibility checks, and the ProMP library. Relation to earlier approaches. The most direct predecessor is Dreher, Wächter, and Asfour’s graph-network approach to bimanual demonstrations. That system builds symbolic spatial graphs from RGB-D observations and performs framewise bimanual action classification and segmentation. Herbert and colleagues effectively extend that line from recognizing the present to forecasting task progression, predicting object motion, and making robot decisions. Lagamtzis and colleagues jointly model action recognition, action prediction, and human motion forecasting with a shared GNN encoder for industrial collaboration. The present paper adds semantic edge features and global task context, shifts attention toward object-centric tabletop manipulation, treats the two hands asynchronously, and evaluates multi-step robot action selection rather than single-step human-intention prediction. Razali and Demiris approach the problem in the opposite direction. Given an action label and initial human-object configuration, their model generates a complete bimanual manipulation sequence, including wrists, objects, fingers, and body pose. Herbert and colleagues instead infer the action and target object from scene history, then use motion prediction to evaluate candidate robot skills online. One work is primarily action-conditioned generation; the other is task-progress forecasting and planning. The contrast with systems such as Universal Manipulation Interface is equally useful. UMI collects embodiment-aligned demonstrations with handheld grippers and trains direct deployable visuomotor policies for dynamic and bimanual behavior. The semantic–geometric graph paper transfers a higher-level task abstraction from conventional human RGB-D demonstrations while relying on separately learned robot primitives for execution. UMI targets motor-policy transfer; this paper targets task-structure transfer. Why the paper matters. The paper’s most important contribution is empirical rather than architectural: it identifies when relational structure pays off. If a task follows one fixed action sequence, a Transformer or even a decoder driven largely by action history may be enough. The real value of the graph appears when the state of the objects disambiguates what should happen next. This sounds intuitive, but the paper demonstrates it across low- and high-variability human tasks and then reproduces the same pattern in physical robot experiments. The work also presents a useful middle ground in the current robotics landscape. Classical task-and-motion planners offer explicit structure but require manually specified operators and geometric constraints. End-to-end VLAs can absorb broad visual and linguistic priors but typically need much more data and make their internal task state difficult to inspect. This model learns a compact task representation from demonstrations while retaining explicit objects, relations, actions, trajectories, masks, and skill candidates. Its transfer mechanism is similarly practical. Freezing the human-trained encoder and adapting only the decoder suggests that object-centric relational dynamics can survive an embodiment change. Part of that success comes from abstracting away the human and robot kinematics in advance: both are represented largely as moving hand or end-effector nodes interacting with objects. That is not universal embodiment invariance, but it is a useful engineering abstraction. Finally, the motion head is not merely an auxiliary training loss. It participates in planning. Many multitask representation papers predict several quantities during training but discard most of them at deployment. Here semantic and geometric predictions jointly influence the selected robot primitive, giving the learned representation a clear system-level role. Limitations and open questions. The first limitation is supervision. Human demonstrations require segmented action labels and a target object for each hand. Object identities and three-dimensional tracks must already be available. The learned representation begins after several difficult perception and annotation problems have been solved. Second, the graph uses object centroids rather than full six-dimensional poses. That is adequate for approaching, lifting, placing, wiping, or clearing objects whose manipulation frame is defined elsewhere by the primitive library. It is inadequate for tasks where orientation is the task: inserting a keyed part, rotating an object in hand, aligning connectors, folding material, or coordinating two hands around a long rigid object. The authors identify six-dimensional, potentially equivariant graph representations as an important extension. Third, deterministic mean-squared-error trajectory prediction is a poor match for genuinely multimodal futures. Clear-up may allow several equally correct next objects and several valid arm assignments. A regressor can average these futures into a motion that corresponds to none of them. Mixture models, latent-variable predictors, or diffusion-based object-trajectory heads would better represent this ambiguity. Fourth, one-hot task and object identities constrain open-world generalization. The multi-task experiments show that the encoder can share structure across a fixed vocabulary, but they do not test unseen object categories or language-specified tasks. Semantic visual embeddings, category hierarchies, or a VLM-based perceptual front end could preserve the graph while relaxing this closed-set assumption. Fifth, a fully connected directed graph has quadratic edge growth. That is manageable for the small tabletop scenes studied here but becomes expensive and noisy in a kitchen or workshop containing dozens of tracked entities. Learned edge selection, interaction neighborhoods, or hierarchical scene graphs would be natural scaling mechanisms. The bimanual formulation itself also remains restricted. Each hand selects a primitive and a single target object, and the final candidates are scored independently for the two arms. Shared graph encoding captures some coordination, but independent final selection may not express tight joint constraints such as two-handed carrying, cooperative insertion, cloth tension, or simultaneous force closure. Those settings call for joint action-pair decoding and explicit inter-arm feasibility. The robot evidence is promising but narrow: two tasks, ten trials per model, and only one task where high variability strongly differentiates the methods. There is no dedicated ablation that removes the ProMP motion-consistency score while keeping the semantic planner fixed. Such an experiment would clarify exactly how much robot success comes from the geometric forecast rather than the graph encoder’s improved action-object probabilities. Reproducibility is another near-term issue. As of August 5, 2026, the official project page labels the code as “coming soon,” and its repository link is still a placeholder. How roboticists might use this approach. The clearest application is as a high-level scheduler for an existing skill library. If a robot already has reliable grasp, lift, place, pour, hold, and wipe controllers, this model can learn how those skills compose under variable object arrangements without requiring a manually written task graph. A second use is as a structured layer above a VLA. The graph model could propose an action, target object, and expected object trajectory. A generative low-level policy could execute the skill, while the trajectory head acts as a consistency check or subgoal specification. This preserves the VLA’s motor flexibility while adding an interpretable object-level task state. For orientation-critical work, the most valuable extension would be to replace centroids with six-dimensional poses and use SE(3)-aware message passing. For highly variable tasks, the deterministic motion head should become multimodal. For dense scenes, interaction edges should be sparsified. And for truly coordinated bimanual behavior, the two hands should be decoded jointly rather than treated as independent final decisions. The experimental results also contain a practical lesson for dataset design: if every demonstration follows the same sequence, there is little reason to expect a relational model to outperform sequence memorization. Demonstrations must include meaningful permutations of object identity, action order, hand assignment, and geometry. Variability is not simply noise to be normalized away; it is the signal that teaches the representation which scene relationships determine task progress. Bottom line. This paper is not a claim that graph networks should replace VLAs or end-to-end imitation policies. It is a strong demonstration that structured object-centric representations remain useful when the core problem is reasoning over variable task progression with limited data. Its decisive idea is to keep semantics and geometry together: actions say what the task means, object motion says how the scene is changing, and the graph makes their relationship explicit. The cooking task shows that this machinery is unnecessary when a sequence can simply be memorized. The clear-up task shows why it matters when the robot must actually look at the evolving world and decide what remains to be done. That is the paper’s lasting contribution: not merely another bimanual predictor, but a compact account of when relational inductive bias becomes operationally valuable—from human demonstration, through task representation, to physical robot planning.