Latest / Embodied AI 101 / SIEVE: Structure-Aware Data Selection for Imitation Learning with VLAs
Transcript
- SIEVE: Structure-Aware Data Selection for Imitation Learning with VLAs. Imitation learning for robotics often relies on collecting large demonstration datasets. Naively, one might assume that more data guarantees a better policy. However, as recently pointed out in the literature, “more data does not necessarily yield better policies” when demonstrations contain redundancy, noise, and uneven coverage. In fact, offline imitation learners frequently suffer from distribution shift – a policy trained on demonstrations will inevitably encounter novel states at test time, making high-quality, well-curated datasets critical. In short, in contrast to internet-scale data for vision or language, robot learning has to be data-efficient and data-centric. This has spurred a line of work on data selection or demonstration curation for imitation learning. One emerging setting is the training of Vision-Language-Action (VLA) models – large policy networks that take visual (and possibly language) inputs and output motor trajectories. Examples include recent transformer-based robotic controllers like RT-1/RT-2 and foundation models (PaLM-E style) for manipulation. These VLA models are typically trained by behavior cloning on collected robot demos. But as SIEVE’s authors note, simply adding more demonstrations to train such VLAs often has diminishing returns, due to overlapping or low-quality examples. Instead, which data we include can matter as much as how much data we have. Existing data-selection methods in imitation learning often operate at a coarse level – for example, scoring entire trajectories by some distance or diversity metric, or even treating each state-action pair independently. In contrast, SIEVE is motivated by the insight that robotic tasks have an inherent temporal structure: long-horizon behaviors are built from reusable sub-behaviors. If demonstrations share underlying primitives or skills, a good data selection scheme should recognize and exploit those structures. As the SIEVE paper puts it, many existing methods “ignore the reusable structures that compose long-horizon behaviors”. Indeed, the idea of breaking down demonstrations into reusable components is well known in robotics. For example, Lioutikov et al. describe movement primitives as a “well-established approach for encoding and executing movements,” forming a library of sub-skills that can be sequenced to solve tasks. SIEVE builds on this intuition: instead of treating each demonstration as an isolated example, it views each trajectory as a composition of reusable visuo-motor primitives and transition interfaces. By identifying these parts across the dataset, SIEVE can deliberately pick demonstrations that cover the needed sub-skills efficiently, rather than wasting effort on repetitive or noisy data. Concretely, SIEVE’s pipeline involves a few key steps:. Segment and Discover Primitives. First, SIEVE automatically segments each demonstration trajectory into contiguous chunks corresponding to primitive behaviors. We can think of this as dividing a pick-and-place sequence into phases like “approach object,” “grasp object,” “move to goal,” and “release.” By clustering these segments across all trajectories, SIEVE discovers a set of visuo-motor primitives – prototypical motion patterns or skills that recur in the data. Each primitive might correspond to a distinct object manipulation or motion style, represented in terms of the sensorimotor sequence. The boundaries between primitives are the transition interfaces – essentially the end-state of one primitive and start of the next. These transitions are also important because they define how primitives can be legally composed. Identify Composition Patterns. Once the primitives are defined, each original trajectory can be represented as a composition pattern – an ordered sequence of primitives (and their transitions) that the demonstration implements. For example, a pick-and-place trajectory might have the pattern (approach, grasp, move, release). SIEVE collects the set of all such patterns present in the dataset. Allocate Budget Across Patterns. Now comes the novel selection strategy. Instead of sampling uniformly or randomly, SIEVE decides how many trajectories to pick for each composition pattern, by optimizing coverage of the identified structure. Formally, it defines a notion of “structural exposure” that measures how well a set of selected trajectories covers the discovered primitives and transitions. It then greedily allocates a limited budget of trajectories to maximize this structural exposure, with a diminishing-returns constraint to avoid redundancy. Intuitively, this means choosing trajectories so as to cover as many unique primitives and transitions as possible, but adding an extra example of the same primitive yields progressively less new information. This submodular-like allocation ensures a balanced coverage: if two patterns share components, SIEVE keeps sampling from whichever pattern covers the least-exposed component next, until budgets are filled. Select Central (Medoid) Trajectories. Within each pattern bucket, SIEVE then selects the actual demonstrations to use. It does so by choosing the medoid trajectory of each pattern cluster – the trajectory that is most “central” to others under some similarity metric. The medoid is a robust representative: it is less likely to be a noisy outlier and more likely to exemplify the pattern clearly. By taking medoids, SIEVE retains “central, stable, and imitation-friendly” examples from each pattern. The result is a subset of the original demos, typically only a fraction of them, that collectively span the discovered primitives and transitions thoroughly. After this selection procedure, the pruned dataset (consisting of one or a few medoids per pattern) is used to train the VLA model by normal behavior cloning. In effect, SIEVE filters out redundant trajectories (e.g. many similar grasps of the same object) and noisy ones, leaving a compact training set that still covers all the required skills. Why does this work? The key is that many robotic tasks exhibit modularity. If a task is compositional – say, moving one type of object after another – then different complex tasks may still reuse the same grasp or move primitives. By curating data that highlights each primitive and transition at least once, SIEVE ensures the model sees all the building blocks. Meanwhile, it avoids overfitting to unrepresentative quirks. This is a departure from prior curators that score whole trajectories (or individual frames) ignoring that two trajectories might share much of their structure. For instance, a naive method might drop a trajectory simply for being long or “outlier,” whereas SIEVE knows if that trajectory introduces a new primitive sequence it could be valuable even if imperfect. The SIEVE authors present several compelling results. Across multiple robot learning benchmarks and datasets, using a range of VLA model architectures, SIEVE consistently outperforms strong baselines. In experiments, SIEVE often beats random subsampling or simpler diversity baselines that don’t account for structure. Remarkably, it can even surpass the performance of training on the full dataset while using only 50% of the demonstrations and 50% of the training steps. In other words, a VLA policy trained on SIEVE’s curated half-data learns faster and ends up better than a baseline policy trained on all the data. This underscores the practical payoff: not only does SIEVE deliver higher policy quality, but it cuts training time and data collection needs. (For perspective, prior work like Demo-SCORE showed two-digit percentage gains in success rate simply by filtering bad demos; SIEVE goes further by using structure to achieve even stronger improvements.). In sum, SIEVE tells us that better data beats more data for these robot learners. As one remark by the authors pithily asks, “Do VLA models need more data, or better data?” SIEVE clearly argues for the latter. It illustrates a broader lesson noted in the field: since robotics inevitably has less data than NLP or vision, we should invest effort in curating and understanding that data, rather than blindly scaling up. SIEVE’s structure-aware selection is one promising step toward that goal. Implications and Usage: For practitioners training VLAs or other imitation models, SIEVE can be viewed as a preprocessing step. Given any large demonstration pool, one would apply SIEVE to segment the data, identify primitives/patterns, and select the representative medoids. The resulting subset is then fed into your usual behavior cloning pipeline. Crucially, SIEVE does not require changing the model architecture or learning algorithm – it simply rebalances the data. Because of its submodular-like greedy selection, SIEVE is also efficient to run: after primitive discovery and clustering (which can be done via standard unsupervised learning), the selection stage is a lightweight optimization. One could also imagine iterative or online variants: for example, after an initial SIEVE pass and training, collect a little more data in weak spots and rerun. Or combine SIEVE selection with active learning: if a robot faces a new scenario missing a certain primitive, one could gather demos targeting that pattern. Relation to Prior Work: SIEVE is part of a growing trend to optimize data for robot learning. Work like Quality over Quantity (QoQ) by Lee et al. (NeurIPS 2026) similarly acknowledges that not all demos are equal. QoQ uses influence functions to score individual state-action pairs by how much they reduce loss on a validation set, treating high-influence samples as higher-quality. SIEVE shares the motivation but takes a different angle: instead of statistical influence, it looks for structural reuse. Another approach, Demo-SCORE (Stanford 2023), has robots curate data by training a success/failure classifier from trial rollouts, then filtering out trajectories predicted to be unreliable. Demo-SCORE improved task success substantially, but again it does not explicitly exploit temporal structure. In comparison, SIEVE’s novelty is in reasoning at the level of primitives and their sequences. SIEVE also resonates with ideas from hierarchical reinforcement learning (options, subgoals) and from classic movement primitive libraries. However, those earlier methods typically focus on learning or representing skills. SIEVE’s contribution is to show that the same concept can guide data selection. Finally, SIEVE complements recent theory on IL dataset quality: Belkhale et al. (NeurIPS 2023) argued that good data should keep learned policies “in-distribution” and avoid compounding error. By ensuring each primitive is well-represented, SIEVE helps the policy stay within familiar dynamics, effectively reducing distribution shift. Conclusion: In short, SIEVE is a structure-aware data curation technique for imitation learning that explicitly exploits the compositional nature of robot tasks. By discovering reusable primitives and allocating selection budgets to pattern coverage, it dramatically improves learning efficiency. Instead of drowning in data, SIEVE helps VLA models learn from just the right data. This approach highlights the importance of data-centric design in robotic AI: better data selection can speed up training, improve policies, and ultimately make embodied agents more capable. Citations: The above summary is based on Wu et al. (2026), who introduced SIEVE. For context on data curation in imitation learning, see Belkhale et al. (“Data Quality in IL”, NeurIPS 2023), Lee et al. (“Quality over Quantity”), and Demo-SCORE. The notion of movement primitives comes from prior robotics literature.