Latest / Embodied AI 101 / A Thousand Tasks, One Day, and the Case Against Learning Everything End to End
Transcript
- A Thousand Tasks, One Day, and the Case Against Learning Everything End to End. Learning a Thousand Tasks in a Day • Kamil Dreczkowski, Pietro Vitiello, Vitalis Vosylius, and Edward Johns • The Robot Learning Lab at Imperial College London • Science Robotics, Volume 10, Issue 108 • 2025. “Learning a thousand tasks in a day” sounds like the headline for a giant vision-language-action model: billions of parameters, years of pretraining, and an industrial-scale fleet collecting robot data. But this paper makes almost the opposite argument. Its central result comes not from making the policy more general, but from making the learning problem more structured. The proposed system, Multi-Task Trajectory Transfer—or MT3—does not attempt to absorb every demonstrated behavior into the weights of one monolithic network. Instead, it stores demonstrations in an external memory, retrieves an appropriate example at deployment time, geometrically aligns the robot to the new scene, and then replays the demonstrated interaction. Adding a new skill means adding a new demonstration, not retraining the policy. That distinction is the key to understanding both the significance and the boundaries of the work. The robot was taught 1,000 object-manipulation tasks from one demonstration per task, collected on one robot in 17 hours. The resulting library covered 31 broad manipulation primitives, 534 more specific motion categories, and 402 physical objects. MT3 then achieved a 78.25% success rate on the demonstrated tasks under randomized test conditions, and 68% when transferring known skills to 100 previously unseen object instances. The authors supported the main design choices with 3,450 controlled real-world rollouts, followed by another 2,200 rollouts for the thousand-task evaluation. But a “task” here has a precise definition. It is one micro-skill applied to one particular object instance: open this box, pick up this mug, wipe this plate, insert this piece of toast into this toaster. These are mostly single-interaction primitives, not unconstrained long-horizon household missions. Multi-stage behaviors such as cleaning a desk are constructed by sequencing several learned primitives with a separate high-level mechanism. That still makes this an unusually ambitious real-world manipulation study. More importantly, it provides a careful empirical answer to a question that the current foundation-model conversation can sometimes obscure: if demonstrations per task are extremely scarce, should we really be asking one neural policy to learn perception, geometric reasoning, control, task identity, and termination all at once?. MT3’s answer is no. The Data Bottleneck Is Also an Architecture Problem. The paper begins with a familiar observation: behavioral cloning works remarkably well when supplied with sufficient data, but “sufficient” often means hundreds of demonstrations per task. The authors contrast their setting with systems such as BC-Z, which used roughly 26,000 demonstrations for 100 tasks; RT-1, with around 130,000 demonstrations over 744 tasks; and MT-ACT, with 7,500 demonstrations for 38 tasks. These are not strictly comparable benchmarks—the robots, tasks, observations, and success criteria differ—but they illustrate the prevailing data regime: approximately 175 to 250 demonstrations per task in several prominent multi-task systems. Why does one-shot behavioral cloning struggle so much? Because the single demonstration does not isolate the important invariants. The policy sees one object geometry, one object pose, one robot approach path, one contact trajectory, and one visual configuration. It must somehow infer which parts should remain fixed and which should change when the object moves or a new instance appears. A monolithic policy must learn several things simultaneously:. how to recognize the target object and distinguish the requested behavior;. how the robot should reposition itself when the object moves;. which portions of the demonstrated motion are incidental;. which contact motions must be reproduced precisely;. when the behavior has finished. With one or three examples per task, this is less a conventional supervised-learning problem than an underdetermined inference problem. The network is being asked to discover the structure of manipulation from almost no task-specific evidence. MT3 does not discover that structure. Its designers provide it. This is the deeper interpretation of the paper’s reported order-of-magnitude gain in data efficiency. It is not simply that nearest-neighbor lookup is mysteriously better than transformers. MT3 removes several hard learning problems from the policy and replaces them with explicit geometric operations. The comparison is therefore between two different allocations of intelligence: one attempts to internalize the entire behavior, while the other combines learned perception with analytical transformations and episodic memory. The Crucial Prior: Alignment, Then Interaction. The paper decomposes every manipulation trajectory into two phases. The alignment phase moves the end effector—or an object already held by the gripper—to a suitable pose relative to the target object. Only the final relative configuration is task-critical. There may be many acceptable collision-free paths for moving a plug in front of a socket, a spoon over a mug, or a towel above a plate. The interaction phase begins from that aligned configuration and performs the contact-rich part of the task. Here the trajectory itself matters. A plug must move along the socket axis. A lid may need to rotate around a particular hinge. A card must pass through a reader in a specific direction. The transition between these phases is represented by what is effectively a bottleneck pose: the initial end-effector pose of the recorded interaction. Once the robot reaches a geometrically equivalent bottleneck in the test scene, it can reuse the demonstrated contact motion. This is a strong inductive bias, but it is also a very natural robotics abstraction. It separates free-space motion—where planning and geometric reasoning are effective—from local interaction, where demonstrations are most valuable. It shortens the learned horizon and removes irrelevant variation in approach trajectories. The authors evaluate this decomposition rather than simply assuming it helps. They construct four two-stage systems by choosing either retrieval or behavioral cloning independently for each phase:. BC-BC learns both alignment and interaction. BC-Ret learns alignment but retrieves and replays interaction. Ret-BC retrieves a demonstration for geometric alignment, then uses a learned interaction policy. Ret-Ret, renamed MT3, uses retrieval for both phases. All four are compared with MT-ACT+, a monolithic behavioral-cloning policy that learns the complete trajectory without an explicit phase boundary. The factorial design is valuable because it disentangles two claims that could otherwise be conflated: that decomposition helps, and that inference-time retrieval helps. What MT3 Actually Does. Recording a Skill. During data collection, the demonstrator first moves the robot into a configuration suitable for beginning the interaction. Recording starts only at that point. The demonstration therefore contains the contact-relevant motion, while its first end-effector pose becomes the target for the future alignment phase. At 30 hertz, the system records RGB-D observations, the six-dimensional end-effector pose, and the binary gripper state. Each demonstration also receives a language description such as “open bottle” or “insert toast into toaster.” The object is segmented and converted into a point cloud using the calibrated head-mounted camera. In the paper’s data-processing pipeline, Grounding DINO supplies the initial object segmentation and XMem propagates it through the remainder of the demonstration. This selective recording is one reason collecting 1,000 demonstrations is possible. The human does not need to provide varied approach trajectories or repeatedly demonstrate recovery behavior. On average, 17 hours for 1,000 entries works out to roughly one new demonstration per minute, including the surrounding object handling and setup. The stored representation can also be compact. MT3 fundamentally needs an initial RGB-D observation, the corresponding object geometry, the bottleneck pose, and the sequence of end-effector motions and gripper commands. The full training-oriented video representation is not required merely to retrieve and replay a task. Stage One: Retrieve the Right Kind of Motion. At deployment, MT3 receives a language command and a segmented point cloud of the target object. Retrieval is hierarchical. First, a language stage filters the database to demonstrations of the requested micro-skill. If the command is “open bottle,” only demonstrations labeled with that motion category remain. In the core experiments, this is not an open-ended language-model interpretation step. The system uses template-based parsing and matching against a known skill vocabulary. This matters. Language is functioning primarily as a database key, not as a source of physical reasoning. The appendix includes a proof of concept in which an LLM maps natural utterances to stored micro-skill names and proposes sequences for longer tasks, but that component is separate from the evaluated MT3 controller. The LLM handles straightforward commands well and becomes unreliable when asked to invent underspecified long-horizon plans. Second, MT3 chooses a particular demonstration within the requested micro-skill. It embeds the test object’s point cloud using a PointNet++ encoder trained through occupancy prediction, and selects the stored demonstration with the highest cosine similarity. Because the point clouds are expressed in the robot frame, the embedding reflects both object geometry and object pose. That is intentional. Geometry similarity increases the likelihood that the retrieved motion will suit the object, while pose similarity makes the subsequent registration problem easier. The downside is a tension between those objectives: the geometrically best object might be in a difficult pose, while the closest pose might belong to an object whose required contact trajectory is less appropriate. Stage Two: Reconstruct the Bottleneck Pose. Once MT3 has selected a demonstration, it estimates the relative transformation between the demonstrated object and the object currently in front of the robot. It begins with a learned unseen-object pose estimator inherited from the authors’ earlier Trajectory Transfer work, then refines the registration using generalized ICP. That transformation is applied to the demonstration’s bottleneck pose. The system can now ask a conventional motion planner to move the end effector to a location that recreates the demonstrated end-effector-to-object relationship in the new scene. The released implementation makes the tabletop assumptions particularly visible. It includes a pretrained PointNet++ geometry encoder, a pretrained four-degree-of-freedom pose regressor, registration refinement, and a corresponding tabletop pose prior. This is not arbitrary six-degree-of-freedom manipulation in unconstrained space; it is a deliberately structured solution for a predominantly planar workspace. Stage Three: Replay the Interaction. After reaching the bottleneck, MT3 executes the retrieved end-effector velocities, expressed in the local end-effector frame, along with the demonstrated gripper commands. This local representation gives the trajectory a useful degree of geometric portability after alignment. But the interaction remains open-loop. MT3 does not continuously observe the evolving contact state, estimate whether the object slipped, or reshape the trajectory as a deformable object responds. It commits to the retrieved motion. That apparent simplicity is precisely the point. With only one demonstration, replay guarantees that the interaction remains behaviorally close to something the user actually showed. Behavioral cloning may produce smoother or more adaptive motions with sufficient data, but in a sparse regime it can leave the demonstration distribution and produce unexpected actions or poor termination decisions. MT3 therefore requires no task-specific gradient update when a new demonstration is added. That does not mean the complete system is untrained: its geometry representation, pose estimator, segmentation system, and other perception components rely on prior learning. The accurate claim is that new task acquisition happens without retraining. The Controlled Study: Decomposition Versus Monolithic Learning. The controlled experiments use a Sawyer arm, a Robotiq 2F-85 gripper, and a head-mounted RealSense D415. Every method receives the same two high-level inputs: a segmented target-object point cloud and a language task description. This reduces the possibility that MT3’s advantage is merely due to seeing cleaner observations than the behavioral-cloning baselines. The BC implementation is also substantial rather than deliberately weak. MT-ACT+ adapts the multi-task action-chunking transformer to point clouds. PointNet++ encodes geometry, a CLIP representation of the task description modulates the point features through FiLM, and a variational latent variable represents multimodal trajectories. The policy additionally receives action history, predicts task termination, and was independently sized for each data regime. For learned alignment, the authors synthesize 1,000 approach trajectories per real demonstration by sampling starting configurations in a large volume above the workspace and drawing linear paths to the bottleneck. Extra perturbations near the target encourage final-pose accuracy. Interaction training also includes pose perturbations intended to improve robustness to small deviations from the demonstrated path. These choices make the result more convincing: the retrieval systems are not being compared with a baseline that was denied obvious augmentation or architectural support. The first experiment holds task diversity fixed and varies demonstrations per task from one to 50. It covers four distinct micro-skills involving articulated objects, deformable objects, scooping, and insertion, with 12 demonstrated tasks and eight unseen object instances. The second experiment fixes the total budget at 150 demonstrations and distributes them over increasingly diverse task sets. The regimes contain 10 tasks with 15 demonstrations each, 30 tasks with five each, and 50 tasks with three each, spanning 10 micro-skills. Together, the studies produce 3,450 physical rollouts across 70 objects. Test objects are moved throughout an 80-by-45-centimeter workspace and, in the controlled setting, can be rotated through the full range around the vertical axis. Three results stand out. First, decomposition consistently improves low-data performance. Even BC-BC outperforms the otherwise comparable monolithic MT-ACT+, showing that the phase boundary itself provides useful structure. Second, retrieval beats behavioral cloning in both phases. Retrieval-based alignment is more reliable than asking a sparse-data policy to infer a precise spatial transformation. Retrieval-based interaction preserves the demonstrated contact motion rather than approximating it from a handful of examples. MT3 consequently achieves the strongest results throughout the evaluated regimes. Most strikingly, MT3 with only three demonstrations per task outperforms every alternative even when those alternatives receive 50. Third, the scaling curves complicate any simple claim that retrieval is categorically superior. Decomposition and retrieval improve rapidly in the one-to-ten demonstration range but begin to plateau. Monolithic BC improves more quickly between 10 and 50 demonstrations and benefits much more from diverse object instances. The likely explanation is representation sharing. A monolithic policy sees complete trajectories containing relatively regular approach structure followed by varied interactions. With enough diversity, it can discover reusable features spanning both phases. Separating the phases deprives BC-BC of some of this synergy: its alignment network sees large numbers of synthetic but monotonous paths, while its interaction network sees fewer and more heterogeneous real trajectories. The paper therefore does not establish that analytical retrieval should replace learned generalist policies. It identifies a regime boundary. When demonstrations per task are scarce, explicit structure wins. When data becomes abundant and diverse, the greater capacity of monolithic learning becomes increasingly valuable. What “One Thousand Tasks” Looks Like in Practice. The thousand-task library spans an impressive range of household interactions: placing, inserting, picking, pouring, removing, wiping, opening, scooping, stirring, stacking, washing, hanging, filling, dusting, folding, ironing, pressing, twisting, operating switches, zipping, sharpening, and more. It is also unevenly distributed. The six largest macro-skill groups—placing, insertion, picking, pouring, removal, and wiping—account for 619 of the 1,000 tasks. Some rare groups contain only one or two tasks. That does not invalidate the scale claim, but it means the aggregate success rate is influenced heavily by common, relatively structured primitives rather than being a macro-skill-balanced estimate. The authors explicitly note that practical constraints produced a nonuniform distribution. After collecting the demonstrations, the team evaluated each of the 1,000 demonstrated tasks twice. It also evaluated 100 unseen tasks—new object instances from known categories—twice each, producing 2,200 rollouts. Scenes contained between five and 20 visual distractors, changing backgrounds and lighting, and object rotations of up to 45 degrees relative to the demonstration. MT3 succeeded on 78.25% of seen-task trials and 68% of unseen-task trials. Here, “seen” does not mean exact replay of the original scene: object position, orientation, clutter, lighting, and background were altered. “Unseen” means category-level transfer to object instances for which that task-object pairing had not been demonstrated; it does not mean inventing an entirely new manipulation concept. The 10-point drop between seen and unseen tasks is informative. MT3 can often reuse a motion across mugs, plates, bottles, and similar containers, but only when the interaction-relevant geometry is sufficiently consistent. It generalizes by selecting the nearest remembered case and treating the new object as though that case were applicable. It does not learn a continuous category-level action model. Where MT3 Works—and Where It Breaks. MT3 performs best when success tolerates small errors in approach angle and contact location. Wiping, stirring, broad placement, and many grasping tasks exceed 80% success. In these tasks, global geometry is informative and the interaction trajectory has a generous basin of success. Failures appear when small local features dominate. A kettle’s body may look approximately symmetric to a global point-cloud encoder even though its small spout determines the correct orientation. The paper reports cases in which this produces a 180-degree registration mistake. Coin slots, plugs, hooks, card readers, and other millimeter-sensitive structures leave too little tolerance for registration error. Category-level transfer fails for a related reason. A pouring trajectory may need to change with the height, width, or rim geometry of the receiving vessel. A card swipe must align to a slot whose location varies between register designs. MT3 cannot interpolate between two retrieved trajectories or generate a new one halfway between them; it selects a single example and executes it. Deformable objects expose an even more fundamental limitation. Two backpacks may look similar yet have different flap stiffness, opening dynamics, or internal friction. Vision alone does not reveal these properties, and open-loop replay cannot adapt as the material moves. Folding fabric, inserting objects into bags, or manipulating elastic items may require continuous visual or tactile feedback. The authors’ failure analysis attributes substantial shares of failed seen-task trials to retrieval mistakes, segmentation errors, and pose-estimation failures—22.3%, 19.5%, and 23.9% of failures, respectively. Tasks involving grasped objects account for another prominent category: if the object is held differently at deployment than during demonstration, the discrepancy propagates through the entire open-loop trajectory. The system also does not test dynamic obstacle avoidance during interaction. Distractors make visual perception harder, but they are not intended to block the demonstrated path. An open-loop replay controller would otherwise continue through an obstruction, while a learned policy would face an out-of-distribution configuration. These weaknesses are serious, but unusually legible. An operator can inspect the retrieved demonstration and visualize the registered point clouds before execution. If alignment is clearly wrong, the system can stop. When MT3 fails during interaction, it generally continues performing a recognizable version of the retrieved motion. That predictability is operationally different from a low-data neural policy whose failure may involve a novel and poorly understood action. The Research Lineage: Retrieval, Alignment, and Replay. MT3 is best understood as the scaling-up of a research program rather than an isolated algorithm. The most direct predecessor is One-Shot Imitation Learning: A Pose Estimation Perspective, by Vitiello, Dreczkowski, and Johns. That work formulates one-shot imitation as trajectory transfer: estimate how the object moved between demonstration and deployment, then transform the demonstrated end-effector trajectory accordingly. It studies calibration, registration error, and spatial generalization over 10 real-world tasks. MT3 turns that single-task formulation into a multi-task system by adding hierarchical retrieval and evaluating it at a much larger scale. DINOBot, from the same lab, also follows a retrieve-align-interact recipe. It uses DINO vision-transformer features at two levels: global image features retrieve a similar demonstrated object, while dense features support pixel-level alignment. MT3 instead emphasizes RGB-D geometry, pose estimation, and a systematic comparison with behavioral-cloning alternatives. The test-time use of demonstrations also separates MT3 from methods such as Behavior Retrieval, SAILOR, and FlowRetrieval, which generally retrieve relevant prior data before or during policy training. It differs from VINN as well: VINN repeatedly performs nearest-neighbor action lookup from image-action pairs during execution, whereas MT3 retrieves one complete trajectory before the task starts. At the opposite end of the design spectrum are generalist BC and vision-language-action policies. Their goal is to internalize cross-task regularities in a parametric model and eventually produce behaviors that are not copies of any one demonstration. MT3 gives up much of that compositional capacity in exchange for immediate acquisition, interpretability, and data efficiency. Edward Johns has also contrasted MT3 with the lab’s Instant Policy work, where a neural policy is pretrained in simulation and then conditioned in-context on a new demonstration. That alternative retains prompt-like task acquisition while allowing a learned network to synthesize actions. MT3 is more explicitly memory-based: retrieval chooses a concrete past behavior and geometry tells the robot where to replay it. How Roboticists Might Use—or Improve—This Idea. MT3 is attractive when a deployment has many tasks, little data for each one, and a user who needs to add behaviors without waiting for retraining. Examples might include laboratory automation, flexible small-batch manufacturing, assistive manipulation, or a household robot customized by its owner. The best task families share several properties: a clearly segmentable target, a repeatable bottleneck pose, a contact trajectory that remains useful across modest geometric variation, and enough tolerance to survive small alignment errors. Consistent grasps and largely obstacle-free local interactions are especially helpful. A practical system inspired by MT3 should probably add several mechanisms around the core pipeline. First, retrieval should be task-aware below the whole-object level. Global shape may be appropriate for placing one vessel inside another, but it is insufficient when a handle, slot, hinge, spout, or button determines success. A language-conditioned geometric representation could emphasize the part relevant to the requested interaction rather than embedding every surface equally. Second, top-one retrieval need not be the final action generator. A system could retrieve several neighbors, register each one, and use their trajectories as proposals. A small generative or in-context policy could then interpolate between them while remaining anchored to demonstrated behavior. This would preserve MT3’s external memory while addressing its inability to synthesize intermediate motions. Third, open-loop replay could become a prior for closed-loop control. During execution, a residual policy, visual servo, model-predictive controller, or tactile feedback loop could make small corrections around the retrieved trajectory. The important design constraint would be to keep the correction bounded enough that the resulting behavior remains interpretable and safe. Fourth, the system needs explicit confidence estimates. Segmentation quality, retrieval similarity, registration residual, planner clearance, and grasp consistency all provide signals that could gate execution. MT3’s modularity makes such gating easier than in a single opaque policy: each stage can expose a diagnostic score and request another demonstration when confidence is low. Finally, retrieval and behavioral cloning need not remain competitors. The scaling results suggest a natural fast-and-slow architecture. A robot could acquire a task immediately by storing and replaying one demonstration. As the memory accumulates enough examples, it could periodically consolidate recurring patterns into a parametric policy. Retrieval would supply rapid adaptation and a fallback behavior; the learned policy would supply interpolation, reactive control, and cross-task transfer. The released repository makes this direction relatively accessible. It includes implementations of MT3, the three hybrid decomposition methods, and MT-ACT+, along with Docker-based training and deployment scripts, pretrained geometry and pose components, demonstration formats, and a runnable MT3 example. The project page separately links the paper, data, and code. The Real Lesson of a Thousand Tasks. The title is defensible, but only if we keep its definitions visible. The robot did not learn 1,000 unrestricted natural-language goals from scratch. It acquired 1,000 labeled, single-interaction skills within a pre-engineered perception, registration, planning, and replay system. The 17-hour figure measures demonstration collection, not the research effort, prior model training, calibration, segmentation infrastructure, or task-library construction that made those demonstrations usable. At the same time, dismissing the result as “just retrieval” would miss the scientific contribution. The paper demonstrates that a nonparametric skill memory can be operated at a scale rarely tested in physical manipulation, across hundreds of objects and thousands of real evaluations. It also shows, through controlled ablations, exactly why the system works in the sparse-data regime: explicit decomposition removes the need to learn task structure, pose estimation replaces learned spatial extrapolation, and replay replaces learned contact dynamics. The most important result is therefore not the number 1,000. It is the shape of the trade-off. If there is one demonstration per task, it can be irrational to train a universal policy and hope it discovers the correct invariances. A structured system may deliver far more capability per minute of human supervision. If there are thousands of diverse demonstrations, however, the constraints that make MT3 efficient become limitations, and representation learning begins to offer advantages that retrieval cannot. For embodied AI, the likely destination is neither pure replay nor pure end-to-end learning. It is a layered system that can remember an experience immediately, reason geometrically about where it applies, adapt it through feedback, and eventually consolidate repeated experience into reusable models. MT3 gives us a particularly clear implementation of the first two capabilities. And by teaching a robot roughly one new manipulation skill per minute, it makes a persuasive case that external memory and old-fashioned structure still belong in the foundation-model era.