Latest / Embodied AI 101 / Local Policies Enable Zero-shot Long-Horizon Manipulation
Transcript
- Local Policies Enable Zero-shot Long-Horizon Manipulation. General-purpose manipulation–tidying a table, organizing a shelf, or cooking–requires chaining many sub-tasks together in novel scenes. Modern robotics still struggles with such open-ended problems. Unlike vision and language, where large models can generalize from internet-scale data, robotic learning is often confined to either narrowly engineered skills or at most short-horizon tasks with limited generalization. Even with simulation-to-real (sim2real) transfer, robots usually train and test on the same task in simulation, making modest changes in object pose or shape. For truly open-world manipulation (e.g. “put the carrot in the pot and close the fridge”), we need a strategy that leverages large-scale training but also composes many simple skills. This paper (ManipGen) presents one such approach: local policies. The core insight is that many manipulation skills can be formulated locally – focused on the target object – rather than reasoning about the entire scene. By training policies that only observe and act in a small neighborhood around an object of interest, these policies become invariant to the global arrangement. They no longer care where the robot or shelf is in the world, or what order other skills have happened, as long as they start near the target. In effect, each skill decomposes into (1) a global motion-planning reach phase and (2) a local interaction phase. The local policy only handles the latter, operating with inputs like a wrist-mounted depth camera and a segmentation mask of the target object. Figure: An industrial robot arm manipulating an object (here shown with a computer mouse). A local policy would focus only on the target mouse and its immediate vicinity, ignoring the robot’s absolute position or the rest of the factory, thereby simplifying the learning problem. This locality affords powerful invariances: as the authors note, a local policy is absolute-pose-invariant, skill-order-invariant, and scene-configuration-invariant. In practice, that means the policy only needs to reason about the relative geometry between its gripper and the object (and maybe any immediately adjacent obstacles), not the whole workspace. For instance, whether the object lies on a table of a certain height or floor, or which skill came before, makes no difference to the local controller once it starts. In simulation, this implies we can train hundreds of these skills in diverse random scenes and then reuse them in many sequences without re-training. ManipGen leverages these local policies together with off-the-shelf vision, language, and planning models to achieve zero-shot long-horizon manipulation. The high-level pipeline is: parse a user instruction with a language model into a sequence of (object, skill) steps; for each step, use a vision model to locate the object and a motion planner to reach it; then execute the appropriate local policy to interact (grasp, push, open, etc.). Remarkably, all low-level skills are trained only in simulation, yet in real-world tests the system solved tasks up to 8 steps long on novel objects and configurations, without any additional real training. Across 50 distinct multi-step real tasks, ManipGen achieved an overall 76% success rate – vastly surpassing prior methods like SayCan (40%), VoxPoser (16%), and others. In short, restricting learning to per-object, contact-rich policies while delegating sequencing and reaching to pre-existing models yields strong generalization without expensive new data. The Challenge of Long-Horizon Manipulation. Imagine telling a robot “Clean up this table by putting all the toys in the bin.” A human breaks this into subskills: pick up each toy, reach the bin, release the toy, and so on, adapting on the fly to where everything lies. Building such a capability in robots is hard because it combines (a) high-level planning (“which object to move next?”), (b) perception (finding objects and receptacles in novel poses), and (c) low-level dexterity (picking, pushing, opening) in cluttered, contact-rich settings. Historically, robotics tackled parts of this with “sense-plan-act” pipelines: segment the scene, run classical planners, and execute. This can work for structured tasks with known objects, but breaks down for more flexible tasks or when precise contact dynamics matter. More recently, research has explored using large language and vision models (“foundation models”) to plan or suggest actions in a zero-shot way. For example, SayCan and TidyBot use LLMs to pick a sequence of high-level skills from a predefined library, and other projects prompt a VLM (Vision-Language Model) to infer sequences of end-effector poses. These approaches lean on massive internet data to generalize across objects and instructions. However, their low-level modalities are limited: they often rely on canned primitives (e.g. “grasp object”, “open drawer”) without learning the contact-rich motions anew for every situation. If those primitives end up imprecise, the plan fails. Indeed, recent comparisons found that say SayCan does okay on basic pick-and-place but struggles with tasks needing tight contact control or unusual orderings. Meanwhile, pure RL-based methods have pushed sim2real transfer for manipulation, usually on specific tasks like assembly or in-hand dexterity. The problem is generating realistic diversity for training: one can randomize physics and objects, but to learn multi-stage tasks you quickly need millions of precise interactions. Some hierarchical RL attempts and language-guided RL (Plan-seq-learn, BOSS, etc.) try to tackle multi-stage tasks, but they mostly train on the task of interest, not truly zero-shot. ManipGen aims for something different: no task-specific training at execution time. The robot should solve a new long-horizon task described by text, even if it was never tried in real life and with new objects and layouts. The smoke test is bold: tasks of up to 6–8 steps with random object placements. The insight to make this feasible is to reduce the learning burden by using local policy abstractions. Local Policies: Exploiting Invariance. The key idea from the paper is “local policies”: specialized controllers whose inputs and effects are confined to a small region around a target object. Concretely, a local policy is defined by (i) an initialization region near some object of interest, (ii) local observations (e.g. a wrist camera depth image and object mask) ignoring the global scene, and (iii) local actions relative to that object frame. In contrast to a monolithic policy that sees the whole environment, a local policy only “watches” the thing it will manipulate. This locality confers powerful properties. First, absolute pose invariance: because the policy never sees the robot’s base frame or the global coordinates, it learns only the relative geometry between gripper and object. Thus, a pick policy trained with the robot at one orientation will work just as well if the robot starts in a completely different pose–as long as its gripper is initialized near the object. Second, skill-order invariance: the output end state of one skill exactly matches the start conditions of the next skill via motion planning, so local policies needn’t know when they are in the sequence. For example, a “place” skill doesn’t have to enforce what came before; it only cares that it starts with an object in-hand at some height and ends with the object on the table. Third, scene configuration invariance: the policy does not observe distant objects or the robot’s base position. It only “sees” the depth image of the region around the target object (from a wrist camera) and possibly a mask indicating the object. Everything else in the room or ordering of tasks is invisible. These invariances dramatically reduce the variability a policy must learn across. In one stroke, the infinite configurations of a home kitchen collapse into a much smaller set of relative configurations. For example, training a door-opening policy: instead of placing the door anywhere in the house, one can fix a randomized handle and just train on its neighborhood. At test time, the robot will segment the target handle and bring its wrist near it; the local open-door policy then works just as it did in sim. In essence, we factor the problem: train specialized “reach and interact” skills for generic objects, then let a higher-level planner sequence those skills for the full task. As the authors put it, this design allows “policies that observe and act in a region local to the target object” to be by construction invariant to absolute pose, skill order, and scene layout. It also means one can use procedural scene generation far more easily in simulation. If a policy only cares about a tiny scene around its object, the simulator only needs to randomize that local neighborhood; the rest of the world can be ignored or kept fixed. This enables training at scale (thousands of objects) without needing billions of global scenes. Training Local Skills in Simulation. Given this framework, the authors train a library of local policies entirely in simulation using supervised and reinforcement learning. They focus on a minimal but surprisingly powerful skill set: Pick, Place, Grasp Handle, Open, Close. These five skills cover a wide range of everyday manipulations: picking up a loose object, placing it down, grasping any kind of door or drawer handle, and pushing or pulling that handle to open or close the receptacle. Each skill is trained as follows:. State-based RL Experts: First, they collect state-based expert policies with reinforcement learning (PPO). In each simulated episode, the robot starts in a random configuration within an ε-ball of the target. For example, for the pick skill, the gripper is initialized near but above a random object (sampled from a large object set of 3,500+ household items), with the gripper pointed toward it. The policy is given privileged state information: the end-effector pose and velocity, the object’s pose, and even a small set of keypoints sampled from the object’s 3D mesh (16 points from farthest-point sampling). They also provide intermediate reward signals and a terminal indicator to speed up learning. The action space is 6D: a delta position and delta orientation (axis-angle) to be executed by a high-frequency task-space impedance controller. Each skill has a tailored reward that encourages the desired motion: e.g. for pick, the reward penalizes distance from a goal grasp pose (and penalizes unnecessary motion or contact with other objects); for open, it rewards reaching the joint limit of a drawer, etc. In all cases the robot learns to move to a target pose (often a pre-grasp pose near the object) and then close the gripper or push to complete the action. Importantly, the training scenes are randomized locally: clutter and obstacles are added near the object so the policy must learn local obstacle avoidance, but the rest of the world (like table properties) is fixed. They also randomly vary object sizes, friction, and (for articulated objects) joint ranges – for example, they generate 2,600+ 3D handles and random box shapes for doors/drawers from PartNet. Distillation to Vision-based Policies: The RL experts are state-based (having privileged information), but the real robot cannot see internal states or keypoints. So the next phase is to distill these experts into vision-based, local policies via imitation (DAgger). Here each policy sees only local sensor data: the depth image from a wrist camera (clamped to 30cm for locality) plus a segmentation mask of the target object from the first frame, and proprioceptive offsets of the gripper relative to its starting pose. The idea is that at execution time, the system will segment the object with a perception model and then feed that mask and the current depth crop into the policy. To make the sim-trained policies robust to realistic noise, they augment the simulated depth with random “edge artifacts” and holes that mimic real sensor imperfections. In practice, they apply a form of DAgger with a replay buffer: the current policy is rolled out on a batch of random objects, and the expert provides actions. A buffer of recent trajectories (across objects) is kept, and training alternates between updating on this replay buffer and collecting new data. They found that a moderate buffer size (100 objects worth of trajectories) worked best; too little and the policy forgot earlier tasks, too much and learning slowed. Crucially, the observation structure enforces locality: the network never sees absolute positions, only local depth arrays and masks, so it acquires the invariances we want. At the end of training, each skill has a generalist visuomotor policy that can handle any object of that type (for grasp-handle/open/close, any handle geometry) from any starting orientation, within the local region it was trained for. The result is a library of five high-frequency local controllers, each conditioned on an object mask and designed to take over once the robot is “at” the object. Deploying Local Skills with VLM Planning. With this skill library in hand, how do we solve an arbitrary long-horizon task described by text? ManipGen uses a three-part pipeline that marries large pretrained models with the local policies:. Task Decomposition with a Language Model. Given a textual task instruction (e.g. “Put the carrot in the pot on the stove and close the fridge”), ManipGen first prompts a vision-language model (they use GPT-4o) to break it into a sequence of subgoals. The system feeds the LLM a description of its available skills (object-agnostic: pick, place, open-handle, etc.) and a description of the scene (e.g., which objects are present). 4o then outputs a list of (object, skill) pairs. For example, for a “Cooking” task it might produce or similar. If the first output misses an object or mislabels one, they have a simple “tagging” loop: the LLM is asked to list all scene objects, Grounded-SAM segments them, and then the LLM re-plans using properly identified object names. In all, GPT-4o leverages its vast commonsense knowledge to decide what should come next, but it does not output motions or parameters; it merely enumerates which object to manipulate with which skill. This gives a flexible, language-conditioned plan with minimal prompt engineering needed. Pose Estimation and Motion Planning. Once we have the next (object, skill), the robot must navigate the arm to that object. ManipGen uses an open-set detector (Grounded-SAM based on Grounding DINO) to segment the target object in the real scene. From the 3D point cloud of that segmented object, they compute a goal end-effector pose. In simple cases (open-space pick-and-place), they align the gripper vertically and center it on the object; in clutter or tight spaces (like a shelf or microwave), they sample candidate gripper poses around the object and pick one with minimal collisions. For articulated objects (doors, drawers), they find the handle and set up the gripper for the best approach. Once a target pose $X_{\text{targ}}$ is decided, they use a neural motion planner (Neural MP) to compute a collision-free joint-space trajectory from the robot’s current state to that goal. Neural MP is a learned planner that runs in ~2 seconds, faster than classical planners in complex scenes. The trajectory is then executed open-loop by a task-space impedance controller until the gripper is in position. Local Policy Execution. When the arm has reached the correct pose, the system invokes the corresponding local policy. At this point the gripper is near the object (within ~10cm). The policy gets its input: the current depth crop from the wrist camera plus the object’s first-frame segmentation mask. It runs at 60Hz through the learned network and sends tiny delta-wrist commands back to the impedance controller. This executes the actual manipulation (grasping, pushing, etc.). The skill runs for a fixed duration (e.g. 6–8 seconds) or until its learned success condition (like the door joint fully opened) is met. Then the pipeline loops: the world state is updated, and the next subgoal from the language plan is addressed. Importantly, aside from prompting and segmentation, everything in this loop was trained in simulation. The policy never saw the real robot or real objects until deployment. Yet because of the local design and the augmentation, the policies transfer robustly. No real-world finetuning or human correction is used. Experimental Results. The paper validates ManipGen in both simulators (to isolate effects) and on real robots with fully novel tasks. The results are striking. Simulated Benchmark (Robosuite). First, they test on a set of standardized long-horizon tasks in the RoboSuite simulator (the “PSL” benchmark tasks: PickPlace with various objects like Bread+Milk, etc.). These benchmark tasks have 2–4 stages and randomized object poses. ManipGen executes zero-shot, using the same language-and-planning pipeline as in reality, and achieves an average 97.33% success rate across tasks. This even exceeds PSL (a recent state-of-the-art method that trained on those tasks) which got around 96%, and far outperforms short-horizon end-to-end RL methods (which often failed >50% of trials). It also beats classical task-and-motion planning baselines: e.g. PDDLStream got 86% and SayCan got 80% on the average of these tasks (Table I). In other words, without any task-specific training, the local-policy approach solved these multi-stage benchmarks as well as or better than methods trained on those exact tasks. Real-World Sim2Real Benchmarks. Next, the authors turn to purely single-step tasks from a real-world sim2real benchmark (FurnitureBench, used in the Transic paper). They train local policies specifically for three test tasks (pushing a block, precisely inserting a peg, etc.) with cross-domain randomness, then deploy them directly on hardware. Compared to prior sim2real methods (including ones using real corrections or demonstrations), ManipGen performs superbly. For “Stabilize” (block pushing) and “Reach and Grasp” tasks, ManipGen gets 95% success, matching Transic (which used human corrections) and far above simple direct transfer (15–65%) or naive behavioral cloning. Most impressively, on the precise “Insert” task (peg-in-hole insertion), ManipGen achieved 80% success with zero real training, compared to only 45% for Transic (which used about 5K correction samples). In fact, on average ManipGen scored 90%, beating all other methods (Table II) – demonstrating that these local policies really can handle tight tolerances if trained properly in sim. Zero-Shot Long-Horizon Tasks (Real). Finally, the ultimate test: 50 end-to-end tasks of up to 8 stages, on a real Panda arm with wrist camera. The tasks include things like: Cook (2 stages: pick a food item and put it in a black pot), Replace (4 stages: fetch from a shelf to board, then board to another shelf), CabinetStore (4 stages: open a drawer, drop object, close it), DrawerStore (6 stages: open a drawer and place two items in it), and Tidy (8 stages: pick scattered toys and drop them in a bin). Each trial randomizes the objects used, their positions, and receptacle positions. Crucially, none of these exact configurations were trained—ManipGen had only seen the local skill-level simulations for pick/place/handle/open/close, but never the full scenario. In these experiments, ManipGen achieved an average 76% final success rate across all tasks. This dwarfs the baselines: SayCan (the strongest prior method) scored only 40% on average, VoxPoser got ~16%, LLMTrajGen 14%, and even a fine-tuned vision-language-action model (OpenVLA) had essentially 0% (it couldn’t handle novel poses without extensive data). Looking at each task: ManipGen solved Cook 90% of the time (SayCan 80%, LLMTrajGen 70%), Replace 80% (SayCan 10%, others 0–10%), CabinetStore 90% (SayCan 70%), DrawerStore 60% (SayCan 20% at best), and Tidy 60% (SayCan 20%). On every single task, ManipGen was by far the best-performing method. It typically completed 4–6 of the 4–8 stages on average, whereas competitors often stalled quickly (e.g. LLMTrajGen could usually only handle the first pick action). Error analysis shows the limits: failures mostly came from perceptual hiccups (e.g., Grounded-SAM sometimes mis-identified the target object) or motion-planning hiccups (jumpy Neural MP paths led to collisions). In very cluttered scenes, the pick policy could also struggle if the object was partially occluded. Nevertheless, the breakthrough is clear: without any real-world fine-tuning, ManipGen’s local-policy pipeline handled complex, multi-stage instructions on unseen objects. Why This Works: Generalization Through Locality. The results were so strong because of the properties of local policies. A few ablations from the paper underscore what matters:. Observation design is crucial. Training with “privileged” state helped the RL experts learn quickly, but the distilled policies still needed the right input features. For instance, including the 3D keypoints of the object (through the point-sampling trick) gave the policy a sense of object shape. Removing that keypoint information dropped pick success from ~97.4% to ~94% in a test setup. Similarly, including the first observation (the initial depth and mask) at every step, and using relative rather than absolute robot pose, improved transfer; leaving those out caused noticeable performance hits. This design ensures the policy knows which part of the depth image is the object (via the mask) and how its hand has moved since start. Strict locality matters. The authors tested training “pick” policies with different initial region sizes (i.e. how far the gripper could be from the object initially). At 8cm radius (“very local”), the success was ~97.4%. Expanding to 16cm hurt only a little (95.7%), but 32cm gave 89.6% and 64cm only 72.5%. In other words, making the problem bigger made learning much harder. Sticking to a small local region lets the policy concentrate on fine control near the object; the reach motion is better handled by the separate motion planner. DAgger on-policy data is key. When distilling the policies, they found that a purely off-policy or purely on-policy approach didn’t work well. A small replay buffer (100 objects) gave the best of both worlds. This ensured the network saw a diversity of objects (so it generalized to new ones) while staying reasonably close to its current behavior (for stable learning). In short, by carefully shaping the learning problem to be just the local interaction (with fixed start-of-skill conditions), the training becomes feasible at scale. The skills become highly general: a single “open” policy handles hundreds of door and drawer shapes, just by focusing on the handle. It also means domain randomization and augmentation (e.g. adding holes and noise to depth) yield real-robot robustness: the real depth from the Intel RealSense often has artifacts, but the policy was trained to be immune to them. Putting It All Together. To summarize how ManipGen operates in the real system: a user says “goals in language.” GPT-4o plans the sequence of subgoals in language form. For each subgoal:. Perception: Grounded-SAM (with extra prompting heuristics) segments the target object in the camera view and generates a precise mask. The 3D points of the mask give the object’s pose (position and orientation). Motion planning: Given this object pose, Neural MP quickly plans an arm path so that the end-effector comes to a pre-contact target pose near the object. Local policy: Once the arm is in place, the corresponding local policy (pick, place, etc.) is run for a fixed duration. Visually, it takes the current depth crop and the initial mask and moves the gripper as needed. For example, a pick policy will grasp the object and lift it. Loop: The gripper’s target state is achieved, we proceed to the next (object, skill) in the plan, until the task is done. Because each policy was trained to succeed from any nearby start pose, minor errors in the reach phase don’t derail them. Because the LLM step is just a high-level planner, the system can flexibly handle many different instructions without retraining. The overall effect is a hybrid of learned skills and classical planning, with learned skills taking care of the contact-rich core of each action. Comparison to Prior Approaches. This strategy differs from and improves on other recent methods in several ways:. Versus end-to-end sim2real RL: Many past works focus on one monolithic task (e.g. assemble this part, open this particular drawer) and train an end-to-end policy. They then try to randomize geometry or use real correction data to transfer. ManipGen instead trains task-agnostic skills once, and composes them. The results above show that even in domains where other methods do use real data (like FurnitureBench), ManipGen’s pure-sim local policies beat them. For example, for the insertion task, prior work needed real corrections to reach 45% success, but the local policy got 80% with zero real data. Versus LLM-planning only methods (SayCan, VoxPoser, etc.): Those methods delegate all low-level control to either hand-designed primitives or to the LLM itself. SayCan, for instance, has fixed primitives for pick/place/open, which often rely on heuristics. ManipGen, by contrast, actually learns exactly how to do each skill robustly. This makes a big difference, especially for contact-rich steps. The empirical gap is telling: in Table III, SayCan often works when interactions are easy (Cook 80% success), but it fails badly on drawers and clutter (only 20% at best). VoxPoser and LLMTrajGen, which try to use models to generate motion goals, similarly struggled. The lesson is that it’s worth spending the simulation effort to learn real servoing skills. Versus hierarchical or language-guided RL: Some approaches like “Plan-seq-learn” or BOSS use a frozen LLM to define a reward or curriculum while training on each task. Those still require on-policy data collection for the specific tasks. ManipGen’s whole training is done up-front in generic environments; no new RL is needed at test time. In experiments, it actually outperforms PSL (Plan-seq-learn) on the Robosuite tasks, despite never being trained on them. In summary, by marrying learned local control with powerful pretrained models for planning, ManipGen achieves what neither alone could: flexible high-level decision-making plus robust low-level execution. Conclusion. The Local Policies idea may seem simple, but it has a surprisingly large payoff. By imposing a strong locality constraint on skills, we turn the combinatorial difficulty of open-world manipulation into a set of tractable simulation training problems. ManipGen demonstrates that thousands of hours of sim-trained local controllers can immediately empower a robot to handle novel multi-step instructions zero-shot. The vision-language models handle the abstract reasoning (“what to do next?”), the motion planner handles gross arm movements (“how to get there?”), and the local policies handle the fine motor tasks (“actually grasp/open/etc.”). Across simulations and real experiments, this approach sets a new bar for zero-shot manipulation. It solves tasks with up to 8 sequential actions and large variations in objects and scenes, without any new real data. It outperforms previous state-of-the-art language-guided or sim2real methods by tens of percentage points. In the authors’ words, “local policies enable a variety of appealing properties” that make broad generalization possible. Future work might expand the skill library (e.g. adding push, pour, or more dexterous motions) and improve perception integration, but ManipGen already shows a compelling path toward generalist robots: train many simple skills in sim, stitch them together with language and planning, and generalize to complex real tasks, all zero-shot. Key Takeaways: Local policies---where each skill only observes its local region---allow simulation-trained manipulation controllers to transfer zero-shot to novel, long-horizon tasks. ManipGen trains thousands of such skills (pick/place, open/close, etc.) via RL, distills them into vision-based controllers, and deploys them with GPT-4 planning and motion planning. The result is a robot that can follow diverse instructions on unseen objects, achieving far higher success on complex tasks than prior methods. This work highlights how the right abstraction (locality) unlocks the power of sim2real at scale for open-ended robot manipulation.