Latest / Embodied AI 101 / REMAC: Turning Robot Failures into Better Team Plans

Transcript

REMAC: Turning Robot Failures into Better Team Plans · Embodied AI 101

• REMAC: Turning Robot Failures into Better Team Plans. REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulation • Puzhen Yuan, Angyuan Ma, Yunchao Yao, Huaxiu Yao, Masayoshi Tomizuka, and Mingyu Ding • Tsinghua University, Carnegie Mellon University, UNC–Chapel Hill, and UC Berkeley • Pattern Recognition, Elsevier • 2026. Publication note: the openly accessible manuscript was first submitted to arXiv on March 28, 2025, and revised as version 2 on August 10, 2026. This episode discusses that revised manuscript and the accompanying journal DOI and code release. The microwave-door problem. Imagine asking a household robot to heat a carrot. The robot navigates to the carrot, picks it up, rolls over to the microwave—and discovers that the door is closed. This is a trivial situation for a person and an awkward one for many language-model-based robot planners. The robot is now holding the carrot, but opening the door requires a free gripper. A fixed plan either fails outright or triggers an expensive replan. A somewhat better system notices the immediate precondition violation, puts the carrot down, opens the door, retrieves the carrot, and continues. An even better system remembers that experience and, on the next attempt, opens the microwave before picking up the carrot. And a genuinely collaborative system sends a second robot to open the microwave while the first retrieves the food. That four-step progression—failure, correction, memory, and parallelization—is essentially the entire REMAC paper in miniature. Its Figure 1 follows exactly this sequence: a bad plan fails because the robot is holding the target object; condition checking enables recovery; reflection improves the next single-robot plan; and finally another robot takes responsibility for the prerequisite action. The interesting part is not that a vision-language model can be prompted to say “put the carrot down.” The interesting part is the system architecture built around that observation. REMAC treats failed preconditions as evidence about the causal structure of a task. It records that evidence, uses it to improve future planning, and eventually turns learned dependencies into opportunities for multi-robot concurrency. This makes REMAC less an end-to-end manipulation policy than a memory-augmented orchestration layer. It sits above an existing skill library, asks foundation models to plan and diagnose, and turns repeated interaction into a progressively better task program. What REMAC actually claims to solve. The paper focuses on scene-agnostic, long-horizon instructions such as “heat the vegetables” or “defrost the meat.” These instructions intentionally omit operational details. They do not say which appliance to use, where the necessary container is, whether a door is open, or which subtasks could run concurrently. REMAC assumes that the system receives an inventory of the current environment: available fixtures, available objects, and available executable commands. From that structured scene description and the high-level goal, a reasoning model must produce a sequence of subtasks for one or two robots. In the paper’s formulation, the system operates with either one or two robots and decomposes the instruction into a finite list of actions executable by low-level policies. That qualification matters. “Scene agnostic” means the human instruction does not have to encode the scene configuration. It does not mean the planner begins with unconstrained pixels and autonomously discovers every object, affordance, and controller. The high-level planner is given a symbolic inventory and a closed command vocabulary. The released examples include commands such as:. navigate to an object or fixture,. pick up an object,. place an object on or in a fixture,. open or close an appliance,. operate a faucet or stove,. wait. The repository connects these command strings to hard-coded planners or learned checkpoint policies. It includes PID-style planners for navigation, picking, and placement, while the paper describes pretrained language-conditioned low-level policies for common appliance and manipulation skills. So REMAC is best understood as hierarchical robot control. The foundation models operate at the task level, while conventional or learned controllers handle motion and manipulation. This is not a vision-language-action model that directly emits joint commands, and it does not learn new motor skills through reflection. Its main contribution is the logic connecting planning, visual verification, execution, memory, and task allocation. The architecture: planner, checker, executor, memory. REMAC separates the system into roles that are often conflated in language-based robot agents. A reasoning-capable language model produces and revises plans. A vision-language model examines images and evaluates conditions. Low-level controllers execute the resulting action strings. A persistent database stores useful plans and explanations of planning failures. In the main experiments, the authors use DeepSeek-R1 for initial planning and reflection, while GPT-4o performs visual precondition and postcondition checking and local plan adjustment. The stated motivation for this split is cost and latency: deliberative reasoning is invoked where global planning is needed, while a multimodal model handles repeated visual checks. Step one: generate an initial plan. The planner receives three kinds of structured information:. the high-level task,. the fixtures and objects present in the scene,. the exact command grammar available to the robot. It then emits an ordered list such as: navigate to microwave, open door, navigate to vegetable, pick up vegetable, navigate back, and place vegetable inside. The released code uses few-shot prompting and strict output templates so that free-form model text can be parsed into executable command lists. The examples encode common domain conventions, including navigating before picking, opening a microwave before using it, and restricting every action to the listed objects, fixtures, and commands. This reduces the interface problem between generative models and robot software, but it also narrows the meaning of zero-shot planning. REMAC does not fine-tune the planning model for each task, yet the prompt still contains substantial kitchen and action-schema prior knowledge. Step two: check the next action’s preconditions. Before executing each subtask, the system asks what must be true for that action to be feasible. For “open microwave door,” the conditions might include:. the microwave is visible or reachable,. the gripper is empty,. the door is currently closed. For “place vegetable in microwave,” they might include:. the microwave is present,. the gripper is holding an object,. the microwave door is open. A vision-language model—or, for selected predicates, a state-based checker—evaluates each clause. If all conditions hold, the action proceeds. If one fails, REMAC asks the model to explain why and generate a corrective sequence. The paper treats this explanation as a reflection on a planning error and stores it for later use. This is more structured than simply showing the model an image and asking, “What should I do now?” The condition list creates a lightweight predicate interface between perception and planning. It encourages the system to identify the particular missing fact that blocks execution. The distinction also improves error attribution. A closed microwave before a placement action indicates a planning problem: a required prerequisite was omitted or ordered incorrectly. That information should modify future plans. Step three: execute and check the postconditions. After the low-level policy runs, REMAC evaluates whether the intended effect occurred. Opening the microwave should result in an open door. Picking up the vegetable should result in the gripper holding it. Placing the vegetable should leave the gripper empty and the target object at the destination. If the postcondition fails, the system retries the action. Importantly, the authors do not add postcondition failures to the planning-reflection database. They classify them as execution failures rather than defects in the high-level plan. Because a failed controller can still disturb the scene, the system may re-evaluate the action’s preconditions before trying again. This separation is conceptually clean:. failed precondition: the plan asked for an infeasible action;. failed postcondition: the action was appropriate, but execution did not achieve its effect. In practice, the boundary will not always be so sharp. A grasp might fail because the motion policy is unreliable, because the planner chose an occluded object, or because the robot approached from an impossible side. Nevertheless, explicit attribution is preferable to placing every failure into an undifferentiated language-model context. A hybrid rather than purely visual checker. The repository reveals an implementation detail that is easy to miss in the paper’s high-level description. Several predicates—including whether the gripper holds something and whether the microwave or stove is activated—are routed to ground-truth or state-based checkers. Other conditions are evaluated from one or two camera views by a vision-language model. That is a reasonable engineering choice. Robot proprioception should generally be used when available, rather than inferred unnecessarily from pixels. But it means the simulation results should not be interpreted as demonstrating fully visual, open-world verification. REMAC’s checker is better viewed as a hybrid state estimator whose interfaces happen to be expressed in language. “Self-evolving” means evolving context, not model weights. The term self-evolving invites a stronger interpretation than the implementation supports. REMAC does not update neural-network parameters, train a policy online, or automatically modify its skill library. The thing that evolves is the planner’s context. During a trial, the system accumulates precondition failures and textual reflections. At the end, it reconstructs the useful executed plan, removes redundant or unsuccessful pieces, and records the interaction. On later iterations, the reasoning model receives the task, scene information, previous plan, and stored reflections. It is then asked to produce a better initial plan. The repository stores records containing the goal, environment information, generated plans, executed actions, precondition and postcondition results, reflections, final plan, and overall outcome. These records can subsequently be loaded as reminders or examples. This is episodic, in-context adaptation. It has several practical advantages over online weight updates. The memory is inspectable, failures can be audited, and an operator can remove bad conclusions. Improvements also require relatively little new data or compute. But the knowledge is only as reliable as its scope. “Open the microwave before picking up the carrot” may be useful in one kitchen and suboptimal in another where a second arm is already holding the door, the appliance opens automatically, or the object can be temporarily transferred. A production version therefore needs memory retrieval and invalidation policies, not simply indefinite accumulation of prose. How one robot becomes two. REMAC does not begin by jointly searching over arbitrary two-robot plans. Instead, it first improves a single-robot solution. Once that solution is successful and reasonably efficient, the system identifies prerequisite actions that another robot could perform in advance. The main robot remains responsible for the plan centered on the primary object. The assistant robot handles enabling conditions: opening the cabinet, opening the microwave, putting a bowl in the sink, or placing a pan on the stove. The assistant’s actions are scheduled so their effects are available before the main robot needs them. This is a clever reduction of the multi-agent planning problem. Rather than asking a model to invent two interacting trajectories from scratch, REMAC starts with a known causal chain and cuts out prerequisite branches that can run in parallel. The implementation makes this strategy especially concrete. It retrieves successful single-agent records, selects the shortest stored useful plan as a seed, and prompts the cooperative planner to divide that plan between two agents. The generated action lists are padded with waits, and the prompt tells the model to avoid placing both robots at the same location simultaneously. It can add waiting or a “navigate aside” action to reduce collision risk. This is centralized schedule generation, not decentralized cooperation. The robots do not negotiate online, share learned beliefs, bid for tasks, or continuously reallocate work under uncertain durations. Coordination is largely encoded in two synchronized lists. That simplicity is both the strength and the limitation. It makes the method understandable and runnable, but it works most naturally when the task graph contains obvious, detachable prerequisites. The benchmark: four causal traps in many kitchens. The authors extend RoboCasa with a two-robot environment and four long-horizon task families. Across the suite, they report 27 environment styles, more than 50 objects, and multiple kitchen layouts and object configurations. Each task is designed around a prerequisite that a superficial planner is likely to omit. The four task families are:. OpenCabinetPnP: open a closed cabinet before placing objects inside. OpenMicrowavePnP: open the microwave before placing vegetables inside. DefrostInBowl: place a bowl in the sink before defrosting meat or fish. HeatOnStove: place a pan on the stove before heating vegetables. The visual diversity is meaningful: the system sees different kitchen geometries, styles, objects, and placements. The causal diversity is more limited. Each task revolves around discovering and respecting one dominant enabling condition. The main ablation evaluates four configurations:. BASE: single robot, with no condition checking or reflective evolution;. CC: BASE plus precondition and postcondition checking;. RE: condition checking plus cross-iteration reflective evolution;. REMAC: RE plus the second robot. Each task and configuration is tested with ten randomized initializations. The paper reports task success rate, subtask completion rate, execution time, and initial plan length. For plan length, two actions executed in parallel count as one step, so the metric measures schedule depth rather than total robot work. Averaging the four task rows in the paper’s Table 1 gives the following picture:. | Configuration | Average task success | Average subtask completion | Average initial plan length | |---|---:|---:|---:| | BASE | 5.0% | 51.9% | Not consistently defined | | CC | 32.5% | 69.9% | 11.625 | | RE | 45.0% | 81.0% | 7.5 | | REMAC | 45.0% | 81.2% | 5.5 |. These averages expose the respective roles of the components. Condition checking supplies the largest initial jump in success. Reflective evolution adds another 12.5 percentage points. Introducing the second robot leaves average success essentially unchanged but shortens the resulting schedule. The paper’s headline of a 40% success improvement is therefore best read as 40 percentage points: average success rises from 5% under BASE to 45% under REMAC. In relative terms, that is an eightfold increase, although the tiny baseline makes relative improvement a less informative statistic. The reported 52.7% efficiency improvement can be reconstructed from initial plan length. Moving from the condition-checking single-robot system’s average of 11.625 steps to REMAC’s 5.5 parallelized steps is a 52.7% reduction. This is a valid makespan-oriented measure, but it depends on the convention that simultaneous actions count as one plan step. Looking instead at reported execution times, moving from RE to REMAC reduces the four-task average from about 11.2 seconds to 8.9 seconds—approximately the 20% reduction explicitly discussed by the authors. The paper also reports that self-evolution prunes 35% to 62% of redundant initial-plan steps. Task-level results are instructive. REMAC matches RE on cabinet and microwave tasks, improves stove success from 50% to 60%, but reduces defrosting success from 50% to 40%. The authors attribute such losses to failure propagation: an additional robot creates additional opportunities for a controller failure to invalidate the joint plan. The right conclusion is not that more robots make the system more capable in every sense. Reflection makes it more capable; the extra robot primarily makes successful plans faster. Which models can reflect?. The authors separately test whether different language models can turn a suboptimal plan and prior reflections into an improved plan. Across the four tasks, the reported average reflection success rates are 12.5% for GPT-4o, 26.25% for QwQ, 37.5% for DeepSeek-R1, 62.5% for o3, 66.25% for Qwen3, and 82.5% for Grok 3. Each task-model combination uses 20 randomized initializations. The broad message is plausible: remembering an error is insufficient if the planner cannot reason through its implications. A model has to translate “the gripper was occupied when the door needed to be opened” into a globally reordered plan, rather than merely inserting more local actions. The exact ranking should be treated as a snapshot, however. API-served models change, and performance is sensitive to model version, prompt formatting, decoding parameters, image handling, and provider-side updates. The experiment supports the value of deliberative reasoning, but it should not be interpreted as a timeless leaderboard. What survived contact with a physical robot. The revised paper includes a real-world OpenMicrowavePnP experiment. The platform is a HEXMOVE-based mobile manipulator with two PiPER arms, a head-mounted Femto Bolt RGB-D camera, two wrist-mounted Gemini 336L RGB-D cameras, and a RealSense T265 used in scene reconstruction. The experiment is run on one dual-arm mobile platform rather than a fleet of two independent mobile manipulators. The task is divided into four phases: open the microwave, place the vegetable inside, add seasoning, and close the door. Over 20 trials, BASE never completes the first phase. CC reaches the final phase in 10% of trials, while REMAC reaches it in 25%. Among final successful trajectories, the reported average execution time falls from 128.5 seconds for CC to 57.2 seconds for REMAC. This is useful evidence that remembered precondition errors can help outside simulation. It is not yet broad real-world validation. The experiment covers one task, one platform, and a final success rate of only one in four. Moreover, timing is conditioned on the small subset of successful trials. Still, the physical results reinforce the paper’s central point: correcting a mistake once is less valuable than converting it into a constraint that improves the next attempt. Where REMAC sits among related approaches. REMAC’s intellectual lineage begins with language-model planners such as SayCan, which ground linguistic preferences in executable robot skills. Grounded Decoding and VoxPoser similarly constrain language reasoning using grounded models or spatial value representations. These systems established the pattern of placing a foundation model above a structured action interface. A second line of work closes the loop around execution. Inner Monologue injects environmental feedback into repeated reasoning. RePlan revises plans after new perceptual observations. CAPE generates corrective actions from precondition errors. REFLECT summarizes past failures for future correction, while ISR-LLM and Reflective Planning iteratively refine long-horizon plans. REMAC explicitly positions its condition checking and episodic memory as an extension of this reflection-and-replanning family. The third lineage is multi-robot task allocation. SMART-LLM decomposes instructions and assigns subtasks according to robot capabilities; LaMMA-P combines language models with PDDL-style planning; classical multi-robot task-and-motion planning represents dependencies and physical feasibility more explicitly. REMAC’s distinguishing move is to delay task allocation until single-agent reflection has exposed the task’s relevant prerequisites. That gives REMAC an interesting position between foundation-model agents and symbolic planning. Its preconditions and postconditions resemble symbolic predicates, but they are generated and interpreted in language. Its reflection database resembles an experience-based domain model, but it has no formal consistency guarantees. Its two-robot plan resembles a dependency schedule, but it is produced through prompting rather than a verified scheduler. What the paper gets right. The strongest design choice is the explicit separation of planning errors from control errors. Too many embodied-agent systems pass every failure back to the same model with an unstructured “try again” instruction. REMAC asks whether the requested action was feasible before execution and whether its intended effect occurred afterward. Even when the resulting classification is imperfect, the architecture creates useful diagnostic boundaries. The second strong idea is to store causal lessons, not just successful trajectories. A final plan tells the robot what worked. A reflection such as “the door must be open and the gripper must be free before insertion” explains why the ordering matters. That explanation can potentially transfer across objects, layouts, and related tasks. Third, REMAC introduces collaboration only after reducing uncertainty about the task graph. Multi-agent planning has a combinatorial tendency: every new robot multiplies allocation, synchronization, collision, and recovery choices. Stabilizing a single-agent plan before searching for detachable prerequisites is a sensible way to control that complexity. Finally, the project provides code, task environments, experiment scripts, and a modified RoboCasa stack rather than only a conceptual diagram. The repository includes separate paths for condition checking, reflective single-agent planning, and two-agent collaboration. Where the evidence is still thin. The first limitation is semantic breadth. The benchmark has strong visual variation but only four causal task schemas. Opening a door, adding a bowl, and adding a pan are good test cases for omitted prerequisites, yet they do not cover richer long-horizon phenomena such as irreversible actions, resource consumption, temporal deadlines, partial observability, human intervention, or conflicting goals. Second, the planner is given a list of fixtures, objects, and legal commands. The hard problem of constructing a reliable symbolic scene description is largely outside the planning evaluation. A robot deployed in a new home would still need to discover that a particular object is a microwave, determine its state, locate its handle, and connect an “open” command to a feasible controller. Third, condition checking benefits from simulator or robot state for selected predicates. That is good systems engineering, but it makes performance depend on instrumentation that may not be available for every object or appliance. A more general evaluation would report results separately for oracle predicates, learned state estimators, and pure image-based checks. Fourth, the main ablation uses only ten random initializations per task and configuration, without uncertainty intervals in the reported table. With success counts as low as two, three, or five trials, modest changes in random seeds could noticeably change the percentages. The absolute success rates—30% to 60% in simulation and 25% for the complete real task—also remain far below what autonomous household deployment would require. Fifth, the multi-agent planner is a simplified synchronous scheduler. The implementation uses waits and prompt-level collision warnings rather than formal spatial reservation, online duration estimation, or task reallocation. If the assistant robot is delayed, blocks an aisle, fails to open a door, or changes the scene unexpectedly, the joint plan can cascade into failure. The paper acknowledges this propagation effect. Sixth, plan length favors parallelism by construction. Counting two concurrent actions as one step measures critical-path depth, which is useful, but does not measure total robot-seconds, energy use, wear, communication cost, or the additional probability of failure. A complete multi-robot efficiency study should report both makespan and aggregate resource consumption. Finally, cross-task evolution is not convincingly demonstrated. The released reflective planner retrieves reminders and examples associated with a task name, while the cooperative stage selects among successful stored plans. That establishes within-task improvement. It leaves open whether knowledge acquired while opening a microwave helps with a previously unseen oven, refrigerator, sterilizer, or industrial enclosure. How roboticists might build on REMAC. The most promising way to extend REMAC is to preserve its modularity while replacing free-form parts with more explicit representations. First, reflections should become typed, testable knowledge. Instead of storing only prose, a system could record predicates and causal edges: opening a hinged fixture requires a free manipulator; placing an object inside requires the container to be open; heating on a surface requires a compatible vessel. Language models could still propose these rules, but a symbolic layer could validate, deduplicate, scope, and expire them. Second, the two-robot conversion should operate on a dependency graph rather than synchronized text lists. Once reflection has discovered prerequisite relations, conventional scheduling, constraint programming, or multi-robot task-and-motion planning can optimize assignment and makespan while enforcing spatial exclusions and resource constraints. The language model can remain responsible for proposing task structure without being solely responsible for correctness. Third, precondition checking should expose uncertainty. A calibrated checker could return probabilities or an “unknown” state instead of forced Boolean answers. Ambiguous observations could trigger a viewpoint change, active perception action, or direct state query. For safety-critical actions, predicates should be fused from vision, proprioception, appliance telemetry, and geometric reasoning. Fourth, the memory system should distinguish task-specific episodes from reusable domain knowledge. Retrieval should condition on appliance type, embodiment, available tools, scene geometry, and controller competence. A reflection learned by a dual-arm robot should not automatically be applied to a single-arm platform without checking embodiment compatibility. Fifth, future benchmarks should vary the causal structure, not just the visual skin. Useful tests would remove expected tools, lock fixtures, introduce broken skills, swap appliance mechanisms, add humans who move objects during execution, or require the robots to abandon an invalid strategy. They should report planning latency, model cost, total robot work, safety violations, recovery quality, and memory transfer to unseen task families. REMAC could also serve as a high-level layer above modern diffusion policies or vision-language-action models. An expressive low-level policy does not eliminate the need to reason about closed doors, occupied grippers, tool availability, or parallel work. Conversely, REMAC’s discrete planner benefits from low-level policies with broader object and embodiment generalization. The takeaway. REMAC’s most important idea is not that robots can “reflect,” and it is not simply that two robots are faster than one. Its central idea is that execution reveals hidden task structure. A failed precondition is converted into a causal explanation. That explanation improves the next plan. Once the plan’s dependencies are known, some of them can be assigned to another robot and executed concurrently. The current system remains heavily scaffolded: it uses structured scene inventories, a closed skill vocabulary, hybrid condition checkers, small task families, and relatively simple two-agent schedules. Its success rates make it a research prototype rather than a deployment-ready household planner. But as an architectural pattern, REMAC is compelling. It suggests a practical bridge between language-model reasoning and classical task planning: let foundation models propose and interpret constraints, let interaction discover which constraints matter, store those discoveries as explicit memory, and use formal robot systems to execute and eventually schedule them. In other words, the system does not merely ask, “What action should I take next?”. It asks, “Why was that action impossible—and can another robot make it possible before I get there?”.