Latest / Embodied AI 101 / When the Appliance Becomes the API: Part–Function–State Models for Robot Planning

Transcript

When the Appliance Becomes the API: Part–Function–State Models for Robot Planning · Embodied AI 101

• When the Appliance Becomes the API: Part–Function–State Models for Robot Planning. Appliances Describe Themselves: Part–Function–State Modeling for Appliance Manipulation Planning • Xinting Yang et al. • affiliations not available in the accessible bibliographic record • IEEE Robotics and Automation Letters, vol. 11, no. 8, pp. 9771–9778 • 2026. Source-status note — August 8, 2026: The DOI and journal metadata for this newly published paper are available, but the primary paper’s abstract, full text, complete author list, affiliations, figures, and tables were not accessible through the public interfaces available during this research pass. I therefore will not fabricate its experimental setup, baselines, numerical results, or implementation details. This episode separates what is bibliographically verified from a technical reconstruction of the Part–Function–State idea, grounded in the title and in closely related primary work on manual-based appliance manipulation. A PDF is needed for a definitive paper-specific account of the experiments and results. The Real Problem Is Not Opening the Door. Imagine placing a household robot in front of an unfamiliar microwave and asking it to warm a meal. At first glance, this sounds like a standard embodied-AI problem. The robot needs to recognize the microwave, localize the door and controls, choose grasp points, generate trajectories, and execute contact-rich motions. Those are all genuine challenges. But none of them answers a more basic question: what do the parts mean?. A handle-shaped region probably opens the door. Beyond that, visual common sense becomes much less reliable. One knob may select temperature; another may select cooking time. A row of visually identical buttons may represent power, defrosting, clock configuration, or preset programs. The same physical action—pressing a button—can start a process, stop it, increment a value, acknowledge a menu item, or do nothing at all, depending on the appliance’s current mode. This is the distinction between manipulating an articulated object and operating an appliance. An articulated-object model might tell the robot that a door rotates about a hinge, a drawer translates along an axis, or a knob has a particular angular range. It describes how the mechanism moves. But appliance operation also requires manufacturer-defined semantics: which part controls which function, what values are available, how those values are encoded, and how the appliance changes state after each interaction. Recent work has made this gap explicit. CheckManual argues that appliances differ from ordinary tools because the functions of their parts are often impossible to recover from appearance or common sense alone. It introduces manuals as a source of the missing information and frames appliance use as a combination of manual comprehension, part grounding, task planning, and physical execution. RealAppliance pushes the same agenda from the simulation and benchmarking side. It introduces high-fidelity appliance assets aligned with their manuals and evaluates models on manual-page retrieval, part grounding, open-loop planning, and closed-loop planning adjustment. The title Appliances Describe Themselves suggests a complementary answer: rather than forcing the robot to infer operational semantics from pixels or reconstruct them from prose, encode those semantics directly in the appliance model. The proposed abstraction is named Part–Function–State, or PFS. That sounds simple. It may also be one of the more consequential ideas in household manipulation: treat an appliance not merely as a geometric object, but as a machine-readable interface to its own operation. Why Geometry and Affordances Are Not Enough. Robotic object models have become increasingly expressive. Modern assets can include meshes, semantic segments, articulation trees, joint limits, collision geometry, masses, and material properties. Perception systems can estimate six-degree-of-freedom poses, segment movable components, predict articulated motion, and infer affordances such as grasping, pulling, pressing, or rotating. Yet an affordance is not the same thing as a function. A robot may correctly infer that a cylindrical component can be rotated. That geometric affordance does not reveal whether the component adjusts temperature, fan speed, volume, or cooking mode. Nor does it reveal the correspondence between angle and meaning. A 30-degree clockwise rotation might mean “medium heat” on one appliance and “defrost” on another. The problem is particularly severe for controls because their geometry is intentionally standardized while their semantics are not. Buttons tend to look like buttons. Knobs tend to look like knobs. Touchscreen regions may have almost no geometric structure at all. Manufacturers communicate the missing information through labels, icons, displays, manuals, and conventions, all of which require additional interpretation. Even a correct functional label is insufficient without state. Suppose the robot knows that a button controls power. It still needs to know whether the appliance is currently on or off. If the button is a toggle, executing the same action in the wrong initial state produces the opposite result. A timer-up button may need three presses when the display reads zero, but only one press when it already reads two minutes. The effect of an action is conditioned on the state from which it is applied. This is why the three terms in PFS form a useful progression:. Part identifies the physical entity to manipulate. Function describes the operational variable or effect controlled by that part. State specifies the current or desired value of that function. Part answers where. Function answers why. State answers what now and what next. Without the first, a plan cannot be grounded. Without the second, the robot does not know which control is relevant. Without the third, it cannot reliably reason about transitions. Reconstructing the Part–Function–State Abstraction. Because the primary text was unavailable, the following is an engineering reconstruction of the representation implied by the paper’s title—not a claim about its exact schema. At minimum, a useful PFS model would need to connect three layers. The part layer would describe the appliance’s manipulable components. Each record might contain a persistent part identifier, geometric region, articulation type, interaction surface, allowable motion, and the manipulation skill needed to actuate it. A door might be associated with a pull-and-swing skill. A push button might require a linear contact. A rotary knob might expose an axis, limits, detents, and preferred grasp region. The function layer would assign operational meaning to those components. Instead of merely calling something , the model might say that it controls start, power level, or timer increment. Importantly, a part and a function need not have a one-to-one relationship. A single multifunction dial may select values, navigate menus, and confirm choices. Conversely, one function may be controlled by multiple parts, such as separate increase and decrease buttons. The state layer would represent the domain of possible values and the currently believed value. Door state might be open or closed. Power might be off or on. Temperature could be a discrete set of presets or a continuous interval. A mode selector might have named states such as bake, broil, defrost, and convection. State records should ideally include units, tolerances, observability, and confidence. A concrete entry might read conceptually like this:. This representation turns an ambiguous visual component into a typed control interface. Given the goal “set the timer to ten minutes,” the planner does not need to infer knob semantics from appearance. It resolves the function, identifies the associated part, finds the state corresponding to ten minutes, and instantiates a rotary manipulation toward the required configuration. The Hidden Fourth Element: Transition. A static Part–Function–State tuple is not yet a planner. Planning concerns changes between states, so a complete implementation needs transition semantics, whether they are represented explicitly or derived from the part model. The planner must know that pressing a button moves the power state from off to on, that rotating a knob changes a parameter according to direction and angle, or that opening a door may disable cooking. It also needs preconditions and side effects. In that sense, the actionable core is closer to:. Consider a start button. Its transition may require the door to be closed and the timer to be nonzero. Pressing it while the door is open may have no effect. Opening the door during operation may stop the appliance and change several states simultaneously. A single local part transition therefore can depend on—and modify—global appliance state. This is a critical issue for evaluating any PFS system. If the representation records only labels and target configurations, it is a useful semantic annotation. If it also captures preconditions, effects, dependencies, and observations, it becomes a lightweight executable model of the appliance. That distinction determines whether PFS merely helps an LLM produce plausible instructions or enables a planner to verify that those instructions are valid. How a PFS-Based Planning Pipeline Could Work. The most natural system architecture would separate semantic planning from geometric execution. First, the robot obtains an appliance description. The phrase “appliances describe themselves” implies that the description is provided rather than inferred entirely from perception. Possible delivery mechanisms include metadata packaged with a CAD or URDF asset, a manufacturer-hosted machine-readable manifest, a QR or NFC identifier resolving to a product model, or an appliance-side communication interface. The delivery mechanism matters less than the contract: the appliance exposes a structured account of its manipulable parts, their operational functions, and valid states. Second, the robot binds that description to the physical instance in front of it. Even perfect semantics do not eliminate perception. The robot still has to estimate appliance pose and associate the manifest’s part identifiers with observed geometry. This is where CAD alignment, visual grounding, fiducials, instance registration, or manufacturer-provided calibration features become relevant. Third, the task instruction is converted into desired functional states. “Warm the food for two minutes” might become a goal specification such as:. An LLM could perform this semantic compilation, but it need not be the planner itself. Once the instruction has been mapped into typed goals, a symbolic or graph-search layer can reason over the appliance transitions. Fourth, the system identifies which parts can produce the required state changes. The cooking-time function resolves to one control, power to another, and start to a third. If more than one interaction sequence is valid, the planner can optimize for path length, reliability, reachability, or minimal state disturbance. Fifth, the planner orders the transitions according to their dependencies. A microwave-like appliance might require opening the door, inserting the object, closing the door, configuring time and power, and then starting. A washing machine might require loading, closing, selecting a program, setting options, and initiating the cycle. The ordering is not purely geometric; it follows the appliance’s control logic. Finally, each semantic transition is grounded into a manipulation skill. becomes a handle grasp and constrained hinge motion. becomes one or more button presses or a knob rotation. becomes a guarded pressing action followed by state verification. The clean architectural boundary is important:. PFS says what physical control corresponds to what operational change. The task planner says which changes must occur and in what order. The manipulation stack determines how to realize each change under the current geometry and contact conditions. Perception verifies whether the intended transition actually happened. PFS would therefore not replace grasping, motion planning, tactile control, or visual feedback. It would supply the semantics those components normally lack. A Worked Example: Operating an Unfamiliar Oven. Suppose a robot is instructed to preheat an unfamiliar oven to 180 degrees Celsius. A vision-only system may recognize the appliance category and detect a door, display, and several controls. But appliance-level common sense is insufficient. One knob might control the heating mode and another the target temperature. The target may be represented by printed markings, an electronic display, or a menu accessible only after pressing a mode button. A PFS model could expose the following information:. The language instruction becomes a goal: heating mode must be compatible with preheating, temperature must equal 180 degrees, and operation must enter the heating state. The planner checks the current state. If the mode is off and the temperature is 100 degrees, it selects transitions to bake, 180 degrees, and heating. It then obtains the corresponding part actions and checks reachability. After rotating the first knob, the robot should verify the mode state. Verification might come from knob pose, display text, indicator lights, appliance telemetry, or some combination. Only then should the robot proceed. This example highlights why “state” cannot be treated merely as a target joint angle. There are at least three related state estimates:. The physical configuration of the control. The semantic value intended by that configuration. The appliance’s internal operational state. These normally agree, but not always. A knob can move without engaging its shaft. A button can be physically pressed while the appliance ignores it because a precondition is unmet. A display can lag behind an action. A robust PFS system needs to distinguish commanded configuration from confirmed functional state. What the Paper’s Core Contributions Appear to Be. Only three high-level contributions can be attributed directly from the verified title and publication record. First, the work introduces or formalizes Part–Function–State modeling as a representation for appliances. Second, it applies that representation to appliance manipulation planning, rather than treating PFS solely as an annotation or recognition task. Third, the phrase “Appliances Describe Themselves” advances a design philosophy in which manipulation-relevant semantics are supplied by—or stored with—the appliance instead of being reconstructed exclusively from visual observations and general world knowledge. The inaccessible primary text prevents verification of several essential details:. | Paper detail | Verification status | |---|---| | RA-L publication, volume, issue, and pages | Verified | | Six-author count | Supplied by the user | | Complete author names and affiliations | Not recoverable | | Formal definition of PFS | Not recoverable | | Whether PFS is a graph, ontology, table, or state machine | Not recoverable | | Whether the planner is symbolic, LLM-based, or hybrid | Not recoverable | | Appliance dataset and task set | Not recoverable | | Simulation and robot platform | Not recoverable | | Baselines and ablations | Not recoverable | | Numerical results and statistical analysis | Not recoverable | | Code or project page | Not located |. Any more specific attribution—such as naming a particular language model, claiming a robot platform, reporting a success-rate improvement, or stating the number of appliances—would be unsupported. Experiments and Results: What a Serious Evaluation Must Establish. The target paper’s actual experiments and numerical results cannot be reported responsibly without the primary text. What can be analyzed is the evidence a PFS planning paper needs to provide. The first level is semantic planning accuracy. Given an appliance model, initial state, and language goal, does the system produce the correct sequence of function-state transitions? This isolates high-level planning from perception and control. Useful metrics would include exact plan success, valid-prefix length, part-selection accuracy, target-state accuracy, and constraint violations. The second level is grounded plan accuracy. A semantically correct plan is not enough if the selected parts cannot be associated with physical geometry. Evaluation should test whether function-bearing parts are correctly localized and whether generated interaction parameters—contact point, axis, direction, displacement, or target joint value—are consistent with the appliance model. The third level is closed-loop transition success. For each planned step, does the robot actually move the appliance into the requested functional state? This metric is stronger than geometric movement. A button press should count only if the appliance responds; a knob rotation should count only if the resulting semantic value is correct. The fourth and most important level is end-to-end task success. Can the robot start from the observed appliance state, plan the full sequence, execute it, recover from minor errors, and achieve the requested operational outcome?. A compelling experiment would compare at least four information conditions:. Visual observations and general model knowledge alone. Visual observations plus ordinary appliance documentation. Parts and articulation metadata without function or state. The full Part–Function–State description. Such an ablation would determine whether the value comes specifically from the structured representation rather than from simply providing more text. Within PFS, the crucial ablation would compare Part-only, Part–Function, and Part–Function–State variants. Part-only modeling should help with physical grounding but leave control semantics unresolved. Adding function should improve part selection. Adding state should particularly improve toggles, repeated actions, mode-dependent controls, and tasks beginning from non-default configurations. Generalization also matters. If the planner is evaluated only on appliance instances used to design the ontology or prompts, PFS may function as a handcrafted lookup table. The stronger test is whether a common schema transfers across manufacturers and novel layouts while allowing each appliance to expose its own part-to-function mapping. Finally, the system should be tested with imperfect state information. Real homes contain appliances whose state is partially hidden, whose displays are occluded, or whose controls have been moved manually. A planner that assumes the manifest’s initial state remains correct will fail in precisely the settings where closed-loop planning is needed. Without the paper’s tables, we cannot determine which of these questions the reported evaluation answers—or how decisively. How PFS Relates to Manual-Based Manipulation. PFS and manual-based planning address the same information deficit from opposite directions. Manual-based methods begin with human-oriented documentation and attempt to extract an actionable model. The CheckManual pipeline explicitly connects appliance parts with functional names and states, then represents manipulation plans in terms of function-bearing parts and desired values. Its ManualPlan baseline resolves multi-page manuals, generates manipulation steps, and aligns functional descriptions with observed parts. The difficulty is that manuals are not designed as robot APIs. They contain prose, diagrams, safety warnings, repeated explanations, tables, references across pages, and assumptions about human common sense. They may describe functions without providing precise geometric grounding. Models must retrieve the correct sections, resolve visual-textual references, and translate instructions into executable state changes. A self-describing PFS asset would move that translation upstream. Instead of asking every deployed robot to independently interpret the same manual, the appliance manufacturer—or an asset curator—would publish the structured result once. That makes PFS resemble a compiled version of a manual:. The manual is the human-readable source document. PFS is the machine-readable operational interface. The planner consumes PFS to generate state transitions. The manipulation stack grounds those transitions in the physical world. RealAppliance’s emphasis on aligning high-fidelity assets with real manuals makes it a natural environment in which such a representation could be constructed or tested. Its benchmark explicitly spans information retrieval, part grounding, open-loop planning, and closed-loop adjustment—the same chain that a reliable self-description interface would need to support or simplify. Why This Matters in the Age of Vision–Language–Action Models. It is tempting to treat structured appliance models as a temporary scaffold that sufficiently large vision-language-action models will eventually make unnecessary. A foundation model, after all, may recognize common appliances, read labels, recall product knowledge, and generate motor actions directly. But manufacturer-specific semantics are not always inferable. If two visually identical buttons have different functions, no amount of generic visual intelligence can determine the correct mapping without an additional information source. A model may make a statistically plausible guess, but household operation often requires exactness. Pressing the wrong control on an oven, induction cooker, or cleaning appliance is not equivalent to choosing a slightly suboptimal grasp. PFS therefore should not be viewed as competition for foundation models. It can serve as structured context for them. An LLM can map an open-ended user instruction into functional goals. A symbolic layer can validate those goals against the PFS state machine. A VLM can ground manifest parts in the current camera view. A VLA policy can execute the atomic interaction. The structured appliance model acts as the shared interface between otherwise heterogeneous components. This hybrid has several advantages. It is more inspectable: the robot can explain that it is rotating the right knob because that part controls temperature and because 180 degrees Celsius is the desired state. It is easier to validate: a safety checker can reject transitions not included in the appliance model. It is more compositional: a general planner can operate many appliances as long as each exposes functions and states through the common schema. And it is easier to update: correcting a product-specific semantic error does not require retraining the robot’s entire policy. The larger lesson is that embodied intelligence does not have to infer everything from scratch. The environment can participate in the intelligence of the system by publishing useful structure. The Hard Problems PFS Does Not Solve. A PFS description could remove a major semantic bottleneck, but several difficult problems remain. The first is manifest-to-instance alignment. Knowing that exists does not tell the robot where it is in the current image. Appliances can move, become occluded, or differ slightly from their CAD models. Accessories may be installed or removed. A self-description is useful only if its parts can be reliably bound to observed geometry. The second is state observability. Some states are visually obvious, such as whether a door is open. Others exist only in a display or internal controller. A planner needs to know not just the state domain but how each state can be observed. Otherwise the robot cannot distinguish successful execution from a silent failure. The third is coupled state. Appliance functions are rarely independent. Opening a door can pause a cycle. Selecting one mode can change the meaning of another control. Buttons may be overloaded according to menu context. A flat list of PFS entries cannot capture these interactions without global constraints or hierarchical state machines. The fourth is temporal behavior. Some operations depend on pressing duration, double presses, waiting, or acting within a timeout. Others initiate long-running processes whose state evolves without further robot action. PFS therefore needs temporal transition semantics, not merely static target values. The fifth is interaction with task objects. An appliance model can describe its own controls, but household tasks involve contents: food must be placed in a microwave, detergent loaded into a washing machine, or a cup positioned under a coffee dispenser. Planning must combine appliance state with object state, spatial relationships, and task-specific constraints. The sixth is trust and safety. A manufacturer-provided description may be stale, corrupted, or mismatched to the physical model. Safety-critical planning cannot blindly trust self-reported metadata. The robot needs consistency checks between the manifest, perception, observed responses, and physical constraints. These limitations do not weaken the PFS idea. They locate it correctly. PFS is a semantic control model, not a complete household-robot solution. The Standardization Opportunity. The most important long-term implication may be outside any individual planning algorithm. Robotics already has widely used ways to describe rigid bodies, joints, sensors, and coordinate frames. What it lacks is an equally common schema for the operational semantics of consumer devices. A practical standard would need to specify:. Manipulable parts and their persistent identifiers. Geometric and kinematic interaction parameters. Human-readable and machine-readable function names. State domains, units, tolerances, and default values. Valid transitions, preconditions, effects, and invariants. Methods for observing or querying state. Safety constraints and forbidden transitions. Versioning and correspondence to product variants. With such a standard, appliance operation could become less like unconstrained visual reasoning and more like using a typed API. Robots would still need to handle uncertainty, contact, and variation, but they would not need to rediscover every product’s control semantics. This also creates an interesting division of labor. Manufacturers know the control logic. Robot vendors know manipulation. Foundation models know language and broad task semantics. PFS can define the boundary across which those capabilities meet. The phrase “appliances describe themselves” is therefore more than branding. It proposes that embodied environments should expose operational metadata in the same way that software components expose interfaces. Final Takeaway. The verified publication record establishes that Appliances Describe Themselves is an eight-page 2026 RA-L article centered on Part–Function–State modeling for appliance manipulation planning. Beyond that, the unavailable primary text prevents a responsible account of its exact representation, planner, experiments, and quantitative results. The idea named by the paper is nevertheless clear and technically important. Household appliance manipulation is not only a perception-and-control problem. It is an interface problem. Robots need a bridge between physical parts, manufacturer-defined functions, and evolving operational state. PFS offers exactly that conceptual bridge. Its strongest possible contribution would be to turn an appliance from an opaque articulated object into a lightweight semantic digital twin: one that tells the robot what can be manipulated, what each manipulation means, which states are valid, and how actions transform those states. The decisive questions for the actual paper are whether its PFS model captures transitions rather than only annotations, whether state is observed and updated in closed loop, whether the representation generalizes across heterogeneous appliances, and whether the experiments demonstrate gains in physical task success rather than only plausible plan generation. Those questions—and the paper’s reported answers—require the primary PDF. Until then, any exact experimental counts or success rates would be invention rather than research.