Latest / Embodied AI 101 / MMSkills: Building Multimodal Skill Libraries for Visual Agents
Transcript
- MMSkills: Building Multimodal Skill Libraries for Visual Agents. Autonomous agents and robots need rich procedural knowledge to tackle complex tasks. In traditional skill frameworks, this knowledge often comes as code or text-only instructions – think of reusable snippets of code, chain-of-thought prompts, or one-shot routines. But when an agent must operate in a visual environment – clicking GUI buttons, editing spreadsheets, or manipulating objects – execution depends on more than just “what to do next.” The agent must also recognize the current state of the world, visually sense progress or failures, and gather just the right evidence before taking an action. In other words, the skill must carry multimodal cues. For visual agents, Zhang et al. argue, “procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next”. MMSkills is a new framework built on this idea, packaging each reusable behavior with its own text instructions and visual references so an agent can apply that skill in context. Why Multimodal Skills?. Picture an agent assistant trying to “set the title of a slide to ‘Sales Data’” in a presentation program. A text-only description might say “click on the title area, then type the words “Sales Data.” But without some visual grounding, the agent may not know which part of the screen is the title area, or it may not verify that the title indeed changed. Naively stuffing entire screenshots into the prompt quickly becomes unwieldy, and naively matching pixel images can lead to “anchoring” on an exact screenshot that rarely matches a live environment. MMSkills addresses this by bundling a skill with just a handful of example images and states. Each MMSkill package includes:. A textual procedure (like a normal step-by-step instruction list) describing the operation in words. A set of state cards (key screenshots of the environment in relevant states). For instance, one card might show how the screen looks before selecting a menu, another the screen after creating a new document. These are often stored as full screenshots plus smaller “focus” crops of the important region. A set of multi-view keyframes, which are exemplar images (or screenshots) from different points in the skill’s execution. These help the agent visually verify that each step has taken effect. When an agent later considers using that skill, it can pick only the state cards and keyframes relevant to the current branch of execution to confirm things. In short, MMSkills “turns public visual-agent trajectories into reusable multimodal skill packages with text procedures, runtime state cards, and multi-view keyframes”. At runtime, a specialized “branch-loaded” agent inspects just those relevant images and states, aligns them with what the agent sees on screen, and distills them into compact guidance. In other words, the agent doesn’t have to carry dozens of large images in its prompt; it only pulls in the small subsets needed for the immediate task. Constructing the Skill Library. But where do these multimodal skill packages come from? The MMSkills paper introduces a trajectory-to-skill generator that takes raw demonstration logs (off-line interaction traces) and turns them into compact multimodal skills. The pipeline has several steps:. Workflow grouping. First, raw agent trajectories (say, logs of a user or model solving tasks in OSWorld or games) are parsed to identify which contiguous actions belong together as a single skill. For example, all actions needed to “open a new browser tab” would be grouped into one unit. The paper calls this grouping by workflow, to separate long task demonstrations into reusable sub-skills. Procedure induction. Next, each grouped sub-trajectory is converted into human-readable instructions. Typically, a large language model (LLM) can be used to describe what happened: e.g. “Click on the File menu, then click New Document.” The text procedure is a concise summary of the steps that were performed, akin to writing a mini how-to guide for that skill. Visual grounding. Simultaneously, the relevant screenshots from the trajectory are labeled as state cards and keyframes. For each step of the instruction, the generator picks out example screenshots – perhaps a “before” screenshot and an “after” screenshot – to form the state cards. It also selects a few keyframes (screenshots at intermediate moments) to illustrate progress. Sometimes both a full screenshot and a zoomed “focus” crop are stored for each state, giving multiple views of the same scene. The result is that each skill package knows what the environment looks like at each juncture of the procedure. Meta-skill auditing. The system then checks and refines each proposed skill package. This might involve using higher-level “meta-skills” or human review to ensure the generated text matches the images and that the instructions are coherent. For example, if the induced procedure is “Open the settings menu” but the screenshots show it clicking on a different button, the audit step would catch that. The end result is a library of MMSkill packages: each one is a compact bundle of text plus a few images. Unlike storing entire raw demonstrations (which can be large and redundant), MMSkills compress each skill into just the necessary clues. The authors report that the released MMSkills library (as of writing) contains 515 such skill packages, covering tasks on desktop operating systems, image editing, coding, and even game environments (Minecraft and Mario). In just the Ubuntu desktop (OSWorld) portion, these skills span 247 different procedures with 879 state cards and 1898 keyframes in total – enough to cover 360 distinct GUI tasks. (For comparison, the OSWorld paper introduced 369 tasks across multiple apps, and on that benchmark humans succeed over 72% of the time while pure LLM agents achieve only ~12%.) The variety of skills is broad: things like “Create a new slide in a presentation,” “Switch to the Linux desktop,” “Search Google for computer vision,” or “Kill a specific Goomba in Super Mario.”. Using MMSkills at Inference. How does an agent actually call on these MMSkills during execution? The key idea is branch loading. When the agent is following a high-level task (e.g. “generate a monthly sales report”), at certain decision points it will decide to invoke a sub-skill from the library. The MMSkills framework spawns a temporary “branch agent” that is given just the selected state cards and keyframes relevant to that skill, along with the text procedure. The branch agent also sees the current live screenshot. It can then align them: for example, it might match the candidate “before” state card to the live view to verify that we are indeed in the pre-skill configuration. If the images line up, the branch agent knows “aha, this is that part of the flow,” and it uses the textual steps (now grounded in visual context) to decide the next action. It then outputs a structured suggestion (“Next: click the green checkmark button.”) or actual low-level action to the main agent. Crucially, the main agent itself never has to load all images or do the matching; it just queries the branch system and receives a succinct guided action. This mechanism avoids the common pitfalls of multimodal prompting: the agent doesn’t have to ingest dozens of irrelevant screenshots, and it doesn’t rigidly bind to one static image. Instead, it gets exactly the visual hint needed at that moment, and can ignore the rest. This “branch-loaded multimodal agent” setup was shown to compare against two baselines: a no-skill agent that just follows the global instruction, and a text-only skill agent that has the same procedures but no images. In all cases, MMSkills led to more efficient, correct behavior. For example, the agent would avoid repeating needless clicks or taking wrong GUI paths because the visual state cards helped it recognize progress. The developers found that with MMSkills, agents made fewer redundant actions and were better at recognizing when a subtask was complete (they could literally “see” it in the state card) – improvements that translate to higher success rates. Performance Gains. On benchmark tasks drawn from the OSWorld suite (real Linux desktop tasks), adding MMSkills gave consistent gains for various language-vision models. The improvement was especially dramatic for smaller models that struggled more without guidance. In one reported example, a powerful agent (Gemini 3.1 Pro) improved from about 44% to 50% success with MMSkills; a medium agent (Gemini 3 Flash) went from 37% to 48%; and a smaller LLM (Qwen3-8B) jumped from only 11% to 25%. Even the very best agent still benefited: no model failed to improve when given access to the skill library. In summary, the experiments showed that “MMSkills consistently improve both frontier and smaller multimodal agents,” confirming that external skill knowledge can complement what the model already understands. In other words, an LLM like GPT or Qwen might “known” parts of these tasks internally, but having explicit multimodal examples significantly boosted its reliability. Importantly, these gains align with recent observations in GUI agent benchmarks: raw LLM/VLM systems struggle to infer intent and state. For instance, the GUIDE benchmark for GUI tasks found state-of-the-art vision-language models achieve only ~45–55% on recognizing user behavior or predicting when help is needed, and adding structured context (like the think-aloud narrations in GUIDE) dramatically helped. MMSkills can be seen as supplying exactly that kind of structured context at decision points. In the OSWorld benchmark as a whole, the best pre-trained model only solved ~12% of tasks (while humans solved over 72%). By contrast, an agent augmented with the MMSkills library saw a marked improvement in the same evaluation setting – a clear indication that these multimodal skill cues are filling in gaps in understanding. Ablation studies in the paper further confirm that each element of MMSkills is valuable. When state cards or visual keyframes were removed, or when the branch-loading was disabled (forcing the agent to see all images at once), performance dropped. This shows it’s not just the text that matters, but the way MMSkills supplies visual evidence. The “branch-loaded” design in particular strikes a balance between giving the agent sight of relevant screenshots and not overwhelming it with pictures. A Concrete Example. To illustrate, imagine the skill “Create a chart in a spreadsheet” in a LibreOffice Calc task. The MMSkill procedure might read:. Along with this text, the skill package might include “before” and “after” screenshots of the sheet (Sheet1 with no chart vs. after the chart appears) and intermediate keyframes (like a cropped image of the range A1:B5 selected, and another showing the chart wizard dialog open). At runtime, the agent sees its live spreadsheet environment. When it reaches the step “Go to Insert → Chart,” it calls the Calc skill’s branch. The branch agent compares the live view to the “before” card and confirms it’s still in the expected state (Sheet1 is active). It then guides the agent: “Click on cell A1 and drag to B5.” After that sub-step, it matches the live image to the “range selected” card, then advises, “Open the Insert menu and click Chart.” Finally it matches the “chart wizard open” keyframe, and steps through finishing the chart. The agent completes the skill knowing it did it correctly, because the final state card (chart placed) matches the environment. Without MMSkills, the agent might have still clicked something, but it would have no concrete sign that a chart was actually inserted, and might have made unnecessary mistakes. Skill Library Scope. The MMSkills library is large and growing. As of the publication, there are hundreds of skills spanning multiple domains. In the Ubuntu desktop/OSWorld set, 247 distinct skills were extracted from 360 tasks. Each skill has several state cards (for instance, the Ubuntu “Search in Settings” domain has 139 cards for 24 skills). In addition, there are 24 MMSkills for the VAB-Minecraft environment and 10 for Super Mario. Overall, 515 multimodal skills cover workflows across web, office apps, system settings, media tools, and even game logic. Each skill package is quite compact – typically one or two pages of text plus a few images – so the whole library remains easily stored on a robot or cloud server. This breadth is important. It means an agent has off-the-shelf procedures for many common subtasks. For example, there are skills for opening a web browser, navigating to Google, switching spreadsheet tabs, attaching an email file, controlling media playback, text-editing steps, and various mini-game challenges. By organizing skills by domain (Chrome, GIMP, LibreOffice, etc.) the system can quickly retrieve the top candidates given the current context. There is even a simple “task-to-skill mapping” that determines which skills might be relevant for a given high-level task. When run on a new task, the agent looks at the task description and its current state, and “top‐K” relevant skills are loaded for possible invocation. In practice, the agent often only has to load 3–6 skills at decision time, each with only a handful of images, so the cost is modest. Related Benchmarks and Context. MMSkills fits into a recent trend of equipping agents with external knowledge. For textual planning, modern systems add retrieved documents or code; for vision, this work shows that small sets of curated images and instructions can serve the same role. The idea of learning reusable skills is also common in robotics – in reinforcement learning or hierarchical planning, people often learn subpolicies. MMSkills differs by hand-crafting the skill library from demonstrations in a structured way, rather than end-to-end learning. It’s more akin to personal assistant plugins or the skills format used in AI agent frameworks, except here each skill has an explicit visual component. On benchmarks, MMSkills demonstrates clear value. In OSWorld (the Linux desktop benchmark introduced in Xie et al., 2025), even the strongest VLM agent initially solved just a small fraction of tasks. That same group (Zhang et al.) now show that adding MMSkills roughly doubles or triples performance for weaker models, and boosts stronger models as well. In fact, one can interpret MMSkills as giving agents a form of “visual chain-of-thought”: instead of the agent reasoning only in language about which icon to click, it can momentarily think in images by referring to the hints. There are also parallels with the new GUI-intent benchmarks (like GUIDE). GUIDE found that models could only predict user intent or recognize state about half the time without extra help. It also found that providing user context (think-aloud explanations of what the user is doing) greatly improved performance. MMSkills essentially provides a similar context automatically: with state cards, it's giving the agent a mini-explanation of what the “ideal” user action looked like. In that sense, MMSkills are a concrete step toward GUI agents that not only execute instructions but also understand and monitor progress. Discussion and Future Work. MMSkills highlights the importance of combining vision and language in agentic skills. The results suggest that large vision-language models have latent procedural knowledge, but external evidences let them apply it more reliably. One tradeoff is crafting the skill library itself, which currently relies on pre-recorded demonstrations and some manual or LLM-based curation. Future work might automate the trajectory-to-skill pipeline more fully or adapt it to real-world robot logs (e.g. videos of robot assembly tasks). Another direction is expanding skill branching policies: how does an agent decide when exactly to call which MMSkill, especially under uncertainty?. In robotics or manipulation domains (beyond GUIs), a similar idea could apply: imagine a factory robot with a library of multimodal skills like “grasp screw,” with text and camera images showing the screw and tool. When the robot sees a partially assembled object, it could match its camera view to the skill’s state card (“screw is loose in fixture”) and then execute the sequence. MMSkills was demonstrated in software environments (Ubuntu, Minecraft, Mario) for convenience, but the principle is general. As robots increasingly interact through screens, simulated sensors, or physical cameras, embedding a human-friendly multimodal procedure library could greatly enhance robustness. In summary, MMSkills provides a structured way to capture “how-to” knowledge for embodied agents by bundling instructions with visual cues. Through an automated pipeline of grouping, induction, and grounding, it turns raw demo logs into a compact, searchable skill library. Agents that leverage MMSkills at runtime become more adept at recognizing when and how to perform subroutines, improving success on challenging tasks. As AI moves toward more complex, stateful environments, such multimodal procedural memory might become as essential as the language models that consume it. References: As reported in the MMSkills paper by Kangning Zhang et al., et al.