Latest / Embodied AI 101 / UI-Mate: Show the Desktop Agent Once, Then Let It Replan
Transcript
- UI-Mate: Show the Desktop Agent Once, Then Let It Replan. UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations • Zihan Ding, Longxu Dou, Qi Gao, et al., Tencent HY Frontier Team • Tencent Hunyuan • arXiv technical report • 2026. A recurring problem in computer-use agents is that the instruction rarely contains the whole task. “Process these candidates,” “update the pipeline,” or “prepare the monthly report” may specify an outcome, but not the organization’s preferred tools, naming rules, approval sequence, spreadsheet layout, or notification policy. A capable agent can fill in those gaps, but it may fill them differently on every run. The result is an awkward distinction between occasionally capable and reliably delegable. Tencent’s UI-Mate attacks that problem from two directions. First, it builds a strong general desktop policy through environment-grounded supervised learning and online reinforcement learning. Second, it lets the user provide a single recorded execution that is distilled into a procedural scaffold. The agent follows that scaffold where it remains relevant, while continuing to choose actions from the live screen rather than replaying coordinates. For a robotics audience, the most useful interpretation is not “behavior cloning from one video.” UI-Mate is closer to a visually grounded hierarchical policy conditioned on an external task program. The demonstration proposes subtasks, milestones, and termination conditions. A learned low-level policy decides how to realize them in the current environment. The live observation retains veto power over the nominal plan. That is an appealing formulation. It is also important to be precise about what the paper actually demonstrates. The Real Claim Behind “One-Shot Doubling”. The headline result is accurate: on a 33-task subset of the new OSWorkerBench, adding one demonstration increases UI-Mate’s strict success rate from 17.17% to 35.35%. That is an 18.18-point absolute gain and slightly more than a twofold relative increase. Checkpoint-weighted progress rises from 67.85% to 81.14%, with improvements on 28 of the 33 tasks. But “one-shot” needs qualification. It means one demonstration is supplied at inference time without performing a target-specific gradient update. It does not mean that an unmodified general-purpose model watches an arbitrary video and immediately learns a new skill. UI-Mate uses a dedicated demonstration-conditioned 27B checkpoint, continued from the general computer-use model and trained on a mixture of ordinary and demonstration-augmented trajectories. Before use, the recording is also normalized, captioned by a vision-language model, segmented into subtasks, and potentially edited by a human. The quantitative experiment is also a self-demo evaluation. Each of the 33 targets receives a successful trajectory of that exact same task, produced by a stronger GUI agent. This tests whether UI-Mate can exploit a known successful execution path. It does not establish transfer from a related but different task. OSWorkerBench separately contains 45 variant-demo pairs, where a human demonstrates a semantically related workflow with different entities, values, states, and execution details. That is the more interesting procedural-generalization setting, but the paper does not report a systematic result across those 45 pairs. A ten-task pilot was reportedly net positive only after adjusting repetition counts to match the true number of target entities, and the authors describe performance as not yet stable enough for a primary claim. This distinction does not invalidate the doubling result. Long desktop workflows are sufficiently brittle that turning a known-good trace into reliable closed-loop behavior is itself difficult. But the result should be read as demonstration-conditioned execution guidance, not yet as broad one-example skill acquisition. The System Is Much Bigger Than the 27B Checkpoint. UI-Mate is presented as five connected components: an environment-grounded data pipeline, a general computer-use training stack, the DemoCUA mechanism, the OSWorkerBench evaluation suite, and a shared execution harness. Tencent has released general 9B and 27B checkpoints, based respectively on Qwen3.5-9B and Qwen3.6-27B, plus a separate demonstration-capable 27B checkpoint. The model observes screenshots and emits reasoning followed by structured mouse and keyboard operations. One response may contain several consecutive UI actions, but the environment supplies no new screenshot until that batch finishes. The paper therefore measures trajectory length in decision turns—observation-to-response cycles—rather than raw clicks or keystrokes. The policy is only one part of the control system. The harness maintains recent history, invokes the model, executes actions, records trajectories, and, in DemoCUA mode, tracks the current position in the demonstrated workflow. That last piece matters. Progress through a long procedure is not left entirely inside transformer activations: the harness exposes an explicit checklist and advances it when the model emits a action. In robotics language, UI-Mate divides state between the learned controller and an external task executive. This is arguably a feature rather than a concession. Long-horizon reliability often improves when persistent task state is represented explicitly instead of expecting the policy to reconstruct everything from a lossy conversational history. Building Training Data Around Executable Environments. The paper’s strongest contribution may be its treatment of GUI data as an environment-construction problem rather than a trajectory-scraping problem. UI-Mate draws task instructions from four sources: existing computer-use datasets, atomic operations extracted from failed or stalled rollouts, tasks grounded in authentic files and static websites, and operations sampled from application capability trees. Failed rollouts are particularly useful because they reveal behaviors current policies do not already solve. Real documents, spreadsheets, and presentations prevent the corpus from collapsing into clean synthetic files whose uniform structure creates visual shortcuts. An automated setup pipeline converts each instruction into a runnable initial state. It identifies required files, creates or retrieves them, configures applications, and initializes task-specific state. Wallpapers, desktop arrangements, sidebars, and application settings are randomized while preserving feasibility. The objective is to vary incidental appearance without changing task semantics. Tencent reports that authentic office resources are roughly six times larger than their synthesized counterparts. Trajectories over those resources average 58.4 steps, compared with 38.5 for synthetic files—a 51.7% increase. The implication is important: task difficulty often lives in the environment, not in the linguistic complexity of the instruction. A trivial sentence can induce a difficult task when the relevant data are buried among merged headers, irrelevant sheets, ambiguous records, and partially matching entities. Rollouts run through a unified interface spanning Ubuntu, Windows, and macOS environments. Before a trajectory enters supervision, a multimodal judge checks whether the task was feasible, the initial state was valid, the task was not already completed, the actions were well formed, and the observations were intact. It then derives independently verifiable deliverables from the instruction and looks for evidence of every deliverable throughout the execution. This avoids accepting a convincing final narration or a superficially plausible last screenshot when an intermediate requirement was missed. The capability tree closes the data loop. Tasks are mapped into a three-level hierarchy: application, coarse capability, and fine-grained operation, with a separate domain for cross-application behaviors. Collection budgets are adjusted using coverage, rollout success, rejection rate, and task length. Oversupplied operations are down-weighted; sparse capabilities receive new tasks; low success combined with high rejection signals a defective task or environment rather than simply a weak model. The authors report a 15.5-point improvement on Multi-App tasks relative to training without capability-aware allocation, although they explicitly describe this evidence as correlational rather than a clean causal isolation. Still, the design principle is strong: count covered behaviors, not just collected episodes. Verifiers Are Part of the Task, Not an Afterthought. Online GUI reinforcement learning requires more than a resettable desktop. Each task needs an initial state, at least one reachable successful state, and an executable evaluator that understands the actual user objective. UI-Mate begins with a basic consistency invariant: the untouched state must fail and a reference completion must pass. A generator constructs the environment and reference state, while a separate verifier agent writes the reward from the task specification without seeing how the reference completion was produced. This separation is inherited conceptually from CUA-Gym’s generator–discriminator design. That invariant is necessary but insufficient. A reference state and its evaluator can share the same semantic error. UI-Mate therefore subjects evaluators to two additional probes. A hard negative modifies a successful state into a plausible but wrong completion that should be rejected. An alternative positive constructs a different legitimate solution that should still pass. These probes test both permissive rewards and overly rigid ones. The need is not hypothetical. An audit of evaluators that had already passed the initial consistency check found roughly 18% remained misaligned with their instructions. The leading failure modes were overly strict matching, semantically empty assertions, and checks applied to the wrong target. Flagged bundles enter a bounded repair loop covering the instruction, environment, reference state, or evaluator. This is directly relevant beyond GUI agents. In any embodied RL system with programmatic rewards, environment generation and reward generation cannot be scaled independently. If the observation space, terminal state, and evaluator semantics are not co-designed, more environments simply create more opportunities for reward hacking. From Imitation to Online, Verifier-Grounded RL. Training starts with supervised fine-tuning on filtered agent rollouts and repaired human trajectories. SFT teaches the action protocol, visual grounding, common application workflows, and cross-application execution. Human recordings are validated against their visible state transitions, repaired when cursor placement or incomplete rendering leaks the target, and relabeled by a teacher model into the same reasoning-and-action format used by agent rollouts. The SFT model is then optimized online using groups of independent rollouts on the same task, generated from the same policy snapshot. An executable verifier returns binary success for each complete trajectory, and Group Relative Policy Optimization compares outcomes within the group without learning a separate value network. Groups where every rollout has the same outcome are discarded because they provide no relative signal. Long-horizon GUI episodes make ordinary trajectory-level credit assignment awkward. Failed runs often contain more decision turns because they wander, retry, or recover. If the same negative advantage is copied to every turn, long failures dominate the update simply because they contain more responses. UI-Mate addresses this with decision-turn centering, computing the group baseline under the distribution of turns rather than treating each trajectory as equally sized. A second normalization handles response length. Failed turns often produce longer reasoning traces, so their advantages would otherwise be repeated across more generated tokens. The method normalizes advantages at token level to keep verbose failures from exerting disproportionate influence. An optional Process Credit Model further localizes the terminal reward. A teacher identifies milestones and alternative valid branches from successful runs, then marks failed trajectories with progress, causal errors, recovery, redundancy, blocked steps, and skipped branches. The resulting weights focus actor loss on decisions most responsible for success or failure. Importantly, PCM does not replace the executable reward; it redistributes that reward over the trajectory. GUI rollout lengths vary dramatically, so training is asynchronous. The learner updates once enough complete trajectories from a task group arrive rather than waiting for every worker. IcePop and SeqClip filters reject tokens whose likelihood under the current policy has drifted too far from the rollout policy, while PPO-style clipping limits retained updates. The paper’s ablation findings are refreshingly restrained. Adaptive curriculum sampling and PCM improve training efficiency, often reaching comparable performance in fewer than half as many updates, but do not consistently improve final success. Historical reasoning has a similarly nuanced effect: retaining old reasoning traces improves long-horizon inference and cross-application state tracking, yet using those traces during RL rollouts accelerates entropy collapse and reduces exploration. UI-Mate therefore benefits from memory at inference without necessarily wanting the same memory regime during policy optimization. DemoCUA: Turning a Recording Into a Closed-Loop Workflow. A raw UI-Mate demonstration contains every pointer and keyboard action, plus screenshots immediately before and after each action. It can come from a human or a successful agent rollout. The recording is first normalized into a common action-and-frame representation. A vision-language model then annotates each event with the visible state, the operator’s intent, the action performed, and the visual evidence used to locate its target. Events are grouped into a small number of subtasks, each with a natural-language goal, an explicit completion criterion, and an ordered list of action descriptions. Concrete pixel coordinates are removed. At execution time, the agent does not receive the full demonstration in equal detail. It sees a progress checklist containing all subtasks, marked as completed, current, or upcoming. Only the active subtask includes its complete goal, completion criterion, and action sequence. When the current screen satisfies the criterion, the model emits , and the harness advances the workflow pointer. This is closely analogous to an options framework. The demonstration supplies a sequence of high-level options and termination predicates. The learned policy supplies the option implementation. Unlike classical replay-based automation, option execution is re-grounded at every observation. UI-Mate is explicitly trained not to trust demonstrations blindly. Its demo-augmented data contain three relationships between workflow and live screen: full alignment, partial mismatch, and irrelevance. In the second case, the model must correct the nominal steps using the screenshot. In the third, it must ignore the demonstration altogether. Training workflows also omit mundane intermediate operations such as focus clicks, scrolling, and popup dismissal. The supervised target still contains the full successful trajectory, forcing the model to infer how to connect the demonstrated milestones. At inference, the complete action sequence is provided because the model has already learned that the screen, not the trace, determines the next executable action. The case studies illustrate why semantic procedure can matter more than low-level action identity. In a Godot task, both the failing and successful agents selected the correct bullet asset. The unguided agent attached it to one character instance in the current level, while the demonstration led UI-Mate to modify the reusable character template. The visible operations looked almost identical, but only the latter changed the intended source object. In a document-grounded visa application, the demonstration taught viewport management: maximize the PDF viewer, reposition it to expose the relevant fields, and cross-check extracted values. The demo-conditioned run found the required $18,000 funding figure and selected the corresponding certificate, scoring 99.5%; the unguided run misread the document and scored 24.5%. Long histories are managed through context folding. Older interaction steps are summarized into a progress note while recent text and screenshots remain verbatim. One current systems limitation is that the workflow is placed at the beginning of the context. Every subtask update changes that prefix, invalidating KV-cache reuse. The authors propose moving workflow state to the end so the episode can grow append-only. OSWorkerBench: Testing Office Work as a Stateful Process. OSWorkerBench contains 100 office tasks across 41 normalized applications and ten job families. Ninety-nine tasks require at least two applications; the mean is 3.26 applications, and some require seven. Slack appears in 65 tasks, followed by Gmail, Google Sheets, Salesforce, and Google Calendar, creating a hub-and-spoke structure resembling real enterprise workflows. The benchmark distinguishes merely opening many applications from actually carrying information across them. Its 49 Multi-App tasks require at least three tools and substantive transfer of multiple dynamic facts, records, or passages. Its 67 Long-Memory tasks require delayed reuse of information or sustained tracking of multiple constraints and unfinished stages. The two subsets overlap but represent different failure pressures: spatial transfer across tools and temporal retention across a workflow. Tasks are synthesized around explicit cross-application dependencies. In one recruiting example, a Greenhouse record determines the candidate, Calendar constrains available interview times, Gmail communicates the invitation, and Slack receives the confirmed schedule. Difficulty is designed along breadth, navigational depth, and reasoning rather than estimated after observing model failures. Evaluation is based on functional state, not trajectory similarity. Eighty-eight tasks use state-based evaluators over initialized application backends, while twelve inspect artifacts such as spreadsheets, images, documents, or conditional outputs. Evaluators contain between one and thirteen weighted checkpoints. Strict success requires every final condition; progress awards partial credit for verified checkpoints. The benchmark genuinely produces long interactions. Kimi-K2.6 has a median of 68 decision turns, while UI-Mate-27B has a median of 71, with 40 UI-Mate trajectories reaching at least 100 turns. These are observation-to-decision cycles, not individual UI actions, and several actions may be batched into one turn. What the Results Say. Under instruction-only evaluation, UI-Mate-27B reports 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench it reaches 41.0% strict success and 76.86% progress, compared with 23.33% and 52.35% for its Qwen3.6-27B base. UI-Mate-9B reaches 34.0% strict success and 66.55% progress, a particularly large gain over the 5.05% and 18.11% of Qwen3.5-9B. The 27B model is the strongest open-weight system in the paper’s comparisons on OSWorld-Verified and WindowsAgentArena, but it does not erase the closed-model gap everywhere. On OSWorkerBench, GPT-5.6-Sol reaches 71% strict success against UI-Mate’s 41%. The open-weight result is therefore impressive primarily as evidence of the training stack’s leverage, not evidence that long-horizon office automation is solved. The core demonstration results are:. | Evaluation setting | Without demo | With one demo | Change | |---|---:|---:|---:| | GameDev normalized score | 76.76% | 81.15% | +4.39 pp | | OSWorld selected 30-task score | 40.27% | 65.75% | +25.48 pp | | OSWorker 33-task progress | 67.85% | 81.14% | +13.29 pp | | OSWorker 33-task strict success | 17.17% | 35.35% | +18.18 pp |. The OSWorld subset was deliberately selected from feasible tasks that UI-Mate failed without guidance but a stronger reference agent could solve. That makes it a useful intervention study with substantial headroom, but not an estimate of demonstration gains over the entire OSWorld distribution. The OSWorker results are averaged over three runs per target, while OSWorld uses five. Demonstrations do not always shorten trajectories. On OSWorkerBench, average length rises from 173.3 to 216.0 steps because unguided agents frequently stop after only part of a repeated or branching workflow. Guidance encourages them to complete remaining branches. On GameDev, by contrast, demonstrations reduce average length by 16.6% while improving score, suggesting that they remove exploratory detours when the required work is already represented in the workflow. Nor are demonstrations uniformly beneficial. Four OSWorld tasks become worse with guidance, and several individual GameDev tasks also decline. A mismatched or misapplied procedure can anchor the policy to the wrong interpretation. UI-Mate’s anti-copying training mitigates that risk; it does not eliminate it. Relation to RPA, Skill Libraries, and Human-Taught Agents. Traditional robotic process automation records deterministic sequences. It is efficient when the interface and data remain fixed, but brittle under layout changes, unexpected dialogs, or altered task logic. UI-Mate retains the procedural economy of recording once while replacing coordinate replay with screenshot-conditioned action selection. ShowUI-Aloha is the closest direct predecessor. It turns human screen recordings into semantic action traces using a recorder, semantic learner, planner, and executor. UI-Mate pushes this idea into a dedicated foundation policy and adds explicit training for partially mismatched or irrelevant demonstrations, plus a subtask executive that exposes only the currently relevant segment. CUA-Skill takes a more engineered route, encoding human-authored procedures as parameterized skills and composition graphs, with retrieval, argument binding, and memory-aware recovery. MMSkills represents procedures through text, state cards, and visual keyframes, temporarily loading relevant multimodal evidence before distilling guidance back into the main agent. These approaches provide more explicit reusable skill objects; UI-Mate instead aims for a lighter contextual representation that a trained policy can reinterpret against the current screen. The interesting convergence is that all three move procedural knowledge outside the model’s static weights. The disagreement is about representation: executable graphs, multimodal state packages, or demonstration-derived subtask plans. What to Take Away—and What Remains Open. For practitioners, UI-Mate makes the most sense when tasks are repeated, expensive to specify fully, and governed by tacit conventions: enterprise onboarding, reporting, content operations, internal tooling, or multi-application reconciliation. A demonstration library becomes a form of user-programmable policy memory. Retrieval can select a relevant procedure; the model can adapt it; executable checks can verify the result. The design also suggests a useful recipe for robotic learning. Distill demonstrations into subgoals and observable termination conditions rather than retaining only action sequences. Train the controller on aligned, partially misaligned, and irrelevant plans. Omit enough low-level detail during training to prevent copying. At deployment, let the plan guide exploration while the current observation remains authoritative. Deployment artifacts are unusually concrete. The public repository connects the checkpoints to an OpenAI-compatible endpoint and a model-agnostic desktop harness. The current native client targets Apple Silicon macOS; Linux uses a PyAutoGUI bridge, while Windows client support is described as in progress. Tencent reports roughly two to three seconds per step for its optimized 27B service on an eight-GPU machine, and ten to twenty seconds for a six-bit 9B model running on one Apple Silicon Mac. The largest scientific limitation remains variant transfer. Showing the exact target procedure from a stronger agent is valuable, but easier than deciding which parts of a related human demonstration should transfer, how many times repeated segments should run, and which target requirements are absent from the example. Until the 45-task variant-demo collection receives a stable aggregate evaluation, the strongest claim is improved execution of demonstrated workflows rather than general one-shot procedural learning. Reproducibility is another concern. The report is detailed about pipeline mechanics but much less revealing about total SFT and RL corpus sizes, training compute, teacher-model costs, and the contribution of individual training stages to the final checkpoint. The full system also depends on internal cloud environments, stronger rollout agents, VLM-based annotation, evaluator generation, repair loops, and occasional human editing. Reproducing the policy from the public weights is straightforward; reproducing the data flywheel is not. The benchmark results also combine model, prompt, history policy, action batching, context management, and harness behavior. The authors acknowledge that comparisons are end-to-end system comparisons rather than clean tests of parameter count or architectural specialization. In particular, UI-Mate’s explicit workflow pointer and checklist are meaningful pieces of agent state, not merely neutral formatting. Finally, strict success remains much lower than progress. UI-Mate-27B’s 76.86% progress on OSWorkerBench becomes only 41% complete success. Many failures are late-stage omissions: one missing field, an unwritten date, or a notification never sent. This is familiar from manipulation and mobile robotics, where a policy can execute nearly every component yet fail the task-level specification. Better terminal checking, milestone-conditioned recovery, and verifier-grounded progress rewards may matter more than another increment of visual grounding accuracy. Bottom Line. UI-Mate’s central idea is simple and consequential: when natural language under-specifies a procedure, give the agent an example—but convert that example into a state-aware workflow rather than a replay trace. The paper combines that interaction model with a serious environment-and-verifier pipeline, strong open-weight checkpoints, and a benchmark built around the failure modes of real office work: delayed information reuse, cross-application handoffs, branching procedures, and unforgiving end conditions. The reported doubling in strict success is real, but it comes from same-task demonstrations and a dedicated demo-trained policy. The harder result—robust transfer from one related human workflow to another—remains open. Even so, UI-Mate offers a compelling blueprint for agents whose competence comes from three layers working together: model weights for general skill, demonstrations for procedural intent, and live observations for closed-loop correction.