Latest / Embodied AI 101 / MobileGym: A Controllable, Parallel Sandbox for Mobile GUI Agents

Transcript

MobileGym: A Controllable, Parallel Sandbox for Mobile GUI Agents · Embodied AI 101

• MobileGym: A Controllable, Parallel Sandbox for Mobile GUI Agents. Modern AI agents are increasingly being trained to operate real-world software interfaces – for example, using natural-language instructions to open apps, navigate menus, and complete tasks on a smartphone. However, existing platforms have struggled to balance realism and reproducibility. Emulators (like AndroidStudio or AndroidEnv) can run Android apps in a repeatable way, but they are heavy and typically limited to open-source or system apps, making large-scale online RL impractical. On the other side, agents tested on real phones face drift (app updates, network changes) and risk of uncontrolled side-effects (e.g. accidentally sending a message or making a purchase), and these physical setups are hard to parallelize. MobileGym is a new simulation platform that bridges this gap with an innovative approach: it runs “everyday mobile use” scenarios inside the browser, with a fully controllable structured state under the hood. In MobileGym, each app’s data and UI are modeled as JSON state, and agent actions modify only a runtime overlay on top of static world data. This means the simulator can snapshot, fork, and fully compare states with bit-level precision, allowing deterministic success/failure judgments and safe resets. Because it bypasses proprietary backends (e.g. remote servers), MobileGym stays lightweight – a single browser instance consumes ~400 MB RAM and starts in ~3 seconds – enabling hundreds of parallel environments on one commodity machine. Crucially, it provides dense, programmatic rewards: each task comes with a scripted judge that inspects the final JSON state instead of relying on vision/LangChain, eliminating ambiguity and even flagging unexpected side effects. In short, MobileGym gives researchers a verifiable, scalable sandbox for training and testing mobile GUI agents, without the drawbacks of emulators or real devices. This deep dive will unpack MobileGym’s design and the key results from the accompanying study. We’ll explain how the environment works, how tasks are defined (including the new “AnswerSheet” protocol), and how the authors trained an LLM-driven agent with parallel RL. We will examine the benchmark results – showing both the difficulty of everyday mobile tasks and the significant improvement from RL training – and the dramatic sim-to-real transfer they achieved. Finally, we’ll reflect on the technical contributions (like the layered state model and deterministic judgments) and what this means for future GUI agent development and safety research. Motivation and Overview. The MobileGym platform was built to address two fundamental problems in mobile GUI agent research: (1) How do you get reliable success signals from a complex app? and (2) How do you train agents at scale on many instances of real app tasks? As the authors note, existing approaches forced a trade-off: “Emulator-based platforms… offer repeatable evaluation but are limited to […] apps and are expensive to run at scale. Real-device benchmarks… are irreproducible and carry risk, preventing parallelization and verifiable evaluation.”. A key insight of MobileGym is that mobile UI tasks can be studied without perfectly reproducing backend data. Instead, MobileGym simulates the interface logic and synthetic content, enough to give agents realistic screenshots and behavior responses. In practice, MobileGym runs a simplified Android-like world entirely in the browser. It implements Android concepts such as task stacks, intent routing, permission dialogs, on-screen keyboards, notifications, and back-button behavior – everything needed for agent interaction fidelity – but it does not connect to real online services. Public data (e.g. social media posts, product listings) are represented as fixed read-only JSON; the agent can modify a lightweight runtime state overlay (like creating an account, sending a message, etc.). By segregating static world data from mutable user data, the system can perform quick snapshots and resets on just the small overlay. All state is captured as structured JSON, which means the entire app/user environment can be serialized, forked, and diff’d with precision. This design yields two breakthrough capabilities. First, every task in MobileGym has a deterministic programmatic judge that looks at the final JSON state to decide success or failure. No more shooting screenshots and trusting a VLM or fuzzy text match – the judge can check exact values in forms, or compare two versions of a contact list, etc. This gives binary success metrics and dense reward signals without noise, while also enabling a new safety-check: after each episode, MobileGym automatically compares initial vs final state to report any “unexpected side effects” (for example, did the agent actually send an email behind the scenes, even if the task only asked to open the mail app?). Since all states are known, Unrealized changes can be flagged – an option impossible when using real devices. Second, because MobileGym runs in the browser with only lightweight logic, it can scale up massively. The platform runs about 256 environments in parallel on a single server using only a few dozen CPU cores and 100 GB of RAM. This multiplier effect means you can try large-scale online RL in a way that would be unfeasible on emulators or phones. The authors show that running a full benchmark of 256 episodes can take just ~6 minutes in MobileGym, orders of magnitude faster than prior setups. And because of the JSON state, it supports group-based RL methods: you can reset multiple rollouts to the exact same initial state “fork” and thus use algorithms like GRPO (Group Relative Policy Optimization) that need repeated trials from identical conditions. To sum up, MobileGym is not a pixel-perfect reimplementation of Android or iOS. Instead, it trades some realism (no user photo albums or network flakiness) for full control, determinism, and speed. The authors believe that for research on GUI agents, this trade is worth it: agents still see realistic-looking screenshots and app navigation, but researchers get the benefits of a well-defined, reproducible training ground. MobileGym thus “turns the browser into a zero-risk, fully deterministic mobile world”. In concrete terms, the companion MobileGym-Bench includes 416 parameterized task templates (256 for testing, 160 for training) across 28 apps (12 common user apps and 16 system utilities). Tasks are things like “post a tweet”, “find the cheapest flight”, “change a setting”, or “query how many steps you walked”. They cover single-app sequences and multi-app workflows, mix operational actions (taps, swipes, typing) with information querying tasks, and vary in difficulty (L1 easy to L4 very hard). Crucially, query tasks don’t ask the agent to “speak” an answer. Instead, MobileGym uses an AnswerSheet protocol: at the end of a query task, the agent must fill in a small GUI form with typed fields (numbers, text or choices). The fields have clarity (for example, “enter the current temperature as a number with 1 decimal place”), and the system checks them exactly. This avoids all ambiguity of parsing free-text replies, and ensures that agents learn the actual information-seeking behavior without loopholes. The combination of the controllable simulator, the rich task suite, and the deterministic evaluation makes MobileGym a unique platform for advancing GUI agents. In tests, the authors found that state-of-the-art vision-language models achieved only 9.4–58.8% success rate on the various tasks, with the hardest level L4 being largely unsolved except by the very largest models. They then took one of these agents (a Qwen-VL model with 4B parameters) and fine-tuned it on 160 training tasks using online RL. Using parallel rollouts and a GRPO algorithm, they boosted its performance dramatically – in simulation its success rate jumped by +12.8 percentage points on the 256-test set, and impressively, this benefit carried over to a set of 59 equivalent tasks on real devices (the real-device success went from ~32% to ~72.9%, a +40.7 point gain, retaining 95.1% of the sim-side improvement). In short, MobileGym not only provides a powerful researchbed, but the policies learned in it really do transfer to the real world. The rest of this article will unpack all these ideas and findings in detail. We will first describe the technical architecture of MobileGym and how tasks are defined. Then we’ll explain the evaluation benchmarks and metrics. Next we examine the experiments: baseline agent performance, the RL training process, and the sim-to-real transfer results. Finally, we highlight the key takeaways for GUI agent development, including future extensions and research enabled by this work. Platform Architecture: A Browser-Based Mobile World. At its core, MobileGym is a browser-hosted simulation of Android-like apps. It is not built on a traditional Android emulator or heavy virtualization. Instead, the system simulates the pieces of Android that matter for GUI interaction (activity stack, intents, widgets, keyboard, back button, etc.) directly in JavaScript/HTML. Static app content (like fake social media posts, product listings, contacts) is coded as JSON databases, and the UI is rendered based on that. The result for the agent is just a sequence of screenshots (PNG images) along with minimal metadata – basically the agent sees the app screens as images and must act with tap/swipe/keyboard actions. The technical novelty is in how MobileGym manages state. All of the information about the world and the user is broken into layers. One layer is the read-only world state: fixed content like news feeds, map data, public posts, etc. Another layer is the runtime user state: the user’s account info, settings, form inputs, etc. (The authors also mention an OS-level state, but effectively actions only modify the user overlay.) When the agent takes an action (e.g. “click this button” or “type ‘Hello’”), only the small runtime overlay changes; the world layer remains intact. This layered model serves multiple purposes:. Efficient Snapshots. Because only the runtime overlay can change, resetting or forking the simulation only requires copying a compact JSON state, not the entire world. This makes operations like “save state at step 5 and try different actions” practical. Deterministic Comparison. At any point, the entire environment state (world data + runtime overlay + OS flags) is captured in a structured JSON. If we roll out an episode of, say, 30 steps, we can later compare the initial and final JSON documents to detect exactly what changed. This power enables the platform’s judges and metrics. Exact Reset/Reproducibility. We can reset the simulation to any snapshot, guaranteeing the environment is identical every time. That means training is reproducible and parallel episodes can start from the same state – a key requirement for some RL methods. Just to emphasize, this comes at a small cost in visual realism. MobileGym deliberately does not try to look exactly like a real phone screen (no dynamic lighting effects, third-party animations, etc.). Instead it focuses on interaction fidelity: if the agent taps where it should, the simulated app responds as it would (e.g. navigating to the right next page, updating a field). As one of the authors notes, “GUI agents only see screenshots and give discrete actions, so a lightweight programmable simulator that returns realistic screens and effect-of-action is enough”. In practice the screenshot images look like clean mock-ups of phone UIs, with realistic text and icons but a slightly cartoonish style. This trade-off keeps the runtime fast and predictable. MobileGym implements key Android mechanisms. For example, it supports the Android task stack, so if the agent launches an intent to open one app from another, it can later press the back button to return. It handles intents and deep links so you can simulate actions like “open the browser to search results” or “share a picture to another app”. It enforces permissions (like camera or location) as dialogs. It also simulates the on-screen keyboard for text input, so typing a message is done by switching to a keyboard and filling letters. All of this happens in the browser in JavaScript, but the agent is oblivious to that – it just sees the app screens update in response to its actions. Action Space and Perception. To keep things standardized, MobileGym defines a fixed 17-action interface for every app (the exact set isn’t exhaustively listed in the paper, but typical actions include “touch at (x,y)”, “long-press at (x,y)”, “swipe up/down/left/right”, maybe “press home/back”, etc., plus specialized actions like pressing “Enter” on the keyboard). The coordinate system for touches is normalized: every app screen is treated as a 1000×1000 grid, so an action like “touch at (600, 300)” means the agent taps at 60% from the left and 30% from the top of the screen. This normalization avoids having to deal with different device resolutions. Crucially, agents never see the JSON state directly. From the agent’s point of view, the observation is just the screenshot image (an array of pixels) plus maybe some trivial metadata. This simulates the real condition where an agent only has vision of the screen. All the logic about which GUI element was tapped, or what text is present, is handled internally by the simulator. Behind the scenes, when the agent taps, MobileGym looks up that (x,y) location in the UI layout to see which button or field was hit. It then updates the JSON state accordingly (e.g. mark a checkbox, submit a search query, open a new page) and renders the new UI. This separation means there is no need for cumbersome computer vision to interpret the screen for the environment – the simulator knows exactly what’s drawn and where. Nevertheless, the agent must still learn to interpret the pixels (or more advanced agent methods like vision-language models are used). State Management and Deterministic Judging. Perhaps the most important feature of MobileGym is how it uses the structured state for deterministic evaluation. Each task in MobileGym ends with a judge – a small piece of code that inspects the final JSON environment state to decide success or failure. For example, a task “send a greeting message” would have a judge that checks whether the target recipient’s inbox now contains a message with the greeting text. Because the judge is a programmatic check on JSON, it is totally repeatable and unambiguous. The authors contrast this with the alternative of using a vision-language model (VLM) to read the screen in words and judge whether the correct info is present – a method prone to false positives/negatives and hard to reproduce. MobileGym’s judges give exact verdicts for success, and also easily compute a dense reward (for instance +1 at final success and maybe intermediate bonuses) for RL training. Even more interestingly, the platform can compute Unexpected Side Effects (USE). After each episode, MobileGym automatically does a full diff between the initial and final JSON states. Any change not required by the task is flagged. For instance, imagine the task was “show me the directions to the post office”. If the agent accidentally sent a text message while navigating, the state diff would reveal that an extra item appeared in the messaging app, which could be reported as an unintended side effect. The authors incorporate USE as an explicit diagnostic metric: if it’s high, it means the agent often does things we didn’t ask for, which is unsafe. Having USE available is a novel safety feature only possible because MobileGym knows the complete state. Finally, the state design enables some advanced training tricks. Because you can snapshot the entire state mid-episode, you can “fork” it to do multiple rollouts from the same starting point. The authors use this to run group policy optimization. Intuitively, if you give the agent the same initial app state multiple times (like same feed or same list of contacts), you can compute relative rewards across those parallel runs to stabilize learning. This underpins using the GRPO algorithm, in which groups of rollouts are compared to each other. Without such exact state control, you couldn’t guarantee identical starting points on parallel runs. In summary, the architecture of MobileGym trades back-end realism for full programmability. It guarantees that if an agent does X, we will see exactly Y change in JSON. This makes the environment predictable, safe, and easy to scale out. Podcast host style: think of it like a “game engine” for mobile apps, where the level designers have given a camera feed to the agent but still control everything underneath. The MobileGym-Bench Task Suite. A major part of this work is the MobileGym-Bench: a benchmark of parameterized GUI tasks designed to evaluate and train agents. It includes 416 task templates, divided into 256 for testing and 160 for training (so agents are tested on tasks they were not explicitly trained on). These templates cover 28 apps. Twelve are “everyday” user apps such as social media, navigation, shopping, messaging, etc., and 16 are system utilities (settings, calendar, contacts, etc.). Each task template is like a mini-scenario with placeholders. For example, a template might say “Send $AMOUNT of money to [contact name] using the banking app”. At runtime, the task instance is filled with specific values: $50 to “Alice”, another time $20 to “Bob”, etc. Likewise, instructions given to the agent can be phrased differently (“Transfer 50 dollars to Alice” vs “Pay Alice 50 dollars”). The environment can also use different underlying data, like different lists of contacts or items. By varying instructions, parameters, and initial data, each template can spawn tens of alternative instances (in total over 27,000 distinct episodes are reachable). This rich variability is crucial to prevent agents from simply memorizing fixed sequences – they must truly understand the instructions and adapt to the specific values. Tasks are categorized along four axes:. Scope (Single vs Multi-App): Some tasks only involve one app (e.g. “find the nearest pizza” in Maps), while others cross apps (e.g. “Receive an email about an event, then add the event to your calendar and set a reminder”). Objective (Operate vs Query vs Hybrid): An operate task means the goal is to perform an action (like sending a message, entering settings, completing a purchase). A query task means it’s asking for information (e.g. “What’s the weather on Friday?”). Hybrid tasks combine both (e.g. “Find which of your friends lives in Seattle and send them a check-in message”). Composition (Atomic vs Sequential vs Transfer vs Deep-Dive): Atomic tasks are one-shot actions. Sequential tasks have a clear sequence of steps. Transfer tasks require earlier steps to enable later ones (like logging in before viewing messages). Deep-dive tasks involve exploring screens in detail (maybe clicking through various menus). Difficulty (L1–L4): After collecting performance data from eight reference models, the difficulty of each template was calibrated. Level 1 is easiest, up to Level 4 which only the best models crack. This multidimensional taxonomy helps analyze where agents struggle. In fact, the paper reports that success rate drops monotonically from L1 to L4 tasks, with Level 4 being a true frontier: out of nine top models, only Google’s Gemini 3.1 Pro managed any 21.9% success on average at L4. All tasks obey a standard screen budget. Each task is allowed a certain number of steps (15–60) to reach success, plus up to 15 extra steps to fill out any final AnswerSheet form. The budgets were set generously higher than the optimal length so agents must decide when they have finished and not waste steps. AnswerSheet: Structured Query Answers. A key innovation for query tasks is the AnswerSheet protocol. Unlike many benchmarks that have an agent speak or type a free-form answer, MobileGym requires the agent to fill in a GUI form of typed fields at the end of a query. For example, a task might be “How many events are scheduled today in the calendar app?” and the agent’s final action is to fill a numeric field labeled “Number of events.” Fields can be numeric, dates, choices, text, etc., and each has a specific matcher in the judge. This solves two common problems: (1) Semantic equivalence: if the answer is “twelve” vs “12”, a textual judge might miscount, but a numeric matcher will know they match. (2) Verbatim leaks: prompts sometimes ask the LLM to “think” out loud. If an agent dumps reasoning text, the benchmark could wrongly mark it correct just because it saw the answer in the text. By using typed fields, MobileGym ensures the agent declarations (not its chain-of-thought) are judged. In short, the AnswerSheet protocol makes sure query tasks cleanly map to structured output, “avoiding failure modes such as rejecting equivalent answers or accepting answers that contain the reasoning chain”. Once the agent submits the form, the platform checks each field with a strict matcher (numeric tolerance, exact text, choice membership, etc.), giving partial credit or applying the final success flag accordingly. This yields fully deterministic scoring for information tasks – a big improvement over, say, past work that used CLIP or GPT-style post-hoc graders that could be wrong 10% of the time. Evaluation Metrics. The authors define multiple metrics to characterize agent behavior and failure modes. The primary metric is Success Rate (SR): the fraction of tasks where the agent satisfied the judge (fully correct answer or complete operation) within the allowed steps. Beyond that, they log:. Progress Rate (PR): for multi-step tasks, this measures how many subtasks or steps the agent completed successfully before stopping. It gives partial credit if an agent solves some parts but not all. False Complete (FC): the agent might declare success too early. This flag is set if it stops with success but the judge says it hadn’t actually completed the goal. Overdue Termination (OT): conversely, if the agent already achieved the goal and then kept taking steps (e.g. clicking irrelevant things) until hitting the step limit, that’s “overdue continuation”. Unexpected Side Effects (USE): as discussed, this is the percentage of episodes where an unintended state change was detected. A lower USE means the agent was careful not to do anything outside the task. By analyzing these metrics across the difficulty levels and dimensions, the benchmark provides a “diagnostics-rich” view of what kinds of errors agents make. Comparison to Prior UI Benchmarks. MobileGym-Bench stands out compared to previous GUI benchmarks. Many existing suites focus only on in-app command sequences (e.g. basic navigation or simple tasks in one app) or rely on matching screenshots. A few, like MobileBench-OL, run on real devices but then suffer from drift. The authors position MobileGym between these extremes: it supports everyday apps and tasks in a repeatable way, but without the costs and risks of actual devices. For example, AndroidLab is an emulator-based benchmark that covers some tasks on Android (mostly utility apps), but each emulator instance is huge (multiple GB) and the UI state is tied to open source apps only. MobileGym, by going browser-based, can cover proprietary-like apps while keeping instances small (~400 MB). Compared to real-device toolkits (e.g. the RealWorld benchmarks), MobileGym gains the ability to fork/reset and to have exact outcome signals (no flakiness), all while still showing screenshots that look plausible. The paper’s Table 1 quantitatively compares these platforms (we’ll mention key points in words): MobileGym’s task set is larger (416 tasks) and includes transfer/difficult tasks, and it alone offers verified sim-to-real transfer in their study. In terms of infrastructure, MobileGym is extremely efficient: on a single server it ran 256 parallel environments using under 10% of CPU and ~100 GB RAM, finishing a full benchmark run in about 6 minutes. This is orders of magnitude faster than running dozens of Android emulators or physical phones for the same tasks. The scalability means researchers can use off-the-shelf hardware to do large-scale RL or many-shot evaluation of UI agents – something that was previously only possible with dedicated device farms. In summary, MobileGym’s contributions on the environment side are:. Controlled, JSON State: Full world/user state as structured data, exposed to the researchers (not to the agent). Deterministic Judging: Programmatic success metrics and dense rewards based on state, not vision. Unexpected Side-Effect Detection: A novel safety metric from state diffing. Parallel Simulation: Hundreds of light-weight browser instances per server. Rich Task Suite: 416 parameterized tasks across varied apps, with an AnswerSheet for queries. Together, these amount to a qualitatively new tool for GUI agent research. The rest of the paper demonstrates how they validate and exploit this platform. Training Agents in MobileGym: Methodology. To show that MobileGym can actually train useful agents, the authors performed an online RL experiment using a vision-language foundation model. They started with Qwen3-VL-4B Instruct, a 4-billion-parameter multimodal foundation model (trained to accept both images and text). At baseline, this model had a low success rate on the tasks (around 9.4% overall on the 256-test set), which reflects the difficulty of mobile app reasoning without any task-specific training. For training, they used an approach based on GRPO (Group Relative Policy Optimization). In broad terms, GRPO is a variant of PPO (Proximal Policy Optimization) designed for group sampling: at each step the agent generates G alternative completions (rollouts) from the same prompt, computes a group-level advantage score, and updates the policy to maximize that. The HuggingFace TRL documentation explains that GRPO “maximizes the advantage of the generated completions while ensuring the model stays close to the reference policy”. The MobileGym setup leverages GRPO’s group nature by forking the environment state: each group of rollouts can share an identical initial screenshot. In practice, they used G=96 parallel environments (forks) for training on the 160-task training set, each group being 96 presumably. The training loop looked roughly like this:. Sample a batch of instructions from the 160 training templates (with random parameter values). For each instruction, fork the current environment to 96 identical copies. Use Qwen3-VL-4B to generate actions step-by-step in each copy (guided by the instruction prompt). All 96 follow potentially different action sequences. At each step, capture the image screenshot and give it (plus the prompt) to the model to sample the next action. Continue each rollout up to the step limit or until the model stops (declares done). Compute rewards: final success (1 or 0) plus any intermediate shaped reward (the paper suggests they use “dense RL rewards” from state differences, which likely include partial progress). Also detect any “unexpected side effect” and possibly penalize it (not explicitly stated, but likely it’s used as a separate metric rather than reward shaping). Using these rewards across the 96 rollouts, update the model policy with GRPO. They repeated this for 10 RL steps (I interpret this as 10 gradient updates, presumably with many episodes per update) on the 160 tasks. The entire process was done with the new MobileGym environment, so each model action was actually driving a browser-based simulation. Because MobileGym runs so many envs per server, this RL training required no special hardware (just one machine with ~100 GB RAM). Precisely how they mapped the Qwen model outputs to actions isn’t exhaustively described, but we can infer: likely the model’s vocabulary included tokens for “touch(x,y)”, “type(‘hello’)”, etc., as in prior work on HATs (HuggingFace Agents Toolkit). There might have been a fixed grammar or set of tokens for the 17 discrete actions and coordinate placeholders. These details aren’t in the summary, but this is a standard technique in Vision-Language RL for UI tasks. The model would generate an action keyword and a coordinate, then the environment would apply it, produce a new screenshot, and feed that back to the model (similar to an RLLM loop). Importantly, the simulator provided dense rewards and known success. This means the RL was not left with sparse text judgments. For example, each time the model clicked something that correctly progressed the task, it could get a small reward. Once the task was done, it got a big success reward, and any stray action (side effect) could have been negative. This reward design (based on structured state diffs) likely greatly accelerated learning, as opposed to just giving +1 for final success. While GRPO is central to their training recipe, the high-level takeaway is that they fine-tuned the vision-Language model on the MobileGym tasks via on-policy RL with parallel rollouts. The multi-instance forking made GRPO feasible, and MobileGym’s efficiency made it practical to do many rollouts quickly. We should note that they refer to “+12.8 percentage points” improvement on the test set after RL, which suggests they measured success before vs after fine-tuning. In raw numbers, the base Qwen3 model was at 9.4% SR; after 10 RL steps it reached 22.2% success (an improvement of 12.8 points). We’ll detail this in the results section. On the technical side, MobileGym uniquely supports this workflow: it lets the model train with closed-loop feedback in a controlled, safe way. In a real-device scenario, doing RL would be infeasible (you can’t let an agent randomly tap around on 96 phones). In an emulator, the slowness would limit you. Here, everything needed for RL is available: fast resets, reliable judging, parallel rollout fork, etc. This demonstrates MobileGym as not only a benchmark but also a training platform. Experimental Results: Performance and Sim-to-Real. The paper reports two main experimental results: (1) evaluation of several base models on the MobileGym-Bench without extra training, and (2) the gains from the RL-trained agent, both in simulation and on real devices. We’ll cover each and highlight the important findings. Baseline Model Evaluation. Using the MobileGym environment, the authors tested nine leading vision-language agents (models like Claude, Gemini, GPT, Qwen, etc., including the 4B Qwen3-VL) on the 256-test tasks. The overall success rates spanned a wide range. The weakest model (Qwen3-VL-4B-Instruct as a zero-shot baseline) achieved only 9.4% success, while the strongest (Gemini 3.1 Pro) reached 58.8%. This 6× gap shows that current models still struggle significantly with even common app tasks, and that model size/quality matters. Across the four difficulty levels, performance fell off sharply. The easiest (L1) tasks saw moderate success rates, but by L4 (the hardest, usually multi-step and multi-app tasks), almost all agents hit very low performance. In fact, only the largest model (Gemini Pro) managed a non-trivial 21.9% SR at L4. This serves as a sobering “frontier”: even state-of-the-art LLMs can rarely fully solve the hardest GUIs without task-specific training. Another interesting finding was on the Unexpected Side Effects (USE) metric. Models with similar overall success rates showed large differences in USE. Some agents tended to make extra unintended changes far more often than others. For example, one model could complete a task mostly cleanly, while another (with comparable success) might hit wrong buttons on some runs, causing extra taps or sent messages. The reported USE ranged from about 4.7% to 14.5% across models. This highlights why USE is valuable: two agents with identical success could differ a lot in safety. The best models had both higher success and lower USE, as MobileGym would certainly prefer. These baseline results establish a few baselines. First, they validate that the benchmark is challenging and discriminative: a 6× range in SR is a good span. Second, they underscore the room for improvement via learning – the fact that even the best model only solved a bit more than half the tasks means that training could help. Third, they show that MobileGym can reveal nuanced behavioral traits (like side effects) that would be opaque in a non-programmatic setting. RL Training Gains. The most compelling result is how RL fine-tuning dramatically improves an agent’s success, and how well that improvement transfers to real phones. Recall they took Qwen3-VL-4B-Instruct and did 10 GRPO updates (with 96 parallel forks) on the 160-train task set. Let’s walk through the numbers:. Simulation (MobileGym) Performance: Before RL, Qwen3’s success rates on the 256-test tasks were very low (around 9.4%). After just 10 RL steps, the success rate jumped to 22.2%. This is a +12.8 point absolute gain. In relative terms, the agent more than doubled its success. This shows that even a small amount of online RL in the new environment can significantly teach the model how to accomplish UI tasks. Real-Device Performance (59-task subset): Critically, the authors did not stop at simulation. They identified 59 tasks whose simulated conditions could be meaningfully replicated on an actual Android device (same apps, similar data). They then evaluated the Qwen model before and after RL on that real-device subset. Amazingly, the success rate rose from 32.2% to 72.9% – a +40.7 point absolute increase. Note that the baseline 32.2% on real matches roughly the 58.8% top SR in sim; likely the Qwen model had some ability but less than Gemini. After training, the real-device success nearly tripled. Perhaps more important is that the simulated performance gain (+42.8 points on the 59 tasks in sim) was almost entirely preserved on the phone side. MobileGym reported that 95.1% of the sim’s benefit was retained on the real device. In practice, 95.1% is an impressively high sim-to-real fidelity. It indicates that what the model learned in the MobileGym simulator really did generalize to the real app UX. Sections of unpredictability – like slight differences in text layout or timing – did not wipe out the training effect. The experiment also suggested that trajectory characteristics matched well. The average number of steps taken by successful episodes was similar between sim and real evaluation, especially for tasks that involved “operate” actions. This hints that the agent’s learned behavior (how it navigates screens and clicks) translated faithfully. Recovery of Missing Info (Qualitative): One qualitative anecdote further supports this. In training, the Reddit app in MobileGym did not require a “flair” tag to post a message (because the simulated environment had no concept of flair). But on the real Reddit Android app, it turned out that one subreddit enforced a required flair before posting. The fine-tuned model trained on MobileGym handled this gracefully: it recognized that a greyed-out “Post” button meant something was missing, found a flair, and filled it in, ultimately posting successfully. The base (untrained) model, by contrast, just clicked on the grey button repeatedly and gave up. This suggests that MobileGym training taught a general strategy (e.g. “don’t stop until active button”) and made the agent robust to an out-of-distribution condition it hadn’t seen in sim. Improved Safe Behavior: Although not quantified in the summary, presumably the RL agent also learned to avoid side effects. The authors note auditing revealed that relying on VLMs to judge real screen texts had ~10.2% error (for GPT-5.4 и Qwen3.6), whereas MobileGym’s judges had none by design. This reinforces that MobileGym allowed perfectly accurate training labels and evaluation. In summary, the real-device result is the headline: +40.7 points on real tasks is a huge improvement, and it came from modest training in simulation. It tells us (1) MobileGym’s fidelity to real apps is high, and (2) policy learning there can generalize. For GUI agent developers, this is a very encouraging outcome. It means one can do most of the heavy training in simulated MobileGym tasks, then only a small real-device calibration or sanity check at the end. This is far more feasible than training entirely on phones. Efficiency and Scalability. While the above focused on success rates, the paper also highlights how efficient MobileGym is. For example, running one full benchmark (all 256 test tasks, presumably with several episodes each number) took about 6 minutes on one server with 256 threads. Using 96 parallel envs, they did 10 RL steps in one afternoon. This kind of throughput is only possible because each instance is lightweight – the environment only uses ~400 MB RAM per instance and has a fast startup. Empirically, 256 envs used under 10% of a 32-core CPU and ~100 GB RAM. By contrast, running 256 Android emulator instances would be unimaginable on the same machine. They also contrast MobileGym to a real-device evaluation: testing even 59 tasks on physical phones is much slower (getting outputs from phones, restarting them, etc., is labor). In MobileGym, those same 59 tasks take seconds to run in simulation. The efficiency is critical to making training and ablations possible. Technical Contributions and Innovations. Having described what MobileGym is and does, let’s distill the key technical contributions:. Browser-Hosted Mobile Simulation. MobileGym shows that it’s viable to host an Android-like environment entirely in the browser for research. By focusing on the interface layer (UI widgets, app logic written in JS) and abstracting away backends, the platform can include everyday app functionality without needing actual app binaries. This approach is novel compared to using emulators. Layered State Model. The separation into a static world layer and a mutable runtime overlay is a crucial design. It ensures that snapshots and comparisons remain lightweight and complete. This makes the environment not just fast, but also verifiable: every action’s effect is captured precisely. The idea of bit-level consistent JSON serialization of state is a technical enabler for reproducible RL. Declarative Task Definition. Each app’s UI flow is defined by a state machine or similar declarative spec, allowing both runtime execution and static analysis. This makes it easier to generate tasks and proofs of difficulty. The paper mentions using extended finite-state machines (EFSMs) to model each app’s navigation. This is analogous to how some web-testing frameworks work (like Selenium for websites), but applied to simulated Android apps. Programmatic Judgement & AnswerSheet. By implementing each task’s success criteria as code, MobileGym guarantees deterministic outcomes. The introduction of the AnswerSheet protocol is a neat trick: it essentially enforces that queries are answered by filling a form. This solves the messy problem of free-text answers and allows type-specific matching. In RL terms, it also means the agent’s “action” can include an answer in a structured way. This kind of answer-form mechanism could be a general idea for other tasks (e.g. robot question answering by symbols). Unexpected Side Effects Metric. This is a conceptual contribution to safety in embodied agents. By comparing initial and final states, the system can flag any accidental change (like sending a message, changing volume, etc.) that wasn’t part of the task. Including USE in the benchmark encourages designing agents that are cautious. It’s likely the first explicit USE metric in a mobile-agent environment. Massive Parallelism. Achieving 100+ envs on a single commodity node is not an algorithmic innovation per se, but it’s a noteworthy engineering feat. They show how the choice of architecture (browser JS vs emulator) directly leads to this capability. It enables the use of parallel RL like GRPO in a practical way. MobileGym-Bench. On the evaluation side, the 416-task suite is itself a community resource (with code/JSON in the repo). Designing a diverse set of everyday tasks is valuable – especially with the instantiation variety (27k instances) and difficulty calibration. The careful inclusion of multi-app workflows and parameterization goes beyond simpler benchmarks. Altogether, these contributions form a package that is more than the sum of parts. It is not just “another simulator” – it redefines what it means to simulate a mobile UI for RL. The results validate the design: they could do RL with an LLM this way and get a remarkable Sim-to-Real lift. Implications for GUI Agents and Future Work. What does MobileGym mean for the field of GUI agent research? Here are some takeaways and future directions:. Accessible Training Environment: Researchers can now train or fine-tune agents on hundreds of mobile tasks without needing a farm of devices. MobileGym lowers the barrier to doing RL/IL on UI tasks at scale. This could accelerate progress in building capable agents that can, for example, help with personal productivity tasks, accessibility, or automated testing. Bridging Sim-to-Real: The high transfer rate (95%) is a very positive sign. It suggests that policy learning in such simulators can indeed generalize to real apps – at least for GUI tasks. Other robotics simulators often struggle with the “sim gap”; here, because the environment is not modeling physics but UI state, the gap is smaller. Future work could explore even more complex real-app scenarios, or test transfer on more tasks (the 59 chosen were apparently those that mapped straightforwardly). Safety and Aligned Behavior: By including things like payment flows, account deletion, etc., MobileGym allows training and evaluating how agents handle high-stakes operations in a zero-risk setting. Researchers could systematically study safety behaviors: e.g. punish USE if it indicates risky ops, train refusal/submission, or curriculum that introduces pitfalls. Because the environment can inject things like fake notifications, low battery, or network errors, one could also test robustness. Reward Shaping and Curriculum: The snapshot/fork feature supports advanced RL ideas. For instance, one could train with contrastive rewards: compare what happens if the agent deviates or picks an “unsafe” action. One could also implement multi-task or hierarchical training: first solve simpler tasks, then progressively harder ones. The authors mention doing state-difference reward shaping or multi-task curricula as future possibilities. World Models / Representation Learning: Each step yields a pairing of visual input and concrete JSON changes. This data could be used to train a UI world model – e.g. a neural net that predicts the next JSON state from the image and action. Or to train image encoders that align pixels with semantics. These datasets (states and transitions) could benefit offline learning or model-based RL. Benchmarks for Vision-Language Models: The MobileGym tasks provide a challenging benchmark that is still closer to real user goals than many other LLM/agent tests. Future LLMs could be evaluated on this suite easily (simply by connecting them to the MobileGym API). The AnswerSheet format is general; one could adapt it to other domains where precise answer fields are needed. Extending Apps and Domains: The modular nature of MobileGym means new apps can be built in. The paper suggests domain-focused expansions: e.g. specialized financial transaction flows, travel booking apps, or security-sensitive scenarios. One could also tie in multi-modal elements: perhaps adding voice commands or AR. Commercial and Practical Applications: For industry, MobileGym could be useful in automated testing of mobile apps (though it’s not officially supported by app stores). It also opens the door to research on “personal assistant” apps that use RL to learn user preferences. The zero-risk simulation means one could try novel UI features and let an agent explore them. Limitations to Note: It’s worth acknowledging what MobileGym does not do. It doesn’t simulate network delays, occasional failures, or true backend uncertainty (e.g. an online service going down). It can’t capture extremely rich multimedia UIs or 3D AR effects. The environment fills in content in a synthetic way, so it won’t teach an agent how to navigate a brand-new app it’s never seen fields for. However, for many everyday tasks, these limitations seem acceptable given the benefits. Overall, MobileGym represents a step toward the vision of RL agents that can handle “real” software interfaces with guaranteed assessment and at scale. The success of sim-to-real in this work suggests that similar strategies could be applied to desktop apps, websites, or even home automation interfaces: wherever the environment can mask complexity but preserve actionable structure. For now, this paper gives us a compelling new platform to experiment on – and the early results are very encouraging. TL;DR: MobileGym is a browser-based mobile GUI simulator that offers 100% deterministic state, parallel rollout, and a 400+ task benchmark with structured answer forms. It lets agents learn everyday app tasks with online RL. The authors show that an LLM agent trained on MobileGym gets a huge +40.7 point boost on a real-phone test set, retaining 95% of simulated gains, proving that controlled GUI sims can drive real-world agents. This platform could become a new standard toolbox for training and evaluating mobile UI agents, with built-in safety metrics and efficiency that encourage more ambitious agent research.