Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps

Apple Machine Learning Research published work describing an online reinforcement learning framework for training user interface navigation agents directly on live iOS applications. The paper's authors are Peitong Duan and Seyed Khashaiar Gatmiry, listed with an equal-contribution asterisk, along with Jeff Nichols, Emmanuel Abbe and Samy Bengio. The team frames the problem around two failure modes: frontier vision-language models can navigate UIs well but cost too much to run every day, while smaller models trained through supervised fine-tuning or offline reinforcement learning tend to memorize fixed trajectories and break down when something goes wrong.
The proposed system splits the agent into two specialized parts. A planning agent handles reasoning about what to do next, and a separate UI grounding model translates those plans into concrete on-screen targets. Decoupling them means each component can be trained and tuned for its own job rather than one model carrying both responsibilities. This is a recognizable pattern in agent design: language reasoning and pixel-level localization have different data and error profiles, and separating them makes it easier to diagnose whether a failure came from a bad decision or a misidentified button.
Reward comes from an LLM-as-a-Judge, which supplies two kinds of signal. Dense, step-level feedback scores intermediate progress, while a terminal reward marks whether the task was completed overall. The agent is optimized with Proximal Policy Optimization, the widely used policy-gradient method, extended here with step-level advantage estimation so the dense signals are credited to individual actions. Step-level credit assignment is the hard part of long-horizon UI tasks, since a single successful tap at the end of a ten-step flow says little about which earlier steps mattered.
The paper also introduces GiGPPO, described as a neighbor-conditioned value estimation method. It is designed to stabilize the critic by drawing on reward signals from visually similar states encountered in concurrent trajectories. In practice, many parallel rollouts pass through near-identical screens, so pooling evidence across those lookalike states gives the value estimate more data per update than a single trajectory would provide. A likely trade-off is that defining similarity between states becomes its own design choice, since two screens that look alike can behave differently.
Supporting this online loop at scale required infrastructure that orchestrates hundreds of concurrent virtual iOS devices. That is what makes live-app training feasible: the environment is real enough to expose genuine UI behavior, and the fleet of virtual devices provides the throughput that policy-gradient training needs. The engineering burden here is substantial, since each device must be kept in a working state while agents take exploratory actions that can leave an app in arbitrary conditions.
The authors report evaluations on UINavBench, iOSWorld and AndroidWorld. Online RL-trained agents substantially outperform both SFT and terminal-only RL baselines while converging faster during training. The headline generalization result is on Android: the 8B RL agents show nearly a sevenfold improvement over the SFT baseline when tested out of distribution on Android environments. On real iOS applications, the 8B agents outperform Gemini-3.5-Flash and tie with Gemini-3.1-Pro, per the reported results.
Why it matters: If these results hold up, teams wanting on-device or low-cost UI agents may not need a frontier model for every tap, since a compact model trained against live apps can approach much larger systems on iOS tasks and retain far more cross-platform skill. The infrastructure and judge-based reward pipeline are the expensive parts, and their absence from typical app-testing stacks is a likely barrier to replication. For developers, the practical signal is that agent reliability depends heavily on recovery behavior and step-level feedback, not just final-task accuracy.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications.