How GUI Agents Fail: GraceBench for Execution Stability Diagnosis
GUI agents are increasingly deployed to automate mobile tasks, but most benchmarks only measure whether a task finishes, not whether the execution path was stable. GraceBench fills this gap by representing each app's reachable interaction space as a UI Transition Graph (UTG) and aligning video-recorded trajectories to it, enabling step-level diagnosis of deviation, looping, and recovery. The authors evaluate eleven models on 169 tasks across ten Android apps and show that success metrics hide serious instability: over half of successful trajectories still contain a deviation or loop, and even the highest-success model recovers poorly. The work matters because it provides a concrete framework for measuring process-level reliability, which is essential for testing and real-world deployment.
GraceBench makes a convincing case that final success rate and completion rate hide meaningful execution instability in GUI agents. The paper supports this with 3,718 trajectories across eleven models, showing that $51.24\%$ of successful runs still contain deviations or loops and that even the highest-$SR$ model recovers poorly. However, the benchmark's scope is narrow—ten mostly Chinese mobile apps with 169 tasks—and the evaluation pipeline leans heavily on automated labels that were human-audited on only a $10\%$ sample. The contribution is valuable but should be read as a proof-of-concept diagnosis framework rather than a definitive cross-ecosystem assessment.
The central empirical finding is robust: outcome metrics alone are insufficient. The data show that trajectories with identical $SR$ and $CR$ can differ substantially in deviation rate, loop burden, and recovery ability, and the outcome-controlled pair analysis supports this claim with large effect sizes. The human audit of generated transition labels on a stratified $10\%$ sample also strengthens confidence in the annotation pipeline, yielding Cohen's $\kappa$ values above $0.76$ for deviation, recovery, and loop detection.
The benchmark's coverage is limited to ten Android applications, dominated by Chinese services such as Douyin and Dianping, which raises questions about generalizability to other geographies and platforms. More critically, step-level diagnostic labels are produced by an automated video-based evaluator and only spot-checked by humans on a $10\%$ stratified sample; while reported Cohen's $\kappa$ is respectable, the remaining $90\%$ of transitions carry unquantified error risk that could systematically bias aggregate metrics like $DR$ and $LBR$. In addition, recovery metrics ($RSR$, $ARL$, $RA$) are conditional on naturally occurring recovery opportunities, so their denominators vary across models and tasks in ways the paper does not fully control, complicating fair comparison.
The comparison to related benchmarks in Table 1 is clear and fair on the chosen axes, accurately noting that prior work lacks simultaneous online execution, multi-path tasks, graph references, stability metrics, and video evidence. The empirical design also isolates the contribution by comparing matched outcomes: when $SR$ and $CR$ are held constant, stability metrics still discriminate models, which directly supports the claim that process diagnostics add information beyond terminal success. That said, the evaluation is strictly bounded to the constructed UTGs, so any incompleteness in graph coverage or state canonicalization could reclassify valid agent behavior as deviation or stagnation.
Full reproduction will be difficult. The authors state that code will be released, but it is not yet available, and the two GUI-specific models were collected on a separate host without retained video or screenshot files. Furthermore, because the apps are live mobile services whose content and layout change continuously, replaying exact states requires the frozen UTG metadata and canonicalization routines, which are only promised for future release. Hyperparameters are summarized at a high level—temperature 0 and a 15-action budget—but per-model prompting details and device configurations are insufficient for independent replication.
As GUI agents are increasingly deployed for mobile task automation, their reliability depends not only on final task outcomes, but also on whether they reach these outcomes through stable interaction trajectories. In automation workflows, unnecessary actions, detours, and execution errors can add noise to collected data, introduce irreversible side effects, or undermine trust, even when the agent eventually completes the task. It is therefore important to systematically diagnose the execution stability, including whether an agent deviates from a feasible path, enters a loop, becomes stagnant, or recovers from unstable actions. Existing benchmarks, however, primarily assess final task success or a small set of intermediate milestones, leaving step-level execution behavior unclear. To bridge this gap, we introduce GraceBench, an online evaluation framework that represents the reachable interaction space of mobile applications as UI Transition Graphs (UTGs), providing a structural reference for step-level execution diagnosis. To support reliable alignment between dynamic UI observations and graph states, GraceBench uses video-grounded trajectory evaluation over this graph-aligned representation, assessing process stability beyond final success with metrics such as deviation rate, loop burden rate, and recovery ability. GraceBench contains 169 tasks spanning 10 applications and constructs 10 UTGs with 2,909 canonical GUI states and 3,116 observed transitions. Across 3,718 trajectories, GraceBench shows that outcome scores hide execution instability: GPT-5.6 Sol achieves the highest success rate (SR) but the lowest recovery ability, 51.24% of successful runs still contain a deviation or loop, and trajectories with the same success rate (SR) and completion rate (CR) can differ substantially in recovery behavior. Code and data will be released on https://iaie-physicalai.github.io/GraceBench.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.