Diagnosing Personalization Failures in Mobile GUI Agents
Personalized mobile GUI agents must decide both how user preferences alter execution and when they apply along a trajectory. Existing benchmarks only report aggregate task success, obscuring where and why personalization fails. This paper introduces PrefDiag-Bench, a diagnostic benchmark with 318 end-to-end tasks and 1,664 stage-wise probes that categorize preferences into five execution types and pinpoint bottlenecks in evidence selection and trigger recognition. The result is a reproducible tool set for moving beyond coarse success rates toward targeted improvement of personalization capabilities.
PrefDiag-Bench makes a credible case that current GUI agents struggle with personalization, but the paper is stronger as a diagnostic instrument than as a definitive system ranking. The benchmark's five-type taxonomy and paired probes reveal sharp capability differences that aggregate metrics obscure, and the controlled evidence intervention demonstrates that evidence selection is a manipulable bottleneck. However, the dataset's synthetic provenance and the protocol of building probes from observed failure cases limit the universality of the reported gaps. Even the best-performing model, Seed-2.0-Pro, achieves only 47.17% on process preferences under explicit user profiles.
The execution-oriented preference taxonomy is a genuinely useful contribution, shifting evaluation from domain labels to how preferences alter agent behavior. The paired design of end-to-end tasks with stage-wise probes enables fine-grained failure attribution, and the finding that evidence selection and trigger recognition are the weakest stages is well-supported by both probe accuracy and joint task-probe failure analysis.
The benchmark's reliability is weakened by a circular construction protocol: diagnostic probes were derived from tasks with observed failures under behavioral logs, which likely inflates failure rates for P2 and P4 relative to a random task sample. The evidence intervention was also evaluated on only 85 P2-enriched tasks selected because at least one model failed both P2 and the task, so the reported 7.06% gain may not generalize. Additionally, the behavioral memory logs are synthetic and LLM-generated, making it unclear whether agents struggle with realistic user behaviors or with stylized distractors.
The empirical evidence supports the central claim that aggregate task success hides large preference-type differences, with Outcome tasks scoring far higher than Ranking and Process. Comparisons to related work are modest and fair, positioning PrefDiag-Bench as complementary rather than strictly superior. That said, because probes were built from observed failures, the diagnostic sample is conditioned on difficulty and does not rigorously establish that the same bottleneck pattern would hold on an independently sampled task set.
Code and data are publicly released, and open-source model configurations are specified with temperature settings and hardware details. However, reproduction is blocked for three of the top-performing systems because they are closed-source commercial APIs, and the synthetic data construction relies on specific proprietary LLM versions. The environment reuses KnowU-Bench infrastructure, which helps standardize setup but requires access to that prior system.
Unlike conventional GUI agents, personalized GUI agents require user preferences to guide sequential execution over changing GUI states. This raises two challenges: determining HOW each preference should alter execution strategy and WHEN it should take effect along the interaction trajectory. Existing benchmarks primarily evaluate overall task success, offering limited insight into these questions. We introduce PrefDiag-Bench with two components: (1) a categorization of preferences into five types, characterizing HOW they affect execution; and (2) a stage-wise diagnostic framework with 1,664 probes, tracing WHEN preferences should take effect and where failures occur. We evaluate eleven GUI agent systems, including end-to-end agents and agentic workflows, under two forms of user context: explicit user profiles and behavioral memory logs. Although existing agents achieve relatively high completion rates on non-personalized tasks, performance is substantially lower on preference-personalized tasks. Even the best-performing model, Seed-2.0-Pro, achieves only 47.17% on Process preferences under explicit user profiles. Agents struggle particularly with Ranking and Process preferences. Further analysis reveals two key capability bottlenecks: selecting relevant evidence from user history and recognizing when preferences should take effect during execution. Controlled interventions that explicitly provide relevant evidence improve end-to-end task success by up to 7.06%. PrefDiag-Bench offers a more systematic evaluation of GUI agents' personalization capabilities beyond overall task success. Code and data are available at https://github.com/Zhixin-L/PrefDiag-Bench.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.