GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution

The ablation evidence is compelling: removing visual evidence causes the largest quality drop, reinforcing the central premise that text traces are insufficient for GUI diagnosis. The repeated-execution design is well justified, as 11.8–20.4% of tasks flip between zero and positive scores across three runs, so single trajectories are noisy accounts of agent capability. Checking predicted behavioral effects against screenshots rather than raw scores alone is a methodologically sound innovation.

First, the optimization loop is enormously expensive and optimizer-dependent. All four roles use Claude Sonnet 5, and the protocol consumes up to $B=10$ rounds with $P=5$ attempts per round and $K=3$ rollouts per task, meaning thousands of GUI executions. The impressive gains may partly reflect the strength of this external model rather than a generally scalable recipe. Second, the behavioral soft gate is relatively permissive: a candidate passes if each prediction has at least one determinate verdict and more supporting than contradicting verdicts, which is a weak majority rule rather than stringent behavioral proof. Third, gains are uneven across backbones: OpenCUA improves by only 1.49–4.68 percentage points, versus 6.05–12.33 for Qwen and frontier models, suggesting the method adds limited value when the native runtime is already tightly coupled to its backbone. Fourth, Appendix D notes that residual errors involve successfully executed actions targeting the wrong object, route, or evaluator-relevant state, which harness-only changes cannot easily detect because these actions may appear valid in screenshots and execution logs despite failing the task.

The direct comparison to Self-Harness and Meta-Harness is fair and well controlled: the authors use the same initial Agent S3 harness, the same Claude Sonnet 5 optimizer, and the same ten-round budget, yet GUI-HARVEST gains 10.16 Test points versus 2.00 and 4.38 respectively. This supports the claim that GUI-specific visual diagnosis outperforms generic text-trace optimization, which the cited papers indeed evaluate on classification, math, and terminal coding rather than GUI tasks. The LFF comparison is properly caveated: the authors acknowledge they could only evaluate released patches, not the full search procedure. The broader benchmark comparisons in Figure 1 and Tables 5–7, however, are not controlled experiments; they juxtapose different frameworks, step budgets, and aggregation protocols, so the highest score in every comparison statement reflects reported leaderboard positions rather than paired causal evidence.

The authors provide code, but independent reproduction would face major barriers. The core algorithm depends on extensive, unstated prompts for four Claude Sonnet 5 roles, a capability manifest defining editable harness files, and the GAFT toolkit for aligning screenshots with actions. Crucial implementation details are relegated to appendices, and the exact system prompts and ledger schemas visible to the optimizer are not reproduced in the main text. Without these, a third party cannot replicate the optimizer's behavior even with the released codebase.