PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
Existing computer-use benchmarks conflate planning, reasoning, and execution, making it hard to pinpoint exactly where agents fail at precise visual control. PointerBench introduces a diagnostic suite of 1,500 static screenshot–instruction pairs that isolates fine-grained GUI grounding into three tracks: spreadsheet geometry, character-level text targets, and professional-application controls. The work matters because it reveals that even top frontier models drop sharply on thin structural elements and text carets, while specialist grounding models lag further behind, exposing substantial headroom before agents can reliably manipulate dense professional interfaces.
PointerBench is a well-scoped benchmark contribution that successfully isolates fine-grained grounding primitives and evaluates them across a broad model zoo. The per-suite breakdowns are informative and the dataset construction is deterministic. However, as the authors acknowledge, the static $1024 \times 768$ setup strips away temporal context, resolution variability, and real environment dynamics, so the results are diagnostic rather than predictive of end-to-end agent success. The paper also offers no human baseline, leaving it unclear whether a 69.6% success rate on text grounding is near ceiling or far from it.
The benchmark’s diagnostic granularity is its main strength. By decomposing grounding into target families—cells, edges, corners, characters, punctuation, carets, icons, and interface controls—the authors surface exactly where frontier and specialist models fail. The evaluation is unusually comprehensive, spanning 37 frontier systems and 39 specialist configurations with native prompts. The deterministic, programmatic generation of Sheets and Text and the released metadata with executable scorers support reproducibility.
First, the fixed resolution and absolute coordinate frame impede ecological validity. Several models (Gemini 3.7 Flash, Qwen3.8 Max) default to a normalized $0$–$1000$ frame and receive misleadingly low official scores; although the authors diagnose this, the leaderboard presentation still ranks them at the bottom. Second, the benchmark is purely static: it does not test robustness to window resizing, multi-step context, or interactive state changes. Third, there is no human performance baseline or inter-annotator reliability report, so the difficulty of each target type is only relativized across models, not against a gold standard. Finally, despite evaluating 39 specialist configurations, the paper offers no insight into why smaller specialists underperform frontier models on text—whether it is architecture, training data, or output representation.
The quantitative evidence supports the central claim that text grounding is the dominant bottleneck: frontier models score 46.6%–69.6% on Text versus 78.8%–98.8% on Sheets, and no specialist exceeds 34.0% on Text. The comparisons to related work (Mind2Web, WebArena, OSWorld, ScreenSpot) are fair and correctly position PointerBench as a complement rather than a replacement. However, cross-model comparisons are confounded by divergent coordinate conventions and prompt formats; the authors mitigate this by using native specialist prompts, but this means rankings mix model capability with prompt/protocol alignment. The per-suite rank reversals (e.g., UI-Venus-2-9B strong on Pro, GTA1-32B strong on Text) are well-documented and strengthen the paper’s argument that aggregate scores obscure specific weaknesses.
The authors pledge to release benchmark metadata, scorers, and evaluation artifacts, including exact endpoint identifiers and per-example outputs, which supports independent auditing. The deterministic generation of Sheets and Text also helps reproducibility. Nevertheless, full replication of frontier model scores depends on proprietary API endpoints via OpenRouter that may shift over time, and the paper does not report temperature, top-$p$, or exact decoding settings beyond token caps and reasoning budgets. Specialist runs require locating 35 distinct checkpoints and replicating native prompts, some with custom coordinate normalization; while Table 15 lists formats, the exact prompt strings and wrappers (e.g., MVP view augmentation) are not provided in the main text. These gaps could block a third party from exactly reproducing the 39-run specialist leaderboard.
Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.