AgentVQA: A Unified Benchmark for Agentic Visual Understanding
AgentVQA unifies 23 fragmented agentic datasets into 1,000 manually verified multiple-choice questions across five domains. By doing so, it provides a scalable offline complement to costly interactive simulator evaluations for vision-language models. The paper shows that even top models achieve only 58.5% accuracy and that failures stem primarily from grounded visual perception rather than reasoning mistakes.
AgentVQA is a well-motivated and carefully executed benchmark. The human-in-the-loop curation is rigorous: as noted in Section 3.1, "Only about 22% of the initial candidates survive the full curation pipeline," yielding a clean set of 1,000 questions with hard negatives. The strong rank correlation with SpatialWorld ($\rho=0.895$) in the matched domain supports its claim as an offline proxy, though the authors note it is "a reliable and substantially lower-cost offline alternative for ranking agentic models." However, the benchmark is small---only 200 questions per domain---which limits statistical power and sub-domain granularity.
The paper makes two especially strong contributions. First, the standardized MCQ format is validated as robust to option ordering and well above random chance (Appendix F confirms that performance is highly consistent across shuffles, with the table caption stating: "Average accuracy across 5 random permutations of option orders. The low standard deviation indicates high stability.", and results are above the $25\%$ baseline). Second, the cross-domain error analysis provides a clear, domain-specific diagnosis: failures are "dominated by grounded scene and state understanding," with the authors noting that "Among the analyzed incorrect Web Agents predictions, 78% were classified as grounding errors." This evidence directly contradicts the assumption that agents fail primarily at high-level planning.
The benchmark's chief limitation is its small size: with only 1,000 questions total and 200 per domain, the evaluation set is modest compared to general VQA benchmarks and offers limited coverage within each domain. The paper also risks overgeneralizing the claim that offline evaluation is a reliable substitute for online testing. While the correlation with SpatialWorld is strong ($\rho=0.895$) for the matched Spatial Planning domain with 12 models, the authors themselves note that "the lower correlations for other AgentVQA domains suggest that model rankings depend on the capabilities being assessed, highlighting the limitations of a single dataset as a proxy for overall agentic performance." Additionally, the evaluation protocol uses inconsistent generation settings: "Models are evaluated with a temperature of 0.8 and 0.2 for reasoning and non-reasoning models, respectively," which may confound cross-model comparisons. The error analysis is also restricted to a single model (Gemini 3.1 Pro), leaving it unclear whether other models fail for the same reasons.
The evidence strongly supports the central claim that general-purpose VQA benchmarks are poor predictors of agentic competence. Spearman correlations are only moderate for MMMU-Pro ($\rho=0.67$) and GPQA-Diamond ($\rho=0.59$) and weak for ChartQA-Pro ($\rho=0.44$). The paper's citation to Yao et al. regarding GPT-4o's low success on interactive tasks is accurate: the source reports that "even state-of-the-art LMs like gpt-4o achieve low task success rates (pass^1) using function calling (\sim61% on \tau-retail and \sim35% on \tau-airline)." The comparison to SpatialWorld is fair for the matched domain, though the small overlap of 12 models limits generalizability. The authors are appropriately cautious about weaker cross-domain agreement, which bolsters credibility.
Reproducibility is mixed. On the positive side, the authors release code, data, and evaluation prompts via GitHub and HuggingFace, and the five-stage curation pipeline is documented in detail. On the negative side, many evaluated models are proprietary or unreleased (e.g., GPT-5.6-sol, Gemini 3.7 Flash), making exact leaderboard reproduction impossible for most researchers. The use of different decoding temperatures introduces variability: the paper states that "Models are evaluated with a temperature of 0.8 and 0.2 for reasoning and non-reasoning models, respectively. For models without manual temperature like GPT series, we use the default." It also notes that "For models that do not natively support video input, we sample 32 frames at evenly spaced intervals," which may lose critical temporal information. Finally, the exact size of the initial candidate pool before curation is not stated in the main text, obscuring the denominator for the reported survival rate.
Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.