Video-Index: A Curated Meta-Benchmark for Video Understanding
Video benchmarks are supposed to certify that models can understand temporal sequences, yet most can be gamed without watching the video. This paper audits 115 multiple-choice video benchmarks with a five-level attack pyramid that grants attackers access to answer options, question text, other items, single frames, or shuffled videos. After finding severe shortcut vulnerabilities—35 benchmarks break without any visual input and near-duplicates plague 63 of them—the authors screen half a million items down to 840 hardest survivors and compose them into Video-Index, a reproducible meta-benchmark backed by agent-assisted selection and lockfiles.
The scale of the audit is impressive—115 benchmarks and over 500,000 items are tested under a unified protocol—and the attack pyramid gives a crisp taxonomy for certifying what a benchmark actually measures. Still, the final Video-Index is extremely small at only 210 items per capability group, and its construction depends on specific proprietary and open-weight models that may not generalize to future attackers. The framework is valuable, but the resulting benchmark should be treated as a high-difficulty probe rather than a comprehensive evaluation.
The attack pyramid and exploitability metric $\varepsilon_{\mathcal{A}}(B)=\max_{a\in\mathcal{A}}\mathrm{acc}_B(a)-c$ provide a clean, nesting framework for evaluating benchmarks. The empirical sweep is striking, and the agent-assisted pipeline with deterministic selectors, red-team gates, and versioned lockfiles represents best practices for reproducible benchmark construction.
The screening protocol assumes that Qwen3-VL-8B and Claude Opus 5 approximate the space of all possible shortcut attackers; if stronger shortcuts exist, the 840 surviving items may still be exploitable. The tolerance $\delta=5\%$ is applied uniformly even though chance accuracy varies wildly across benchmarks with different option counts, and the 300-item sample size means margin estimates are noisy for some sources. More critically, winnowing 505,518 items down to 840 selects for a narrow, adversarially filtered subset, and it is unclear whether these 'hardest' items cover video understanding diversity or simply form a brittle, model-specific challenge set. Finally, the claim that newer benchmarks break earlier could partly reflect that they are attacked by stronger contemporary models rather than purely worse design.
The comparison to prior work is fair: the authors note that prior audits probe one benchmark with one shortcut and discard it, whereas their pyramid tests every benchmark at every level. The near-duplicate analysis using the Vendi score is methodologically sound, grounded in Friedman and Dieng's definition of diversity via eigenvalue entropy. However, the enormous gap between Claude Opus 5 (56.8%) and the best open-source model (19.0%) on Video-Index suggests the meta-benchmark may still reward language-heavy reasoning that proprietary LLMs excel at, rather than isolating pure video understanding.
The paper laudably commits to lockfiles and deterministic selection, yet independent reproduction faces major hurdles. The audit requires access to Claude Opus 5 and Gemini 3.5 Flash, proprietary systems whose weights and exact prompts are unavailable, and the reference accuracies that define breaking levels depend on their 32-frame or 128-frame behavior. Full experimental details, agent prompts, and the exact red-team gate code are deferred to an appendix not provided here, leaving critical hyperparameters unspecified.
A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.