Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning
FrameEvidence is a diagnostic benchmark for multi-image spatial reasoning that tests whether vision-language models correctly answer questions when the reference frame or the set of visual evidence is manipulated. It introduces 960 Chinese multiple-choice questions arranged in 320 matched triplets, where FQ1 varies the spatial reference frame while holding images fixed and FQ2 deletes or adds images to test evidence necessity and invariance. The core finding is a sharp divergence between single-item accuracy, which reaches 50\% for GPT-5.6, and joint correctness across matched variants, which remains below 11\% for $\mathrm{FQ1\text{-}GEM}$. This design exposes how standard accuracy obscures brittle reasoning and answer instability.
The paper presents a carefully designed diagnostic suite, but its empirical story is one of near-universal failure on joint metrics. GPT-5.6 achieves 50.00\% item accuracy under three-shot prompting yet only 10.42\% $\mathrm{FQ1\text{-}GEM}$, and even the best model solves fewer than a quarter of necessity-transition groups. While the benchmark itself is methodologically sound, the experiments reveal that current VLMs largely fail at systematic frame grounding and evidence-aware reasoning. The authors are commendably transparent about limitations—scene overlap across splits, a single human reference, and confounding camera-heading mentions in non-camera prompts—but these concessions weaken the generalizability of several conclusions.
The matched-intervention design is the paper's strongest asset. By requiring correctness across three reference frames or across evidence-deletion and addition pairs, the authors make it impossible to game the benchmark with static answer biases. The analysis of execution instability—showing that a repeated GPT run changes 46.53\% of answers despite ostensibly identical inputs—is a valuable cautionary contribution. Likewise, the fixed-marginal diagnostic cleanly demonstrates that high $\mathrm{DIS}$ scores can reflect answer persistence rather than genuine evidence invariance.
A first concern is scale and construct validity: with only 96 test groups (288 questions), the evaluation is a narrow slice of spatial reasoning, and camera-heading cues leak into 30 of 48 non-camera FQ2 prompts, confounding the frame-manipulation claim. A second concern is severe methodological asymmetry: GPT-5.6 and Claude Fable 5.1 are evaluated with unset temperature, 2,048 output tokens, and undisclosed model versions, whereas open-weight models use temperature 0 and 32 tokens, rendering cross-model comparisons unreliable. Third, adaptation via LoRA consistently improves item accuracy while drastically degrading $\mathrm{DIS}$—Qwen LoRA drops from 43.75\% to 12.50\%—suggesting that fine-tuning induces over-cautious insufficiency responses rather than robust evidence reasoning. Finally, the human baseline reports only a single participant's aggregate summary without item-level responses or documented sample size, making it an unverifiable reference point.
The evidence broadly supports the central claim that item accuracy masks fragility, but the comparisons across models and methods are uneven. The fixed-marginal diagnostic shows that most models underperform independent random rearrangement on $\mathrm{NTS}$ while overshooting it on $\mathrm{DIS}$, confirming that successes are misaligned across variants. However, because gateway models are tested with different decoding parameters and undisclosed weights, the leaderboard-style table conflates architectural differences with experimental conditions. The paper does not compare against prior frame-specific benchmarks like COMFORT or FoREST on shared instances, so readers cannot gauge whether FrameEvidence is harder or simply different.
Reproduction would be difficult. The authors release data, an offline scorer, and LoRA checkpoints, but the API-served models use black-box aliases whose weights and reasoning traces are unavailable. Crucially, GPT-5.6 uses an unset temperature and 2,048 output tokens, while all other models use temperature 0 and 32 tokens, so the reported numbers are tied to this specific API configuration. Scene and template overlap between splits means that test performance may reflect memorization of layouts rather than general spatial reasoning, and the paper notes that exact-image clusters are shared across splits. Full reproduction therefore requires not only the released data but also the exact API state at the time of querying, which is not version-locked.
A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctness persists when either changes. We introduce Frame Evidence, a benchmark of 320 matched triplets and 960 Chinese multiple-choice questions from captured and rendered scenes. One family compares three grounded reference frames over fixed images. The other pairs image deletion annotated as insufficient with answer-preserving addition. Joint metrics require correct answers across all three frames and paired evidence variants. Across 20 configurations on 288 held-out questions, GPT three-shot reaches 50.00\% item accuracy, but the best three-frame group accuracy is 5/48 (10.42\%), reached by GPT two-shot. Few-shot prompting and task-specific adaptation improve item accuracy without consistently improving joint correctness. Input controls show where images support matched success, while repeated executions reveal answer instability beneath similar scores.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.