Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models
Cast shadows are a clean, observable signature of a hidden physical variable: the direction of light. The authors introduce ShadowBench, a benchmark built from real photographs that asks VLMs to read, infer, and validate cast-shadow direction under carefully matched composites. They find a sharp dissociation—VLMs read visible shadows well yet infer and validate them at chance—and show that routing a VLM through an image generation model supplies the missing prior, motivating reconstruction-based objectives for illumination-aware VLMs.
This is a sharply designed diagnostic paper that isolates a single physical variable—cast-shadow direction—and exposes a stark gap between VLMs’ ability to read a visible cue and their inability to use it for inference or validation. The IGM-borrowing experiment is convincing, though the resulting pipeline is only a test-time workaround and the benchmark’s scope is intentionally narrow. A few copy-editing errors (e.g., “1919 open VLMs” instead of 19) are present but do not undermine the scientific contribution.
The benchmark design is elegant and rigorous. Matched alpha-composited states—shadow-correct, shadow-removed, and wrong-direction—ensure that direction is the only changed variable, and the bias-free 2AFC cleanly separates discrimination from response bias. The read-to-infer dissociation is dramatic: Gemma-4-26B reads three-way direction at $86.6\%$ but falls to $43.8\%$ when the shadow is removed, defaulting to a single direction on $84\%$ of scenes. The opacity-sweep control confirms models actually perceive the shadow as a graded quantity rather than missing it entirely.
The borrowing pipeline is an existence proof, not a practical fix: it requires calling a separate IGM at test time and lifts inference only to $58.9\%$ on average, still far below the human $85.7\%$ ceiling.
The benchmark’s scope is deliberately narrow—ground-cast shadows from a single dataset—so generalization to shading, inter-reflections, or multiple light sources is untested. The repeated typo “1919 open VLMs” instead of 19 suggests lapses in proofreading, though the underlying numbers in the tables are clear. Finally, the claim that IGMs already hold the missing prior rests on a small set of tested editors; while FLUX.1-Kontext-dev is open, the proprietary alternatives are black boxes.
The evidence supports the central claim well. The three tasks form a sensible ladder of difficulty, and the presence-versus-direction 2AFC control cleanly shows that models fail on direction specifically, not on detecting shadow absence. The CoT ablation is decisive: neither zero-shot nor context-aware scaffolding improves comprehension, which undercuts any explanation based on insufficient reasoning. Comparisons to related work are fair—the authors correctly note that prior benchmarks probe spatial reasoning, occlusion, and plausibility but do not isolate shadow direction—and the human baselines establish a strong ceiling.
The benchmark construction is described in enough detail to recreate from the OBER dataset, but the authors do not state whether code, prompts, or curated composites will be released. The pipeline relies on proprietary APIs for some VLMs and the open FLUX.1-Kontext-dev editor, yet generation hyperparameters are not reported. With only 458 curated scenes, experiments are cheap to rerun, but independent reproduction would be blocked by the need to rebuild the alpha-matting pipeline and the absence of released data.
Shadows are observable consequences of latent physical causes, arising from the interaction between illumination, object geometry, and scene structure. A model that understands shadows should be able not only to read a shadow's direction, but also to infer where a shadow should fall when it is absent, and to judge whether a shadow is physically consistent with the scene. We introduce ShadowBench, built from real photographs, which poses these three questions to vision-language models (VLMs). From each scene, we build a correct-shadow, a shadow-removed, and a human-annotated wrong-direction state. Evaluating 19 open VLMs (2B to 78B) and frontier proprietary models, we find that VLMs read a visible shadow's direction well above chance and improve with scale, yet infer the absent direction only weakly and fall to chance at judging physical consistency even under a bias-free two-alternative test that removes the yes-bias of single-image naturalness questions. Only Gemini-3.1-pro exceeds chance on validity, and even then only modestly and far below humans. Chain-of-Thought (CoT) does not help. The missing ingredient is not reasoning but knowledge that image generation models (IGMs) already hold. Routing a VLM through an IGM lifts its inference well above its direct attempt and helps it flag the wrong-direction shadows it otherwise accepts. Cast-shadow direction is a single facet of illumination understanding, but the sharp gap it reveals between VLMs and IGMs suggests image-generation or reconstruction objectives as a route toward illumination-aware VLMs.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.