Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning
Hob-VL introduces a diagnostic benchmark for visually grounded Boolean reasoning, asking vision-language models to evaluate ten-literal Boolean combinations over images and to identify objects satisfying Boolean descriptions. The core idea is to pair each expression with a mechanically verified De Morgan equivalent and to test both symbolic and natural-language presentations, isolating compositional reasoning from surface memorization. It matters because real-world visual queries—warehouse checks, agricultural monitoring, medical imaging—depend on reliably combining multiple visual facts under logical constraints.
Hob-VL is a carefully engineered diagnostic benchmark that exposes severe weaknesses in VLMs' visually grounded Boolean reasoning. Across eight disabled/minimized-thinking configurations, Boolean accuracy is essentially coin-flip level (48.52% to 50.57%), and even a thinking-enabled GLM configuration—while improving on symbolic reasoning—still fails to reach 65% Boolean accuracy and remains riddled with inconsistencies. The benchmark's executable annotations and paired equivalent expressions make it a valuable tool, but the results suggest current VLMs lack reliable compositional Boolean reasoning over visual inputs.
The benchmark construction is rigorous and transparent. Every base expression is paired with a mechanically verified De Morgan rewrite that preserves truth values across all $2^{10}$ assignments, and prompts are deterministic renderings of the same abstract syntax tree in either symbolic or natural language. The validation pipeline is extensive, replaying rewrites and recomputing logical annotations across all 6,000 expressions. The evaluation protocol is also unusually thorough: it reports not just item accuracy, but pairwise correctness, conflict rates, label frequencies, and output validity.
Several design and presentation choices limit the strength of the conclusions. First, the natural-language prompts are dramatically longer than their symbolic counterparts (median 1,911.5 versus 371 whitespace-delimited words), so format comparisons conflate representation with verbosity and potential length-related degradation. Second, the thinking-enabled GLM configuration uses a 4,096-token output cap while disabled-thinking configurations use 2,048 tokens; this uncontrolled confound makes it impossible to attribute gains solely to reasoning rather than to increased generation budget. Third, the photographic track rests on only 46 images from 45 scene families, raising questions about visual diversity and generalization, while the generated scenes use only three shapes and six colors. Finally, many invalid responses in the thinking-enabled condition are simply token-limit failures, particularly in natural language.
The evidence clearly supports the central claim that disabled-thinking VLMs perform near chance on Boolean composition, but the comparison between thinking-enabled and disabled GLM is weakened by the differing token caps and by the fact that thinking-enabled gains are uneven—strong for symbolic evaluation and identification, yet minimal for natural-language Boolean questions. The conflict-rate analysis is telling: even with thinking enabled, GLM answers both members of equivalent symbolic pairs correctly only 45.13% of the time, and natural-language conflict reaches 34.37% among valid pairs. Some models show extreme label bias (e.g., DeepSeek valid Yes rate of 1.78% on symbolic Boolean questions), confirming that aggregate accuracy can mask degenerate strategies. Related-work comparisons are fair and well-contextualized, distinguishing Hob-VL from CLEVR, GQA, and VQA-LOL by its exhaustive truth-table annotations and paired equivalent-expression design.
Reproducibility infrastructure is a notable strength: the authors provide hashes for prompts and images, append-only evaluation histories, and explicit commands to regenerate all tables without new API calls. The paper also honestly discloses that model API backends may shift and that temperature was not explicitly set, recording null rather than a deterministic value. However, the absence of fixed temperature and the use of provider-default sampling mean repeated runs may not match exactly. Additionally, the thinking-enabled GLM condition differs in both thinking mode and output-token limit from its disabled counterpart, which complicates direct comparison and independent reproduction of the thinking effect unless both caps are tested.
Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves uneven gains while retaining substantial errors and inconsistencies. Hob-VL exposes these failures through executable reference answers and matched evaluations.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.