Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
When text and images conflict in vision-language models, prior work has characterized behavior as modality-specific biases such as "blind faith in text." This paper proposes a predictive alternative: a two-parameter Bayesian cue combination rule in which each modality contributes conditionally independent evidence weighted by a constant integration bias, plus a prior. The authors show that this simple linear model accurately predicts how VLM preferences shift across dozens of input degradation conditions and even transfers to entirely new corruption types without refitting, offering a unified account of why text sometimes dominates and vision sometimes wins.
The paper makes a strong empirical case that VLM modality preferences can be approximated by a linear weighted combination of unimodal evidence and a prior. Its fits are quantitatively impressive ($R^2$ of 0.80–0.86 for three of four models), and the out-of-distribution and transfer experiments are genuinely convincing. However, the model is descriptive rather than normative: the fitted weights capture empirical tendencies without explaining why a given architecture should have a specific bias. As the authors acknowledge, "The modality weights $\beta_T$ and $\beta_V$ are fitted empirically rather than derived from a normative account, because models differ in how they weight text and vision evidence in ways our framework does not yet predict."
The experimental design and generalization tests are the paper's strongest assets. By independently degrading text (random word replacement) and images (Gaussian blur) on controlled conflicting-VQA examples, the authors cleanly separate evidence strength from integration bias. Most compelling is the weight-transfer experiment across corruption types: weights fit on blur and word noise predict preferences on pixel occlusion and textual relevance degradation almost as well as refitted weights. As reported in Section 4.4, "transferred weights come within 0.021 of the refit value for every VLM." The interpretable decomposition is also valuable; for instance, Table 3 shows that Qwen3.5 weights visual evidence more heavily ($\beta_T/\beta_V = 0.56$–0.68) while still often following text because clean-input textual evidence is stronger by a factor of 1.4–3.8, neatly reconciling apparent "text bias" with a vision-favoring integration rule.
The central leap from predictive fit to computational-level theory is not fully justified. VLMs use deep cross-modal attention that heavily entangles text and image features, so the conditional independence assumed in Assumption 1 is unlikely to hold mechanistically; the success of the additive model shows only that a first-order approximation suffices, not that modalities are integrated as independent cues. The approximation also breaks down noticeably for Gemma3 ($R^2$ 0.69 vs. 0.80–0.86), which the authors note: "Gemma3's lower fit shows that the approximation is less complete for some models." A second concern is the behavioral measurement. Preferences are derived from length-normalized log-likelihoods of pre-specified candidate answers rather than open-ended generation. Appendix B.5 validates this with multiple-choice reasoning, but agreement is only 75.3–82.0% overall, and the models sometimes select answers based on position bias rather than content. Finally, the study is limited to relatively small open-weight models (3–4B parameters) and three datasets, one of which required synthetic text generation.
The evidence supports the core cue-combination hypothesis well. The comparison against alternative integration rules in Table 1 is fair and illuminating: the "best single probe" baseline—selecting whichever unimodal modality is more confident—trails the linear model by 0.22–0.35 in $R^2$, confirming that VLMs blend evidence rather than using winner-take-all arbitration. As stated in Section 4.3, "The best single probe trails by 0.22--0.35 in $R^2$, which supports the argument that VLMs integrate evidence rather than deferring to the more confident modality." The unconstrained linear comparison is also informative: for three of four VLMs the prior coefficient stays near one, but for Qwen3.5 it drops to 0.53–0.66, indicating a deviation from the strict Bayesian parameterization that the authors candidly report. Comparisons to related work are adequate though brief; the paper positions itself against prior modality-bias heuristics and layer-wise oscillation findings but does not quantitatively pit its model against those alternatives.
The study relies on open-weight checkpoints and provides extensive implementation details: model names in Appendix F.2, dataset construction in Appendix A, fitting procedures and hyperparameters in Appendix C.2, and prompt templates in Appendix B. However, there is no explicit mention of code or data release in the provided text. Exact replication of the RealworldQA conflicting passages and textual relevance degradations depends on API access to Gemini 3 Flash with specific generation settings (e.g., "temperature 0.7, an output limit of 512 tokens, and a thinking budget of 1,024 tokens"), and the output is not guaranteed to be deterministic across API versions. Pixel normalization and degradation hyperparameters (blur radii, noise levels, occlusion fractions) are fully specified, which lowers barriers to approximate reproduction. The main remaining blocker is the lack of a confirmed public code repository or frozen dataset artifact.
Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.