Visual Grounding First, Multimodal In-context Learning Follows
Multimodal in-context learning remains weak because MLLMs often hallucinate and fail to ground textual outputs in the visual evidence provided in demonstrations.
This work argues that the root bottleneck is poor visual grounding rather than a lack of ICL-specific training data.
The authors propose ProCoRe, which uses proposition-level contrastive reinforcement learning on non-ICL image–caption pairs to sharpen fine-grained visual discrimination.
The result is stronger multimodal ICL without catastrophic forgetting, plus a new personalized-captioning benchmark that forces models to rely on demonstrations instead of lexical priors.
ProCoRe is a compelling and well-supported contribution that reframes multimodal ICL as a visual grounding problem. The proposition-level reward design is elegant, and the empirical gains across classification, captioning, and multiple base models are substantial. The paper makes a strong case that improving grounding generalizes better than fitting ICL-formatted benchmarks.
The central thesis that weak visual grounding is the fundamental bottleneck is supported by targeted attention analysis (TAS), qualitative attention maps, and the failure of generic hallucination-mitigation methods to match ProCoRe. The proposition-level decomposition is methodologically sound: by isolating shared from unique propositions and applying contrastive rewards only to image-specific content, the model avoids shortcut captions that omit shared context. The ablations also cleanly disentangle the reward mechanism from the contrastive data, showing that supervised fine-tuning on the same pairs yields only marginal gains.
The proposition-decomposition step relies on GPT-4o-mini during training; although the authors show smaller open models can substitute it, the main experiments anchor reproducibility to a closed API, and the exact API behavior may drift.
The causal claim that grounding is the root cause of better ICL is largely correlational: higher TAS and lower OpenCHAIR align with gains, but the ablations do not fully exclude confounding text-generation improvements. The PICL benchmark is also hampered by ROUGE-L inflation from generic descriptions and by a GPT-4o-mini identifier judge, despite high human agreement. In addition, several baselines are architecturally tied to specific shot counts, so their poor cross-benchmark transfer is expected and makes the generalization gap somewhat unsurprising.
The evidence broadly supports the claims: ProCoRe consistently tops the leaderboard on MS COCO, PICL, TrueMICL, and ISEKAI across 1- to 8-shot settings, and it does so without using ICL-formatted training data. Comparisons are reasonably fair—1,000 training samples for all trainable baselines—but the ICL-dependent methods are at an inherent disadvantage on out-of-distribution benchmarks because they are explicitly optimized for the ICL format of the training set, whereas ProCoRe optimizes a more general visual skill. The cross-model and cross-scale results (Qwen, LLaVA, InternVL at 4B–14B) strengthen external validity, and the reduction in forgetting on standard VQA tasks is a genuine differentiator.
Reproduction is partially blocked by the lack of a public code release; the paper does not mention or link to one. Training details are otherwise explicit: batch size 16, LoRA rank $r=128$ on all LLM linear layers, learning rates ($2 \times 10^{-5}$ for Qwen/LLaVA, $2 \times 10^{-6}$ for InternVL), reward coefficients $\lambda_I=2$, $\lambda_T=1$, $\lambda_s=4$, and GRPO optimization. The datasets (InpaintCOCO, EQBEN) are public and the decomposition prompt is provided, but anyone replicating the work must recapitulate the GPT-4o-mini API calls or swap in a local LLM, which the authors show is feasible but not default. Main tables report single-run results justified by low ICL variance, so training stochasticity is not characterized.
LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks. For this reason, we identify 'visual grounding' as the key bottleneck in multimodal ICL; MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data for improving multimodal ICL. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones. Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.