Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts
Vision-language models lag at visual in-context learning because they exploit textual shortcuts rather than learning from demonstration images. This work argues the root cause is insufficient training data that forces pure visual induction, and proposes MIME—a synthetic, task-agnostic dataset where the rule is never stated in text—and DTAD, a label-dropout mechanism that shifts attention to visual tokens. The result is an 8.1–18.0 point jump in vICL accuracy across multiple architectures, with an 8B model matching frontier proprietary systems, suggesting vICL-specific data should become a standard ingredient in VLM training.
The paper presents a clear diagnosis and an elegant, data-centric cure: VLMs fail at visual in-context learning because they are rarely forced to induce rules from images alone. MIME and DTAD are strongly supported by controlled ablations and attention analysis, and the cross-model gains are broad enough to recommend the approach as a training standard, though the headline frontier comparisons rely on proprietary APIs. The preliminary study is blunt: "Qwen3-VL-8B reaches near-perfect accuracy (99.8%) when the demonstrations are given as text... Its accuracy falls from 78.3% to 44.8% when the task-specific instruction is replaced with a generic one that gives no hint about the task" (Section 2).
The ablations are exceptionally tight. The authors show that trained models genuinely depend on the demonstrations: shuffling answers or images cuts accuracy by 33–52 points, and replacing demonstrations with a foreign rule produces the largest drop for DTAD, indicating the model follows the induced rule rather than falling back on priors. The crucial comparison between stating and hiding the rule nails the causal role of the data design: "Stating the rule reduces the gain to +12.1 / +12.3 points over the untrained model, relative to +17.0 / +22.0 when the rule is hidden" (Section 5.2 / Table 6).
The headline gains are largest under the generic instruction protocol, which deliberately mirrors MIME’s training distribution by removing all textual hints; this design choice makes the task easier for MIME-trained models than for baselines that expect task descriptions. The authors acknowledge that "We adopt the generic instruction protocol as our primary protocol and use it henceforth throughout the paper" (Section 4.1), while the official protocol yields smaller lifts (Appendix Table 10). DTAD is also not uniformly better than naïve fine-tuning: on LLaVA-OV it lags by 1.2 average points, and the setup is small-scale: "Our fine-tuning experiments are similarly limited in scale. We train 4B–8B models with LoRA adapters, and leave full-parameter training with larger backbones unexplored" (Limitations).
The evidence that ICL structure matters more than multi-image exposure is compelling: when retraining LLaVA-1.5, MIME doubles vICL accuracy while a same-size Mantis set barely helps and regresses POPE by 30 points. However, the paper’s dismissal of prior attention-rebalancing methods is sharper than the literature supports. The authors report that "Existing methods barely change the base model’s performance. On Qwen3-VL-8B, DARA and MGI stay within ±0.2 points of the untrained model" (Section 4.2), yet the original TrueMICL work claims DARA yields "up to a 10% improvement on downstream tasks" (Chen et al., Sec. 1). This contradiction may reflect different backbones or benchmarks, but the paper does not analyze it, making the comparative claims less certain.
Training hyperparameters are reported in full—"We fine-tune each model with LoRA ($r=8$) for one epoch on MIME, with a learning rate of $2 \times 10^{-4}$ and an effective batch size of 16" (Section 4.1)—and the five MIME sources are named, but the exact 23,578 generated instances and the generation code are not publicly released. Reproducing the frontier-model comparisons requires API access to Claude Opus, and the proprietary TrueMICL and VL-ICL-Bench evaluation kits are standard but not trivial to set up identically. The LLaVA-1.5 full retraining relies on the public 665K instruction mix, which is a plus.
Vision-language models (VLMs) struggle with in context learning compared to their textual counterparts. Our preliminary study shows that VLMs rely on textual shortcuts and underutilize the visual demonstrations. We therefore hypothesize that VLMs are insufficiently trained to perform visual in-context learning (vICL). To this end, we construct MIME, a task-agnostic vICL training set without textual shortcuts. We ensure the instruction does not state the rule linking the demonstration images to their answers, so the model must infer it from the visual demonstrations. Fine-tuning on MIME improves vICL accuracy by 8.1 to 18.0 points across three VLMs, preserves single-image performance, and improves multi-image and text only in-context learning. In contrast, stating the rule in the training data sharply reduces this gain. We further propose DTAD (Demonstration Text Attention Dropout), which randomly withholds demonstration labels during training so that the VLM must infer the rule from the demonstration images. While achieving comparable vICL accuracy to fine-tuning on MIME, DTAD yields more consistent attention to the demonstration images. On TrueMICL, Qwen3-VL-8B trained on MIME matches frontier models such as Claude Opus 5, and surpasses them with DTAD. Finally, adding MIME to the instruction-tuning mix of LLaVA 1.5 13B more than doubles its vICL accuracy without hurting single-image abilities, whereas a same-size multi-image set improves it only slightly. These results suggest that vICL data should be a standard component of VLM training.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.