Same Reward, Different Skills: When Multimodal RL Learns to Look
Multimodal reinforcement learning with verifiable rewards (RLVR) is widely assumed to improve visual reasoning when models are rewarded for correct answers. This paper exposes a critical gap: benchmark gains can be achieved without learning from images at all, and prolonged training can even erode visual grounding. The authors propose "visual resolvability"—designing tasks where visual evidence is strictly necessary for reward—and show that standard GRPO on carefully constructed counterfactual coordinate scenes drives genuine visual discovery, unlike conventional benchmark training.
The paper makes a timely and important contribution by disentangling benchmark performance from actual visual skill acquisition in multimodal RLVR. Through systematic visual-access controls—training on gray canvases versus real images—it demonstrates that a 3B model recovers roughly half to four fifths of real-image benchmark gains without ever seeing training images, while prolonged real-image training can cause "grounding corrosion." The proposed visual resolvability principle and the accompanying counterfactual task design provide a concrete framework for ensuring RL learns intended visual operations rather than shortcuts.
The experimental design is rigorous and the controls are convincing. The paper distinguishes three levels of visual dependency—cued readout, find-and-bind, and discovery—and shows that ordinary benchmark training mainly improves cued readout while blind conditions match real-image training on discovery. The gray-canvas control during training on the constructed corpus is devastating: at matched step 30, the blind-trained model recovers essentially none of the discovery gain, proving that the visual gain is attributable to actual image use during learning. The counterfactual scene construction and pair-accuracy metric $PairAcc(h)=\frac{1}{N_{\mathrm{pair}}}\sum_{j=1}^{N_{\mathrm{pair}}}C_{j,a}(h)\,C_{j,b}(h)$ elegantly prevent image-independent guessing.
The paper's broader conclusions about multimodal RL may be overstated by the specificity of the coordinate-scene testbed. While the discovery task is carefully validated, all demonstrations use synthetic coordinate plots with labeled points—a highly structured domain where relations like "smallest y-coordinate" are unambiguous and caption-susceptibility is easily measured. It remains unclear how "visual resolvability" would be operationalized for natural images where visual necessity is harder to verify and tasks may not decompose cleanly into identification-plus-readout. The authors acknowledge this scope limitation, but the leap from synthetic scenes to general design rules for multimodal RL is substantial. Additionally, the reliance on Qwen2.5-VL models limits the claim's generality to other architectures.
A second concern is that the intervention requires solving both visual necessity and learnability simultaneously, which may be expensive to engineer at scale. The paper shows that simply mixing one-third constructed data with ordinary data preserves most gains, but constructing the scenes themselves demands programmatic control that is unavailable for many real-world benchmarks.
The evidence strongly supports the narrow claim that benchmark gains conflate readout with discovery, and the paired counterfactual evaluation is methodologically superior to standard accuracy metrics. The comparison to related work is fair but somewhat brief in the main text; the paper correctly positions itself against methods that modify objectives, rewards, or data selection, arguing instead that task construction is the fundamental lever. However, the paper could engage more deeply with whether recent perception-aware reward methods would actually fail on their constructed tasks—the briefing table suggests they might succeed if rewards are image-dependent, but the core argument is orthogonal rather than competitive.
Reproducibility is strong. The authors provide anonymized code, scene programs, the counterfactual grounding suite, training recipes, and evaluation scripts that can rebuild most paper values without a GPU. They document training hyperparameters including GRPO with 5 rollouts, learning rate $10^{-6}$, KL coefficient $0.01$, and the exact reward function $r(o,y)=\frac{1}{2}\mathbf{1}\{\operatorname{correct}(o,y)\}+\frac{1}{2}\mathbf{1}\{\operatorname{format}(o)\}$. The only barriers to full reproduction are compute (training requires GPUs) and the dependence on the specific Qwen2.5-VL base model; the evaluation outputs themselves are released.
Reinforcement learning with verifiable rewards (RLVR) improves vision-language benchmark scores even without visual information during training. With images at test, blind-trained models recover roughly half of the real-image gain at 3B and nearly four fifths at 7B. Prolonged real-image training can erode grounding while benchmark gains persist. Both findings expose the same gap: an image in the prompt is not an image in the learning signal. Our design rule, visual resolvability, asks that visual evidence be necessary for a correct answer and that the task remain learnable. We test it on counterfactual coordinate scenes in which the question stays fixed and the target is never named, so a correct answer requires finding the target in the image. With standard GRPO and correctness-and-format rewards, a 7B model raises its accuracy at finding the target (discovery) from 0.425 to 0.875 on held-out scenes denser than any it trained on, and it improves on question types it never trained on. Two controls locate the source of the gain. Replacing test images with gray canvases drops discovery to zero; training on gray canvases instead, at matched step 30 and in each of four seeds, yields essentially none of the gain even when the model is then tested with real images. The learned skill carries over to grounding tasks built independently of the training corpus. A caption that answers the training question, added to the same images, reward and budget, cuts the gain by nearly two thirds. Changing what reward requires changes what RL learns.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.