Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
The paper tackles a critical problem in automated image-editing evaluation: multimodal large language models (MLLMs) acting as judges may be swayed by irrelevant presentation cues rather than the edit itself. Because visual interventions can alter the very quality being assessed, the authors introduce EditJudgeBias, a counterfactual benchmark of 1,196 real samples where 13 cues are injected across four evaluation stages and rigorously verified to preserve underlying quality. They audit five judges along three dimensions—invariance to cues, agreement with humans, and pairwise stability—and show that every judge shifts beyond its own noise floor, with fabricated social proof inflating scores and display-order swaps reversing up to 60.9% of verdicts.
The paper presents a compelling and timely critique of MLLM-based image-editing evaluation. Its central argument—that agreement with humans is insufficient without invariance to quality-preserving perturbations—is well-motivated and substantiated by extensive experiments. The EditJudgeBias benchmark is carefully constructed, with quality preservation verified through calibrated validators, positive controls, and human inspection. The empirical findings are striking: even a sticker covering $\leq$4% of the frame causes larger rating shifts than whole-image photometric manipulations, and display-order swaps reverse preferences at rates far exceeding re-query noise. The observation that different judges top different robustness rankings effectively undermines any single-score summary of judge reliability. Limitations are transparently disclosed, including the finite cue set, evolving API versions, and the fact that human preservation annotators were aware of the study hypothesis.
The benchmark design is the strongest contribution. By enforcing verified quality preservation through calibrated MLLM validators, positive controls, and human checks, the authors solve the attribution problem that plagues prior perturbation-based audits. The three-dimensional evaluation framework—invariance, agreement, and stability—provides a nuanced diagnostic that reveals distinct failure modes. The quantitative results are robust and large in magnitude: fabricated bandwagon opinions inflate ratings across all judges, and the position-bias reversal rate of 60.9% for qwen3.5 dwarfs its 10.5% retest ceiling. The finding that a small sticker outranks whole-image photometric cues underscores how locally disruptive visual noise can hijack model attention.
Several concerns temper the conclusions. First, the quality-preservation claim for the inset cue is weak: two validators flag it more often than sham, and a post-hoc human criterion flags 11/30 examples, yet the authors retain it in tables and only partially exclude it from aggregate counts. Second, human preservation annotators were aware of the study hypothesis, raising the risk of confirmation bias in labeling cues as harmless. Third, the repertoire of 13 cues, while broad, cannot exhaust the space of possible presentation biases, and the reliance on specific API versions without exposed checkpoint identifiers limits temporal generalizability. Mitigation is also underdeveloped: the authors explicitly defer systematic mitigation strategies to future work, leaving practitioners without concrete remedies.
The evidence broadly supports the claims. The comparison to related work is fair and thorough: the authors correctly note that FRAME and MM-JudgeBias focus on generated images or general VQA rather than the editing-specific challenge of assessing instruction adherence given a source image. The use of judge-specific sham and retest noise floors—expressed as $F_{j,d}^{(b)}$—instead of a universal zero baseline is methodologically sound and strengthens causal interpretation. However, some effect attributions rely on the summative $J(x) \in [3,30]$ score rather than individual rubric dimensions, and the borderline status of the inset cue suggests that not all interventions are unequivocally quality-preserving. The replication arm with qwen3-vl-32b-instruct largely confirms the primary panel.
Reproducibility is mixed. The authors provide an open-source repository with code, prompts, and configurations, but explicitly exclude benchmark images, human annotations, and judge responses. Because several models are queried through external APIs without exposed checkpoint identifiers, exact response replication is impossible. The experimental design partially mitigates this by estimating judge-specific retest noise and using paired inference with confidence intervals, but an independent lab would need to rebuild the image corpus and collect new API outputs to approximate the study. Hyperparameters are documented ($T=0$ where configurable), though gpt-5.5 uses a provider-fixed setting that cannot be controlled.
Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model training. However, systematically auditing whether these judges are influenced by cues irrelevant to editing quality is challenging because visual interventions may themselves alter the quality being evaluated. A judgment shift can therefore be attributed to bias only when the intervention is verified to preserve the underlying editing quality. To address this challenge, we introduce EditJudgeBias, a counterfactual benchmark with verified quality preservation, comprising 1,196 real editing samples and 13 cues injected across four evaluation sites. We verify quality preservation for the requested edit using calibrated multimodal validators, controls, and human inspection. We then audit five MLLM judges along three complementary dimensions: invariance to quality-preserving cues, agreement with human judgments, and stability of pairwise preferences. Importantly, observed shifts are evaluated against each judge's own zero-dose and re-query noise floors rather than against zero. Experiments show that quality-preserving cues move every judge beyond its own noise. Fabricated majority opinions increase ratings, irrelevant visual elements cause larger shifts than whole-image manipulations, and swapping candidate order reverses up to 60.9% of pairwise decisions. Edit-region cues also tend to reduce human agreement. The three measures characterize judges differently, showing that robustness cannot be captured by a single metric.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.