Mitigating Video Hallucinations via Attention-Guided Representation Perturbation
Video large language models often hallucinate answers that are not supported by the input footage. Contrastive decoding can reduce this by comparing the original video against a weakened "negative" view, but existing negatives are either predefined or optimized for query reconstruction rather than the model's actual visual reliance during answer generation. ARP-CD introduces a training-free fix: it treats the attention mass on visual tokens at the answer-prediction position as a differentiable optimization target, then uses bounded gradient ascent to perturb the input visual hidden states and create a negative branch for contrastive decoding. The authors evaluate the idea across four Video LLMs and four benchmarks, supplementing headline results with ablations and a mechanistic analysis of first-token logit margins.
ARP-CD is a well-motivated and carefully evaluated training-free method. It achieves the best result in 12 of 16 model–benchmark settings and its ablations isolate the value of the attention-guided direction. However, the gains are sometimes modest, LLaVA-NeXT-Video actually degrades on VideoHallucer, and the iterative optimization adds a 3.8–6.3× latency overhead versus vanilla decoding. These issues limit its immediate practicality for black-box or latency-sensitive deployments.
The core mechanism is plausible and well grounded: the authors tie low visual attention to hallucination risk and use that signal to shape the negative branch. The experimental design goes beyond headline metrics with four carefully matched ablation controls and a mechanistic analysis of first-token logit margins. As the paper notes, ARP-CD "yields positive mean $D$ values exceeding those of all ablation controls, indicating a stronger average contrastive correction toward the correct answer," showing that the optimization direction matters beyond raw perturbation magnitude.
The improvements are uneven and occasionally marginal; for example, for LLaVA-NeXT-Video-7B the paper notes that "all evaluated interventions underperform vanilla decoding on VideoHallucer," yet offers no diagnostic for this failure. The method also carries a meaningful latency cost, running "faster than MACD in every setting but slower than the other baselines." More fundamentally, aggressively suppressing visual attention can remove relevant evidence along with spurious cues, but the paper does not quantify how often this happens or characterize failure cases.
The comparisons are generally fair. The critique of MACD is accurate: the MACD paper itself states that it uses "a detector (e.g., YOLOv11), and assign soft masks to each object and each frame" and maximizes query reconstruction loss, which ARP-CD's QueryRec ablation directly shows is less effective than an answer-position attention objective. Still, the authors report no statistical significance tests, and some gains (e.g., sub-percentage-point margins on MVBench) could be noise. The benchmarks also focus on short-answer and multiple-choice tasks, leaving open whether the approach benefits open-ended captioning or long-form generation.
Although the algorithm is described in detail, critical hyperparameters—the chosen decoder layer $\ell$, step size $\eta$, maximum steps $T$, and RMS budget $\epsilon$—are never disclosed. The paper states that "we backpropagate the attention objective from layer $\ell$ to the input perturbation $U$ ... Starting from $U_0 = 0$, we perform up to $T$ steps of RMS-normalized projected gradient ascent," but without numeric values. No code release is mentioned. These omissions make independent reproduction difficult, especially since contrastive decoding is sensitive to the correction strength $\alpha$ and plausibility threshold $\beta$.
Video large language models (Video LLMs) often generate responses unsupported by the input video. Contrastive decoding (CD) can mitigate these hallucinations, but its effectiveness depends on how the negative branch weakens visual evidence. Existing methods rely on predefined perturbations or masking guided by question reconstruction, offering limited control over the model's visual reliance during answer generation. We introduce Attention-guided Representation Perturbation for Contrastive Decoding (ARP-CD), a training-free framework for constructing contrastive negatives directly in representation space. ARP-CD uses the attention mass assigned to visual tokens at the answer-prediction position to guide bounded gradient updates of the language model's input visual hidden states. These updates reduce visual attention and produce a negative input, whose predictions are contrasted with those of the original branch throughout generation. Experiments across four Video LLMs and four benchmarks show that ARP-CD outperforms representative training-free baselines in most model--benchmark settings, reducing hallucinations while improving general video understanding. Further analyses show that attention-guided perturbation directions improve negative quality and that the resulting negatives provide effective contrastive references for correcting the original predictions.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.