SIEVE: Selective attention-value Suppression for Vision-Language Models Unlearning
SIEVE tackles selective unlearning in vision-language models (VLMs), where a model must forget specific sensitive image-question-answer associations about an individual while retaining permitted facts about the same person. The core idea is to suppress attention-value representations for forget examples toward zero and anchor retain-example values to a frozen reference model, coupled with sequence-level negative cross-entropy. This matters because VLMs can recover biographical information from faces, creating privacy risks, and prior methods struggle to disentangle closely related forgotten and retained knowledge without degrading utility.
SIEVE addresses a genuinely hard problem—selective multimodal unlearning at the association level—and presents a well-motivated method with promising empirical results on forget-set metrics. However, its strongest claims are partially undercut by notable utility drops in some settings and a limited experimental scope that omits several recent VLM-specific competitors discussed in the related work. On the whole, it is a reasonable but not yet conclusive contribution to private multimodal model updating.
The selective unlearning formulation is precise and well-motivated: the paper correctly identifies that forget and retain examples can share the same image and identity while differing only in the queried information. The attention-value intervention offers a novel, mechanistically grounded target, and the ablation study provides credible evidence that value suppression and reference-based matching work synergistically to preserve retained utility.
The empirical support is narrower than the claims suggest. All experiments rely on a single benchmark—MLLMU-Bench—where sensitive data is synthetic, and the evaluation omits several recent VLM unlearning methods such as VL-Eraser and PAVA surveyed in the related work. Retained utility degrades markedly in some settings; for example, LLaVA-1.5-7B Textual-QA Retain generation drops from 0.625 to 0.418, a gap the text does not adequately explain.
The theoretical analysis in Appendix A offers norm bounds under strong Lipschitz assumptions but provides no guarantee that the designated information is actually unlearned or that the alternating forget-retain optimization converges. Finally, the zero suppression target is adopted without principled justification beyond empirical ablations.
The evidence supports the existence of a forgetting–utility trade-off, but the comparisons are incomplete. Table 1 shows SIEVE often achieves the lowest forget-set classification and cloze scores, yet the Retain generation metric drops sharply in textual-QA settings, which the paper frames as competitive without acknowledging the magnitude. Crucially, the experiments omit direct comparison against several concurrent VLM unlearning methods cited in Section 2, leaving the state-of-the-art claim incompletely substantiated.
The implementation details in Appendix B are unusually thorough—Table 3 lists adaptation modules, LoRA rank and scaling, loss weights, optimizer settings, and random seed—and the benchmark MLLMU-Bench is public. However, no code release is mentioned, and the alternating forget-retain training requires a frozen reference model to be run alongside the trainable model, raising practical questions about memory overhead and training time that are not quantified. The AI use statement discloses employment of generative AI for dataset cleaning and manuscript polishing, which is commendable but means independent replicators must verify the exact preprocessing pipeline.
The ability of vision-language models (VLMs) to associate visual identities with biographical information creates a need for selective unlearning of personally identifiable information (PII) while preserving permitted knowledge about the same individual. This setting is challenging because both sensitive and retained information can share the same visual inputs and intermediate representations. We introduce SIEVE, a simple and effective framework for selective VLM unlearning. SIEVE directly regularizes attention-value representations while also controlling model outputs. SIEVE suppresses attention values for forget examples toward a constant zero, while preserving retain-example representations by matching them to a frozen reference model. These objectives are combined with sequence-level forget and retain supervision, enabling targeted forgetting without largely affecting retained knowledge. Extensive experiments show that SIEVE achieves state-of-the-art performance on unlearning with multiple model-modality settings, while maintaining competitive retained utility. Ablation studies further show that value suppression and negative cross-entropy contribute complementary forgetting signals, while reference-based value matching substantially reduces utility degradation. These results demonstrate that attention values provide an effective intervention point for selective multimodal unlearning when sensitive and retained knowledge are closely related.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.