Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment
Long-text image-text alignment underpins fine-grained scene understanding, yet existing models suffer from saliency collapse: they over-rely on visually prominent text while neglecting informative non-salient context. This paper proposes RoSA, a finetuning framework that decomposes image-text pairs into complementary salient and contextual views and enforces auxiliary contrastive objectives over both. By treating saliency as a multimodal training prior with no inference overhead, RoSA improves retrieval robustness when salient cues are weakened or removed.
The paper identifies a genuine and overlooked failure mode—saliency collapse—and proposes a clean, well-motivated auxiliary-learning solution. RoSA consistently advances robustness under synthetic salient-text reduction and yields modest but reliable gains on standard benchmarks, though it inherits the biases and domain gaps of external saliency and LLM decomposers. It is a solid contribution with clear practical appeal for CLIP-based retrieval pipelines.
The preliminary study is especially strong. Through length-matched random controls and a text-only oracle, the authors rigorously isolate saliency collapse from confounders such as query length and lexical frequency, and demonstrate that non-salient context is informative yet underutilized. The plug-and-play results in Table 3 are equally compelling: adding $\mathcal{L}_{\mathrm{RoSA}}$ to Long-CLIP, GOAL, StructXLIP, LoRA, DoRA, and SigLIP2 yields consistent gains across architectures.
Despite its conceptual elegance, RoSA delivers only modest and occasionally mixed gains on standard full-caption retrieval: Table 2 shows near-zero or negative margins on several DCI and Sketchy metrics, and the INSECT benchmark exhibits very low absolute recalls across all methods. The evaluation of saliency collapse is largely synthetic—progressively deleting salient tokens—which may not reflect natural query variation. Furthermore, the method depends critically on a 2014 object-saliency detector and an LLM text decomposer whose biases and out-of-domain behavior are acknowledged but not deeply quantified. Notably, the main text omits the values of the loss weights $\lambda_{\mathrm{sal}}$, $\lambda_{\mathrm{ctx}}$, and $\lambda_{\mathrm{anc}}$ that govern the auxiliary objectives.
The evidence convincingly demonstrates that saliency collapse exists and that auxiliary salient/contextual alignment mitigates it. However, the comparison landscape is narrow: all baselines are CLIP-derived finetuning methods, and the paper does not benchmark against modern native long-context encoders or generative retrieval systems. In addition, the cross-modal mismatch analysis (Appendix Table 12) reveals that filtering low-agreement salient or contextual pairs often degrades performance relative to the full noisy set; a stricter reading is that some gains may stem from data augmentation rather than from faithful multimodal decomposition.
The authors commit to releasing code and checkpoints, and they describe the backbone (CLIP ViT-B/16 with KPS extension), training duration (10 epochs), and hardware, which facilitates reproduction. Yet several implementation details needed for an exact reproduction—most importantly the loss weights $\lambda_{\mathrm{sal}}$, $\lambda_{\mathrm{ctx}}$, and $\lambda_{\mathrm{anc}}$, the exact optimizer configuration, and the full LLM prompt—are relegated to supplementary material that was truncated in the provided text. Because preprocessing relies on specific Qwen2.5 and FASA versions, reproducing the exact decomposed views independently requires additional care.
Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: https://github.com/intelligolabs/RoSA
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.