CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
This paper tackles a fundamental question in multimodal large language models (MLLMs): should the vision encoder be fine-tuned or frozen during instruction tuning? The authors identify visual preference conflicts—where diverse linguistic instructions pull encoder parameters in conflicting directions—as the root cause of instability in existing visual fine-tuning (VFT) methods. They propose CoVFT, a context-aware framework that extracts multimodal context vectors and routes visual tokens through mixture-of-experts layers to decompose these conflicts, achieving consistent gains across 12 benchmarks.
The paper presents a compelling case for context-aware visual fine-tuning, backed by systematic evidence that standard VFT methods (Full FT, LoRA, BitFit) yield unstable improvements over frozen encoders in MLLMs. The proposed CoVFT framework delivers state-of-the-art and consistent gains, with the notable result that a 7B model with CoVFT surpasses its 13B frozen counterpart. The analysis of visual preference conflicts and contextual vector clustering provides valuable insight into why naive fine-tuning fails, though the dense routing strategy and BERT-based context extraction could benefit from deeper theoretical justification.
The benchmark design is rigorous: under a unified LLaVA-1.5 pipeline, the authors demonstrate that existing VFT methods win on only 6-9 of 12 tasks versus freezing, whereas CoVFT wins consistently. The contextual vector extraction (CVE) produces semantically meaningful clusters—such as relationship reasoning versus region grounding—validated by PCA visualization and correlation analysis with expert routing ($r=0.76$). The method shows strong data scalability, matching the full-data baseline with only 75% of training data, and generalizes across architectures (SigLIP, DINOv3, InternVL).
The evaluation is limited to 13B parameters and million-scale data, leaving behavior at larger scales uncertain despite the authors' speculation. The dense routing strategy in CoMoE, while effective, increases computational overhead (~4% training time, 13% memory) and lacks theoretical justification beyond empirical observation—sparse routing is dismissed primarily due to under-training concerns rather than fundamental analysis. The reliance on a frozen BERT encoder for context extraction adds an external dependency and inference latency (~1.14ms) that is acknowledged but not fully resolved. Additionally, the claim that deeper layers exhibit stronger preference conflicts (Fig. 1c) is supported by parameter distance metrics, but the causal link between this divergence and final task performance could be more tightly established.
Evidence supports the core claim that CoVFT outperforms existing methods, achieving 61.08% average accuracy versus 59.29% for full fine-tuning and 58.93% for frozen encoders across 12 diverse benchmarks covering general VQA, OCR, and vision-centric tasks. Comparisons to related work are fair: QA-ViT and Q-MoE are re-implemented under the same LLaVA-1.5 framework, with CoVFT achieving 61.08% versus 59.12% and 59.29% respectively. The ablation studies effectively isolate components, showing that text-guided aggregation (CVE) outperforms image-only (59.77%), text-only (60.55%), and simple concatenation (60.44%) variants. However, the comparison to methods like SVPT (which was designed for classification) on multimodal tasks may slightly exaggerate the instability of prior art, though the overall trend holds.
Reproducibility is strong: code is publicly released, and the paper provides detailed hyperparameters (learning rates $1\times 10^{-3}$ for pre-training, $2\times 10^{-5}$ for instruction tuning; batch sizes 256 and 128). The standard LLaVA-1.5 pipeline with CLIP-ViT-L/14-336 and Vicuna-1.5 is used, and the 12 evaluation benchmarks are well-established. However, exact random seeds are not specified, and hardware details are limited to '8× NVIDIA H100 GPUs' without specifying memory or interconnect details that might affect training dynamics. The method involves several design choices (CoMoE placed in layers 11-22, 4 experts) justified by ablations, but sensitivity to these exact settings versus slight variations is not fully explored.
Multimodal large language models (MLLMs) achieve remarkable progress in cross-modal perception and reasoning, yet a fundamental question remains unresolved: should the vision encoder be fine-tuned or frozen? Despite the success of models such as LLaVA and Qwen-VL, inconsistent design choices and heterogeneous training setups hinder a unified understanding of visual fine-tuning (VFT) in MLLMs. Through a configuration-aligned benchmark, we find that existing VFT methods fail to consistently outperform the frozen baseline across multimodal tasks. Our analysis suggests that this instability arises from visual preference conflicts, where the context-agnostic nature of vision encoders induces divergent parameter updates under diverse multimodal context. To address this issue, we propose the Context-aware Visual Fine-tuning (CoVFT) framework, which explicitly incorporates multimodal context into visual adaptation. By integrating a Context Vector Extraction (CVE) and a Contextual Mixture-of-Experts (CoMoE) module, CoVFT decomposes conflicting optimization signals and enables stable, context-sensitive visual updates. Extensive experiments on 12 multimodal benchmarks demonstrate that CoVFT achieves state-of-the-art performance with superior stability. Notably, fine-tuning a 7B MLLM with CoVFT surpasses the average performance of its 13B counterpart, revealing substantial untapped potential in visual encoder optimization within MLLMs.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.