Geometric Similarity in VLM Low-Level Vision Representations
Vision-language models are increasingly used as backbones for all-in-one low-level image restoration, yet how they internally organize pixel-level perceptual representations remains unclear. This paper introduces GeoSim, a unified four-level framework that probes global similarity, local geometry, sparse feature decomposition, and topological structure across 24 tasks in autoregressive and diffusion transformer VLMs. By revealing that task representations are mechanism-centric and that cross-model agreement is primarily global rather than local, the study offers an actionable interpretability lens for designing transferable unified restoration systems.
The paper makes a timely contribution by shifting focus from task performance to the representational geometry of low-level vision VLMs. The GeoSim framework is empirically ambitious—covering 24 tasks and seven models—but the authors appropriately bound their conclusions as interpretability findings rather than causal proofs of transfer. Its central weakness is that the link between measured geometric similarity and actual downstream knowledge transfer remains speculative, despite the paper's framing around latent transferability.
The multi-scale framework is methodologically sound and the coverage is impressive: 24 tasks, seven models, and four complementary analysis levels. The finding that task geometry is mechanism-centric rather than family-centric is a useful conceptual advance, and the observation that cross-model universality is primarily global—preserving relational structure without local neighborhood agreement—is nuanced and well-supported by $dCor$, $NNGS$, and $PAR$ metrics. The sparse autoencoder decomposition adds genuine feature-level insight, showing that features are broadly reused across tasks within each model.
The analysis relies on a single selected stable layer per model, which may miss important layer-wise dynamics. Standardized prompts and dataset-specific sources introduce potential confounds: some observed structure may reflect prompt formatting or dataset bias rather than intrinsic perceptual organization. While persistent homology is included as Level 4, the paper relegates most topological results to the appendix and does not deeply integrate them into the main conclusions. The causal relationship between geometry and transfer is assumed rather than tested.
The evidence robustly supports the claim that low-level tasks possess stable within-task geometry and that cross-model agreement is primarily global, but the comparisons remain descriptive rather than predictive. The within-model results benefit from repeated sampling ($b=10$, $n=100$), yet the cross-model SAE comparison uses distribution-level MMD without establishing feature correspondence, inherently limiting the universality claim. Related work on RSA, CKA, and Taskonomy is cited fairly, though the paper could more directly contrast its geometric findings against prior transfer-learning taxonomies to clarify what GeoSim adds beyond confirmation.
Reproducibility is reasonably strong: the authors provide code, list exact models and datasets, and report hardware. However, several details—such as the complete prompt templates and exact SAE hyperparameters—are only partially specified in the main text, forcing reliance on the appendix and external repository. Additionally, large models like Emu3.5-Image (roughly 34B) and Qwen-Image-Edit (roughly 20B) require substantial compute, and the topological verification depends on stochastic UMAP projections whose stability across seeds is not discussed in the main text.
Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers (DiTs). Yet, adapting them efficiently for all-in-one low-level image restoration remains a challenge. Crucially, the field lacks an understanding of how VLMs organize hidden-layer representations and whether these structurally distinct paradigms share a common geometric organization for pixel-level perception. Such shared organization is a prerequisite for building highly transferable, unified restoration VLMs and adapters. In this paper, we systematically investigate representational similarity across 24 low-level tasks spanning 5 categories. We propose GeoSim, a unified four-level framework that analyzes task-conditioned representations from global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. Our formulation applies to the analysis of hidden states in AR models and feature maps in DiTs across same- and cross-task/model settings. Our results reveal the organizing principles of low-level visual representations while exposing their limits in cross-task and cross-model agreement. Ultimately, GeoSim provides an interpretability lens for probing latent transferability in low-level vision and diagnosing model limitations in task- or model-specific scenarios.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.