LoRA Superposition across Pretrained Models
The paper attacks the combinatorial cost of maintaining separate LoRA adapters for every pretrained backbone–task pair. Its core idea, LoRA superposition, is to sum the low-rank updates learned independently on different backbones and apply the same composite update unchanged to every contributing model. The empirical finding is that frozen backbones implicitly select their own component from the sum, enabling a single shared adapter without explicit routing or cross-model alignment. If the phenomenon generalizes, it could enable modular adapter libraries that work across pretrained models without joint retraining.
The paper presents a striking empirical finding: across 56 backbone–task pairs, the exact sum of solo LoRAs nearly matches the performance of individually trained adapters (within 0.28 percentage points on average), while a target-excluded sum collapses to the unadapted baseline. This strongly supports the proposed component-wise non-interference hypothesis. However, the theoretical explanation remains post-hoc and descriptive; the “implicit projection” is not characterized or predicted a priori. Moreover, the study is confined to seven ViT-B/16 CLIP backbones with identical architectures, leaving open whether the phenomenon generalizes to more diverse or numerous models.
The experimental diagnostics are well chosen. The leave-one-model-out (LOMO) control is particularly convincing: removing a backbone’s own component from the sum produces a mean gain of only 0.06 percentage points, directly supporting the claim that each backbone relies on its own update. The cross-backbone response matrix and the near-orthogonality of solo LoRA vectors—mean absolute cosine similarity of $1.34 \times 10^{-3}$ across different backbones—provide converging geometric evidence. The honest comparison to Joint shared further contextualizes the result.
A central limitation is scope. All seven backbones share the identical ViT-B/16 architecture and are all CLIP-style vision-language models pretrained on similar web-scale data; this homogeneity means the results may not transfer to architecturally distinct models or more diverse pretraining regimes. The authors rightly caution that "small pairwise responses do not necessarily imply that the sum of many foreign components remains negligible, because weak interactions may accumulate as the backbone set grows." Mechanistically, the paper offers a geometric interpretation—each backbone induces an implicit projection onto its own useful directions—but this is not validated beyond the observed near-orthogonality. It remains unclear whether orthogonality is a cause of non-interference or merely a correlate of the specific training setup.
Additionally, the superposed update has rank up to $K r = 112$, yet it is stored as dense unfactorized matrices; this yields a single shared adapter but does not obviously reduce total storage relative to the factorized solo LoRAs unless many backbones are involved. The paper notes that "the exact sum is a composite update whose rank can grow with the number of contributing backbones," yet compression to rank 16 is left to future work.
The internal evidence supports the narrow claim well. The comparison between Exact sum and Solo is nearly matched in accuracy, while LOMO sum serves as an effective negative control. The authors honestly note that "the comparison is not rank-matched: Exact sum has rank at most 112 per module, whereas Joint shared has rank 16." The positioning against prior work is generally fair: task arithmetic indeed assumes a common pretrained backbone, and Kawamoto et al.’s cross-model transfer explicitly requires learning "layer-wise orthogonal matrices that rotate task vectors into the target model’s parameter space, while preserving their norm and rank" from a few labeled examples. However, the paper does not empirically compare against these methods, so the practical advantage of superposition over a lightweight alignment step is asserted rather than demonstrated.
The paper provides substantial experimental detail, including the seven OpenCLIP backbones, eight datasets, LoRA configuration ($r=16$, $\alpha=32$, $Q$ and $V$ projections only), optimizer settings (AdamW, $10^{-5}$ learning rate, weight decay $0.1$, batch size 64, $\sim$2000 steps), and the three-seed reporting protocol. Nevertheless, no code release or data preprocessing scripts are mentioned, which would block straightforward replication. Reproducing the full 56-condition matrix also requires training 56 separate solo LoRAs plus the joint baseline, demanding non-trivial compute; the lack of exact training step counts per dataset and unspecified prompt-ensembling templates for the text encoder further add friction.
Low-rank adaptation (LoRA) is usually tied to the pretrained backbone on which it is learned. We ask whether LoRAs trained independently for the same task on different pretrained backbones can be summed and applied unchanged to all contributing backbones. We call this construction LoRA superposition. Across seven ViT-B/16 CLIP backbones and eight image-classification tasks, the exact sum improves all 56 backbone-task pairs and achieves a mean accuracy gain of 30.21 percentage points over the unadapted backbone. The exact sum achieves a mean accuracy only 0.28 percentage points below that of individually trained solo LoRAs. In contrast, a leave-one-model-out (LOMO) sum produces a mean gain of only 0.06 percentage points over the unadapted backbones. Backbone-specific solo LoRAs are nearly orthogonal in parameter space, while pairwise cross-backbone responses are strongly concentrated on the diagonal. These results support a component-wise non-interference hypothesis, under which the frozen backbone acts as an implicit selector of its own component without explicit routing or cross-model alignment.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.