Local Support Learning
Catastrophic forgetting remains a critical barrier in multi-phase training of large language models. This paper recasts forgetting as a geometric phenomenon at the level of each weight matrix: standard gradient updates act on all inputs, not just those from the new task, corrupting prior mappings. The authors propose Local Support Learning (LSL), which pairs task-specific adapters with per-matrix Gaussian Mixture Model gates that restrict updates to activations near the current training distribution. Empirically, LSL retains pretrained math, coding, and instruction-following capabilities in models up to 7B parameters while learning new domains efficiently.
LSL delivers a compelling geometric argument for why standard gradient updates forget—namely, that they lack locality in the input space—and couples it with a practical, lightweight gating mechanism. However, the formal analysis is confined to a single weight matrix, the GMM gate rests on an unverified assumption that a small generic reference dataset suffices for thresholding, and the evaluation covers only three fine-tuning domains and a handful of sequential phases. The method is promising, but its theoretical grounding and the breadth of validation remain limited.
The local-support perspective is sharply articulated. The paper shows that a gradient step $\Delta W^{(s)} = -\alpha \frac{d\mathcal{L}}{dv}^{(s)} x^{(s)T}$ alters logits for any prior input not orthogonal to $x^{(s)}$, so global updates are inherently suboptimal for retention. The per-matrix gating design is strongly justified: replacing it with a single router after the embedding layer recovers only 50–77% of LSL’s new-task improvement because per-matrix gates use contextual features at every depth and isolate errors. The ablation distinguishing locality (Union of Spheres versus GMM) from architecture (MLP) confirms that closed-gate behavior on OOD data is the key driver of retention.
The theoretical motivation in Appendix E elegantly handles a single weight matrix under bounded-domain assumptions, but the leap to deep networks is purely empirical. The authors candidly acknowledge this limitation. A practical worry is inference scaling: because the adapters cannot be merged into the base weights, inference cost grows linearly with the number of phases. In addition, the evaluation is narrow—only the Qwen2.5-Instruct family, three specialized fine-tuning tasks, and three retention benchmarks—leaving open questions about behavior on much larger models, longer phase sequences, or diverse model families. The authors also note that the method assumes high-quality base representations, tying retention to model scale in ways that may limit applicability to weaker embeddings.
Internal evidence supports the critique of prior subspace methods. In Appendix C the authors measure that at most 1.7% of a standard LoRA adapter’s energy falls within the top-$k$ singular-value subspace of pretrained weights, concluding that forgetting there is weak. This directly undermines the motivation for orthogonal-projection approaches and justifies the shift to local support. Comparisons to LoRA and LwF are fair in the paper’s setup, and the MLP gate ablation acts as a useful control showing that generic classification without a locality guarantee fails. Still, the dismissal of replay buffers as impractical at pretraining scale is somewhat overstated—the paper does not compare against even a small buffered baseline, which could be competitive given LSL’s own memory overhead.
The authors provide substantial implementation detail in Appendix D—hyperparameter sweeps, random seeds, optimizer settings, and exact Hugging Face model names—and state that source code is available. Key training decisions are documented, and detention benchmarks are kept separate from hyperparameter selection. However, GMM fitting specifics such as the number of EM iterations, initialization randomness, and convergence tolerance are omitted, and because EM can get stuck in local optima this is a genuine obstacle to exact reproduction. The efficiency benchmark uses an Nvidia B200, which is not widely accessible, though the core retention experiments should be replicable on standard hardware.
We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.