How Depth Growth and Looping Shape In-Context Learning in Language Models
Depth growth and looping reuse computation across layers to train or run Transformers more efficiently. This paper shows that both methods create an inductive bias toward learning from context over memorizing fixed associations in weights, accelerating in-context learning and propagating induction heads through depth. The evidence spans models up to 3B parameters and connects architecture design to the mechanistic origins of few-shot performance.
This is a thorough empirical study linking training-time depth reuse to behavioral and mechanistic signatures of in-context learning (ICL). The authors compare dense, grown, and looped models on language modeling, formal-language adaptation, and controlled image-classification tasks, then trace attention heads through training and ablate them in final checkpoints. The evidence is generally consistent and well-documented, though the causal arrow—whether depth reuse directly causes stronger ICL or merely correlates with it through confounding training dynamics—is not fully disentangled.
The multi-task evaluation strategy is a real strength. By pairing natural-language benchmarks with RegBench formal-language learning and Omniglot protocols that explicitly pit ICL against in-weights learning, the authors isolate context use from general capability improvements. The mechanistic tracing is also compelling: in the grown model, ten of the twelve strongest final induction heads descend from just two heads in the initial shallow network, producing repeated depth patterns akin to looping.
Three limitations stand out. First, statistical robustness is thin: with one training seed per arm for the main language-model comparisons, it is hard to know whether reported gaps reflect systematic inductive biases or lucky initializations; the authors note that bands estimate demonstration-seed rather than training-seed variance. Second, the RegBench comparison includes an unfair architectural mismatch: the looped model was selected with width 256, while dense and grown models use width 128, because hyperparameters were tuned on validation rather than forced to match. Third, the $S_{\mathrm{ICL}}$ metric is a difference of early and late token losses, so a higher score can arise from worse early performance rather than better late performance; the authors flag this ambiguity in Appendix B.2, yet the framing still leans on it as a signature of context use.
The dense baselines are carefully matched for effective depth, data order, optimizer steps, and inference FLOPs within each size, making the core comparisons fair. However, depth growth inherently saves training FLOPs and looping inherently reduces unique parameters, so any claim that the architectural choice alone biases ICL—rather than simply trading off capacity—would ideally control for total training compute rather than just final inference FLOPs. The cross-size trends (360M, 1.7B, 3B) are suggestive but not cleanly comparable because tokenizer, context length, and training schedules vary. Related work is discussed comprehensively and citations to concurrent studies are accurate.
Experimental detail is extensive: appendices specify architectures, growth schedules, tokenizers, learning-rate curves, and evaluation protocols. The authors state they use public datasets and the nanotron training framework, but they do not provide a code repository or released checkpoints in the main text or reproducibility statement. This omission is notable because independently replicating the exact growth schedules, layer-duplication mechanics, and head-ablation pipelines would require substantial engineering. The 360M looped model also has a flagged initialization difference (Appendix A.1), which introduces a subtle confound for that size.
Looped and depth-grown language models reuse computations across depth and have attracted interest for their reasoning capabilities. Building on prior suggestive evidence of improved in-context-learning (ICL), we investigate how such reuse favors ICL, including how it shapes the acquisition, retention, and mechanisms of this ability. Our analysis spans pretrained language models up to 3B parameters and controlled tasks. The experiments provide evidence for a shared inductive bias toward learning from context: grown and looped models rely more on information in context and, compared with a full-depth dense reference, acquire ICL earlier in controlled tasks and retain it more strongly when learning fixed associations in the weights is also possible. Tracing attention heads through training reveals that depth growth propagates early induction heads across layers. Ablations show that grown and looped models rely more strongly on induction-ranked heads for both few-shot performance and later-token prediction. Together, these findings provide evidence for a shared inductive bias toward learning from context and suggest that (soft) parameter reuse shapes how this ability develops and which attention mechanisms support it.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.