LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Looped Mixture-of-Experts models reuse parameters across depth, yet prior work treats each loop visit identically. This paper shows that loop visits learn strongly asymmetric residual scales—the first visit updates representations far less than the second—and proposes LatentLoop, which compresses the first pass into a latent sequence and reallocates saved FLOPs to model width. Extensive scaling experiments up to $7$B parameters show consistent validation and downstream gains, framing asymmetric compute allocation as a principled way to improve looped MoE efficiency.
The paper identifies a genuine asymmetry in looped MoE updates and turns it into a concrete architecture, but its impact is bounded by modest absolute gains and unexplained mechanisms. The learned coefficient evidence is clean: "the first coefficient falls well below the second, while the second remains near one by the end" (Section 3.1). Still, the reported $19.0$\% compute savings are fitted-extrapolation estimates, and the downstream average gains over MoE Loop range from only $0.17$ to $1.18$ percentage points (Table 2).
The residual scaling analysis is the strongest part of the paper. Jointly optimized per-visit coefficients reveal a consistent pattern at $1.3$B and $3$B: "The final checkpoint coefficients are approximately $(0.27, 0.87)$ for $1.3$B and $(0.36, 1.09)$ for $3$B" (Section 3.1). Fixed-coefficient sweeps confirm causality: reducing the first visit to $0.25$ improves NLL, whereas identical cuts to the second visit hurt it. The $15$-configuration scaling ladder is also a solid empirical foundation: "At all $15$ matched budget endpoints, LatentLoop consistently achieves lower validation NLL than MoE Base and MoE Loop" (Section 4.2).
The headline compute savings rely on fitted power-law frontiers derived from just $15$ points across three sizes, and the "predicted compute savings are $19.0$\% and $7.9$\% relative to vanilla MoE and the looped baseline" at an extrapolated NLL of $1.988$ (Section 4.2). The actual per-endpoint NLL deltas are small—at $3$B/$40$B the gap to MoE Loop is only $0.0094$—so the practical benefit may be limited. The authors themselves admit that "the mechanisms underlying this learned asymmetry remain incompletely understood, particularly the residual coefficient patterns observed with three and four loop visits" (Limitations). Additionally, the benefit of independent attention blocks appears only when paired with sequence compression; for ordinary MoE Loop, untying attention while matching parameters makes NLL worse (Table 5), which weakens the claim that the design choices are broadly principled rather than compensatory.
The internal ablations fairly establish that sequence compression is the best of the four tested compute-reduction strategies, and the scaled decoder outperforms a simple scalar gain (Section 4.4, Table 4): "A single scalar therefore recovers most of the observed improvement, while channel and position adjustments provide a smaller additional reduction." However, the comparison to SpiralFormer is weakened by mismatched training conditions: Appendix D explicitly notes that "SpiralFormer\u2019s implementation was not publicly available" and the systems "use different training data, tokenizers, training budgets, and other implementation details" (Appendix D). Within the paper\u2019s own matched setting, the evidence supports lower NLL, but the downstream improvements are uneven across benchmarks—some tasks show negligible or negative deltas—so the average gain is driven by a subset of tasks.
The authors provide architecture tables, hyperparameters, and budget matching formulas in the appendices, and list a public code repository. Yet reproducing the main results is challenging: the $15$-configuration scaling ladder requires "about $35$ hours on $64$ NVIDIA H$200$ GPUs" just for the $7$B tier (Appendix B), and the budget matching depends on a bespoke four-step protocol—"adjust the attention projection dimensions to match KV cache, then the model width to match computation per step, and next the number of stored experts to match the parameter budget" (Section 2.2)—whose sensitivity is not fully characterized. The exact causal incremental decoding logic for the latent cache and the training mainline/branching schedule are also non-trivial implementation details.
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose \textbf{LatentLoop}, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding $10^{21}$ FLOPs, LatentLoop consistently lowers validation NLL. At a fitted NLL of 1.988, predicted compute savings are 19.0\% and 7.9\% relative to vanilla MoE and the looped baseline. It also improves average scores on nine benchmarks at all sizes in 0-shot and 5-shot settings. These results suggest asymmetric compute allocation is a promising direction for looped MoE models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.