Decoding Looped Transformers Better for (Almost) Free
Looped Transformers reuse the same parameter block across recurrent iterations, producing intermediate states that are usually discarded at decode time. This paper proposes LoopCD, a training-free contrastive decoding framework that treats an early recurrent state as a weak predictor to guide the final prediction, either in logit space (one extra output pass) or hidden-state space (zero extra overhead). The core insight is that recurrence naturally provides aligned weak-and-strong pairs without auxiliary models, and the authors show this can both boost accuracy at full depth and—more interestingly—allow halving the iterations while matching baseline performance, cutting forward FLOPs by roughly 23–48%.
LoopCD is a clean, practical inference-time technique that extracts genuine value from an architectural byproduct of looped Transformers: intermediate recurrent states are naturally aligned weak predictors. Empirical gains are consistent across four diverse model families, and the FLOP-reduction regime is the most compelling result. However, the decoding rule itself is mathematically identical to existing contrastive decoding methods; the novelty lies primarily in sourcing the amateur from the model’s own early recurrent pass rather than an external model. The headline accuracy improvements are strongest in math and code but more modest on average for multiple-choice tasks.
The observation that looped Transformers produce a built-in weak-to-strong trajectory is well validated. The first recurrent iteration trails the final prediction by 4.0 to 23.7 points on the seven-benchmark mean, and accuracy climbs steadily across iterations. This empirical regularity holds across Ouro, Huginn, Parcae, and Looped-Qwen3 despite their very different training objectives and architectures, suggesting the phenomenon is robust. The analysis section is a strength: the decomposition of the contrast vector into parallel (temperature-like) and orthogonal (re-ranking) components, and the finding that gains concentrate on uncertain decisions (least-confident fifth), provides a simple mechanistic account of why the method works.
First, the method requires extensive per-model, per-task hyperparameter tuning. Fixed strength $\omega$ ranges from 0.2 to 1.0 and reference states vary by architecture and depth (Appendix B.4, Table A4), and reduced-depth settings use distinct hyperparameters again. This tuning burden undermines the plug-and-play appeal of a training-free method. Second, results are mixed on some benchmarks: GSM8K exhibits declines of up to $-1.29$ points (Appendix C.1, Table A5), and on Looped-Qwen3 reasoning, pass@1 drops while pass@10 rises, indicating guidance can harm per-sample accuracy even as it broadens coverage across sampled trajectories. Third, the core update $z' = (1+\omega)z_R - \omega z_1$ is mathematically equivalent to standard contrastive decoding and DExperts; the paper avoids direct comparison against running standard contrastive decoding with any auxiliary amateur model, leaving it unclear whether the internal-state source is superior to an external one.
The evidence supports full-depth gains for mathematical reasoning and code generation, where AIME 2024 pass@1 rises by up to 11.45 points and HumanEval pass@1 by up to 9.15 points (Tables 1 and 3). However, on the seven multiple-choice benchmarks the mean improvements are often below one point (Table 2), and on some tasks individual benchmark scores decline (e.g., WinoGrande down $-2.45$ for Ouro-1.4B adaptive). The lack of comparison to DoLa or to standard contrastive decoding with a small auxiliary model is a notable gap; the paper only compares against the unguided baseline. The FLOP accounting is careful and transparent, breaking out coda, head, and loop fractions, but the "almost free" label strictly applies only to LoopCD-Hidden and only when the coda is shallow.
The paper documents model configurations, layer counts, prompts, shot counts, and evaluation protocols in Appendices A and B, and pairs all guided runs with matched unguided baselines using the same checkpoint and prompt. However, no public code repository is mentioned, and reproducing the results would require reimplementing the specific reference-state selection and strength grids, which differ by model and task. Additionally, the logit and hidden-state variants sometimes use different prompts and shot counts (Tables A2 and A3), meaning direct variant-to-variant comparisons are not always strictly apples-to-apples. The FLOP analysis is theoretical prefill-only; wall-clock speedups would depend on memory bandwidth and kernel fusion that are not measured.
Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.