INTACT: Inducing Latent Reasoning on a Frozen Backbone
INTACT attacks the inference cost of chain-of-thought reasoning by replacing early textual reasoning steps with compact continuous latent vectors while keeping the backbone LLM frozen. A small 107M-parameter encoder iteratively builds a latent prefix from the frozen model's own hidden states, first distilling the effect of written reasoning steps and then refining via group-relative policy gradients. Across Llama and Qwen backbones from 1B to 8B, the method improves accuracy and shortens generation length, and a single encoder trained on math, science, and code transfers to twelve held-out benchmarks including commonsense without adaptation.
INTACT credibly shows that latent reasoning can improve a frozen backbone. Stage 2 raises GSM8K accuracy from $75.8\%$ to $79.3\%$ on Llama-3.2-3B and from $77.6\%$ to $79.7\%$ on Qwen2.5-3B while using fewer tokens, and it surpasses the evaluated latent baselines CODI, MUX and LOTUS at 3B and 8B. However, gains are highly backbone-dependent: at 1B the method trails Aug-SFT by over 12 GSM8K points, and even at 3B it does not uniformly beat supervised fine-tuning (Nemo-SFT exceeds it by $2.1$–$3.7$ points). Moreover, wall-clock latency savings are modest ($19\%$) because the backbone still decodes the remaining reasoning autoregressively and the encoder requires four separate forward passes. A subtle but notable issue is that Stage 1 trains with a position-shifted layout that inference never reproduces, creating a train–test mismatch in prefix construction.
The progressive substitution objective is well-motivated and thoroughly validated. The substitution loss $\mathcal{L}_{\mathrm{sub}}^{(i)} = D([\mathbf{x},\mathbf{z}_{1:i}] \parallel [\mathbf{x},\mathbf{v}_{1:i}]; [\mathbf{z}_{i+1:S},\mathbf{y}])$ distills behavior without directly aligning latent and text vectors, and the on-policy preservation term is validated against weaker anchors. Strong controls confirm the latent prefix is question-specific rather than a generic prompt: shuffling blocks at inference reduces GSM8K accuracy from $84.4$ to $77.0$, and replacing the prefix with one from another question drops it to $70.9$, below the frozen backbone. Likewise, the constant-prefix control shows the conditioned encoder leads a learned constant prefix by $9.2$ points on science after Stage 1 and by $13.1$ on commonsense after Stage 2.
The most significant technical concern is a train–test mismatch in Stage 1: the latent span is systematically shifted during training so the continuation aligns with the teacher trace, but this displacement "grows from 3 positions at $i=1$ to 14 at $i=4$, roughly a quarter of the prefix, so Stage 1 trains on a layout that inference does not reproduce." At inference the prefix uses contiguous positions, so the encoder is optimized under a different positional regime than the one it encounters at test time. Efficiency gains are also weaker than token counts suggest: despite emitting $65$ fewer tokens on Llama-3.2-3B, end-to-end latency falls only $19\%$ because prefix construction costs $127.5$ ms and the longer context raises per-token decoding cost from $16.03$ to $16.67$ ms. Finally, the method still leaves substantial textual decoding intact, and at 1B it underperforms both SFT and competing latent baselines, suggesting the frozen-backbone constraint is especially costly for smaller models.
The authors are careful about provenance, marking quoted results with † and ‡ and noting where no comparable checkpoint exists. At 3B and 8B, INTACT clearly outperforms the evaluated latent-reasoning methods: on Llama-3.2-3B GSM8K it scores $79.3$ versus CODI $60.8$, MUX $65.0$ and LOTUS $70.0$. But the comparison to backbone-updating supervised baselines is less favorable: "INTACT does not uniformly outperform supervised fine-tuning," and Nemo-SFT leads on GSM8K at every scale above 1B by several points. At 1B, INTACT "remains below Aug-SFT and the latent baselines," which undermines claims about broad scalability. The evidence supports superiority among frozen-backbone latent methods at larger scales, not universal dominance over all training recipes.
Hyperparameters and implementation details are documented in depth. Appendix C and Table 10 report exact settings for both stages: $K=4$ blocks of $c=4$ vectors, AdamW learning rates ($10^{-4}$ for Stage 1 and $10^{-5}$ for Stage 2), batch sizes, preservation weights ($\beta$ ranging from $0.28$ to $0.34$ across backbones), RL temperatures, and gradient clipping. The evaluation protocol—including bare prompts, sampling parameters, answer extraction heuristics, and token budgets—is specified in Appendix C.4. Nevertheless, the paper does not mention release of code, checkpoints, or data pipelines, which would be important for independent reproduction given the intricacy of the two-stage training, the position-shifted Stage 1 layout, and the question-dependent prefix construction.
Latent reasoning replaces natural-language chains of thought with compact continuous "soft-token" representations, reducing the number of intermediate tokens that must be decoded autoregressively. We use a small external encoder to induce latent reasoning in a frozen backbone model. The encoder iteratively maps the backbone's hidden states to new latent blocks, constructing a latent reasoning prefix, after which the frozen model writes its remaining reasoning and its answer as text. Our method, INTACT, trains this encoder through distillation to reproduce the effect of written reasoning steps while keeping the model's behavior close to that of the frozen backbone, and then reinforces it on answer correctness. Across four frozen backbones from 1B to 8B, INTACT improves both GSM8K and average out-of-domain accuracy, including GSM8K gains from 75.8% to 79.3% on Llama-3.2-3B and from 77.6% to 79.7% on Qwen2.5-3B. It also reduces end-to-end latency per problem by 19% on Llama-3.2-3B and outperforms the evaluated latent-reasoning baselines at the 3B and 8B scales. Beyond the mathematics setting, the same recipe trained on a multi-domain corpus improves the frozen backbone on twelve evaluation benchmarks spanning mathematics, science, and commonsense, without task-specific adaptation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.