On Language Drift during RLVR Post-Training
Recent reasoning models show impressive capabilities thanks to RLVR post-training, but their chain-of-thought traces increasingly drift into unusual, non-standard language. This paper formally proves that RLVR optimization allows unbounded language drift while supervised fine-tuning imposes a hard finite bound, and shows empirically that such drift arises specifically when RLVR must discover novel behaviors on tasks beyond the base model's existing capabilities. The results suggest a fundamental tension: improving chain-of-thought monitorability by constraining drift necessarily limits the expected reward a model can achieve.
The paper makes a valuable contribution by grounding the empirical phenomenon of illegible CoTs in frontier LLMs within formal theory and small-scale experiments. The KL-divergence-based distinction between RLVR and SFT is cleanly proved, and the empirical link between novel-task behavior discovery and drift is intuitive and partially validated. However, the central drift-versus-reward trade-off in Theorem 3 is only shown theoretically, and the restriction to 1–1.5B models on a single dataset limits how strongly the findings transfer to frontier settings.
The formal separation of RLVR and SFT is rigorous. The authors define language drift as $\operatorname{LD}(\pi)=\mathbb{E}_{x\sim P^{(\textit{HL})}_{\mathcal{X}}}[D_{\textit{KL}}(P^{(\textit{HL})}(\cdot|x)||\pi(\cdot|x))]$ and rigorously exploit the fact that RLVR "places no optimization pressure on the CoT $z$, allowing arbitrary language drift away from natural language in the reasoning trace" while SFT must minimize cross-entropy with a fixed target distribution, yielding Theorem 1 ($\sup \operatorname{LD}(\pi_n^{(\textit{RL})})=\infty$) and the bounded analogue for SFT. The trace-legibility proxy for drift and the careful iso-performance checkpoint selection strengthen the experimental design.
The empirical evidence is far from conclusive for frontier regimes. The authors acknowledge that the observed drift in their 1–1.5B models "does not reach the degree of illegibility displayed in some frontier-model internal reasoning traces," leaving a significant scale gap. More critically, they admit that "we do not include any empirical demonstration of our theoretical findings in Section 4.2 (Theorem 3)," so the practically relevant curve linking constrained drift to reward loss remains completely unmeasured. Without such validation, the strength of the claimed safety-capability trade-off is unclear.
Within its narrow setup the evidence is internally consistent and comparisons are fair. The paper controls for validation accuracy by evaluating both best-performing and iso-SFT/RLVR checkpoints, and it productively contrasts behavior-sharpening models (Qwen) against models that must discover novel behavior (Llama, Gemma). However, the reliance on a single dataset and three small base models limits external validity, and the paper does not systematically compare its KL-divergence definition of drift against operationalizations used in earlier monitorability work beyond citing them as related motivation.
The authors publish their code on GitHub and report hyperparameter sweep ranges, which aids replication. But full reproducibility is hampered by missing details: the exact chosen hyperparameters per model are not tabulated, legibility evaluation depends on specific post-trained instruct checkpoints, and reporting omits GPU hours, wall-clock times, and any random seeds beyond the five training seeds. The use of publicly available GSM8K data and deterministic sampling during evaluation partially offsets these gaps.
Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.