Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning
Test-time reinforcement learning (TTRL) lets language models adapt on unlabeled problems using their own samples as supervision, but a diffuse initial policy wastes the limited adaptation budget concentrating probability mass instead of converting existing capability into reliable output. This paper proposes entry-state sharpening: use data-free training before TTRL to place the checkpoint in a concentrated state that subsequent label-free adaptation can exploit more efficiently. The core insight is that checkpoint preparation should be treated as a state-control problem—where entry entropy is a label-free readout and reachable capability is the constraint—rather than as a choice of upstream training recipe.
The paper presents a compelling conceptual reframing of checkpoint preparation for TTRL as a state-control problem, backed by striking observational correlations across five data-free Qwen3-4B checkpoints. Entry entropy rank-orders endpoint conversion efficiency with Spearman $\rho=-0.90$ ($\rho=-0.99$ after controlling for reachability), and an in-domain self-distillation intervention adds causal plausibility. However, the small $n=5$ roster, single model family, and fixed 15-step horizon mean the evidence is suggestive rather than definitive, and the authors candidly acknowledge that controlled sweeps are still required to establish causality.
The central premise is that "entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman $\rho=-0.90$; $\rho=-0.99$ after controlling for entry reachability)," yet the authors note that "a controlled within-method preparation-strength sweep is still required to establish the causal effect of calibrated entry-state sharpening on final TTRL performance."
The distinction between reachable capability (best@16) and conversion efficiency ($\eta=\mathrm{mean@16}/\mathrm{best@16}$) is well-grounded in prior work, and the observational finding is remarkably consistent: checkpoint ranks by $\eta$ are identical across seeds ($\rho=1.00$). The direct self-distillation experiment adds causal plausibility by moving the base checkpoint from 1.186 to 0.183 nats and improving conversion efficiency from 0.941 to 0.945. The sharpest evidence is the contrast between two data-free objectives: "R-Zero remains diffuse at 3.39 nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches 0.07 nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data."
The authors deliberately frame the problem broadly as "treating checkpoint preparation as a state-control problem rather than as a choice of upstream recipe," which correctly decouples the intervention from any single algorithm.
The study is observational across checkpoints with heterogeneous training histories, not a controlled sweep within a single method. With only five checkpoints, exact permutation $p$-values are bounded below by $0.017$, so the significance testing is coarse and the authors rightly warn that the result should be read as "a strong, capability-controlled ordering rather than a powered significance test." The entry-state proxy is also imperfect: token-level entropy "does not fully capture diversity at the answer or solution level," and the self-distilled checkpoint already shows that aggressive sharpening erodes reachability (best@16 falling from 0.820 to 0.811), confirming that entropy minimization is not the goal and that the proxy can miss coverage trade-offs. Finally, the single model family and 15-step horizon limit generalizability.
The evidence supports the core claims within its scope, though it is stronger for correlation than causation. The paper fairly positions itself against related work, distinguishing upstream preparation from internal TTRL modifications such as entropy regularization loops. Its characterization of cited sources is accurate: Yue et al. write that "RLVR does not introduce fundamentally new reasoning capabilities and that the reasoning capacity of current RLVR models remains bounded by that of its base model," which aligns with the authors' premise that TTRL converts rather than creates capability. The in-domain intervention bolsters causality, but the authors are careful not to overclaim, noting that "the self-distilled checkpoint is marginally highest on AMC" and that they "do not interpret this small difference as evidence for a universally superior preparation recipe."
The TTRL protocol is described in useful detail: 15 GRPO updates with 32 unlabeled problems per step, actor learning rate $5\times10^{-7}$ with cosine decay, rollouts via vLLM at temperature 1.0, validation at steps 0,5,10,15, and eight NVIDIA A100 40 GB GPUs. The authors state that "no ground-truth labels enter adaptation," which clarifies the label-free regime. However, no code, preparation checkpoints, or exact self-distillation filtering scripts are released. Reproducing the preparation stages requires reimplementing R-Zero, SPIRAL, and the in-domain pseudo-label filtering from descriptions in cited papers or from public repositories, which introduces implementation variance. The fixed protocol aids cross-checkpoint comparability but leaves hyperparameter tuning and selection criteria opaque.
Test-time reinforcement learning (TTRL) adapts language models on unlabeled test problems using supervision derived from their own samples. This makes the checkpoint's \emph{entry state} consequential: a diffuse policy provides noisier self-supervision and may spend much of a limited adaptation budget merely concentrating probability mass before reliably expressing capability it already possesses. We propose \textbf{entry-state sharpening}: use data-free training \emph{before} TTRL to prepare a general-purpose checkpoint in a state that subsequent label-free adaptation can exploit more efficiently. The idea is not tied to one training recipe; different data-free objectives can move the same base model to different entry states. Across five data-free checkpoints derived from Qwen3-4B and evaluated under an identical 15-step TTRL protocol, entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman $\rho=-0.90$; $\rho=-0.99$ after controlling for entry reachability). The contrast across objectives is striking: R-Zero remains diffuse at $3.39$ nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches $0.07$ nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data. An in-domain label-free self-distillation intervention further shows that the entry state can be deliberately sharpened. These results motivate treating checkpoint preparation as a \emph{state-control problem}: use data-free training to improve TTRL readiness, with entry entropy as a label-free control signal and reachable capability as the constraint.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.