Does Scaling Reinforcement Learning Really Require More Training?
This paper asks whether the best policy from an RL training run is necessarily one of its visited checkpoints. It introduces policy-space scaling—the idea that completed RL histories contain stronger, constructible policies—and proposes SURGE, which fuses two checkpoints via delta-first spectral decomposition without additional training or per-query inference cost. Across two 1.5B math trajectories and one 7B code trajectory, SURGE exceeds the best native measured checkpoint while using fewer reasoning tokens.
The paper identifies a genuinely underexplored axis for scaling: offline reuse of already-computed RL updates. By preserving an anchor's dominant update directions while injecting a donor's complementary components, SURGE reliably outperforms every measured native checkpoint on the evaluated trajectories. Still, the claim rests on only three histories—two 1.5B math models and one 7B code model—trained with the same GRPO recipe, and the central calibration rule is a post-hoc heuristic rather than a principled optimum. The conceptual reframing is valuable, but the empirical ceiling is drawn from a very small sample.
The methodology is carefully isolated. The authors rigorously justify delta-first SVD over absolute-weight decomposition, and the ablation grid shows that neither scalar interpolation nor random subspace projection recovers SURGE's gains. Crucially, an anchor-only truncation at the same block size loses ten AIME24 accuracy points, proving the donor's complementary component matters beyond simple token reduction. The retained-energy calibration also sensibly handles architectural shape differences, letting a single heuristic transfer from 1.5B math to 7B code.
The evidence is limited to only three RL trajectories, all following the same single-stage GRPO recipe, so generalization to other optimizers, scales, or domains is unproven. The spectral calibration target $\tau=0.95$ was identified from the initial math study and then frozen; the authors correctly note it is empirical and carries no guarantee of reward optimality. More importantly, they explicitly decline to claim per-benchmark statistical significance, and the high evaluation variance on AIME24 ($54.17 \pm 6.40$ across four rounds) makes point estimates of small gains difficult to trust. Additionally, anchor and donor selection relies on trajectory statistics that are not fully automated.
The evidence inside the fixed histories is well controlled. Geometric ablations show that random orthonormal bases ($47.29\%$) and trailing singular subspaces ($44.17\%$) both underperform SURGE, even when they use fewer tokens, isolating the importance of the leading RL-update block. The comparison to related work is conceptually clear, separating policy-space scaling from test-time scaling and from model-merging literature such as Task Arithmetic and TIES-Merging. However, the paper does not run head-to-head empirical comparisons against the nearest trajectory-extrapolation methods—such as RELEX, AlphaRL, or extrapolative weight averaging—leaving it unclear whether SURGE uniformly dominates or is complementary to those approaches.
Reproduction is feasible in principle but has friction. The authors document training recipes (JustRL, GRPO/veRL, DAPO-Math-17k), decoder hyperparameters, and checkpoint steps, and they provide pseudocode for SURGE. The fusion is lightweight—about 37–40 seconds on one RTX A6000 for 196 matrices. The main barriers are the absence of an explicit code release and the need to reconstruct or obtain the exact 19 checkpoints per trajectory; donor-anchor pair selection also depends on statistics gathered along the trajectory that are not fully specified.
Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yield policies stronger than the checkpoints visited by its optimizer. We call this policy-space scaling: expanding the deployable policy set accessible from a fixed RL history, without extending training or increasing per-query inference computation. We instantiate it with SURGE (Scaling Up RL Gradient-free via Eigenspace fusion). SURGE combines two checkpoints from the same RL run: a high-accuracy anchor and a competitive donor that generates shorter responses. It expresses both checkpoints as changes from their shared initialization, then spectrally decomposes the anchor's update to retain its dominant component and incorporate the donor's complementary component. With a fixed target for how much of the anchor update to retain, SURGE determines the block size from the weights without testing candidate policies. We evaluate two 1.5B mathematical-reasoning histories, DeepSeek and Nemotron, and one 7B coding history, OLMo. SURGE improves benchmark-average accuracy over both input checkpoints while using fewer reasoning tokens than the anchor. It reaches 54.17% on DeepSeek AIME24 against a measured native maximum of 50.83%, and 83.7% on OLMo HumanEval+ against 82.8%. These gains exceed the observed training curves. Geometric controls support the importance of RL-update structure beyond weight displacement or token reduction alone. Each constructed model runs as a single policy. Our findings identify stored RL history as a reusable scaling resource: the capability available from a training run need not end at its best checkpoint.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.