When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
Momentum is a pillar of modern optimization, yet its benefits in reinforcement learning with verifiable rewards (RLVR) remain unclear. This paper studies a KL-regularized bandit formulation of RLVR and proves that, with exact policy gradients, larger EMA coefficients strictly accelerate convergence within a safe overshoot-free range. Its central insight is that stochastic sampling obscures this gain unless the coefficient scales with the learning rate so that $\eta/(1-\beta)$ stays constant, a principle validated in bandits and in LLM post-training on Qwen models.
The paper's core contribution is a rigorous characterization of when EMA momentum helps in RLVR: deterministic acceleration is guaranteed by Theorem 4.2, while stochastic gains are recovered only when $\eta/(1-\beta)$ is kept in a safe, constant regime. However, the empirical narrative is undermined by a glaring inconsistency: the abstract claims that increasing $\beta_1$ on Qwen2.5-32B improves accuracy by 'nearly 22 percentage points,' whereas Section 6.3 describes the same result as 'nearly 2 percentage points.' Beyond this typo, the theory relies on a response-level softmax bandit that strips away autoregressive generation dynamics, leaving some gap before the GRPO experiments can be considered fully explained.
The exact-gradient analysis in Section 4 is elegant and complete: within a non-trivial safe range, increasing $\beta$ strictly improves the objective after $T$ steps, and the proof avoids overshoot by tracking a scalar residual $f_t(\beta)$. In the stochastic case, the authors carefully control importance-sampling noise through an auxiliary process and derive a second-order expansion that cleanly separates deterministic gain from sampling variance, yielding testable scaling predictions.
The response-level softmax bandit replaces sequence generation with a single categorical choice, so the relevance to token-level autoregressive LLM training is an abstraction that remains largely unjustified. More concretely, the paper contains a striking factual discrepancy between its abstract and its main experimental section: the abstract touts a nearly 22 percentage-point improvement on Qwen2.5-32B, while Section 6.3 reports a gain of roughly 2 percentage points, a ten-fold difference that must be resolved.
Within the bandit model, the evidence matches the predictions quantitatively: fixed $\beta$ produces a vanishing signal-to-noise ratio while scaled $\beta$ drives the win rate toward one. The LLM results are directionally supportive, but the experiments are narrow—one dataset, one family of models, and a single training horizon—so the claim that 'choosing momentum coefficients substantially above the conventional value of 0.9 can yield greater benefits in practical RLVR training' is less established than the bandit theory.
No code or data repository is advertised in the main text, and the detailed hyperparameters are relegated to Appendix I.2, which lies beyond the provided excerpt. The bandit setting is fully specified in the main text, but the LLM experiments—involving Qwen3-4B-Base and Qwen2.5-32B with GRPO—would be difficult to reproduce without the exact prompt formatting, reward functions, and training scripts.
Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient $\beta$ controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing $\beta$ within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate $\eta$ decreases, the expected gain with fixed $\beta$ vanishes faster than these fluctuations, whereas increasing $\beta$ to keep $\eta/(1-\beta)$ within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing $\beta_1$ improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.