On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

The token-level analysis is rigorous and well-executed. The statistical comparison clearly demonstrates that while entropy and KL divergence distributions overlap nearly completely between base and RLVR models, $\Delta \log p$ exhibits a distinct bimodal pattern that separates the two. The selective token replacement experiment provides strong causal evidence: replacing only 10% of tokens selected by $\Delta \log p$ recovers RLVR performance, whereas entropy requires significantly more replacements. The gradient analysis (Lemma 3.1) correctly derives that policy gradients concentrate on low-probability tokens via the $(1 - \pi_\theta(y_t))$ term, providing a theoretical foundation for the observed sparsity.

The theoretical justification for extrapolation (Theorem 4.1) assumes an idealized tabular softmax bandit with Natural Policy Gradient updates, which abstracts away critical aspects of the actual training setup including: the clipping mechanism in PPO/GRPO/DAPO, the response-level reward structure, and the KL divergence regularization. The authors acknowledge this limitation but don't quantify how the mismatch affects the theoretical guarantees. Additionally, the test-time extrapolation method requires maintaining two full models (base and RLVR) and computing logits from both at each step, which incurs significant memory and compute overhead that limits practical deployment. The causal mechanism linking low-probability tokens to 'reasoning-critical' tokens remains somewhat correlational—while the replacement experiments show these tokens matter for performance, the paper doesn't fully establish *why* these specific tokens are reasoning-critical beyond their probability values.

The evidence supports the core claim that $\Delta \log p$ outperforms magnitude metrics for identifying sparse updates. The comparison to related reweighting methods (Table 3) shows their approach outperforms PPL-based reweighting (Deng et al., 2025) and Dominate (Yang et al., 2025b), though the Dominate baseline uses a more restrictive clip-high ratio (0.24 vs 0.28) which the authors note reduces exploration. The evaluation is somewhat narrow, focusing primarily on math reasoning (AIME-24, AMC) with limited generalization analysis (Minerva in Appendix C shows gains but with smaller effect sizes). The comparison to Yang et al. (2025b) is slightly asymmetrical—while Yang argues low-probability tokens 'over-dominate' and should be downweighted, this paper argues they should be upweighted, yet both report improvements over baselines, suggesting the optimal weighting may depend on training stability or model scale.

The paper provides adequate experimental detail for reproduction. The authors use publicly available models (Qwen2.5-Math-7B, Qwen3-8B-Base, DAPO-32B, ORZ-32B) and standard datasets (AIME-24, AIME-25, AMC). Training hyperparameters are detailed in Appendix B, including learning rates (1e-6), batch sizes (512 prompts, 16 responses each), and reweighting coefficients ($\alpha=0.2$ for Qwen2.5). Figure 9 shows reproducibility analysis with 4 independent training runs demonstrating consistent convergence. However, the code and exact training scripts are not explicitly released or linked in the provided text, which would be necessary for full reproducibility. The test-time extrapolation requires careful tuning of two hyperparameters ($\tau$ and $\gamma$) with grid search ranges provided in Appendix A.3.