Improving Math Reasoning through Value-guided Informative Search
APIVIS addresses the vanishing-learning-signal problem in RLVR mathematical reasoning by mixing direct and Gumbel-searched completions within each rollout group. The core idea is to treat chunk-level search as a policy-improvement operator with a provable monotonicity guarantee, while a selective InfoSFT auxiliary loss preserves training signal when all group returns are identical. Empirical gains on Qwen3-1.7B and Qwen3-4B are consistent but rest on single-seed runs and roughly 3× the compute of direct RLVR.
The paper makes a rigorous conceptual advance by framing search as an explicit policy-improvement operator rather than a diversity heuristic, proving that monotone value reweighting of sampled reasoning chunks yields a rollout policy with expected verifier reward no lower than the current policy, degrading gracefully with value-estimation error. Empirically, APIVIS improves average Avg@16 over direct GRPO by roughly 2.5–3.2 points at two model scales, but these claims rest on single-seed training runs, an unmaintained frozen value model with inconclusive ranking quality at the 4B scale, and roughly 3.2× the GPU-hours of the direct baseline.
Theorem 1 gives a clean exact-improvement bound for reweighting sampled chunk candidates by a non-decreasing transform of their action values: the expected value under the improved distribution is at least the prior policy’s state value, and with approximate values the degradation is bounded by $2K\epsilon$ across $K$ searched states. The mixed-policy rollout design—three direct and three searched completions per group—is principled because it anchors group-relative advantages to the current policy while allowing search gains to create contrast. The InfoSFT auxiliary loss directly targets GRPO’s collapse when all trajectories share identical terminal rewards, preserving a learning signal by distilling only those searched tokens whose backed-up value exceeds the pre-search state or the policy’s direct candidate.
All headline comparisons rely on a single training seed, so the reported 2–4 point average improvements lack any estimate of run-to-run variance; checkpoint selection additionally follows a validation rank-1 protocol on a 35-prompt split before full-benchmark evaluation, risking implicit selection bias.
The compute asymmetry is sharp: at 4B scale APIVIS consumes 160.51 GPU-hours versus 50.57 for direct GRPO, and the authors explicitly note this is wall-clock accounting, not a token-matched comparison. Finally, the value model remains frozen throughout training, yet the 4B step-100 diagnostic shows essentially zero top-1 lift and an inconclusive Spearman correlation, undermining the theorem’s assumption that estimates stay within a small $\epsilon$ of true $Q^{\pi}$ as the policy evolves.
The headline Table 1 supports the central claim that search must be coupled to policy learning rather than used for generation diversity alone: both LATR and TS-LLM underperform direct GRPO on average, whereas APIVIS outperforms all baselines. However, the ablations in Table 4 attribute most of the gain to swapping MCTS-alpha for the Gumbel improvement operator (+3.03 points), while InfoSFT adds only +0.70 points. This supports the theoretical narrative but suggests the selective-supervision component is a modest refinement rather than a transformative addition.
Appendix C reports extensive hyperparameters—including the DAPO token-mean GRPO variant, search budgets ($C=4$, $S=8$, $K=10$), and InfoSFT weighting—but no code, data pipeline, or frozen value-model checkpoints are released. Independent reproduction is blocked by the absence of the pretrained VGS-AI/DeepSeek-VM-1.5B value model and its inference setup, as well as by the lack of exact generated-token counters; the paper provides only wall-clock GPU-hour aggregates. Exact chunk-boundary heuristics are described, but unspecified implementation details such as Gumbel sequential-halving phase allocation and KV-cache sharing would need to be reverse-engineered.
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the mathematical reasoning capabilities of large language models. Recent work introduces search into RLVR rollouts to increase trajectory diversity, but diversity alone does not ensure that the search-induced rollout policy improves upon the current policy. To address this gap, we propose APIVIS, a training-time framework that adapts finite-budget Gumbel search to chunk-level mathematical reasoning. APIVIS combines direct and searched responses within each rollout group, allowing improvements found by search to produce informative relative rewards. It further applies selective supervision to search-improved tokens, preserving a learning signal when uniform group rewards render GRPO ineffective. We show that exact value-guided selection improves the expected verifier reward at each searched state and that this guarantee extends to the complete rollout policy, with a corresponding approximate guarantee under bounded value-estimation error. Experiments on widely recognized mathematical reasoning benchmarks and different model scales demonstrate substantial improvements over competitive search-based methods, validating the effectiveness of APIVIS.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.