VASPO: Self-Distillation with Privileged Value Estimation
On-policy self-distillation gives reasoning models dense token-level supervision via a teacher that sees reference solutions, but this signal can drift toward style and ignores whether the student can finish correctly on its own. VASPO repurposes the same privileged reference inside PPO to train both a prefix-value critic and a bounded self-teacher that reweights selected positive advantages. The paper reports average gains of roughly 2.6 percentage points over PPO on math benchmarks and leading accuracy on Knights and Knaves.
VASPO is a technically clean actor-critic design that conditions the PPO critic on privileged solutions to improve continuation-value estimation, while a bounded self-teacher modulates only selected positive GAE advantages. The gains are consistent across Qwen3-4B and OLMo3-7B, and the positional diagnostics offer a plausible mechanistic story. However, the headline improvement is driven primarily by the privileged critic rather than the teacher modulation: on Qwen3, dropping self-teacher modulation reduces the math average by only about 0.5 pp (55.74 vs 56.28 in Table 3), whereas dropping the privileged critic collapses performance to OPSD levels (35.59). This makes VASPO read as privileged value estimation plus a thin reweighting layer rather than a balanced fusion of two signals.
The paper's strongest element is its empirical characterization of where privileged information helps. Section 2.3 documents complementary positional dynamics: the privileged critic reduces squared error most toward the response end, while the self-teacher diverges strongly from the actor early, weakly in the middle, and rebounds at the end. The algorithmic details are also carefully constrained: the teacher weight $w_t = \operatorname{clip}(e^{d_t}, 1 - \epsilon_w, 1 + \epsilon_w)$ is bounded, the mask $M_t$ applies only to tokens with positive advantages $A^z_t > 0$, and the fused advantage satisfies $1 - \epsilon_w \leq A^{z,\mathrm{f}}_t / A^z_t \leq 1 + \epsilon_w$, which preserves the sign of the critic-derived update.
A central limitation is that the core contribution is lopsided in practice: the privileged critic accounts for the bulk of the gain, while self-teacher modulation adds comparatively little. The positional analysis, while suggestive, relies on fixed-response and frozen-actor diagnostics; it does not establish that these patterns causally drive training improvements rather than offer a post-hoc description. Additionally, feeding both the critic and the teacher the same full solution introduces correlated noise risk.
The authors acknowledge that privileged solutions "may still harbor flawed intermediate reasoning steps that could introduce noisy supervision." On Knights and Knaves, gains are smaller and absolute accuracy on the 11-role out-of-distribution test is below 41% for all methods, limiting the breadth of the claimed robustness.
The main experiments compare VASPO against PPO, DAPO, OPSD, and RLSD using shared data, initialization, and pretraining protocols, which supports internal fairness. VASPO leads on the six-benchmark math average, though against the strongest GRPO-based teacher baseline RLSD the margin on Knights and Knaves is narrow (60.81 vs 60.34) and shrinks further on 9-role puzzles. The evaluation pipeline uses a two-stage verifier at test time (math-verify then CompassVerifier-3B), adding complexity; while shared across methods, the auxiliary verifier could inject systematic biases. The headline metric averages across benchmarks with different sample sizes (avg@32, avg@8, avg@16), which is standard but conflates pass@K performance with raw accuracy differences.
The authors publish code and provide detailed hyperparameters in Appendix D.4 and F, including AdamW with actor/critic learning rates of $10^{-6}$ and $5 \times 10^{-6}$, global batch size 128, eight responses per prompt, and PPO clipping $\epsilon_{\mathrm{clip}} = 0.2$ with reweighting clip $\epsilon_w = 0.2$. Training was implemented in verl and required 200 rollout updates on $8 \times$ H200 GPUs, so reproduction demands substantial resources.
The training-time profile shows student rollout is the dominant cost at 191.97 s per step, with the critic update adding 66.05 s. A notable omission is the exact count of unique training prompts seen over the 200 steps; without this, it is hard to assess sample efficiency or confirm that the reported last-checkpoint rule avoided implicit test-set tuning.
On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.