One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation
On-policy self-distillation improves reasoning by training a model on its own trajectories with dense token-level guidance from a reference-conditioned teacher. The authors identify a critical flaw: because any single reference is just one valid realization of reasoning, conditioning the teacher on the full reference makes the target distribution overly sensitive to that specific solution's wording, ordering, and path. They propose Monte Carlo Privileged Distillation (MCPD), which averages the teacher's next-token distributions over multiple randomly masked partial views of the same reference to produce a more stable marginal target. Across math and science benchmarks, MCPD consistently outperforms standard full-reference distillation.
The paper delivers a clear diagnostic and a principled fix: it formally identifies reference-induced target instability, proves that probability-space averaging over partial views is the exact minimizer of an expected-view KL objective, and shows consistent accuracy gains across four backbones and multiple benchmarks. However, the work stops short of explaining why random token masking preserves useful privileged information rather than corrupting it, and it does not test whether supplying multiple complete alternative references at training time would resolve the instability more directly.
The theoretical justification is unusually rigorous for a language-model training paper. Proposition 1 shows that averaging teacher predictions in probability space is not an ad-hoc smoothing heuristic but the exact target arising from minimizing the expected teacher-to-student KL over random privileged views: $\mathcal{J}_t(p) = \mathbb{E}_M[\mathrm{KL}(q_t^M \| q_t^\star)] + \mathrm{KL}(q_t^\star \| p)$. The empirical evaluation is also tightly controlled; by holding the problem, student trajectory, and evaluation positions fixed, the authors isolate reference choice as the sole driver of observed sensitivity differences.
The authors candidly note that reduced cross-reference sensitivity is purely empirical and not theoretically guaranteed: $q_t^\star(\cdot \mid r^a) = q_t^\star(\cdot \mid r^b)$ does not follow from variance reduction alone. More practically, the partial-view operator is a random token-level mask with no structural constraint, meaning a masked reference can degrade into a mathematically incoherent fragment that offers little useful guidance. The method also incurs a linear compute cost in the number of views $K$, and while the overhead is modest at $K=4$, wall-clock time and memory scale directly with larger ensembles or longer references.
The evidence generally supports the claims. The matched-pair sensitivity design using Jensen-Shannon divergence is symmetric, bounded, and well-motivated, and the entropy analysis in Appendix A usefully rejects the target-flattening alternative: MCPD's mean entropy is comparable to OPSD (0.33 vs. 0.34 nats) and is actually lower on 75.6% of problems. The comparison to AVSD is fair on headline numbers but lacks a mechanistic or qualitative analysis of where and why consensus-based gating fails relative to marginalization.
Hyperparameters and training recipes are reported in solid detail: LoRA rank 64, learning rate $5 \times 10^{-6}$, masking ratios (30% for Qwen3, 40% for DeepSeek), batch size 32, and exact hardware (four A100 80GB GPUs). Algorithm 1 provides a clear forward-pass specification. Still, no public code release or checkpoints are mentioned, and several steps rely on model-specific chat-template flags such as enable_thinking=True. Without released code, exact random seeds for mask banks, or masked-token insertion logic, independent reproduction would require non-trivial implementation guessing.
On-policy self-distillation (OPSD) improves reasoning by training a model on its own trajectories with dense token-level supervision from a privileged copy conditioned on a reference solution. Existing methods typically use a single teacher distribution conditioned on the complete reference. We identify a limitation of this formulation: a reference is only one realization of the reasoning process. Although it provides evidence about the answer, it also encodes an ordering, phrasing, and solution path. Full-reference conditioning can therefore produce correct targets that depend excessively on information unavailable to the student at inference. We show that replacing one valid reference with another, while holding the problem and student state fixed, can substantially alter the teacher distribution, a phenomenon we call reference-induced target instability. To address this instability, we introduce Monte Carlo Privileged Distillation (MCPD), which models privileged context as a distribution over partial reference views and distills the marginal teacher distribution. MCPD evaluates the teacher under multiple views and averages the resulting next-token distributions. We show that this probability-space average is the exact target induced by minimizing the expected teacher-to-student KL divergence over privileged views. Across mathematical and scientific reasoning benchmarks, we show that MCPD produces more reference-consistent supervision and consistently improves over full-reference OPSD.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.