Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
On-policy distillation (OPD) trains a student on its own rollouts using token-level supervision from a teacher, but its effectiveness relies on the teacher's ability to recover from any student prefix. The authors show that this recoverability collapses on incorrect rollouts: the teacher follows the student's errors rather than correcting them, even though it can still recognize them. Their method, CriticOPD, restores recoverability by having the teacher critique an incorrect rollout so the student can rewrite its solution from the last error; the teacher scores the rewrite with the critique as privileged information, while the student trains on the revision without it. On six mathematical reasoning benchmarks, CriticOPD improves standard OPD by roughly 4–6 points across two student scales, with the largest gains on competition-level problems, and the student needs no feedback at inference.
The paper offers a compelling conceptual diagnosis backed by careful measurements. The notion of recoverability is well defined, the asymmetry between teacher continuation failure and error recognition is striking, and the algorithm is elegant. Nevertheless, the evaluation is limited to mathematics with verifiable answers, the cost comparisons rely on favorable early stopping, and the diagnosis leans on an unvalidated LLM judge.
The diagnostic analysis in Section 2 is unusually rigorous. The recoverability metric is precisely formulated, and the evidence supports the core asymmetry: error-recognition precision stays stable at 73–81% even as continuation accuracy plummets. The ablation in Table 2 is decisive—removing the critique drops performance by 5.90 points, while answer-only or generic feedback are far less effective—and the hidden-state occupancy-mismatch analysis in Appendix B.6 goes beyond standard position-based comparisons.
The method is narrowly applicable because it requires verifiable reference answers both to trigger critiques and to ground the critic, as the authors acknowledge. Extending it to open-ended domains would need a separate reward model or self-critique, which is untested.\n\n Several methodological issues deserve scrutiny. The diagnosis relies on an unvalidated model judge (Qwen3-8B in thinking mode) that may disagree with humans, and even this judge flags 19–27% of teacher critiques as invalid. The reported 0.42x training cost relative to OPD is largely an artifact of stopping at step 40 instead of 80; wall-clock time per step is higher and grows as rewrites lengthen. Finally, students trained with CriticOPD generate responses nearly twice as long as OPD students, so part of the accuracy gain is confounded by length inflation, though the authors attempt to control for this.
The evidence supports the central claims well. Figure 6 shows that teacher recoverability on rewrites jumps from 0.14 to 0.67 after the cut point when the critique is present, and Table 1 reports consistent gains of 4.06 and 4.57 average points at the two scales.\n\n Comparisons to FastOPD and RelayOPD are fair and favor CriticOPD, but the comparison to Critique-GRPO is unequal because the latter uses a different objective, a different rollout budget, and is evaluated at an earlier step. A same-objective, compute-matched baseline would strengthen the comparison.
Reproducibility is solid on paper but expensive in practice. The authors release code, list all prompts verbatim in Appendix A, and report detailed hyperparameters in Table 3. However, exact reproduction requires four A100 GPUs, a specific vLLM/verl software stack, and sizable model checkpoints. Variable per-step timing—ranging from roughly 170 s early in training to 870 s later—means total cost estimates depend heavily on the selected checkpoint and the variable length of student rewrites.
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.