More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
This paper confronts the evaluation crisis in agentic self-distillation, where a flood of recent methods claim gains over GRPO using inconsistent protocols and reused baselines. The authors conduct a unified benchmark of ten representative approaches across four tasks, three model sizes, and 132 training runs totaling over 12,000 H800 GPU-hours, measuring not only final accuracy but also training stability and wall-clock cost. Their central finding is sobering: no method consistently beats GRPO, many suffer severe hidden instability, and extra compute rarely translates into better agents.
The paper delivers a rigorous and much-needed corrective to a fast-growing literature. By training every method under a common protocol and evaluating each checkpoint with eight rollout seeds, the authors demonstrate that reported gains are largely setting-dependent and that aggregate win counts can conceal catastrophic regressions. They argue convincingly that independent evaluation should take priority over proposing another training algorithm.
The experimental execution is exemplary. The sheer scale of 12,124.4 recorded H800 GPU-hours, 2,772 validation sweeps, and 1,152 evaluation sweeps provides an unprecedented common basis for comparison. The authors go beyond final accuracy to analyze training trajectories, peak-to-final declines, and wall-clock cost, exposing instabilities that endpoint-only studies miss. Their finding that the highest aggregate performer leads no individual setting underscores why breadth of evaluation matters.
While the benchmark is admirably comprehensive, its scope is necessarily limited to ten methods and three Qwen3.5 scales, leaving open whether the patterns generalize to other model families. The study sometimes conflates method-specific collapses with broader claims about self-distillation: severe failures in GRPO+OPSD or ICSD on specific tasks may reflect poor hyperparameter transfer rather than inherent algorithmic limits. The paper also documents instability meticulously but offers limited causal analysis beyond noting that additional training cost does not consistently yield better performance.
The evidence strongly supports the central claim that prior literature overstates broad progress. The authors effectively dismantle the common practice of reusing baseline scores from earlier papers, showing that under a common protocol, eight of ten alternatives fail to improve GRPO’s twelve-setting mean. The demonstration that fixed-step advantages can vanish under matched time budgets is a particularly important empirical contribution that undermines many prior claims of efficiency.
Reproducibility is a major strength: the authors release a public GitHub repository, project page, and detailed experimental records covering configurations, per-step metrics, and repeated-evaluation summaries. Algorithmic details are spelled out across multiple appendices with explicit hyperparameters and pseudocode for all eleven methods. However, independent reproduction would still require substantial H800 GPU resources, and some method-specific settings had to be adapted or extended to benchmarks not tested in the original papers, which could introduce subtle implementation variance.
Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: https://aim-uofa.github.io/Self-Distillation-AgentRL/
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.