Function-Space Temporal Self-Distillation for Policy Optimization
Reinforcement learning with verifiable rewards suffers from coarse scalar supervision and moving-target instability in self-distillation. FSPO eliminates the persistent teacher model by storing sparse historical next-token distributions as fixed function-space targets, combining them with privileged disagreement correction and reliability-adaptive grounding. Across scientific reasoning and tool-use benchmarks, it achieves roughly 4% higher average accuracy than SDPO under a 5-hour budget while cutting peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.
FSPO is a well-engineered and conceptually clean extension of on-policy self-distillation. Its central innovation—materializing historical privileged predictions as sparse fixed function-space targets rather than maintaining a persistent parameter-space teacher—is convincingly shown to stabilize training and reduce memory. The empirical average gains of about 4 percentage points are solid, though they mask task-level variance, and the proliferation of new hyperparameters is a notable practical downside.
The function-space temporal consolidation idea is elegant and strongly supported by ablations. The authors show that sparse cached targets outperform parameter snapshots, EMA teachers, and frozen initial policies, with several alternatives performing worse than ablating temporal consolidation entirely. This indicates that simply preserving past parameters is insufficient, and that selective sparse function-space replay is doing meaningful work.
Despite the consistent average gain, the improvements are uneven across tasks and models. On Qwen3-8B Tool Use, FSPO matches but does not exceed SDPO at the 5-hour budget, and at the 1-hour budget it trails SDPO on both backbones for that task. More importantly, FSPO introduces numerous sensitive hyperparameters—including disagreement margin $\gamma$, refresh delay $H$, staggered fraction $\zeta$, coverage weight, and cyclic schedule parameters—and the ablations reveal sharp sensitivity: strong reference grounding with $\lambda_{\mathrm{CE}}=1.0$ collapses average accuracy to 55.3%, near the distillation-only baseline. The paper also does not explain why top-$K$ sparsification with a pooled tail bucket suffices to approximate the full teacher distribution without systematic bias.
The baseline comparisons rely on numbers taken directly from the SDPO paper rather than independent reruns. While the authors claim matched settings, this prevents strict control for implementation details and random seed variation. There is also no theoretical analysis of convergence under the moving-target or fixed-target regimes.
The evidence supports the headline claims about memory and average accuracy but is limited in scope. With only two model backbones and five tasks, generalization to larger scales or different domains is unverified. The comparison to SDPO is fair in that FSPO explicitly addresses a limitation acknowledged by that work—the EMA teacher overhead—but the use of imported baseline results weakens causal confidence. The paper would be stronger with at least one independently rerun strong baseline and a comparison to simpler trajectory-replay variants like behavior cloning on stored successful rollouts.
Full reproduction would be difficult. The paper lists hyperparameters in Table 5 and describes the setup in Appendix A.4, but it does not mention released code or data preprocessing scripts. The structural signatures used for coverage-aware memory retention are only loosely specified, and the temporal cache state depends on sampled rollouts and staggered refreshes, which introduces run-to-run variability. Experiments require four NVIDIA GH200 GPUs and wall-clock time budgets, creating a high hardware barrier.
Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.