Reverse-KL Policy Distillation: Statistical Guarantees and Limits
This paper initiates the statistical theory of reverse-KL policy distillation under misspecification, asking whether the teacher's full action distribution improves efficiency over action-level feedback and whether online student trajectories are necessary versus offline teacher data. The authors establish minimax-optimal rates for full-feedback settings, develop clipped importance-weighting and trajectory-truncation algorithms, and prove a striking separation: offline action feedback can match full-feedback rates, while online action feedback requires exponentially more data. The results give a first theoretical footing for empirical design choices in modern LLM distillation.
The paper offers a valuable first characterization of reverse-KL distillation, though its claim of a "rather complete picture" overstates the results. The authors prove tight minimax bounds for online and offline full feedback (Theorems 3.2 and 3.3) and show offline action feedback attains comparable rates (Proposition 4.3), yet they provide only a lower bound—no upper bound or algorithm—for online action feedback, leaving Table 1 with a conspicuous blank. The new algorithmic tools are theoretically sophisticated but computationally prohibitive for the neural-network policy classes used in practice.
The strongest results are the optimal online full-feedback algorithm and the conceptual separation between online and offline action feedback. Algorithm 1 uses clipped prefix importance weights and trajectory-loss truncation to correct the visitation mismatch that causes standard DAgger-style refitting to fail (Proposition 3.1), achieving the optimal $O(B[L^\star + \log|\Pi|/T])$ rate under a relative second-moment assumption. The insight that teacher-generated actions reveal preferences while student-selected queries can hide them is elegant and well-motivated.
The characterization for online action feedback is incomplete: Table 1 lists no approximation coefficient and only a lower-bound estimation coefficient $\Omega(B_\infty m/\log(em))$, with no matching upper-bound algorithm. The algorithms are also computationally infeasible for rich policy classes. Algorithm 1 enumerates the entire finite class to perform an exponential-weights update, and the offline estimator requires exact policy-ratio computations; the authors acknowledge that practical implementations are future work. Furthermore, the boundedness assumptions—particularly the trajectory likelihood-ratio bound $P^\pi(\tau) \le e^{B_\infty} P^{\pi^\star}(\tau)$—can be extremely restrictive because $B_\infty$ may grow linearly with horizon $H$, effectively reintroducing horizon dependence.
The evidence supports the claims for the settings that are fully characterized. The online full-feedback upper bound matches the lower bound at $\Theta(B)$, and the offline clipped ERM achieves the same optimal rate. Comparisons to related work are accurate: fetched sources confirm Rohatgi et al. (2025) obtain constant approximation for Hellinger distance with computationally inefficient algorithms and an $\Omega(H)$ barrier for next-token methods, while Xu et al. (2026b) obtain constant approximation for forward KL with estimation error scaling under a boundedness factor. The paper correctly notes that reverse KL naturally favors teacher-coverage assumptions, whereas forward KL requires student coverage.
The theoretical procedures are fully specified: Algorithm 1 uses exponential weights with $\eta = 1/256$, and the offline estimator is a clipped ERM over $T$ teacher trajectories. However, independent reproduction is blocked by the finite policy-class assumption and the need to compute exact prefix importance ratios $P^\pi(\xi_h)/P^{\pi_t}(\xi_h)$ and enumerate all candidates at each round. No code, data, or computational experiments are mentioned. For continuous policy classes or deep networks, the guarantees do not directly apply.
Reverse Kullback-Leibler (KL) distillation has shown empirical promise for training language models, but its statistical guarantees and limits remain poorly understood. In particular, the effects of two design choices remain unclear: Does access to the teacher's full action distribution provide an advantage over observing only the probability it assigns to the sampled action? Must training trajectories come from the student (online distillation), or can teacher-generated trajectories (offline distillation) suffice? Our work takes a first step toward understanding the theory underlying these questions. We study the sample complexity of online reverse-KL distillation under two feedback models: the teacher reveals either its full action distribution or only the probability of the student-generated action. We establish minimax bounds for the former and show that substantially more data may be required for the latter. In contrast, offline reverse-KL distillation can attain the same minimax rate whether the teacher reveals its full distribution or only its probability of the teacher-generated action. The key distinction is that teacher-generated actions themselves reveal the teacher's preferences, while queries at student-selected actions can leave those preferences unobserved. This highlights an additional statistical challenge for online distillation compared to offline distillation. These statistical guarantees do not follow simply from optimizing standard distillation objectives: our lower bounds show that common online objectives can fail when candidates are evaluated only on past student-generated trajectories. We develop trajectory-loss and importance-ratio truncation techniques together with policy-space learning to achieve optimal online distillation guarantees.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.