GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL
Complex reasoning training hinges on two supervision modes—sparse RL rewards and dense distillation—but combining them naively causes optimization conflicts.
GIFT introduces a token-level geometric filter that decomposes the teacher gradient into a conflict-measuring parallel component ($a_i$) and a novel-guidance orthogonal component ($b_i$), then progressively anneals which tokens receive distillation.
This yields consistent improvements on hard math and logic benchmarks, with relative gains reaching 9.5% on the most difficult tasks.
GIFT offers a well-motivated, theoretically grounded solution to the allocation problem in hybrid RL-distillation training. The geometric decomposition into parallel and orthogonal gradients yields an efficient closed-form scoring rule, and the progressive coverage annealing provides a clean mechanism for teacher withdrawal. The empirical gains are real but concentrated: improvements are largest on the hardest AIME25 problems and more modest on easier benchmarks and logical reasoning. The reliance on privileged teacher distributions and the absence of global convergence guarantees limit the theoretical mileage.
The geometric decomposition is the paper's strongest contribution. By deriving a closed-form, $O(M)$ token-level scoring rule (Proposition 1) that avoids full vocabulary gradient materialization, GIFT makes its filtering computationally tractable. The ablation studies are unusually thorough: they isolate the exact-quota lexicographic program against random selection, entropy-KL heuristics, and disassembled variants (conflict-only, residual-only, conflict-free), confirming that both safety ($a_i$) and novelty ($b_i$) signals are required, especially on harder tasks. The acknowledgment that ATTITTUD is the direct intellectual predecessor, and the crisp articulation of three novel adaptations for the RLVR context, lend intellectual honesty.
The theoretical framework is strictly local. Theorems 1–3 characterize first-order behavior at a fixed point and do not establish global policy improvement or convergence; the irreducibility claim in Theorem 2(ii) is explicitly scoped to token-local logit-space rescalings and disclaims shared-parameter or function-space novelty, considerably narrowing the practical meaning of "complementarity."
Second, GIFT assumes access to a privileged teacher distribution conditioned on reference solutions—a strong assumption whose cost and feasibility in low-resource domains are not discussed. The paper also notes that while per-token loss weights are fixed under coverage annealing, the aggregate auxiliary gradient norm need not decrease monotonically because the retained set concentrates on the largest $b_i$, creating a potential late-stage teacher surge that undermines the intended smooth withdrawal.
The main results in Table 1 support the claim that geometric filtering outperforms scalar-modulation and unfiltered distillation baselines across Qwen3 1.7B, 4B, and 8B, with the strongest relative improvements on AIME25 (up to 9.51%). The ablations in Tables 3–5 rigorously validate design choices, showing that heuristic proxies like entropy-KL or gradient-cosine selection underperform the full GIFT rule. However, the paper omits statistical significance tests and seed variance, making it hard to judge whether margins of 1–3 points on easier benchmarks are reliable. Comparisons to RLSD, RLCSD, and OPDVR are fair in that those methods structurally cannot introduce directions outside the RL gradient span, but no wall-clock time or compute-normalized comparison is provided, so the cost-effectiveness of GIFT's extra teacher forward passes and scoring remains unquantified.
The paper promises public code and trained weights upon acceptance, but nothing is available at submission time. Training uses public datasets (DeepMath-103K, Knights-and-Knaves) and standard benchmarks (AMC23, AIME24/25, Logic-RL), which aids future reproduction. Key hyperparameters are partially visible: $\lambda=1$, coverage annealing with $H=10$ and $T_{\text{exit}}=78$, and group sizes for GRPO are referenced in Appendix H. Yet critical details—exact learning rates, batch sizes, rollout counts per prompt, and the precise teacher conditioning protocol for OPSD—are relegated to the appendices without full exposition in the excerpt. The closed-form scoring (Proposition 1) is explicit enough to reimplement, but end-to-end reproduction will require the promised codebase.
Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose \textbf{GIFT} (\textbf{G}radient-\textbf{I}nformed \textbf{F}il\textbf{T}ering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component \textbf{parallel} to the RL gradient, which measures first-order conflict and provides a \textbf{safety} filter, and a component \textbf{orthogonal} to it, which quantifies novel directional guidance and provides a measure of \textbf{value}. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive \textbf{coverage annealing} schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically-filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5\% on difficult mathematical reasoning tasks. Code: https://github.com/lilifree/GIFT
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.