PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning
Offline supervised fine-tuning for tool-using agents improves target tasks like software engineering but typically degrades existing capabilities such as general reasoning, tool use, and code generation. This paper proposes Privilege-Guided SFT (PG-SFT), which uses the next action in a trajectory as training-time privileged information to estimate turn-level information gain and adapt supervision strength. Instead of uniformly anchoring to the base model, PG-SFT interpolates between the base distribution and the demonstrated token based on how action-deterministic each turn is. The work matters because it shows that balancing capability acquisition and retention requires deciding where and how strongly to deviate from base behavior, not merely limiting overall drift.
PG-SFT offers a well-motivated, adaptive alternative to uniform SFT regularization for offline agent fine-tuning. The paper succeeds in demonstrating that turn-level privileged signals can achieve a more favorable acquisition-retention trade-off than global KL constraints or trust-region clipping. Nevertheless, the study is scoped tightly to two 4B Qwen backbones, a single SWE-bench-style agent domain, and a handful of training runs, which limits the generality of its conclusions. The authors appropriately caution that their empirical measurements do not establish causality between distributional drift and downstream forgetting.
The experimental design carefully isolates the supervision objective by holding data and supervised tokens constant across all methods. The fixed-intensity ablation is especially convincing: matching PG-SFT's average learning intensity with a uniform coefficient produces similar teacher-forced NLL and Base KL, yet yields worse downstream target performance and mixed retention, confirming that the adaptivity of turn-level allocation is crucial. The privileged signal itself is intuitive and grounded in the structure of agent trajectories, where a turn is the smallest complete decision unit.
The primary limitation is narrow empirical scope. All experiments use 4B-parameter Qwen models on software-engineering agent trajectories, and the target benchmark is a small, difficulty-screened 89-instance subset of SWE-bench Lite that is not comparable to standard leaderboards. Baseline comparisons are partially confounded by differing optimization budgets: PSFT uses a custom schedule and learning rate, and SFT+Base-KL uses only 32 steps versus 156 for PG-SFT on Qwen3. The authors acknowledge that distributional drift and capability retention are only correlated, not causally linked. Additionally, the privileged hint assumes that the demonstrated next action is the key determinant of turn content, which may fail for exploratory or erroneous trajectories.
The core evidence supports the claim that adaptive supervision outperforms uniform conservatism: PG-SFT improves Macro Avg over Base while SFT and SFT+Base-KL degrade it. However, some baseline comparisons are weakened by mismatched training budgets. The dismissal of DFT is cursory—only a single target-task score is shown with conjectural explanations. The three non-target benchmarks (GPQA, BFCL, LiveCodeBench) provide a useful breadth check, but four tasks total are insufficient to claim robust broad-capability retention. The paper's observation that fitting, drift, and retention are distinct properties is well-supported by the offline statistics, yet the small model scale and limited domains mean the conclusion should be treated as a promising proof of concept rather than a universal principle.
Reproducibility is partially strong: code is openly released and training hyperparameters are meticulously documented in Appendix B, including LoRA rank, learning rate, batch size, and the exact 1K/5K trajectory splits. The reliance on a single random seed (42) and absence of repeated runs or error bars are notable weaknesses. The SWE-bench evaluation protocol uses a non-standard 89-instance subset with conditional pass@3 scoring, which the authors admit is not comparable to the official leaderboard. The custom chunked full-vocabulary KL implementation (Appendix A) adds engineering complexity that must be replicated exactly. The high-end hardware (two NVIDIA RTX PRO 6000 Blackwell GPUs) is specified but may limit independent verification.
Supervised fine-tuning (SFT) on offline agent trajectories is the standard approach for training specialized tool-using agents, but forcing models to imitate reasoning and actions token by token may harm other capabilities (e.g., general reasoning, tool calling, code generation) of the base model. In this work, we focus on studying \emph{how to better balance the trade-off between acquiring new capabilities and preserving existing ones during agent trace SFT}. By comparing several baselines in our setup, standard SFT improves the target benchmark while lowering several non-target benchmark scores; meanwhile, simply constraining distributional drift using KL penalty or limiting the update magnitude did not avoid this regression trend. Motivated by recent token-wise adaptive learning objectives, this work proposes \textbf{Privilege-Guided SFT (PG-SFT)} to leverage turn-level information gain of agent trajectories as an indicator to adjust supervision strength. PG-SFT yields a more favorable observed trade-off on the evaluated benchmarks, substantially reducing distributional drift and broad capability degradation at the cost of slight degradation in target-task performance. Our findings suggest that balancing the acquisition--retention trade-off depends not only on whether the model is anchored to its base behavior, but also on where and how strongly supervision should depart from that behavior.}
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.