Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning

The theoretical groundwork is solid and specific. The paper derives exact KL thresholds for losing a prescribed margin (Theorem 1), characterizes the feasible logit set via an order cone (Theorem 2), and proves a finite-step condition for aggregate violation reduction (Proposition 3). The separation from KL matching is crisp: "The two objectives therefore assign different costs to reference deviations and define different preservation targets" (Section 5.1). Experiments are well-controlled: matched seeds, shared hyperparameters, a held-out cache for diagnostics, and entropy-matched temperature controls to rule out trivial sharpness explanations. The consistency across three seeds is strong: "Across the three training seeds, SRP-SFT remains above CE and LP-SFT" (Section 7.1).

The headline setting ($K=10, \rho=1, \mu=1$) is not the best-performing configuration. Sensitivity sweeps find higher peaks at $\rho=2.5$, $K=5$, and $\mu=3$, yet the paper reports the pre-fixed main setting, which raises questions about how much the gains depend on tuning. More critically, "the task score and the realized margin statistics move in opposite directions" (Section 7.4), meaning stronger preference preservation does not reliably improve downstream metrics. Cross-backbone transfer is uneven: "Mathematics is more backbone-dependent: SRP-SFT leads MATH-500 on both Qwen bases, Ministral, and SmolLM3, but trails on Gemma (61.59 versus 63.94) and Granite" (Section 7.6). The main study is limited to one-epoch training on $\leq$4B models, and the ten-epoch trajectory for a different $\rho$ value shows early decline, leaving open the convergence picture at the headline $\rho=1$.

The comparison to LP-SFT is fair and matched: identical initialization, data order, hardware, optimizer, and seeds. The held-out cache diagnostics are a strength. SRP-SFT "reduces mean margin contraction from 0.268 for LP-SFT to 0.110 and reduces inversions from 7.41% to 5.65%" while LP-SFT "is closer in local KL, at 0.049 versus 0.075 for SRP-SFT" (Section 7.3), directly supporting the thesis that lower KL does not imply better margin preservation. An entropy-matched temperature scaling control shows "Global scaling recovers less than half of SRP-SFT's gain, so the gain is not a byproduct of a sharper output distribution" (Section 7.5). The 14-method loss screen (Table 4) robustly supports the chosen hinge formulation over ranking-only or exact-gap-matching alternatives.

The training protocol is carefully documented: one epoch, 478 AdamW steps, learning rate $2\times 10^{-5}$, cosine schedule with 15 warmup steps, batch size 128 on eight A800s. Appendix D contains detailed loss configurations and the evaluation protocol. "Reference token IDs and logits are cached for fixed teacher-forced examples... Each update gathers student logits and recomputes hinge activity without another reference pass" (Section 4.4). Algorithm 1 is provided in the appendix. However, the main text does not mention a public code release, and the method depends on a cached reference logit pipeline whose exact preprocessing would need to be reproduced independently. The headline hyperparameters ($K=10, \rho=1, \mu=1$) are fixed before evaluation, but the sensitivity sweeps reveal different peaks, so reproducing the exact headline numbers requires using the stated pre-fixed values rather than the best-tuned ones.