Finetuning with Sampling: SFT Learns Better Than You Think
The paper tackles how to teach LLMs new skills after pretraining without erasing old ones. While RL resists catastrophic forgetting by staying on-policy, it struggles when the model cannot generate successful trajectories on its own. The authors flip the problem: instead of changing the training loss, they use an MCMC projection sampler to reshape off-policy expert traces so they look more like samples from the base model while preserving correctness. This lets ordinary supervised finetuning match or beat RL and self-distillation on science, math, and medical tasks.
The paper argues that standard SFT can rival RL and self-distillation if off-policy expert data is first transformed to be more on-policy via MCMC projection sampling. The core theory—the information projection $p_{\mathcal{C}} := \arg\min_{\pi \in \mathcal{P}_{\mathcal{C}}} \mathrm{KL}(\pi \| p)$ and its monotonic approximation via Metropolis-Hastings—is sound but largely standard; the value lies in the empirical demonstration that vanilla SFT on such boosted data generalizes better and forgets less than strong baselines across chemistry, math, and medical reasoning on 3B–7B models. The approach is elegant, but its reliance on an approximate correctness oracle and a greedy acceptance shortcut tempers the theoretical guarantees.
What holds up best is the empirical evidence that data distribution matters more than algorithmic bells and whistles. Figure 5 shows that increasing MCMC steps monotonically shrinks $\mathrm{KL}(\pi_k \| \pi_{\mathrm{base}})$ while improving downstream accuracy, directly linking the sampler's theoretical behavior to finetuning outcomes. The pass@k analysis also makes a compelling case that the method learns genuinely new capabilities rather than merely sharpening the base model: "some evaluation tasks go from zero pass rate in the base model to pass rates that are up to 67.2% or 53.1%." Finally, the cross-model ablation confirms that the sampling must be native to the learner: "finetuning, e.g., Qwen on the same base task data but boosted for Olmo severely underperforms all baselines."
The strongest caveat is that the theoretical sampler and the implemented sampler are not the same. Appendix C.2 admits they replace the MH acceptance criterion with a greedy heuristic: "We approximate the swapping condition by swapping whenever we generate a higher likelihood candidate than the current." This invalidates the MCMC convergence guarantees and turns the algorithm into a greedy local search, yet the paper still leans on Propositions 2–3 as justification. A second limitation is the oracle dependence: the equivalence class $\mathcal{C}$ is assumed known, but in practice medical correctness is judged by GPT-5-mini, and chemistry/math rely on automatic verifiers that may not generalize to open-ended domains. Finally, cost remains a barrier: Equation (9) places the token cost at roughly $|\mathcal{D}| \cdot N_{\mathrm{MCMC}} T^2 / (4B)$, which grows quadratically with trajectory length and can be prohibitive for large posttraining datasets.
The claims are generally backed by fair head-to-head comparisons with relevant on-policy baselines. The gap with RL methods is most striking on chemistry and medical tasks, where Sampling SFT outperforms OPSD and vanilla SFT. On math, however, the picture is mixed: Sampling SFT achieves a stronger MATH500 boost than GRPO or UFT, but it lags UFT on AMC (0.277 vs 0.293), and the best result comes from combining Sampling SFT with RL (Sampling SFT + RL), suggesting sampling alone may not supplant RL on the hardest domains. The paper accurately cites Chen et al. (2025) in attributing RL's forgetting resilience to on-policy data: "the use on-policy data underlies the robustness of RL to forgetting in practical settings." Comparisons to Shenfeld et al. (2026) / OPSD are also apt, as both leverage demonstration-conditioned generation, though the authors correctly distinguish their data-centric approach from SDFT's teacher-student KL loss.
Reproducibility is partially aided by public code and released prompts, but several gaps remain. Hyperparameter ranges are given and the official repository is linked, yet exact random seeds, checkpointing schedules, and H100/H200 hours are omitted. More importantly, the Appendix C.2 approximation means reproducing the exact boosted dataset requires reimplementing an unstated greedy acceptance rule rather than textbook MH. Medical grading relies on an undisclosed GPT-5-mini configuration, introducing variability unless the temperature and system prompt are pinned. Scaling beyond the reported 3B and 7B models is also unverified.
Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting. At the same time, SFT can learn from off-policy expert data, whereas RL must rely on a model's ability to find successful trajectories with repeated sampling. In our work, we seek to leverage the strength of on-policy learning while utilizing the privileged information contained in off-policy data. However, rather than modifying the learning objective to accommodate this data, we instead tailor the data distribution to better suit the learner. We introduce a Markov chain Monte Carlo (MCMC) sampling algorithm that progressively transforms off-policy traces to be more on-policy given a reference model for finetuning. Across tasks like scientific skill acquisition, mathematical reasoning, and open-ended expertise, our sampling algorithm enables SFT to rival prevailing posttraining techniques, often generalizing better and forgetting less than strong on-policy baselines. In addition, the resulting finetuned models exhibit strong distributional performance and are capable of learning beyond sharpening the base model distribution. At a higher level, our approach presents sampling as a model-native operator that shapes data for learnability, offering broader utility as a general-purpose primitive throughout the posttraining stack.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.