Sharpening Tax in Post-Training
Does RL post-training create new agentic capabilities or just redistribute probability mass toward high-reward behaviors that a base model already possesses? This paper shows that, across 14 base/post-trained checkpoint pairs and three process-centric agentic benchmarks, pre-trained base models equipped with a lightweight harness often surpass their post-trained counterparts in solution coverage (pass@$K$) given enough test-time compute. To diagnose this coverage-accuracy trade-off, the authors propose Sharpening Tax, a single-number metric for the loss in test-time scalability after post-training, and present Posterior-Tempered Group Sampling (PTGS), a Bayesian sampler that mitigates the tax during RL training.
This is a timely and well-executed study that extends the sharpening debate from math and coding to multi-turn agentic tasks. The empirical finding that harness-equipped base models overtake post-trained models in coverage is carefully documented across four model families. The Sharpening Tax formulation ($\text{Tax}_S(K)$) is clean and predictive. However, the observational design—relying on open-source checkpoints with undisclosed training recipes—limits causal claims about RL specifically, and the PTGS intervention is validated on only a single model in two toy RL environments, leaving its generality open.
The core observation is robust: base models with a simple harness catch up and surpass post-trained models in pass@$K$ as $K$ grows, across four families and three benchmarks (Figure 2). The distributional analysis is compelling—post-training bimodalizes per-task success rates, pushing mass from the intermediate region to the always-pass and always-fail extremes (Figure 4). The Sharpening Tax has a clean probabilistic interpretation as the expected number of failed attempts before the first success (Proposition 1), and PTGS demonstrates concrete joint gains in accuracy and coverage (Table 5).
The harness design is a confounder: the authors deliberately add scaffolding for base models that 'hurts the post-trained models' performance,' so the crossover is partly engineered rather than purely emergent. This makes the 'surprising' framing misleading. Second, PTGS is tested only on Qwen2.5-7B in two toy RL environments—far too narrow to support claims about general agentic RL. Third, the paper infers that post-training 'merely amplifies a few rewarding behaviors,' but this is an unsupported leap from coverage metrics to mechanism; low coverage could equally reflect poor elicitation of genuinely new capabilities that are too concentrated to dominate pass@$K$.
The paper fairly positions itself against prior work limited to math and coding, and the shift to process-centric benchmarks is a genuine improvement over answer-only evaluation: 'task success is determined by the environment state that the model reaches through its intermediate actions, not by matching a final answer string.' However, the central claim that post-training does not expand agentic reasoning boundaries confuses coverage (pass@$K$) with capability. A post-trained model might acquire new skills that are simply too concentrated in a narrow mode to dominate coverage against a broader but less reliable base model.
The authors release code and provide detailed PTGS hyperparameters, and the tax metric is easy to compute. However, full reproduction of the main findings is blocked by two issues: the exact post-training recipes are proprietary ('The exact training recipes of these models are not fully disclosed'), so the causal mechanism cannot be independently verified, and reproducing the main results requires 128 rollouts per task across 14 models on three benchmarks, demanding substantial inference budget. The PTGS experiments are more reproducible since they start from a known base model, but only cover two simple environments.
An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.