How Much Can Language Models Gain from Test-Time Computation?
Test-time scaling is often proposed as a cheaper substitute for larger models, but its practical returns remain murky because prior evaluations usually focus on one domain and rarely charge selection costs to the budget. This paper introduces SELF-POT, a cross-domain benchmark and evaluation framework that charges every model call—including selection, critique, and voting—in actual dollars under fixed output-token caps. Across 350 sealed tasks in mathematics, programming, and agentic workflows, the authors show that extra inference yields wildly different returns depending on the domain and selection rule, and that the real bottleneck is often converting generated coverage into a valid submission rather than generating a correct candidate in the first place. By reframing test-time potential as a measured conversion profile tied to price and failure handling, the work connects empirical scaling behavior to concrete deployment trade-offs.
The paper offers a carefully controlled empirical contribution: it isolates candidate generation from selection, charges all inference in dollars, and compares multiple low-cost models against a strong Claude Opus 5.5 anchor across four domains. Its central finding is actionable and well supported: failure handling and selection rules can reverse apparent test-time scaling losses. On programming, joint selection under the frozen rule produced an 8.6 pp loss against Direct, while simply retaining the first nonempty candidate when judging failed turned this into a 3.6 pp gain at identical cost. However, the scope is intentionally limited to simple protocols—majority voting, same-model joint selection, and a single Self-Refine iteration—leaving open whether more sophisticated verifiers or larger sample budgets would alter the picture.
The experimental design is admirably rigorous. The test set is sealed and hashed before any evaluation run, tasks pass a model-independent eligibility check, and the framework holds the model fixed across solver, selector, and critic roles while accounting for every token. The replay analyses on saved candidate pools are the strongest element: by separating coverage (whether any generated candidate is correct) from conversion (whether the selection stage submits it), the authors show that many aggregate 'gains' or 'losses' are really selection artifacts. For instance, public-example selection on programming converts 453 of 457 covered pools, outperforming the model judge while cutting API cost.
The protocol suite is narrow and the agent results are especially fragile. The workflow configurations for GPT-OSS and MiniMax suffer stage failures on 117/160 and 159/160 tasks respectively, so these cells mainly measure interface reliability and malformed outputs rather than the value of multi-plan reasoning. The paper also leans heavily on post-hoc replays that swap selection rules on saved candidate pools; while diagnostic, these are explicitly not equal-cost trials and therefore cannot be treated as deployable protocol alternatives. The absence of larger sample counts, learned verifiers, or iterative sequential strategies means the claim that 'gains depend on domain and selection' is well supported, but the claim about the absolute ceiling of test-time potential is not.
The evidence supports the paper's key descriptive claims, and the comparison to prior work is fair but understated. The authors correctly note that earlier studies mostly evaluate one domain at a time, budget in FLOPs or samples, and compare models only within the same family, whereas SELF-POT unifies dollars, sealed tasks across four domains, and cross-family anchors. The trade-off is that some domains have small counts—only 30 terminal tasks and 60 math tasks—which limits precision and cross-domain generalization. The comparison to Claude Opus 5.5 is also complicated by runtime details: Opus's coding score rises from 93% to 99% simply by adding numpy to the sandbox, which undermines any strong claim that a low-cost model with scaling has 'matched' the anchor.
Reproducibility is moderately strong on protocol specification but weaker on artifact availability. The authors document pseudocode, budget caps, model pricing, failure handling, and four post-sealing infrastructure amendments with unusual transparency. Yet independent reproduction would still face hurdles: the exact candidate pools are not released, provider-specific rate limits and interface fixes (e.g., Bedrock validation-error handling) altered effective behavior after sealing, and the paper notes that 'nominal cell allowances do not imply identical realized inputs and stage limits across interface versions.' Charging every call logically rather than physically, while methodologically clean, also means the reported dollar costs are ledger estimates, not reproducible invoices.
How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.