Calibration and Risk-Budgeted Decisions with System One Models
LLM workflows rely on fast binary decisions like moderation and reranking, but autoregressive generation is too slow for tight latency budgets. This paper studies System One models—lightweight scorers that emit bounded decision scores—and proposes Learn Then Test (LTT) to certify thresholds that maximize the fraction of inputs receiving the utility-maximizing action while controlling the False Omission Rate (FOR) with a finite-sample guarantee. The authors show across six datasets that naive thresholding at the target risk level exceeds the budget in roughly half of settings, whereas LTT violates it in only 7.8% of cases, making a strong case that a probability-like interface is insufficient for safe deployment under asymmetric risk.
The paper presents a compelling and timely argument: aggregate calibration metrics are insufficient for high-stakes deployment, and rigorous finite-sample risk certification is necessary. The theoretical specialization of LTT to conditional binary risk is clean and properly attributed, and the empirical evaluation is broad across three task families. The central conclusion is well-supported: as the authors state, "safely deploying automated decisions under asymmetric risk needs task-specific validation and rigorous risk-budgeting, rather than relying on the assumed universal calibration of any System One model."
The strongest element is Proposition 1, which decomposes the population FOR into $R(\tau)=\mathbb{E}[S\mid S<\tau]+\mathbb{E}[m(S)-S\mid S<\tau]$, making explicit why naive thresholding fails when scores are uncalibrated. The empirical sweep is unusually comprehensive, and the results are striking: the authors show that "LTT produces substantially fewer and smaller risk exceedances (just 7.8% frequency, mean excess 0.1%) because its statistical certification structurally defaults to non-deployment (ND) rather than operate unsafely."
Serious reproducibility barriers exist because the experiments rely on undocumented proprietary API endpoints (e.g., GPT-6-Luna, Span-01) and an unreleased fine-tuned checkpoint, with no code repository or prompt templates provided.
Methodologically, the guarantee relies on calibration and deployment examples being drawn i.i.d. from the same distribution $P$, yet the benchmarks "do not establish performance under deployment or adversarial shift." Moreover, exact binomial certification imposes a fundamental sample-size requirement: under strict budgets the authors note that LTT "frequently defaults to non-deployment ($\bot$), trading policy availability for safety," limiting practical utility when calibration data is scarce.
The evidence strongly supports the core claim that naive and empirical rules are unsafe, while LTT controls risk. The cross-task comparison is fair in showing no single model dominates, and the authors correctly observe that "Jev's ECE degrades from 0.034 on WildGuardMix to 0.29 on SciDocs. Universal claims of 'calibrated probabilities' obscure this severe variability." However, the paper omits post-hoc calibration baselines such as temperature scaling, which Guo et al. found "surprisingly effective at calibrating predictions," despite citing that work to argue training-time calibration is unreliable.
Independent reproduction is blocked by proprietary API endpoints without released prompts and by the lack of a public code repository. While the Kev-ft protocol is detailed—"rank-16 LoRA adapters and 256-dimensional pointer head on a frozen Qwen/Qwen3.5-0.8B-Base backbone. Training uses two epochs, seed 0"—the threshold construction depends on "$\mathcal{T}$ is constructed from up to 200 sorted fit-score midpoints on $D_{\mathrm{fit}}$, plus the endpoints $\{0,1\}$," which is not shared, and proprietary scorer behavior cannot be audited.
LLM workflows and agents rely on a set of binary decisions like allow/block an input, or discard/retain a retrieved document. When human deferral is unavailable, uncertain cases must still receive an action. We study System One decision scores in this setting by defaulting to a conservative fallback when uncertain, and controlling the false omission risk within the subset of inputs assigned the utility-maximizing action. Across six datasets covering moderation, safety, and reranking, we compare general System One models with task baselines and a local open-weights fine-tuned System One model. We show that naive baselines for threshold selection exceed the target risk in roughly half of settings with defined test risk, compared with 7.8\% for Learn Then Test (LTT), which leverages finite-sample statistical guarantees. Our results show why relying on a probability-like interface is insufficient for deployment decisions and how risk-budgeted selection trades useful action against observed error.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.