Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters
Language models are increasingly used for discrete decisions like tool selection or intent classification, yet prompting LLMs for these closed-set choices is slow and sensitive to option order. This paper introduces Crivo, a family of compact encoders scaling from 17 million to 1 billion parameters, and runs a controlled study to determine whether scoring each option in a separate encoder pass outperforms reading all options jointly in a single sequence. The work matters because it tests the limits of parameter efficiency and architectural choices for high-speed decision nodes that must generalize across changing label sets.
The paper delivers a careful, self-critical empirical study of compact decision encoders. Its central finding—that per-option scoring outperforms joint scoring at matched compute, but only when initialized from a cross-encoder reranker—adds nuance to recent efforts that default to joint readers for speed. The released Crivo models are convincingly efficient: the 17M variant answers a median CPU request in 13.5 ms, and the 68M/150M models exceed a 435M DeBERTa zero-shot classifier on 209 external typed-decision tasks. However, the authors rightly flag that their own development panels overestimated private leaderboard scores, and that a rule engine without learned parameters matches the smallest model on same-skill private tests, which tempers enthusiasm for leaderboard-only claims.
The strength of the paper is its controlled 32M-parameter grid isolating the reader mechanism from initialisation, objective, and training compute. By crossing joint versus per-option reading with reranker versus raw encoder initialisation, the authors show the advantage is an interaction effect tied to format matching, not an intrinsic property of per-option scoring. External evaluation is similarly rigorous: S1-Arena results are paired, bootstrapped, and audited for string overlap, with unseen and seen tasks reported separately.
The per-option advantage, while consistent on the public suite, fails to reach significance on unseen external tasks: the compute-matched gain on unseen S1-Arena tasks is +0.011 [−0.009, +0.031] and on unseen BTZSC datasets is +0.025 [−0.034, +0.084]. Because the grid is run only at 32M on a 150k-row subset, it is unclear whether the effect persists at scale or with the full training recipe. More troubling is the private-test evidence: the authors’ development panels overestimated same-skill scores by 5.6–7.1 points, revealing that panels drawn from the training-data pool are no longer independent estimates—a classic adaptive-holdout problem. Finally, the joint models exhibit meaningful option-order fragility, with 12–23% of answers changing under permutation, whereas DeBERTa changes only 2% and per-option models none.
The external benchmark comparisons are fair within the limits of the audit, but the evidence for generalisation is thinner than headline numbers suggest. On S1-Arena, Crivo-68M and Crivo-150M beat DeBERTa-v3-large by roughly 5 accuracy points overall, yet on the 145 unseen tasks the margins shrink to about 3 points and do not survive Holm adjustment, so the safest reading is parity with the 435M baseline. BTZSC offers only four unseen datasets, making it secondary evidence. The paper does not run its closest same-interface competitors—Bekko, Dinah-1, GLiClass, or Verdict—on the external benchmarks, so the comparison set is generic rather than tailored.
Reproducibility is strong on code and metrics but incomplete on data. All model weights, training and evaluation code, run outputs, and figure-generation scripts are released, and hardware specifications are precise down to the warm-up protocol. Hyperparameters are itemised per model and stage. The main limitation is that the training mixes are not redistributed because of upstream licensing, so an independent lab must reconstruct the mixture from the listed sources. Moreover, the board-influenced recipe adjustment after v1 models constitutes a mild form of adaptive holdout reuse that the authors disclose.
Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately. In a three-seed study at 32 million parameters, per-option scoring beats joint scoring on a held-out sample of a 37-benchmark decision suite by 5.8 points of chance-corrected skill at matched optimizer steps and by 2.8 at approximately matched training compute; two external benchmarks agree, though not significantly on their unseen tasks. Crossing reader with initialisation shows that the advantage requires a reranker initialisation, whose query-candidate pair input matches per-option reading; from the raw encoder the two readers are indistinguishable. Per-option scoring needs 5 to 13 times more encoder passes: at most twice the median latency, but more than four times the mean. Of six released joint encoders (17 million to 1 billion parameters), the 68- and 150-million-parameter models exceed a 435-million-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 overlap-audited System1-Arena tasks and are at least on par with it on the 145 tasks unseen in training. The smallest answers a median request in 14 milliseconds on 16 processor threads, against 728 for the classifier. Reordering options changes 12 to 23% of the joint models' answers (per-option scoring: none). Up to 68 million parameters, they lead their size range on a public leaderboard's private, mainly in-distribution tests.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.