The Value of System 1 Models in Medicine
Many clinical decisions are rapid multiple-choice tasks that require uncertainty estimates for safe deployment. This paper evaluates four System 1 models, zero-shot non-generative classifiers that return answer probabilities in a single forward pass, on 86 clinical tasks across six capability tiers spanning text and imaging. The authors find that these models excel at medical knowledge questions and report extraction, operating in under half a second and for a few joules per decision, but they fall sharply on clinical judgement and volumetric imaging tasks. The results map a clear capability gradient and make a pragmatic case for where fast, cheap decision models can fit into clinical workflows.
The paper makes a strong case that System 1 models are ready for high-volume clinical text processing while honestly exposing their sharp limits. The authors conclude that "System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image." The breadth of the evaluation, 179,111 cases across 20 languages and 14 domains, lends weight, though the slice-based volumetric protocol and image downsampling likely understate what vision-specific architectures could achieve.
The tiered taxonomy and massive scale of the benchmark are the paper's greatest strengths. The capability gradient is clear and internally consistent: mean AUROC falls from $0.94$ for medical knowledge to $0.69$ for volumetric findings, with imaging tasks scoring $0.09$ lower than text tasks ($P < 0.001$). The efficiency claims are directly measured rather than estimated: "The median energy of one decision was 23.5 joules for Clef-flash (IQR, 17.2 to 51.3) and 80.9 joules for Clef (IQR, 68.4 to 125.5)." The calibration analysis is rigorous; "Withholding the less confident half of the answers raised accuracy on text tasks from between 0.76 and 0.83 to between 0.96 and 0.99."
The volumetric imaging evaluation suffers from a methodological mismatch that the authors acknowledge but do not fully mitigate. They note that "Because the System 1 interface accepts at most four images per request and no volumes, CT and MRI studies were therefore sent as single slices every 3 to 10 mm, and a small finding can lie between two slices." The resulting AUROC for pulmonary embolism on CTPA was $0.54$, essentially chance, yet this is reported as a capability limit rather than a protocol artifact. Similarly, "radiographic images had to be downsampled, for example mammograms at 1,024 pixels on the long side, at which microcalcifications cover few pixels."
The zero-shot constraint and one-sentence prompt description are realistic for deployment but mean the paper does not establish whether prompt engineering or few-shot examples could close the gap on clinical judgement tasks. The authors also note that "public datasets can be part of the training data, which raises results on text tasks in particular," a caveat that somewhat undermines the headline zero-shot competitiveness claim.
The evidence strongly supports the text-tier claims but is weaker for imaging. "The best System 1 model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks," with a median difference of $-0.17$ on imaging. The comparison with clinicians is sobering: "Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56)." The generative comparison within model families is well-controlled, System 1 models were $3$ to $5$ percentage points less accurate but $11$ to $76$ times faster, though it covered only 32 tasks and 100 cases per task. The authors fairly temper their conclusions by noting that larger frontier generative models were not tested.
Reproducibility is strong for the open-weight models but limited for the proprietary APIs. The authors state: "Code and all model outputs are available in the project repository at https://github.com/twin-x-eu/system1-models-in-medicine." Local runs of Clef and Clef-flash agreed with their hosted counterparts, and energy was measured directly via NVML. However, the two proprietary models are only accessible through OpenRouter with undisclosed weights, and "The providers of the hosted models (GPT-6 Luna Decisions, Jev 1.14) publish no per-request energy figures for them." Generative comparisons also relied on default reasoning settings that are not fully specified.
Many clinical decisions are a choice among a few fixed options, taken quickly and in large numbers. Such decisions also need an uncertainty estimate to know when to defer to a clinician. Generative language models answer slowly in free text and state their confidence poorly. In contrast, System 1 models answer zero-shot with a probability for each answer. However, their performance across clinical tasks is unknown. Four System 1 models, two proprietary and two with open weights, were evaluated zero-shot on 86 public tasks from 14 clinical domains in 20 languages, spanning examination questions and medical imaging. Tasks were assigned to six capability tiers before analysis. The primary outcome was the area under the receiver-operating-characteristic curve (AUROC). Secondary outcomes included calibration, language effects and comparisons with published results and generative models. Across 179,111 cases, the mean AUROC of the best model per task was 0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings and 0.69 for volumetric findings. Imaging tasks scored 0.09 (95% CI, 0.06 to 0.11) lower than text tasks (P < 0.001). The best model matched or exceeded the best published result on 13 of 26 text tasks and on 8 of 35 imaging tasks. Triage nurses outperformed the best model (accuracy, 0.85 vs. 0.56). One decision took 0.34 to 0.44 seconds and 24 to 81 joules on open-weight models run locally. Accuracy on identical questions fell by up to 39 percentage points from English to Yoruba. Generative models of the same family were 11 to 76 times slower, with 3 to 5 percentage points higher balanced accuracy. System 1 models decide quickly, cheaply and with usable confidence when the answer is stated in a text or recalled from medical knowledge, but not when it must be judged or read from an image.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.