Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination
The paper tests whether open LLMs that a hospital could self-host are ready for clinical use, evaluating four models (8–27B parameters) on the Spanish MIR residency exam under its real scoring and ranking rules. The authors go far beyond headline accuracy to probe calibration, the value of abstention under a wrong-answer penalty, the soundness of clinical reasoning, and whether vision models actually use the images they receive. Their headline message is that a model can rank high enough to choose almost any specialty while remaining overconfident, correct for the wrong reasons, and blind to the clinical plates it is shown.
This is a strong, pragmatic contribution that advances medical LLM benchmarking from accuracy toward safety-relevant properties like calibration, faithful reasoning, and image grounding. The experimental design is admirably realistic—scoring under the MIR’s own penalty ($S = 3c - w$), mapping scores to actual residency ranks, and controlling for contamination with a post-release exam. However, the work is held back by an unvalidated LLM judge for reasoning, small samples for vision and rationale analysis, and no declared code or data release.
The evaluation protocol mirrors real deployment constraints by penalizing wrong answers and translating net scores into national ranks and specialty cut-offs. The image-use ablation is particularly clean: showing the clinical plate, withholding it, or swapping in a mismatched plate leaves accuracy unchanged ($p = 0.65$ and $p = 0.50$), demonstrating that the vision modules are not improving decisions. The paired comparison of Gemma-3-27B and MedGemma-27B isolates medical fine-tuning at fixed scale, and the statistical analysis uses appropriate paired bootstrap and McNemar tests.
The rationale grading relies on a Llama-3.3-70B judge that has not been validated against physician ratings and is shown to misread negated reasons (Table 7), yet it drives the headline claim that 13–38% of correct answers rest on unsound reasoning. The generate mode used for this analysis also drastically changes model behavior—Gemma-3-27B drops 15 points in accuracy compared with label mode—so the reasoning sample may not reflect how the model performs when not forced to explain every option. The rank-to-specialty mapping rests on sparse public data and assumes a typical academic record, making the precise specialty placements noisy estimates.
Additionally, the medical-tuning conclusion is drawn from a single model pair (Gemma/MedGemma), and the vision analysis covers only 171 questions from recent exams, limiting generalizability to other architectures or earlier image sets.
The evidence broadly supports the claims. The calibration metrics (ECE, Brier, AUROC) are standard and clearly favor the conclusion that medical tuning lowers overconfidence without improving discriminative self-knowledge: MedGemma-27B’s AUROC versus Gemma-3-27B is essentially unchanged at +0.002. The contamination control—showing comparable or better post-release accuracy—strengthens the finding that gains are not mere memorization. Comparisons to prior work are fair and explicit: the authors note that earlier studies report accuracy or raw marks, whereas they are the first to score under the MIR’s penalty and allocation rules. The one gap is that the judge’s error rate on negated options means the reported 13–38% 'wrong reason' rate is likely inflated by an unmeasured amount.
Methods are documented in considerable detail—model versions, precisions, hardware (48GB NVIDIA Quadro RTX 8000), prompt design in Spanish, and the deterministic answering protocol are all specified—yet the paper does not declare either a code repository or a public release of the extracted corpus, which would block an independent reproduction of the score-to-rank curves, judge-based grading, and image-ablation results. Hyperparameters are partially described (deterministic generation, disabled thinking mode), though exact temperature, context window, or patching strategy for the vision encoder are not stated. Reproducing the rank projections would also require the authors’ private mapping of per-specialty score cut-offs, which they compiled manually from trade press.
Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.