Large language models show systematic failures in incorporating evolving medical knowledge
The paper introduces ChronoEvo, a benchmark of 1,709 real clinical-guideline revisions spanning 12 semantic change types, to test whether LLMs update recommendations when given evolving medical evidence. Its central finding is a persistent dissociation: models frequently recognize that a recommendation has changed yet fail to fully incorporate those changes, leaving a roughly 32-percentage-point recognition-to-incorporation gap even when revised evidence is explicitly provided. This establishes faithful knowledge incorporation as a distinct and under-measured dimension of clinical LLM reliability.
The study convincingly establishes that recognizing a guideline revision is behaviorally separable from faithfully deploying it in a recommendation, and the evidence is robust across 14 models, two automated evaluators, and an adjudicated clinician validation of 300 responses. However, benchmark construction, extraction, verification, and primary semantic evaluation all rely on GPT-5, creating a meaningful risk of shared-model bias that the small, specialty-limited clinician sample only partially addresses. The core claim is well-supported, but the work remains descriptive rather than mechanistic, and the future-dated upstream sources may limit immediate reproducibility.
The ChronoEvo benchmark is well-constructed: revisions are extracted from real longitudinal guideline trajectories, subjected to an independent GPT-5 verification stage requiring confidence ≥0.80, and lineage-split to prevent data leakage. The recognition–incorporation gap replicates across 14 models spanning open-weight and proprietary families, and even the strongest frontier model retains a 20.1-percentage-point gap, while multivariable regression shows that strength-or-certainty changes carry significantly higher adjusted odds of outdated-content retention.
The most serious methodological weakness is the circular reliance on GPT-5 for benchmark construction, extraction, verification, and primary evaluation, which the authors acknowledge introduces "potential shared model-dependent biases." The independent clinician validation is limited to 300 enriched cases across two surgical specialists, so it cannot fully rule out evaluator-specific artifacts or generalize across clinical domains. Moreover, the study is purely behavioral and offers no analysis of model internals or error mechanisms, leaving it unclear whether the gap stems from attention failure, reasoning limitations, or instruction misunderstanding.
The empirical evidence strongly supports the central claim because the gap is large, statistically robust across bootstrap and permutation tests, and corroborated by human clinicians. The related-work framing is fair: the authors correctly note that prior medical benchmarks largely assume fixed knowledge states and that knowledge-editing research focuses on factual associations rather than downstream clinical recommendation updating. One caveat is that the negative salience-intervention result is interpreted as evidence that prioritization alone cannot close the gap, yet only one prompting variant was tested, leaving alternative retrieval or reasoning strategies unexplored.
Reproducibility is mixed. The authors commit to releasing code on Zenodo and the upstream TempoMed-Traj dataset is public, but the 1,709 verified ChronoEvo revisions are not yet deposited and source guideline documents are withheld due to publisher licensing. Open-weight model experiments are reasonably specified—greedy decoding, 2,048 token input limit, 256 new tokens, no stochastic sampling—but full reproduction depends on proprietary GPT-5 and GPT-5.6 Sol access for semantic evaluation, whose outputs may not be stable over time.
Medical knowledge evolves as clinical guidelines revise recommendations, yet whether large language models (LLMs) faithfully incorporate such changes remains unclear. We introduce ChronoEvo, comprising 1,709 recommendation-level revisions across 12 forms of medical knowledge change, to distinguish recognition of revised guidance from incorporation into clinical recommendations. In the primary model, 90.8% of responses recognized revisions and 80.2% were clinically correct, but only 58.5% fully incorporated guidance, yielding a 32.3-percentage-point recognition-incorporation gap. This dissociation persisted across 14 models and was corroborated by two clinicians' adjudicated assessment of 300 responses, yielding a 40.3-percentage-point gap in the validation sample. Changes in recommendation strength and patient eligibility were associated with retention of outdated guidance. A revision-salience intervention reduced outdated-content retention and improved clinical correctness but did not significantly improve full incorporation. These findings identify faithful incorporation of evolving medical knowledge as a distinct challenge for clinical LLM reliability, even when updated guideline evidence is available.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.