Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation
DLoss tackles objective misalignment in radiology report generation: standard cross-entropy treats diagnostic and narrative tokens identically, so models can sound fluent while omitting critical findings or inserting false ones. The core idea is to pool decoder probabilities across sequence positions with different operators—noisy-OR for supported diagnoses and tempered log-sum-exp for unsupported ones—and add this as an auxiliary loss without changing architecture or inference. Evaluated across six 3D MedVLMs on ~50,000 abdominal CT scans, DLoss improves clinical metrics on most benchmarks, though gains vary widely by backbone.
DLoss is a conceptually well-grounded fix for token-level cross-entropy misalignment in radiology generation. The paper presents a clean, architecture-agnostic objective with zero inference cost, and the direction of improvement is consistent across most backbones. However, the strongest models see only marginal gains, and the evaluation is confined to automatic metrics on abdominal CT. As the authors note, "The controlled diagnostic target space is derived from training reports, so the auxiliary loss directly supervises only the concepts and expressions represented there" (Section 5), which bounds its generalizability.
The asymmetric pooling directly encodes clinical semantics: $q_{i,d}^{+} = 1 - \prod_{t \in \mathcal{P}_i}(1 - m_{i,d,t})$ rewards supported diagnostics appearing at any position, while $L_{i,d}^{-} = \tau^{-1}\log(\sum_{t \in \mathcal{P}_i}\exp(\tau \log m_{i,d,t}))$ suppresses spurious peaks everywhere (Section 3.2). This adds negligible overhead: "The largest measured mean increase in step time is 4.4% (BTB3D); peak allocated and reserved memory increase by at most 1.6% and 2.9%, respectively" (Section 4.2, Table 2). The cross-model evaluation on six backbones and two test sets is a genuine strength.
The gains are highly backbone-dependent. On Merlin and RadFM, DLoss improves GREEN by only 0.83 and 0.50 points, while on SPECTRE it adds 7.84. More alarmingly, the ablation shows that "Using only the supported (S-only) or unsupported (U-only) target loss degrades performance across all metrics, supporting the complementary contributions of both components in this setting" (Section 4.3, Table 3). In Appendix G, replacing unsupported-target pooling with mean pooling not only crashes scores but also increases empty generations from 0.09% to 2.52%, revealing instability when the asymmetric design is altered. The reliance on a private external dataset also prevents independent verification.
The matched CE-vs-CE+DLoss protocol is fair, and the paper covers both lexical and clinical metrics (GREEN, RadGraph-F1). DLoss is thoughtfully positioned against related work: "DLoss instead defines supported and unsupported diagnostic targets from each reference report and pools the decoder probabilities for each target across positions" (Section 2). Yet there is no direct benchmark comparison against methods such as semantic rewards or clinical RL on these exact backbones and datasets, leaving open how much of the gain is unique to this pooling formulation versus simply adding auxiliary clinical supervision.
Code is released, but the external dataset is private. Reproducing the full study requires re-running "LLM-based semantic tagging and synonym consolidation" (Section 3.1) with Qwen3-30B-A3B, which is stochastic and expensive. Backbone-specific loss coefficients span an order of magnitude ($\lambda_{+}$ from 0.001 to 0.01, Appendix E.1), selected by an unspecified "small grid search", and different LoRA ranks (16 to 512) complicate fair comparison. The nine-seed BTB3D analysis (Appendix F) is welcome, but it shows two of nine seeds where BLEU-4 or GREEN drop, indicating some training variability.
Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize text agreement over diagnostically critical terms. As a result, models can miss key diagnoses and introduce wrong ones. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To encode these requirements, we introduce Diagnostic Loss (DLoss), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DLoss separately aggregates probability mass over vocabularies associated with diagnoses supported by the reference report and with unsupported alternatives. Using asymmetric pooling across token positions in the report, it encourages supported diagnoses to receive sufficient probability in at least one position while suppressing spurious probability spikes for unsupported diagnoses across all positions during training. The loss can be applied to existing 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We train and evaluate DLoss on 50,000 abdominal CT scans across several 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Across these settings, DLoss improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, showing the value of asymmetric sequence-level supervision for radiology report generation. The code for DLoss is available at https://github.com/gmum/DLoss.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.