A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models
Multimodal large language models can answer clinical questions about retinal OCT scans, but without spatial grounding clinicians cannot verify whether a model attended to real pathology or relied on spurious shortcuts. This paper introduces RetinaGroundBench, the first benchmark for biomarker visual grounding in retinal OCT, together with $AggIoU$—a metric tailored to MLLMs that fragment lesions into multiple boxes—and RetGrounder, an 8B domain-adapted model that outperforms general-purpose models up to 38B parameters. The core message is that grounding remains largely unsolved: even the best open-source model achieves only 23.7% macro $F_1$, and VQA ability does not imply localization skill. The work matters because trustworthy clinical deployment of MLLMs requires precisely this kind of verifiable spatial evidence.
This paper makes a timely and clinically motivated contribution by mapping how poorly current open-source MLLMs localize retinal OCT biomarkers, despite reasonable VQA accuracy. The benchmark design is careful—exhaustive per-instance annotation, a metric ($AggIoU$) tailored to MLLM fragmentation behavior, and a domain-adapted 8B model that beats 38B generalists—but the benchmark's small size (815 B-scans from two datasets) and reliance on bounding boxes for diffuse lesions limit generalizability.
The benchmark construction is unusually rigorous: unlike prior datasets, 'these classes are not a curated subset of lesions of interest, but the complete set of pathological findings observed in each image, so that the ground truth reflects a comprehensive assessment of every scan' (Section 2.1). $AggIoU$ is well-motivated by the empirical observation that MLLMs fragment single lesions into multiple partial predictions, and the ablation cleanly shows that grounding supervision supplies most of the localisation gain while VQA data bolsters presence detection. The cross-model landscape—27 open models across four families—yields the clear result that 'RetGrounder achieved the highest recall (27.5%) while maintaining competitive precision (48.0%), resulting in a macro F1 of 33.9%—a relative improvement of 43% over InternVL3.5-30B-A3B' (Section 2.2).
The most significant limitation is scale and diversity: 'the benchmark comprises 815 B-scans drawn from two public datasets (OCTDL and OLIVES) acquired on a limited range of imaging devices... it may therefore not capture the full diversity of scanner hardware, acquisition protocols, and patient demographics encountered in clinical practice' (Discussion). A related concern is the mismatch between annotation format and pathology morphology: 'the bounding-box paradigm itself presents some limitations for the representation of lesions. For instance, drusen and hard exudates commonly present as multiple small lesions scattered across the retina, and there is no canonical way to group such instances into one box versus several' (Discussion).
Additionally, reproducing RetGrounder is blocked because its fine-tuning mixture includes 34,112 in-house OCT B-scans that 'cannot be shared due to data-sharing restrictions' (Data availability), so the reported 43% improvement is contingent on inaccessible data.
The evidence robustly supports the central claim that VQA and grounding are dissociated capabilities. 'The stark contrast between grounding and VQA performance indicates that these two capabilities are not tightly coupled in current MLLMs' (Discussion). The paper documents this with per-class breakdowns—showing, for example, that drusen reaches 82.7% VQA balanced accuracy while grounding $F_1$ stays at 28.7%, whereas fibrosis grounds moderately ($F_1$ = 47.5%) yet sits near chance on VQA (BAcc = 50.9%). 'RetGrounder dominates grounding yet ranks behind the leading general-purpose models on VQA; conversely, the Qwen3.5 and Qwen3.6 models lead VQA without achieving proportionally strong grounding' (Discussion). Comparisons to prior work are fair and well-scoped: the authors correctly note that LMOD+ and other ophthalmic benchmarks evaluate only anatomical structures or image-level VQA, not biomarker grounding. The human inter-observer baseline ($F_1$ ~41%) is a useful addition that contextualizes model failure without overstating it.
Reproducing the benchmark evaluation is straightforward: the authors release annotations, code, and 'a self-contained pipeline that reproduces the visual grounding evaluation end-to-end—dispatching inference for every evaluated open-source model family, parsing their predictions, and computing AggIoU from a single command—is available at https://github.com/ronnief1/RetinaGroundBench' (Code availability). However, reproducing RetGrounder is blocked because 'the in-house training data used to fine-tune RetGrounder cannot be shared due to data-sharing restrictions' (Data availability). Training hyperparameters for the LoRA adapters are specified (rank 64, alpha 128, 77k steps, initial learning rate $10^{-5}$), and the public components of the training mixture are documented, so a partial reproduction with substitute data is possible but not guaranteed to match the reported 33.9% $F_1$.
Multimodal large language models (MLLMs) can answer clinical questions about medical images, but trustworthy use requires visual grounding: models should localize the pathology they report rather than rely on spurious shortcuts. Whether MLLMs can visually ground clinical findings in retinal optical coherence tomography (OCT), the most widely used imaging modality in ophthalmology, remains largely unexplored. To address this gap, we introduce RetinaGroundBench, the first benchmark for biomarker visual grounding in retinal OCT. It comprises 815 B-scans exhaustively annotated by ophthalmologists with 4,537 bounding boxes covering 13 biomarkers across 8 diseases. Because MLLMs often fragment a single lesion into multiple partial predictions, we propose AggIoU, a metric that evaluates whether the union of overlapping predicted boxes covers the annotated region. Across 27 open-source MLLMs (0.8-38B parameters), we find that visual grounding remains largely unsolved: the best general-purpose model achieves a macro F1 of just 23.7%, with consistently low recall, despite reaching up to 73% balanced accuracy on biomarker presence questions. To narrow this gap, we developed RetGrounder, an MLLM fine-tuned using an OCT-specific strategy integrating grounding and textual information. Despite its relatively small size (8B), RetGrounder improves grounding F1 to 33.9%, a 43% relative improvement. Our benchmark provides a reference point for measuring progress in the field of OCT diagnosis with MLLMs.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.