How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions
Retrieval scores for clinical RAG are often computed against a single annotated chunk per query, conflating retriever quality with the completeness of the ground-truth annotations. This paper introduces RAG-MAMA, a breast-lesion question-answering system, and proposes mapping canonical evidence spans to all containing chunks so that overlapping segments with equivalent facts are not scored as competing candidates. The authors report nearly perfect retrieval metrics ($\text{Recall}@3 = 0.94$) on an 18-query development set and cautiously frame the work as a small methodological baseline, though the narrow scale and lack of independent validation severely limit its conclusions.
The paper presents a conceptually sound argument that chunk-level ground truth can bias retrieval evaluation, and it backs this with a clean, small-scale ablation on chunking parameters. However, the evidence is extremely limited: only 18 simple queries, retrieval-informed curation of evidence spans, and no direct comparison against standard annotation protocols or external benchmarks. While the honesty about scope is commendable, the work remains a preliminary methodological note rather than a validated clinical RAG evaluation.
The central methodological idea is well motivated and clearly articulated: defining relevance at the evidence span rather than the chunk level directly addresses the annotation-coverage bias created by overlapping segmentations. The authors also admirably scope their claims, explicitly framing the study as a small-scale baseline and acknowledging that complex, multi-evidence questions are out of scope. This intellectual honesty helps readers calibrate their expectations.
The development set is prohibitively small: 18 simple queries written directly from 18 documents provide almost no statistical power, and a single failure moves $\text{Recall}@3$ from $0.94$ to $0.89$. More troubling, the ground-truth curation process appears to have been informed by the retriever's outputs ('retrieval-informed curation'), which risks circularity by encoding the system's own behavior into the relevance judgments. The generation evaluation is also thin: DeepEval claim-level metrics judged by deepseek-v4-pro are presented without human calibration, inter-annotator agreement, or any analysis of error modes.
The clinical applicability is narrow. The benchmark consists solely of one Yes/No or WH-question per document, blissfully avoiding the ambiguity, multi-hop reasoning, and conflicting guidelines that characterize real breast-lesion management decisions. The scope gate that defers malignant cases is reasonable, but it further shrinks the problem space.
The internal ablation supports the specific claim that chunk size and overlap affect the number of chunks hit by evidence spans, but it does not compare the proposed evidence-span protocol against a standard single-chunk-per-query baseline under identical conditions. Related work is cited descriptively—MedRAG, Almanac, and others are mentioned—but no empirical head-to-head or even metric correlation is provided. As a result, the paper demonstrates annotation sensitivity within its own setup without proving that its proposed protocol generalizes better or is more reliable than conventional practice.
While the pipeline architecture is sketched in Figure 1 and the retriever components are named (BGE-Large, BM25, BGE cross-encoder, Reciprocal Rank Fusion, Qwen 3.6 27B), critical implementation details are absent: exact prompt templates, temperature and top-p settings, RRF fusion parameters, the specific BGE cross-encoder checkpoint, and how evidence spans were aligned to chunks algorithmically. Neither code, queries, evidence-span annotations, nor the 18-document corpus appears to be publicly released, and the 'separate experiment' on generation provides no index configuration or sampling protocol. Reproduction would require significant guesswork.
Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.