OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

The strongest element is the precise definition of clinical over-refusal as the systematic failure to preserve the Partially Supported category, which captures a genuine safety gap that aggregate metrics miss. The deterministic seven-module design separates evidence verification, missing-information detection, safety classification, and abstention, and the ablation study shows that these modules serve complementary functions rather than providing interchangeable gains. The fully open-source benchmark, evaluation code, and prompts support reproducible follow-up work.

The principal limitation is circularity between the benchmark and the proposed agent. MTB-AuditAgent’s rules and evidence base were built using the same public sources and label definitions that shape the benchmark, and the 50-case expert subset was used to refine the agent, so the reported concordance is post-hoc rather than prospective validation. The authors acknowledge this but still present benchmark accuracy as the primary evidence of superiority.

Additional gaps restrict generalizability. Only GPT-4o and GPT-4o-mini are tested, so the claim that 'all LLM configurations' over-refuse applies to a narrow model family. The expert study is tiny (n=50, enriched for hard cases), and one annotator’s perfect agreement with the benchmark undermines its intended independence. The custom Safety Score formula $\text{SS}=100\times(0.30(1-R_{\text{miss}})+0.20\text{F1}_{\text{abs}}+0.20R_{\text{MI}}+0.15C_{\text{WD}}+0.10C_{\text{Hall}}+0.05C_{\text{OC}})$ embeds subjective harm weights, and the deterministic approach may not scale to other cancer types or evolving guidelines without extensive manual curation.

The evidence supports the existence of over-refusal on the evaluated benchmark, but the paper does not compare against other rule-based or medical-specialized LLM baselines, so the gap between MTB-AuditAgent and alternative clinical systems remains unclear. Related work is cited accurately: CARES and Health-ORSC-Bench do identify over-refusal in medical contexts, and the broader LLM safety literature is represented fairly. We verified that OR-Bench is a large-scale over-refusal benchmark and that CARES evaluates both harmful compliance and false-positive refusals in healthcare, matching the authors’ descriptions. However, the paper’s scope is limited to synthetic NSCLC cases, and no real-world MTB retrospective or prospective validation is provided, so the clinical translation of the findings is still speculative.

The authors release benchmark cases, prompts, evaluation code, and predictions on GitHub, which is a strong reproducibility practice. LLM baseline hyperparameters are reported (temperature=0, max_tokens=512). Yet independent reproduction of MTB-AuditAgent is hampered by incomplete specification of the deterministic rule set: while the seven modules are described conceptually, the exact decision logic, thresholds, and rule tables are not provided in the paper. Moreover, the benchmark lacks a held-out test set that was never exposed during system development, and the expert annotation subset was used for refinement, meaning external teams would need to assemble a new clinical panel to reproduce the concordance claims. The reliance on fully synthetic cases simplifies data sharing but also reduces external validity.