IndoNutriQA: An Indonesian Clinical Nutrition Question Answering Benchmark

The dataset construction is admirably transparent: the authors voluntarily adopt HIPAA Safe Harbor de-identification, release structured metadata, and honestly report that each reference answer has only one nutritionist author. Their statistical protocol goes beyond typical benchmark papers, adding 5-fold cross-validation stratified by source report, learning curves, and bootstrap CIs to a small test split. Most impressively, the LLM-as-judge calibration against three expert raters and the subsequent human error analysis directly advance the methodology of clinical NLP evaluation: they show that an LLM judge tracks human factuality well but misses unsupported clinical additions that humans catch, and they report this without spin.

The single-annotator design is a hard ceiling: the authors plainly note that a conventional inter-annotator $\kappa$ over the references is undefined, and they estimate that 6–10% of audited reference answers are contestable. The fixed eight-question NCP/PES protocol, while enabling controlled comparison, deliberately excludes open-ended clinical dialogue and leaves whole sections of the source reports untested. A verifiable citation error in the related work—claiming PubMedQA has 11K expert-labelled instances versus the actual 1K—undermines confidence in the dataset literature review.

On the modeling side, the fine-tuned baselines are uncompetitive: mT5 achieves essentially zero exact match and plateauing near-zero F1 even with more data, and the RAG system collapses because the mt5-base generator fails. This reinforces the authors' position that the corpus is an evaluation benchmark, not a training set, but it also means the paper provides almost no evidence about how a stronger Indonesian generative model would perform. Finally, the LLM judge's safety dimension is admitted to be unvalidated ($\kappa=0.133$ with raters), and the self-preference test is underpowered, so claims about clinical safety should be treated with caution.

The positioning table (Table 7) offers a fair and useful comparison across language, domain, answer type, and source, correctly distinguishing IndoNutriQA from exam-based and template-generated corpora. Stratified analysis by disease category also supports generalization claims: the top 5-shot LLM performs within a narrow F1 band across represented categories. Yet the held-out test split entirely omits cardiac and renal/urologic reports, so the benchmark cannot verify performance on those conditions. The cross-lingual transfer claim—that English medical pretraining helps more than Indonesian general-domain pretraining—is supported by MedGemma-4B-IT's strong results but is explicitly admitted to rest on only two English-medical models, one of which (BioMistral-7B) does not follow the pattern, making the evidence suggestive rather than conclusive.

Data release is exemplary: the authors publish the anonymized dataset, evaluation scripts, and cached LLM predictions under open licenses, which removes the need to re-query paid APIs to reproduce tables. However, the main paper and appendices lack critical implementation details for the fine-tuned and RAG baselines. Exact training hyperparameters—batch size, learning rate, number of epochs, optimizer settings—are not stated for the IndoBERT or mT5 runs, and the RAG configuration is described only as 'a single dense retriever over recursive chunks feeding an un-fine-tuned mt5-base generator,' leaving chunk size, overlap, retriever model, and top-k unspecified. An independent researcher could approximate these experiments, but true reproduction would require additional correspondence with the authors.