An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

The strongest contribution is the systematic use of graph-grounded evaluation to expose failure modes that reference-based metrics miss. The authors show that fine-tuned models suffer whole-output corruptions and triple omissions on 12% to 23% of WebNLG17 outputs, whereas prompted systems fail on only 2% to 10%, and these omissions grow rapidly for fine-tuned systems as graphs expand from one to seven triples. The two-judge protocol with adjudicated human labels on 110 graphs is a commendable attempt at validation, and the finding that the judges reliably detect contradictions and entity corruptions but miss unsupported elaboration is precisely the kind of diagnostic insight that makes the evaluation actionable.

Several limitations weaken the comparative fairness and generalisability of the results. First, the fine-tuned baseline is restricted to T5-large and a single LoRA-adapted encoder-decoder (T5-Gemma2-4B-4B), with no fully fine-tuned decoder-only LLM included; this makes it impossible to disentangle the effect of prompting versus architecture from the effect of fine-tuning versus not fine-tuning. Second, the prompted LLMs are evaluated on a single generation run per model, so sampling variance is unquantified, and the human annotation covers only 110 of 1,862 WebNLG17 graphs with a single primary annotator whose adjudicator had access to LLM explanations—a setup that risks anchoring bias and yields wide confidence intervals on agreement statistics. Third, the benchmark datasets are not independent, because DART subsumes WebNLG17 and WebNLG20 extends it, raising the possibility of data memorisation in judge LLMs, and the inconclusive GPTScore results are left unexplained.

The tabulated evidence largely supports the headline claims: Tables 1–3 show Nemotron-3-Ultra and Gemma-4-31B matching or exceeding T5-large on METEOR, BLEURT, and BERTScore while trailing on BLEU, BARTScore, and PARENT, confirming that the gap has narrowed to a few points. The BinEval faithfulness and coverage scores in Table 4, combined with the per-question breakdown in Table 5, provide solid support for the assertion that prompted models produce fewer hard errors. The human validation in Table 6 backs this with Cohen’s kappa up to 0.75 for GLM on T5-large faithfulness, though the low kappa of 0.07 for DeepSeek on Nemtron-3-Ultra shows judge agreement is not uniformly strong. Comparisons to prior work are handicapped by acknowledged protocol differences—Table 8 notes that reproduced T5-large BLEU scores are lower than previously published values—making strict SoTA comparisons unreliable.

The experimental pipeline is documented in considerable detail: T5 fine-tuning hyperparameters, LoRA settings ($r=32$, $\alpha=64$, dropout $0.05$ on $q_{\text{proj}}$ and $v_{\text{proj}}$), 3-shot cosine-similarity retrieval with E5-base-V2, temperature 0.0, and max tokens 384 are all specified. The evaluation metrics use standard libraries (SacreBLEU, NLTK, rouge-score) and checkpoints (BERTScore with bert-base-uncased, BARTScore with facebook/bart-large-cnn), and the full prompt template is reproduced in Table 10. However, the authors do not indicate whether code, data splits, or model outputs will be released; the reliance on closed-source GPT-5-mini and the absence of repeated runs with different seeds means independent reproduction would require substantial compute resources and may yield different absolute scores. Crucially, no variance estimates are provided for the prompted LLMs, and the BinEval judge prompts in Tables 12 and 13 would need to be exactly replicated to ensure comparable results.