When More Metadata Misleads: Probing Field-Level Grounding in LLM-Generated Museum Descriptions
Museum description generation is usually framed as a data-to-text task where richer metadata improves grounding, yet heterogeneous and unevenly documented catalogue records break that assumption. This paper introduces a two-phase framework—observational across 10,250 artefacts and controlled across 500 objects—to isolate how metadata quantity, field type, and image access shape what LLMs include, omit, and infer. The central finding is that field identity, not just field count, determines faithfulness: context fields are omitted while content fields trigger unsupported elaboration, and images can increase inference rather than grounding. The authors argue that aggregate reference similarity masks these failures and motivate field- and claim-level evaluation.
The paper makes a strong methodological contribution by moving beyond aggregate input-coverage metrics and showing that metadata-conditioned generation is a schema-alignment problem. Its two-phase design cleanly disentangles confounded natural variation, and the matched-count arm structure provides credible evidence for field-type effects. However, the automated nugget pipeline achieves only moderate agreement with human annotators ($\kappa_w = 0.67$), and the study is bounded by a single dataset and four proprietary models, which limits confidence in the absolute hallucination rates despite robust relative patterns.
The controlled probing framework is the clearest strength: holding object identity fixed while varying only field count and type isolates effects that observational data confound. The discovery that content fields (Subjects, Technique) and context fields (Find spot, Place) produce diametrically opposed error distributions—high coverage plus hallucination versus high silence plus fidelity—is replicated across regions and models, and is supported by paired permutation tests at $p<0.001$. The Phase 1 finding that surface metrics such as BERTScore invert documentation completeness ($\eta^2 = 0.33$) is an important warning against naive reference-based evaluation in this domain.
The claim-level factuality pipeline, though validated, shows only 73% human–pipeline agreement on hallucinated claims and $\kappa_w = 0.67$ overall, which means subtle boundary errors are likely misclassified. Because the evaluation treats catalogue metadata as the sole ground truth, visually or parametrically plausible claims that go beyond the record are labelled hallucinated even when historically accurate; this is acknowledged in the Limitations but still risks overstating error rates. Additionally, the image-ablation results are model-dependent—LLaMA-Vision exhibits a drastic 8.6 pp coverage drop without images under full metadata, against a 0.6 pp aggregate—suggesting that visual effects may not generalise cleanly across model families.
The evidence convincingly supports the central thesis that field type shapes failure mode: matched-count permutation tests across the S_ctx and S_cont arms show significant splits (5.6–8.5 pp) in coverage, hallucination, and bonus wrong at every level. Comparisons to data-to-text benchmarks (E2E, WebNLG, ToTTo) and summarization faithfulness literature are well-placed, though the paper does not directly benchmark its nugget pipeline against contemporary claim-level metrics such as FActScore beyond citation. The regional analysis is correlational by design and should be interpreted cautiously, as field availability itself varies by place, but the authors appropriately note that regional comparisons hold only at matched field type and count.
Experimental detail is unusually thorough: Appendix A provides exact model versions, prompt templates, NLI hypothesis schemas, temperature settings ($0$), and the stratified sampling procedure. However, no code repository or data splits are released, and the nugget verification relies on the proprietary Gemini 2.5 Flash API with a 200-token limit and undisclosed prompt-engineering details beyond the shown templates, which complicates independent reproduction. Because the study uses a public benchmark dataset (TimeTravel), the inputs are recoverable, but reproducing the exact claim-level labels would require re-implementing the two-stage pipeline and incurring API costs.
Metadata-conditioned generation is often framed as a data-to-text task in which additional input fields should improve grounding. Museum description generation challenges this assumption because catalogue records are heterogeneous, partially visual, and unevenly documented across geographic regions. We study how metadata composition shapes the generation of artefact descriptions through a two-phase framework. Phase 1 analyses natural metadata variation in a large museum collection, where field count, field type, image availability, and Place co-vary. Phase 2 introduces controlled field probes that hold object identity fixed while varying metadata quantity, field type, and image access. Our results show that metadata quantity alone does not explain generation behaviour. Field type determines whether models ground, omit, or infer information, while images can both support visual grounding and increase unsupported interpretation. More broadly, our findings show that metadata-conditioned data-to-text evaluation should assess field- and claim-level support, rather than relying only on aggregate input coverage or reference similarity.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.