From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries
Wearable devices generate rich physiological records, but turning that structured data into patient-readable narratives risks semantic drift that can distort clinical meaning. This work formalizes structured-to-narrative summarization for menstrual health monitoring and proposes a constrained generation plus retrieval-augmented pipeline to preserve clinical alignment across five defined semantic drift conditions. Across 4,500 generated summaries, the authors find that explicit grounding constraints improve alignment, especially uncertainty preservation, yet they come at a measurable cost in readability and increased medical terminology.
The paper offers a clear conceptual advance in patient-facing health summarization by introducing a five-category alignment taxonomy and a controlled, systematic evaluation across three model tiers and three generation regimes. Its statistical rigor and transparency about metric brittleness are commendable. Nevertheless, the study remains a synthetic benchmark: inputs are semi-synthetic and unaudited by clinicians, all alignment metrics are automatic without human validation, and the fidelity gains demonstrably degrade readability, leaving the patient-facing value proposition unresolved.
The five-dimension alignment taxonomy—spanning faithfulness, epistemic misalignment, diagnostic overreach, omission, and urgency—provides a practical and actionable framework that moves beyond generic factuality checks. The fully matched experimental design, use of effect sizes rather than raw $p$-values, and careful post-hoc corrections to metrics demonstrate methodological care. The authors rightly conclude that 'Explicit constraints alone account for most metric improvements,' while retrieval's contribution is more nuanced and model-dependent.
The most serious limitation is the total absence of human validation: every alignment score is algorithmic, and the tone-sensitive urgency metric relies on a single LLM judge that the authors note is 'more subjective than the other four.' The NLI model used for faithfulness and omission is itself known to suffer from domain-shift degradation and annotation artifacts, and the omission metric mechanically penalizes paraphrasing, as shown by Phi-4-mini's false regression under constraints. Additionally, the semi-synthetic dataset, while internally coherent, has not been screened by clinicians, and the readability heuristics (FKGL, ARI) conflate syllable count with conceptual difficulty, partially undermining the accessibility analysis.
The evidence robustly supports the claim that prompt-based constraints reduce epistemic misalignment across all tested models. However, the case for RAG improving faithfulness depends on a post-hoc definitional expansion of $M_{\mathrm{faith}}$ to count retrieved lay definitions as valid premises; before this correction, RAG introduces apparent violations. The within-model comparisons are fair, but the absence of an external baseline or any fine-tuned/decoding-based method means the results characterize this specific prompting setup rather than establishing a generalizable state of the art.
The paper includes unusually thorough appendices: exact prompts with system/instructions (Appendix B), the full causal DAG with population-derived parameters (Appendix A), and retrieval specifications ($k=3$, TF–IDF cosine threshold 0.12). The authors pledge to release dataset generation code, which is critical since the evaluation is anchored to their semi-synthetic records. Full reproduction is still hindered by the lack of a released evaluation codebase and output corpus, along with dependencies on proprietary endpoints (GPT-4o generation, Claude 3.5 Haiku judging) and a specific Hugging Face NLI model that may be updated.
As wearable and digital health systems increasingly generate longitudinal physiological records, communicating these observations in patient-accessible language remains a significant challenge. Large Language Models (LLMs) offer a promising mechanism for translating structured clinical information into readable narratives, but this process may introduce semantic drift, where clinically precise meaning is distorted during simplification. We investigate evidence-grounded health summarization for menstrual health monitoring using a constrained generation framework that combines structured evidence conditioning with Retrieval-Augmented Generation (RAG) using a prompt-based constrained generation framework. We evaluate semantic alignment using Natural Language Inference (NLI), atomic factuality metrics, and readability measures. Across three LLMs and 4,500 generated summaries, explicit grounding constraints consistently improve clinical alignment, particularly by improving uncertainty preservation and, once faithfulness is evaluated against retrieved evidence, generally preserving or improving faithfulness, while effects on omission remain model-dependent. We also find that improved clinical alignment comes at the cost of increased reading difficulty and medical terminology. These findings highlight the need to jointly evaluate faithfulness, uncertainty preservation, and readability when developing patient-facing medical LLM systems.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.