A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Earnings call transcripts are long, jargon-dense documents that are hard to analyze at scale, and citation-grounded LLM outputs are costly to evaluate because they rely on expert annotators. This paper proposes numeric evidence evaluation (NEE), an automatic metric that checks alignment of salient financial figures across claims, citations, and source documents to bypass human judgment. The authors also introduce ECTs-100, an automated benchmark of long-context Q&A pairs built from S&P 500 transcripts, and test a failure mode they call conscious incompetence—whether LLMs abstain when evidence is missing. Their findings matter because they show that strong groundedness scores mask real correctness gaps and a troubling tendency to hallucinate (~48%) under informational insufficiency.
The paper is a focused, practical contribution to trustworthy financial NLP. NEE directly addresses the scaling bottleneck of expert annotation, and the three-level groundedness hierarchy is conceptually clean.
Yet the experimental scope is narrow: all baselines use a single proprietary model (GPT-4.1), the correctness benchmark is auto-generated by the same model family, and the key proxy is not validated against human judgments. It is a useful benchmark and method sketch, but the empirical claims should be treated as preliminary.
The strongest element is the clear operationalization of groundedness into three complementary checks—claim–citation, claim–context, and citation–context—which yields a more granular diagnosis than prior work. The automated ECTs-100 pipeline is a sensible scalpel for correctness evaluation, and the conscious incompetence experiment exposes a concrete, high-stakes failure mode that prior benchmarks often ignore. The authors are also commendably transparent about limitations.
First, the correctness benchmark ECTs-100 is constructed by prompting GPT-4.1 to generate questions from salient figures, then evaluating GPT-4.1 answers against that synthetic ground truth. This creates a circularity risk because the model being benchmarked may share biases with the reference generator. Second, the numeric exact-match proxy is overly permissive: the authors themselves note that identical figures can match even when they refer to different entities or reporting periods, and the conscious incompetence condition removes only the target citation sentences while leaving the rest of the lengthy transcript.
Third, all baselines use the same proprietary model, so the study compares citation strategies rather than model families, limiting generalizability.
The evidence supports the headline finding that groundedness is high while correctness is moderate to low: citation-context groundedness exceeds 0.99, yet claim F1 on ECTs-100 is only 0.6468. However, the paper does not empirically validate that NEE scores correlate with human or expert judgments, explicitly admitting this gap. Comparisons to related work are accurate but limited; absent a human-annotated subset or cross-domain transfer experiment, the claim that NEE is a reliable proxy remains theoretically motivated but unproven.
The authors provide GitHub links to the ECTs-100 data and source transcripts, and the experimental setup is described at the API-call level with default hyperparameters. Because no training is involved, reproduction hinges on access to GPT-4.1 and the exact extraction prompts.
The prompts for NEE and question generation are summarized rather than fully disclosed in the main text, and reliance on a proprietary model means results may not be stable across versions. Independent replication is possible in principle but blocked for researchers without API access or budget for large-scale calls over 100 long documents.
Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations. Empirical results show that LLMs perform well in groundedness but face notable limitations in correctness, with informational insufficiency presenting an additional challenge.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.