Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction
Cross-lingual financial analysis requires models to both recognize shared accounting concepts across languages and exhaustively recover all corresponding numerical facts. This paper introduces MultiFinGround, a benchmark built from real FY2025 ESEF reports in English, French, and Greek, aligned through the IFRS Accounting Taxonomy. The core finding is that strong concept recognition does not imply complete evidence grounding: even top models reach near-perfect recognition accuracy while missing target facts in over a quarter of correctly recognized cases. This matters because benchmarks that only test recognition can obscure critical failures in numerical evidence completeness.
The paper makes a compelling case that concept recognition and exhaustive evidence grounding should be evaluated jointly. Its paired task design isolates a genuine failure mode—models correctly identify an accounting concept yet systematically omit some of its numerical realizations, especially in Greek. However, the paper's conclusions are tempered by limited expert audit coverage, substantial generation failures in some evaluated models, and language-sector confounds that prevent clean causal attributions.
The benchmark construction is rigorous: it leverages real issuer filings and a shared IFRS taxonomy rather than synthetic translations, and the paired T2a/T2b design elegantly exposes the recognition–grounding gap. The evidence for the central claim is robust across eight reference models and multiple diagnostic conditions (label language, missing English evidence, harder distractors). Expert validation shows high inter-annotator agreement, and the sensitivity analysis demonstrates that audited annotation errors move scores by at most 1.4 percentage points.
Only 10% of items were expert-reviewed, leaving 90% of the benchmark reliant on potentially noisy issuer-provided iXBRL tags. The paper candidly admits that language, issuer, sector, and document style are confounded, so claims about Greek-specific difficulty are suggestive rather than causal. More troubling is the evaluation protocol: several models suffer catastrophic failure rates—Qwen3.8 exhausts its output budget on up to 34% of items, and Plutus returns empty answers on 80% of T1—making comparisons against these models unreliable. The fact that all four finance-specialized models underperform their generalist bases on T1, T2a, and T2c is striking but under-analyzed; the authors attribute it to settings mismatch without probing whether instruction formats or reasoning traces are responsible.
The core evidence supports the claims well: Figure 2 and Table 5 show conditional omission rates are high and Greek recall is systematically lowest for seven of eight reference models. The lexical-overlap baseline (70.2% on T2a) and harder 20-option sets (where top models still exceed 95%) demonstrate that recognition performance is not merely surface-level word matching. Comparisons to prior work in Table 1 are fair but largely taxonomic; because no existing benchmark explicitly pairs concept recognition with exhaustive fact selection on matched multilingual contexts, direct empirical comparisons are impossible, and the paper wisely does not claim superiority, only complementary coverage.
Reproduction is partially enabled but not fully streamlined. The benchmark relies on public regulatory filings, and the authors commit to releasing annotations, source metadata, and reconstruction instructions. Prompts, formal task definitions, model configurations, and the strict invalid-output protocol are documented in detail across Appendices A–C. However, no code repository or artifact URL appears in the paper, and reconstructing the exact table windows and preprocessing pipeline from the description in Appendix G would require significant engineering. Independent reproduction would be blocked primarily by the lack of released code, the need for API access to multiple commercial models, and the compute to serve large open-weight models.
Comparing financial disclosures across languages requires identifying shared accounting concepts and recovering all numerical facts that express them. Existing benchmarks evaluate aspects of these capabilities, but provide limited insight into whether both succeed on the same multilingual evidence. In particular, it remains unclear whether language models can recognize these concepts across languages and exhaustively recover their corresponding numerical evidence from the same contexts. We introduce MULTIFINGROUND, a benchmark designed to investigate this recognition-grounding gap in multilingual financial reports. Constructed from FY2025 disclosures of 90 issuers in English, French, and Greek, MULTIFINGROUND aligns independently authored reports through a shared accounting taxonomy and provides traceable fact-level annotations. The benchmark evaluates financial fact identification, concept recognition, exhaustive fact selection, and concept discrimination without candidate labels. Across 15 evaluated models, eight API and open-weight reference models achieve 97.3-99.4\% concept recognition accuracy, yet miss at least one target fact in 26.6-41.9\% of correctly recognized groups. Further analyses reveal substantial variation in concept discrimination and limited gains from the evaluated finance-specialized models. Our findings demonstrate that strong concept recognition can coexist with incomplete evidence recovery, highlighting the importance of evaluating semantic recognition and evidence completeness together.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.