Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models
Multilingual language models often fail to recognize that two names written in different scripts refer to the same person. This paper introduces a controlled benchmark of 601 audited identity pairs with attested Devanagari and Latin forms, plus a 374-item Malayalam replication set, to isolate cross-script entity matching failures. The authors find that Qwen2.5-7B's same-person accuracy collapses from 35.1% in Latin-Latin to 8.5% in Devanagari-Devanagari under Marathi prompts, then use probes, calibration, and activation patching to diagnose whether the model retains latent label information and where it can be causally intervened upon.
The paper presents a carefully controlled and thorough diagnosis of a genuine multilingual failure mode. The authors distinguish script-dependent discrimination from response bias using signal-detection measures, and their three-level analysis—behavioural gap, latent information via probes, and causal localization via activation patching—yields a coherent and actionable account. As they summarize, "A fact-independent intervention at this site raises held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy."
The experimental design is a clear strength: holding the question and relation fixed while varying only script-associated surface forms cleanly isolates the effect. The replication in Malayalam—a different script and Dravidian language family—robustly reproduces the phenomenon, with native-script accuracy falling to 5.3% while "decisions sit near the floor at 5.3% while ranking sits near the ceiling at 0.925." The decomposition into discrimination versus criterion placement via scalar calibration is methodologically sharp, recovering 87% of the raw model–probe balanced-accuracy gap and showing that much of the behavioural shortfall is not a complete absence of relevant internal information.
The primary limitation is scope: the detailed mechanistic analysis is essentially a single-model (Qwen) case study; as the authors note, "detailed mechanistic results concern Qwen" and other models receive only exploratory causal analyses. The intervention, while raising accuracy more than fivefold, still leaves two-thirds of held-out same-person pairs misclassified, and much of the gain comes from criterion shifting rather than restored script-invariant discrimination; indeed, "Entity-averaged AUROC changes are small and their intervals include zero." Additionally, the causal analysis relies on very small held-out gates (12–24 items per language), yielding descriptive rather than strongly inferential flip rates, and the shared test items across analyses undermine independence.
The evidence supports the core claim that script gaps reflect both discrimination loss and miscalibration, though the relative contributions vary. The probe comparisons are well-controlled: Latin-trained probes evaluated on Devanagari states exceed a string-feature baseline under similarity-matched negatives, with "a difference of +0.112 [+0.067, +0.163] in Marathi," indicating the signal extends beyond superficial orthographic cues. The comparison to automatic romanizations—which reduce token fragmentation but fail to recover performance—effectively undermines a pure tokenization hypothesis, although the authors rightly note that "The tested reductions in fragmentation are therefore insufficient; spelling fidelity and familiarity remain relevant."
The authors release code and corpora, report compute details in Appendix AA, and use a fixed split with seed 20260813. However, several factors would block straightforward reproduction. The causal gate is small (12–24 held-out items), and the paper explicitly warns that "these analyses share test items and are not independent replications." Moreover, "bfloat16 reductions depend on batch shape, which moves margins by up to 1.5 points near the decision boundary," introducing a source of non-determinism that could flip borderline decisions and make exact replication difficult despite the released artifacts.
Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with attested Devanagari and Latin forms, holding the question and relation fixed. Qwen2.5-7B's same-person accuracy falls from 35.1% in Latin to 8.5% in Devanagari under Marathi prompts, while different-person accuracy remains above 98%. Evaluation across five models reveals distinct response biases, and reduced tokenization fragmentation does not reliably close the gap. Replicating the design on 374 audited Malayalam pairs built from the same entities reproduces the effect in a second script and language family: same-person accuracy falls to 5.3%, about half the matched Devanagari rate, while the margin still ranks matching above non-matching names at 0.925 AUROC. Latin-trained probes recover label information from Devanagari states, exceeding a string-feature baseline under similarity-matched negatives. Scalar calibration recovers 87% of the raw model-probe balanced-accuracy gap, identifying decision criterion placement as a major contributor. Activation patching localises causal sensitivity to the second entity in Qwen's middle layers; one shared sensitivity direction transfers about half the full-state margin correction. A fact-independent intervention at this site raises held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy. Together, these results distinguish script-dependent discrimination from criterion placement and identify a causal site for improving identity judgments.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.