Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
Standard Character Error Rate charges every OCR edit equally, whether it is a harmless typo or a silent factual error like changing a name. This paper introduces semCER, which keeps CER's grapheme alignment and denominator but weights each edit by a severity score from an LLM judge, eliminating the need for word segmentation. Validated against human readers in English, Vietnamese, and Chinese, semCER better ranks damaging errors and breaks ties where CER sees identical edit counts, offering a more nuanced tool for OCR and vision-language model evaluation.
The paper presents a compelling and well-motivated metric that addresses a genuine gap in OCR evaluation. The decomposition of CER into severity-weighted grapheme edits is elegant, the empirical validation against human raters is careful, and the release of data and a distilled scorer makes the work practical. However, the reliance on a proprietary LLM judge whose checkpoint changed mid-study introduces reproducibility concerns, and the human validation is limited to only three languages despite claims about 16.
The theoretical framing is clean: semCER generalizes CER by charging each grapheme edit $s_i \in [0,1]$ so that $\text{semCER}(r,h) = \frac{1}{m}\sum_{i=1}^{q} s_i$, recovering CER exactly when all $s_i=1$ and never exceeding it. The strongest empirical result is the ordering of single edits, where severity-sorted lists catch reader-rated damage far better than uniform CER or judge-free baselines. The distilled 0.8B student reproducing the judge's distributions is also a solid engineering contribution.
The paper's boldest cross-lingual claims rely entirely on judge outputs in 13 languages without human validation, leaving construct validity unverified beyond English, Vietnamese, and Chinese. 'Readers validated the severity in three languages. In the other thirteen, the grapheme load comes from the judge alone, and the judge's severities may shift with a language's resource level.' The preference test was drawn conditional on disagreement between CER and semCER, which inflates the apparent value of semCER by construction. The authors further note that 'no reader rated an edit's severity inside a line with other damage,' leaving open whether severity aggregation holds when errors interact.
The comparison to baselines is generally fair but uneven. Normalized CER actually tracks whole-transcription reader ratings as well as or better than semCER in English ($\rho=0.93$ vs $0.89$), which the authors acknowledge by recommending normalized CER for whole-transcription scoring and semCER for tie-breaking. 'On whole transcriptions, normalized CER tracks readers as well as semCER or better. Its Spearman's rho is 0.93, 0.90, and 0.76 against semCER's 0.89, 0.87, and 0.76.' The three-class charge trails semCER significantly, supporting fine-grained probabilistic severity over coarse bins. However, the equal-count comparison across 16 languages falls short of the pre-specified criterion, with only a minority of VLM-language pairings surviving Holm correction.
The authors release the 61 million judged edits, aggregated reader ratings, and the distilled scorer under open licenses. However, reproduction is blocked by the proprietary LLM judge (DeepSeek-V4-Flash Preview) whose checkpoint changed mid-study; 'The provider replaced the judge's checkpoint during this work, so a severity is comparable only within one checkpoint.' The distilled scorer mitigates API dependency but propagates any judge bias. Human annotation protocols are described clearly, but reader validation covers only three languages, and 'Judging the corpus cost about $2,000,' which may hinder independent replication at the same scale.
Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.