Measuring the Verifier's Floor: How Much Apparent Rare-Word Text-to-Speech Error Is ASR Error?
Text-to-speech systems are commonly judged by feeding synthetic speech into an automatic speech recognizer and measuring word error rate against the input text, a procedure that inevitably mixes ASR mistakes with true synthesis errors. This paper isolates the ASR component of that verifier's floor by running the same three recognizers on the original human recordings from LibriSpeech and FLEURS, then comparing three scoring rules: the standard ASR round-trip, a single-recognizer mask that ignores words missed in human speech, and a three-recognizer adjudication. The work matters because it shows that a large share of apparent rare-word TTS error is purely recognizer failure, and it provides principled recommendations—such as reporting per-frequency ASR baselines and avoiding the single-recognizer mask—for fairer TTS evaluation.
The paper presents an elegant, tightly controlled measurement of how much ASR round-trip evaluation inflates rare-word TTS error rates. By leveraging corpora that contain parallel human recordings, the authors show that adjudication with three recognizers lowers the apparent any-of-$N$ error on very rare LibriSpeech words from 10–13% to 3–5%, and they convincingly demonstrate that simply masking out words the evaluator misses in human speech is biased. They are appropriately cautious about the limits of their evidence: the effect sizes are large enough to make the central claim credible, but the absolute counts are too small for precise error-rate estimation, and the absence of a listening test leaves ambiguity about whether remaining adjudicated errors are truly mispronunciations or merely spelling variants.
The frequency-stratified diagnostic is the strongest contribution. The dependence of ASR error on word rarity is severe and well quantified: the evaluator misses 32.9% of very rare words in human LibriSpeech recordings and 66.7% on FLEURS, against only 1.5% and 3.2% for common words. The three-recognizer adjudication is robust across both TTS systems and both corpora, cutting the standard round-trip error by factors of 2.8–3.3. The most actionable result is the empirical demolition of the single-recognizer mask: on the words that $C$ misses in human speech, alternative recognizers find TTS errors in 3.9–4.9% of cases, but only in 0.2% of the words that $C$ hears, showing that the mask silently discards the very items most likely to be genuine synthesis failures.
The analysis suffers from low statistical power and post-hoc data enrichment. The entire rare-word trend rests on only 10–14 adjudicated errors per system in the very rare band, and the authors enriched the test set with 200 additional rare-heavy sentences after initial results proved unclear, a decision they explicitly label exploratory. This risks inflating the apparent frequency effect through selection. A second concern is the assumption that any recognizer hearing a word in synthetic speech constitutes a correct TTS output; the authors acknowledge recognizers can hallucinate context-appropriate transcriptions, which means the adjudicated error is potentially an underestimate. Finally, the paper notes that about half of the remaining errors may be spelling variants, yet without a listening test it is impossible to know whether the true TTS error is near zero or near 5% for very rare words.
The evidence strongly supports the paper's central methodological argument—that ASR error is a major confounder in rare-word TTS evaluation—though the support for specific numeric ratios is thin due to low counts. Comparisons to related work are fair and precise: the authors distinguish their per-frequency, per-word, multi-recognizer design from the corpus-level ASR-versus-listener comparison in Yang et al.'s Appendix C, and they correctly note the complementary problem raised by Luo and Wan that ASR can also hide real TTS errors by normalizing away reading mistakes. They avoid overclaiming by stating they do not re-run the TTS systems from Asaria et al. and by treating human-speech error as a diagnostic baseline, not a strict bound.
Reproducibility is partially promised but currently incomplete. The authors state they will release code, sentence lists and all transcripts, which is necessary for independent verification, yet no repository is linked in the manuscript. Experimental details are sparse: F5-TTS and Kokoro are run without disclosed hyperparameters beyond $N=8$ and $N=4$ candidates, and the only ASR decoding detail provided is greedy decoding. Crucially, the word-matching rule was revised after inspecting alignments from these sets, a post-hoc tuning step that is hard to reproduce without the exact revision history. The biggest barrier to reproduction, however, is epistemic rather than technical: because the paper lacks a listening test, any attempt to replicate its conclusions must resolve ambiguous spelling-versus-pronunciation cases subjectively.
Text-to-speech (TTS) is usually evaluated by transcribing its output with an automatic speech recognition (ASR) system, so the word error rate also counts recognizer errors. We measure these errors on human recordings of the test sentences. On LibriSpeech and FLEURS, the evaluation recognizer misses 33-67% of very rare words even in human speech, and all three recognizers miss 22-42%. Checking every word with two more recognizer families lowers the apparent TTS error on words that at least one recognizer hears two- to three-fold (very rare LibriSpeech words: from 10-13% to 3-5%), yet very rare words remain the hardest. Ignoring words the recognizer misses in human speech is biased: the other recognizers find TTS errors far more often on these words (4-6% vs. 0.2-0.3%) and overturn about half of the errors this mask keeps. As half of the remaining adjudicated errors may be spelling variants, a listening test is needed.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.