Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning

The attention-fusion ablation (Table 3) produces a meaningful finding: using video as query and text as key/value ($\mathbf{e}_{\text{vid}\to\text{txt}}^{\text{cross}}$) slashes WER to 21.06%, whereas the reverse direction collapses to 80.83%. This suggests that temporally anchoring linguistic content via visual queries is the critical mechanism, not bilateral cross-attention. The subjective evaluation (Table 2) is also encouraging: WYS outperforms all baselines on every MOS criterion, with particularly large gains over LipVoicer in naturalness (4.00 vs. 3.56 on LRS2) and audio quality (3.92 vs. 3.49).

First, the paper advertises state-of-the-art textual accuracy, yet LipVoicer achieves lower WER on both datasets (18.90% vs. 20.13% on LRS2; 22.24% vs. 23.73% on LRS3). The authors rebrand this as a favorable trade-off for naturalness, but that is a goalpost shift: if WER is the primary metric for the one-to-many problem, WYS is not the leader. Second, the gulf between inference-text and ground-truth-text conditioning (Table 7a: WER 20.13% vs. 9.98%) reveals that the pipeline is bottlenecked by lip-reading errors, not by the generative model itself. The problem is therefore only shifted, not solved. Third, the full fusion embedding $\mathbf{e}_{\text{syn}}$ delivers merely a 0.93 percentage-point WER improvement over the single-direction $\mathbf{e}_{\text{vid}\to\text{txt}}^{\text{cross}}$ (21.06% → 20.13%), raising doubts about whether the added self-attention and opposite-direction cross-attention justify the extra parameters and complexity.

The evidence supports the specific claim that WYS improves LSE-C/D and subjective MOS, but it weakens the broader narrative of solving the one-to-many ambiguity. The comparison to LipVoicer is technically fair in that both use text derived from lip reading, yet the rhetorical contrast—classifier-free vs. external classifier—is overstated. WYS still depends on external frozen models (Auto-AVSR lip reader, BERT, HiFi-GAN), so the architectural simplification is one of mechanism, not of reliance. The paper also omits several baseline results in Table 1 (SVTS, FTV, AlignDiT on LRS2; SVTS on LRS3) without explanation, which hampers cross-method comparison. The robustness experiment (Table 8) is valuable but confirms that when lip-reading WER exceeds 50%, WYS degrades to ~78.7% WER, indistinguishable from LipVoicer, undermining its reliability claim.

Training details are relatively complete: single RTX PRO 6000 GPU, Adam optimizer with lr $2\times 10^{-4}$, batch size 64, 1M iterations, CFG null probability 0.1, and Euler ODE solver with $T=100$ steps at $\omega=2.0$. Pre-processing uses standard face-alignment landmarks for a 96×96 lip crop. However, several practical details remain vague: the exact temporal margin for word-level text alignment (Appendix C mentions only a 'small margin'), whether the HiFi-GAN vocoder is identical across all compared methods, and whether baseline numbers are retrained or taken from prior papers. A GitHub project page is listed but the paper gives no explicit code-release commitment or checklist. Independent reproduction is therefore plausible in principle but may stall on unspecified implementation minutiae and the need for the same frozen Auto-AVSR lip-reading checkpoint.