Synthetic-to-Real Code-Switched ASR: A Spontaneous Nepali-English Podcast Benchmark
Nepali–English code-switched ASR suffers from script confusion and switch-point fragility, yet human annotated conversational audio is costly. This paper curates a 15.45-hour spontaneous podcast benchmark and adapts a 1.2B-parameter multilingual encoder–decoder model using only 0.95 hours of synthetic dual-engine TTS data. The core finding is that supervised fine-tuning drives almost all cross-domain transfer, while subsequent GRPO and script-aware rewards add only marginal, hard-to-disentangle gains. The released data and the cautionary ablation make the work practically useful.
The paper's most durable contribution is the rigorously audited 13,044-utterance benchmark, with strict speaker disjointness, dual-annotator verification, and frozen test integrity. Methodologically, the two-stage synthetic-to-real pipeline is sound, but the empirical headline is muted by the authors' own ablations: after SFT, GRPO changes only 1.69% of test utterances and improves switch-point error rate by a scant $-0.07$ percentage points. The ablation showing that token-level script gating degrades performance ($MER$ rises from $18.92\%$ to $19.14\%$) is a helpful negative result. Taken together, the study is an honest, carefully tested empirical account whose main lesson is that in a frozen-encoder encoder–decoder, RL-based script control is largely redundant once SFT has converged.
Benchmark construction is exemplary: spontaneous podcast clips were individually audited to remove machine-translation leakage, acoustic loops, and diarization glitches, achieving $98.4\%$ inter-annotator character agreement. The evaluation protocol sensibly decouples lexical accuracy from orthographic fidelity using purpose-built metrics such as $MER=(E_{\text{w,en}}+E_{\text{c,ne}})/(N_{\text{w,en}}+N_{\text{c,ne}})$, $SPER$, and $SCR$, and tests significance with paired bootstrap and episode-clustered block resampling. Monolingual retention checks on Nepali and English corpora further demonstrate responsible experimental hygiene.
The improvements attributed to GRPO are vanishingly small on a per-utterance basis: of 10,214 test utterances, only 173 differ after policy optimization, and the authors candidly admit that "without an extended SFT compute control, the marginal gain cannot be separated with certainty from extra gradient steps." Because rollout groups exhibit "near-zero script variance (>75% of mini-batches had zero within-group script variance)," the policy search space is effectively collapsed, making the script reward term irrelevant by construction rather than by architectural necessity. The synthetic training set is also tiny—517 utterances ($0.95$ h) from just two Nepali male timbres—limiting acoustic diversity and raising questions about domain coverage. Finally, the dev/test split is confounded by conversational role: dev hosts average $2.65$ switch points per utterance versus $2.11$ in test, so boundary-optimization methods may tune on a denser code-switching regime than they encounter at test time.
Statistical testing is thorough: paired bootstrap ($B=1{,}000$) and episode-clustered block bootstrap appropriately account for speaker non-independence, yielding tight confidence intervals that exclude zero. Comparisons to prior work are qualitatively fair: the authors correctly distinguish their encoder–decoder setting from the decoder-only speech LLMs used by Kumar et al., who report large WER reductions with GRPO on synthetic speech. However, no direct baseline re-implementation is provided, so the claim that encoder–decoder architectures behave differently under GRPO rests on indirect evidence, and the authors acknowledge that "evaluating multi-seed training variance and benchmarking external foundation models (e.g., Whisper) remain valuable future work."
Reproduction is well supported. Code, configurations, and evaluation manifests are released on GitHub; transcripts and reconstruction scripts are on Hugging Face under CC BY-NC-SA 4.0. Hyperparameters are exhaustively documented—e.g., AdamW with $lr=1.0\times 10^{-5}$ for SFT and $1.0\times 10^{-6}$ for GRPO—and training is lightweight ($\sim$21 min on one RTX 4090). The chief practical barrier is data provenance: benchmark audio is reconstructed from public YouTube sources via yt-dlp, so strict bit-exact reproduction is vulnerable to CDN re-encoding or video removal, although the authors provide acoustic tolerance fallbacks (Pearson $\rho>0.999$) and SHA-256 checksums.
Nepali-English code-switched conversational speech presents severe challenges for automatic speech recognition (ASR), chiefly through script confusion (inappropriate mixing of Devanagari and Latin orthography) and elevated error rates at language switch points. Supervised fine-tuning (SFT) can adapt multilingual foundation models, but human code-switched audio is expensive to collect and annotate. We curate a new benchmark of 13,044 spontaneous podcast utterances (15.45 h total; 10,214 test, 2,830 dev), audited and verified to eliminate machine translation leakages and acoustic loops. We investigate a two-stage synthetic-to-real adaptation framework: Stage 1 applies SFT on 0.95 h of speech synthesized via a multi-speaker, dual-engine TTS pipeline; Stage 2 explores Group Relative Policy Optimization (GRPO) with transcription and script rewards. We find that synthetic SFT provides the primary cross-domain transfer to spontaneous human speech, reducing Mixed Error Rate (MER) from 23.40% to 21.52% (-1.88 pp) and switch-point error rate (SPER) from 35.89% to 28.77% (-7.12 pp). Subsequent GRPO yields small, statistically significant adjustments on 173 test utterances (1.69%), driving English script confusion (SCR_en) from 6.45% to 6.36% (p <= 0.001) and SPER to 28.70% (p = 0.022 clustered). Crucially, ablation reveals that in this SFT-converged encoder-decoder setting, an auxiliary script reward term yields performance identical to MER-only optimization because rollout candidate groups exhibit near-zero script variance on synthetic data. On monolingual benchmarks, SFT improves Nepali MER by 2.71 pp (stabilizing at 18.42% after GRPO, a 2.38 pp gain over baseline) while preserving English lexical WER (3.38% vs. 3.33%). Verified transcripts, timestamps, episode metadata, and reconstruction scripts are publicly released.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.