How Well Do Decision Models Transfer to Vietnamese? An Empirical Study of Text Classification
Decision models such as JEV promise efficient zero-shot classification by predicting over predefined labels instead of generating free-form text, yet their behavior on Vietnamese tasks has been largely uncharted. This study benchmarks four specialized decision models across 13 Vietnamese classification settings spanning topic, emotion, hate-speech, sentiment, intent, and NLI tasks. The authors systematically test English, Vietnamese, and bilingual label descriptions, and they introduce three realistic linguistic perturbations—teencode, typographical noise, and diacritic removal—to probe robustness. The core takeaway is that JEV dominates overall but remains vulnerable to accent stripping, while bilingual label descriptions do not reliably help any model.
The paper delivers a practically needed, wide-coverage benchmark of decision models on Vietnamese, and its diacritic-robustness findings are especially actionable. However, the comparison is heavily confounded by model-access asymmetry: JEV is a hosted proprietary API with undisclosed architecture and scale, while the other three models are local open-weight encoders of roughly 300M parameters. GLiNER's repeated collapse to near-random scores—for example, 5.10 macro-$F_1$ on UIT-VSMEC and 6.25 on ViHSD—together with extreme label concentration (assigning a single label to >99% of inputs on multiple datasets) strongly suggests schema or configuration error rather than inherent failure. These issues make it hard to trust that observed gaps reflect true capability differences.
The breadth of tasks and the linguistically grounded perturbation suite are genuine strengths. The study documents that diacritic removal is by far the most damaging noise type, producing mean macro-$F_1$ drops of 9.51–15.47 points—considerably more than teencode or typos—which is an important, actionable finding for Vietnamese text-classification deployments. The label-description analysis also moves beyond blanket recommendations by reporting task-dependent effects and by including per-class confusion analyses that reveal concrete failure modes such as JEV's tendency to over-predict compound intents on PhoATIS.
A primary flaw is the failure to control for or even discuss model scale and access modality. JEV is queried via a closed API with unknown parameters, while LAYA, GLiNER, and Quyet are ~300M-parameter local checkpoints; JEV's large margins may therefore reflect frontier-scale capacity rather than the superiority of decision interfaces per se. GLiNER's near-total prediction collapse on several datasets (e.g., assigning a single label to 99.7% of instances) is never investigated as a possible implementation bug, undermining confidence in the reported rankings. The LLM baseline in Section 4.2 is borrowed from prior work with likely mismatched prompts and protocols, rendering cross-study superiority claims tentative. No statistical significance tests are reported, and the dismissal of bilingual descriptions rests on aggregate means that ignore large per-task swings—for instance, Quyet gains +21.06 macro-$F_1$ on ViNLI with Vietnamese labels while losing −14.23 on ViHSD.
The internal tables generally support the headline ranking that JEV leads on most tasks, but GLiNER's consistent label-concentration failures imply implementation error rather than a meaningful baseline, weakening the evidentiary foundation. The comparison to general-purpose LLMs imports previously reported zero-shot results from Thin et al. (2024) with no assurance of prompt parity or API version alignment, so claims such as "JEV outperforms Qwen3-32B" should be treated cautiously. The perturbation methodology is well-specified and the controls on label order are appropriate, yet the paper's conclusion that bilingual labels offer "no consistent improvement" is drawn from aggregate means that mask instructive per-dataset variance.
Reproducibility is mixed. The authors identify exact open-weight checkpoint URLs, perturbation package versions (nom-vn v0.2.37, pyvi v0.1.1), a fixed random seed of 42, and dataset sources. However, no public code repository or data-preparation scripts are mentioned; the JEV API call configuration (decoding parameters, exact schema formatting, retry logic) is largely omitted; and the GLiNER template that produced catastrophic label collapse is not provided, making independent diagnosis impossible. Because JEV is only available through a hosted API, exact reproduction requires external service access that the authors cannot guarantee.
Recent advances in specialized decision models have expanded the possibilities for zero-shot classification, yet their effectiveness on Vietnamese remains underexplored. This study evaluates four specialized decision models across 13 classification settings spanning six task groups, examining zero-shot performance, sensitivity to label-description language, and robustness to Vietnamese linguistic variations. The evaluation considers English, Vietnamese, and bilingual label descriptions, alongside common linguistic perturbations involving teencode, typographical errors, and diacritic removal. Under English label descriptions, JEV achieves the strongest overall performance, ranking first in 11 classification settings. The effects of label-description language vary across models and datasets, with bilingual descriptions providing no consistent improvement. Robustness experiments reveal that diacritic removal causes the most substantial performance degradation, reducing mean macro-$F_1$ by 9.51--15.47 points across model configurations, considerably more than teencode or typographical errors. These findings highlight the importance of label-description choices and Vietnamese orthographic variation when evaluating specialized decision models for zero-shot text classification.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.