Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
The paper tackles the scarcity of multilingual safety and reasoning benchmarks for LLMs by proposing a reasoning-guided translation framework. It separates translation strategy selection via a reasoning model from large-scale generation by a cheaper translator, then validates, repairs, and independently screens outputs in a task-specific manner. Using seven English sources across 11 target languages, the authors construct a source-aligned collection and expose language-dependent vulnerabilities such as inconsistent refusals and substantial reasoning degradation.
The paper delivers a scalable and auditable workflow for multilingual benchmark construction, but its strongest claims about semantic fidelity and downstream model behavior outpace the evidence. Structural coverage of $99.94\%$ is carefully reported, yet without completed human bilingual review, assessments of preserved benchmark meaning depend on automatic proxies and an LLM judge the authors admit is lenient.
Separating reasoning-guided configuration selection from production generation is a pragmatic, cost-efficient design. The failure-handling strategy—regenerating failed records directly from English sources rather than silently discarding them—is methodologically sound and prevents artificially inflating coverage statistics.
The configuration pilot uses only 70 Thai instances per setup, so the selected GPT-5-mini configuration may not generalize across all 11 languages or seven heterogeneous sources. Downstream comparisons are further weakened by a fixed generation budget that truncates non-English responses more heavily, potentially masking true multilingual gaps behind tokenization artifacts. The semantic fidelity pipeline also rests almost entirely on automatic proxies and a cross-family judge that rates $99.6\%$ of translations acceptable—a near-ceiling rate the authors appropriately treat as screening rather than proof of equivalence.
The authors fairly concede priority on mathematical benchmark translation to MCLM and MMATH, positioning their contribution as an integrated workflow rather than novel task coverage. While structural validity is supported by strong evidence, semantic equivalence remains undersupported; the paper itself cautions that automatic checks do not establish equivalent task difficulty across languages.
Prompts, validation schemas, and final counts are documented, and the collection will be released with metadata flags. However, reproduction is hampered because intermediate repair attempts were not consistently logged, repair provenance is detailed only for WildChat and WildJailbreak, and the pipeline depends on API models without pinned versions or full disclosed prompts for the reasoning-guided selection stage.
Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94\% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.