Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
LLM-as-a-Judge is widely used for multilingual evaluation, yet prompting strategies are usually validated only in English. This paper tests whether those strategies transfer across judge models and languages by benchmarking nine prompting strategies with four open-weight judges on 27 languages and four tasks. The central finding is that judge identity dominates prompting effects, with strategy effectiveness reversing across models and collapsing in low-resource languages, underscoring that prompt design must be validated per judge and language.
The paper presents a large-scale, well-controlled empirical argument that prompting strategies for LLM judges do not transfer across models or languages. A two-way ANOVA on over 1M pointwise scores shows judge identity explains roughly seven times more variance ($\eta^{2}=0.087$) than prompting strategy ($\eta^{2}=0.012$), while the judge $\times$ strategy interaction ($\eta^{2}=0.056$) exceeds the strategy main effect, indicating that strategy effectiveness is conditional on the judge rather than intrinsic to the prompt.
The experimental design is a clear strength: by scoring gold-reference outputs, the authors cleanly isolate judge behavior from generation quality, and the 27-language coverage—from high-resource English to low-resource African languages—fills a genuine gap. The per-judge reversal of target-language prompting is particularly striking: Monolingual prompting raises Gemma 3’s mean score by +0.83 and lowers Aya Expanse’s by −0.32 on identical inputs, with a maximum per-language drop of −2.16 on Yoruba. Releasing 1.07M pointwise scores and prompt templates further strengthens reproducibility.
A major limitation is the gold-reference ceiling effect, which compresses variance in QA and reference-based MT and likely attenuates strategy discrimination; while the authors acknowledge this and provide a supplementary AfroBench check, the main ANOVA and cross-strategy comparisons remain subject to truncated ranges.
A second concern is that human-alignment validation for non-English settings is narrow: SSA-MTE covers multilingual MT, but SummEval and WMT MQM are English-only, so claims about low-resource validity rest primarily on internal score patterns rather than human correlation in those languages. Finally, native-speaker verification of target-language prompts was only performed for a subset of languages, meaning some Monolingual effects may reflect translation quality rather than judge architecture.
The factorial evidence supports the core claim because the design directly manipulates judge, strategy, language, and task while holding content fixed. However, comparisons to prior work would be stronger if the authors benchmarked their best LLM-judge configurations against established metrics such as BLEU or COMET in low-resource settings to show practical superiority, rather than only reporting that Role Prompting achieves the highest aggregate human correlation ($\rho=0.483$). The paper fairly notes that English-only prior studies did not vary prompt or judge jointly, but the absence of baseline metric comparisons leaves open whether any of these judge–strategy pairs are actually useful replacements for reference-based evaluation.
Reproducibility is strong on data release—the authors publish 1.07M scores, rubric-level dimension scores, and templates—but weaker on experimental detail and access. Full reproduction requires substantial GPU compute (A100/H100) and specific open-weight checkpoints. Hyperparameters such as temperature and top-p are reported, yet the exact translated prompts for all 27 languages are only partially verified, and the main analysis relies on a gold-reference design whose ceiling effects are confirmed only in a limited AfroBench subset. Releasing the complete multi-language prompt set and a broader system-output evaluation would remove the biggest practical barriers to independent replication.
LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.