Investigating Multi-Cultural Value Selection Biases in LLMs
Most cultural evaluations of LLMs reduce user identity to a single country label, ignoring that real users hold multi-layered identities. This paper introduces a composite-identity setup where a system-prompt persona names one country while a user turn signals a second identity, either explicitly by naming a country or implicitly through query language. By probing three open-weight models with WVS multiple-choice questions and 1.6 million open-ended responses, the authors show that language and country cues shift model values in different, non-interchangeable ways—a finding with direct implications for how multilingual and multicultural personalization should be engineered.
The paper mounts a methodologically careful, large-scale demonstration that LLMs do not treat language and explicit country cues as interchangeable cultural signals. The core result—that WVS multiple-choice answers shift away from a language-cued country's baseline while explicit country mentions move open-ended values toward it—is robustly quantified via Jensen–Shannon distance and a frozen 512-cluster taxonomy. However, the absence of human ground truth for composite identities means the study characterizes model-internal behavior rather than alignment with genuine multicultural values, and the model-family disagreements in closed-ended probing remain under-explained.
The factorial design is a genuine advance: fixing one identity in the system prompt and varying a second through explicit or implicit cues isolates the contribution of each signal far better than prior single-cue studies. The scale is impressive—over 1.6 million open-ended responses—and the gap-closure metric $\Delta_q(s,c)$ gives an intuitive, comparable measure of cue influence. The openly documented prompts and model choices, together with the honest discussion of country-as-culture and language-as-identity limitations, lend credibility to the empirical claims.
First, the WVS language-cue baseline is always in English, whereas the cue condition is in the target language, so the observed negative shift may be dominated by cross-lingual distributional drift rather than a rejection of the cued culture; the authors note this but do not disentangle it. Second, the model families disagree sharply in the multiple-choice task: Gemma moves toward the named country for 10 of 13 cases, while Llama and Mistral move away from it for 13 and 12 cases respectively, yet no architectural or training explanation is offered. Third, the entire open-ended pipeline—translation with Aya, value extraction with Llama-3.1-8B-Instruct, and clustering into 512 clusters—propagates automated errors, especially for low-resource languages, and validation relies on LLM judges rather than humans.
The evidence strongly supports the central claim that country and language cues should be treated as distinct signals. The comparison to prior work is fair: the authors distinguish their composite-identity setup from prior single-cue studies and correctly note that earlier work treats language and country as rival proxies rather than simultaneous, potentially conflicting cues. The AIRiskDilemmas ablation adds an important negative result: expressed values in free text only weakly predict value-laden choices ($\rho = 0.22$), implying that open-ended alignment does not automatically transfer to decision-making tasks.
Experimental details are solid: models, vLLM settings, prompts, and taxonomic construction are documented in the appendices. However, the authors do not explicitly state that code or data will be released, and reproducing the full pipeline would require replicating a chain of external services—Google Translate, the Aya back-translation model, and the Values-in-the-Wild annotator—whose exact API versions and checkpoints may drift. The reliance on a frozen but automatically induced 512-cluster taxonomy also means that independent reproductions may differ in label granularity unless the exact embedding model and clustering seed are preserved.
Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs under simultaneous identity cues, with a country persona in the system prompt and a second identity signaled in the user turn, either by naming a country (explicit cue) or through the query language (implicit cue). We test World Values Survey (WVS) multiple-choice probing and open-ended sociopolitical generation. In multiple-choice probing, language cues shift answers about three times more than explicit cues but away from the associated country's distribution, and explicit cues do not consistently move answers toward the named country. Across 1.6 million open-ended responses, naming a country closes 30-44% of the gap to the model's own responses for that country, and explicit cues produce more differentiated value changes than language cues. Our results suggest that country and language cues should be evaluated as distinct signals rather than interchangeable proxies for culture.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.