WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias
Existing cultural bias benchmarks rely on costly manual annotation by native speakers, which limits multilingual scalability and introduces subjectivity. WikiOriginQA tackles this by using Wikipedia article creation dates across 342 language editions as an objective proxy for knowledge origin, then automatically generating short-answer QA pairs with LLMs. The resulting dataset supports the claim that LLMs are skewed toward Anglophone knowledge and that propagation into English Wikipedia is a key predictor of model performance.
WikiOriginQA offers an elegant automation strategy, but its conclusions are tentative. The dataset contains fewer than 650 questions across nine language sets, and the authors explicitly caution that intrinsic question difficulty may confound the apparent benefit of English propagation. While the extraction precision is supported by a Japanese manual audit, the generated questions exhibit evaluation artifacts—especially under exact-match scoring—that weaken causal claims about cultural bias.
The core methodological insight—using temporal metadata from 342 Wikipedia language editions to infer cultural provenance—is both novel and highly scalable. The authors validate this signal empirically: a manual review of the Japanese subset found that 88.9% of extracted items genuinely originated from Japan. Additionally, the performance gap favoring English-linked items replicates across GPT-5, LLM-jp, and multiple Qwen3 sizes, suggesting the Anglosphere skew is not an artifact of a single model family.
The dataset is too small to support broad generalizations: financial constraints limited construction to roughly 81–120 questions per language, and no power analysis is provided.
Second, the evaluation protocol is brittle. Exact-match accuracy penalizes correct answers that differ in romanization or translation, and the error analysis reveals that a non-trivial share of "incorrect" labels are actually spelling variations or valid alternate names. Finally, the central causal claim—that knowledge propagation into English drives higher accuracy—is confounded by entity difficulty; the authors acknowledge this but do not disentangle the two effects.
The evidence supports an Anglosphere performance skew, yet comparisons to prior work remain qualitative. The authors correctly note that KoLasSimpleQA’s reliance on $n_{\text{ill}}=0$ ignores propagated knowledge, whereas WikiOriginQA’s temporal signal handles it; this distinction is verified in the KoLasSimpleQA paper, which defines language-specific entries strictly as those lacking inter-language links. However, WikiOriginQA does not benchmark head-to-head against KoLasSimpleQA, CaLMQA, or BLEnD, so claims of superior scalability are not empirically validated. Moreover, the 88.9% precision claim rests solely on Japanese data, leaving other languages unaudited.
While the final dataset is released on Hugging Face under Apache 2.0, independent reproduction of the full pipeline is hampered by dependence on the closed GPT-5 API for question generation, iterative refinement, translation, and seven-criterion filtering. Exact prompts are summarized in appendices but the authors do not state whether generation code, random seeds, or per-example model outputs are released. Evaluation hyperparameters for open-weight models are also sparse, mentioning only default settings and a low reasoning effort for GPT-5.
Existing datasets on cultural knowledge have been typically created by human annotators familiar with each culture. However, the process requires considerable human effort, presenting a significant challenge to scalability. To address this challenge, we introduce WikiOriginQA, a QA dataset automatically constructed by objectively representing the origins of knowledge through the creation dates of different Wikipedia language editions. Experimental results with the created dataset confirm that LLM performance tends to be biased toward Anglophone cultures. Furthermore, the results suggest the importance of knowledge propagation into the Anglosphere. These findings demonstrate the validity of our automated approach, enabling cultural bias evaluations with superior scalability across diverse languages without extensive manual annotation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.