Analyzing performance of Retrieval Augmented Generation models for Question Answering Tasks
This paper investigates how Retrieval-Augmented Generation affects question answering across five South Asian languages. The authors construct a Wikipedia-based multilingual dataset, generate questions with Qwen 3 32B, and compare non-contexted, contexted, and RAG-based answering using four LLMs. They find that RAG improves QA unevenly—substantially aiding non-popular cultural questions in Bangla and Tamil while underperforming direct contexted answering in Hindi and Nepali—suggesting that retrieval quality and language-specific data alignment matter more than the choice of RAG framework.
The paper addresses an understudied problem—multilingual cultural QA in low-resource South Asian languages—and provides a usable dataset alongside a clean comparison of non-contexted, contexted, and RAG-based answering. However, the study lacks statistical rigor: accuracy is reported as point estimates without variance, confidence intervals, or significance tests, and the evaluation relies on brittle string matching against gold titles rather than semantic equivalence. As stated in Section VIII-A, "The overall accuracy results show a clear gap between non-contexted and context-supported answer generation," yet the authors never establish whether the gaps between contexted and RAG answering are reliable.
The dataset construction is transparent and pragmatic: the authors scrape Wikipedia in five languages, filter duplicates and stopwords, and validate question quality through a small human pilot before scaling generation. Their main empirical conclusion is carefully hedged; they observe in Section IX that "Figure 10 shows that non-popular culturally grounded questions receive higher RAG gain than popular culturally grounded questions across the evaluated languages," and explicitly warn that "This means that RAG cannot be treated as an automatically superior method for multilingual QA," a nuanced takeaway rarely emphasized in RAG benchmarking.
The evaluation protocol is methodologically fragile. Exact-match accuracy against expected titles ignores valid synonyms, transliteration variants, and morphological inflections, making the metric especially punishing for inflected languages like Tamil and Hindi. The comparison between "contexted" and "RAG" answering is conceptually muddled: contexted answering feeds the model the gold two-line context, whereas RAG retrieves chunks from a vector store built from the same contexts, so the experiment largely measures the noise introduced by retrieval rather than the value of external knowledge augmentation. The authors note in Section IX that "However, contexted answering remains stronger in Hindi and Nepali," but without reporting retrieval hit rates or relevance scores they cannot explain why retrieval fails. The answer-generation prompt in Section VI-A further rigs the task by demanding that models "Try to limit the answer to one word, and under no circumstances should you use more than five words," which suppresses legitimate surface-form variation.
The figures support directional trends but do not justify detailed comparative claims. The paper reports neither confidence intervals nor repeated trials, so statements such as "LlamaIndex performs slightly better in some cases" lack statistical support. Comparisons to prior work are descriptive rather than quantitative: the authors cite GraphRAG and DO-RAG gains but never benchmark their own exact-match scores against established multilingual QA datasets or leaderboards. They also claim in Section III-A that "Our choices included our native Bangla, along with Hindi, Urdu, Tamil and Nepali, covering the wider range of South Asia," which overstates the geographic and topical coverage of a single-source Wikipedia scrape centered on country landing pages.
Critical implementation details are missing. The authors disclose the embedding model (intfloat/multilingual-e5-large-instruct) and vector store (ChromaDB), but omit chunk size, overlap, top-$k$ retrieval count, and similarity threshold. They state in Section V that "Separate ChromaDB directories were maintained for each framework implementation in order to maintain framework-specific indexing behavior independent of each other aligning with reproducible research practices," yet no code, configuration files, or dataset release is mentioned. Because one model (Grok 4.1 Fast) is closed-source and API-dependent, and because API system prompts are not provided, independent reproduction is blocked even for motivated researchers.
Retrieval-Augmented Generation (RAG) models have become almost a cornerstone of open-domain question answering tasks; in which the quality of answers generated in the process, greatly relies on the effectiveness of retrieval mechanisms. In the realm of LLM, the last two years have seen tremendous progress in RAG architectures, with many models reporting to have improved performance through various design innovations. The cultural question and answering systems of South Asia, in particular, present a specific challenge since the region is home to the vast number of languages and traditions, which are closely interrelated. Along with these deeply intertwined historical narratives increase the need for culturally grounded and context-aware interpretation. Despite major architectural advances in LLMs, claiming significant improvements, challenges remain when answering these region specific questions in native languages. In this study, we aim to extract high-performing factors to guide the development of optimized hybrid RAG systems. We further explore whether the integration of RAG changes performance differently for culturally grounded and non-cultural questions in a South Asian multilingual setup, and whether RAG produces a higher overall percentage of correct answers than contexted answering. We also examine how LLM performance changes when RAG methods are integrated into the question-answering pipeline in comparison with non-contexted and contexted answering and, evaluate whether the introduction of RAG benefits popular or non-popular culturally grounded questions more. Not only does our work evaluate the current methods but also carves a path for future designs of more accurate and efficient question-answering pipelines.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.