CrisiLangs: A Multilingual Resource for Crisis Warning Generation
Crisis warning generation is currently English-centric, hampering emergency response in linguistically diverse Europe. This paper introduces CrisiLangs, a multilingual dataset extending 1,000 expert-reviewed English event-message pairs into Italian, Bulgarian, German, and Greek via professional native speakers. The authors benchmark the resource by jointly fine-tuning Llama-3.1-8B-Instruct across all five languages, showing gains over zero-shot baselines but also revealing that generation quality and the utility of in-prompt guidelines vary substantially by language.
The paper makes a practical, well-motivated contribution by filling a clear gap in multilingual crisis NLG data. The dataset design—human-translating outputs while machine-translating inputs—is sensible, and the experiments directly test whether fine-tuning and explicit guidelines help. However, the benchmark is limited by a minimal human evaluation protocol (one annotator per language) and a narrow comparison against only a zero-shot baseline, leaving open how the approach compares to domain-adapted translation or language-specific models. As noted, "The fine-tuned models score higher than the baseline on all three metrics in every language, temperature and prompt configuration," yet the human study uncovered that "In Bulgarian, the generations using the guidelines led to the rejection of all three candidates in 13 items because of errors in the generated text."
The strongest aspect is the dataset construction rationale: focusing human effort on the model output rather than the input reflects realistic deployment constraints and prioritizes evaluation quality. The temperature sweep and the inclusion of a human preference study provide useful guidance for practitioners, successfully surfacing failures that automatic metrics miss entirely. For example, the authors observe that "the Bulgarian +guidelines generations score close to the plain ones, yet in 13 of the 50 items none of the candidates was acceptable," demonstrating the necessity of human validation. Similarly, the decision to human-translate only the messages is justified by the argument that "Human translation was limited to the warning messages. This was a deliberate design choice: the message is what the model has to produce and what the human evaluation judges, whereas the event description and the guidelines only enter as input."
The evaluation has serious methodological limitations. Human preferences rest on a single expert per language with no inter-annotator agreement, making the per-language conclusions suggestive rather than reliable. The training data is also very small relative to the English source: "Finally, the resource provides 1,000 expert-reviewed event-message pairs per language; by comparison, CrisiText contains more than 100,000 English pairs." This raises concerns about coverage, especially for rare event types such as tsunamis (9 events) or volcanic eruptions (12 events). Additionally, the benchmark omits strong baselines such as language-specific fine-tuned models or domain-adapted machine translation systems, and it does not test the promising cross-lingual setting where the model receives English events and generates target-language messages directly.
The automatic metrics (ROUGE-L, BLEU, BERTScore) consistently favor the fine-tuned model over the zero-shot baseline, but these are reference-based measures that reward lexical conformity rather than communicative effectiveness or safety. The authors acknowledge that "No configuration is consistently better, so the automatic metrics do not settle which configuration to prefer," which honestly frames their limitations. However, the paper does not compare against related generation or translation approaches from prior work, such as those by Staiano et al. or Castaldo et al., making it difficult to assess whether fine-tuning on 1,000 pairs per language is the best available strategy. The evidence for the utility of the dataset is therefore positive but circumscribed.
Reproducibility is partially supported: the dataset is released on HuggingFace, the base model is openly available, and Appendix A reports detailed training hyperparameters including "rank 16, alpha 16, dropout 0" and a learning rate of $3\mathrm{e}{-4}$ with a cosine schedule. Nevertheless, the fine-tuned model weights, generation outputs, and annotation spreadsheets do not appear to be released, which would block independent reproduction of the human preference results. Moreover, because "Translation and preference annotation were carried out by a single native speaker per language," exact replication of the subjective evaluation is effectively impossible without access to those same individuals or comparable crisis-communication experts.
Communicating with the public during an emergency entails issuing warning messages promptly. In Europe, it often requires producing them in more than one language. Recent work has shown that language models can be trained to generate warning messages as a crisis unfolds, but the available data covers only English. To fill this gap, we present CrisiLangs, a multilingual extension of CrisiText (an English dataset of crisis events paired with warning messages written according to expert communication guidelines) to Italian, Bulgarian, German, and Greek. Expert native speakers translated the warning messages of 1,000 events drawn from 411 scenarios, for a total of 5,000 event-message pairs across five languages. We test the resource to fine-tune a multilingual model on all five languages jointly, with and without the emergency guidelines in the prompt, and compare it with a zero-shot baseline. The fine-tuned models outperform the baseline on ROUGE-L, BLEU and BERTScore in every language and configuration. In a human preference study, low decoding temperatures receive the most preferences in all five languages and in both configurations, while the effect of the guidelines varies by language.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.