CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing
Most benchmarks test whether LLMs can produce or understand mixed-language text, but not whether they decide when, where, and why to switch for narrative effect. CSWBench fills this gap with a creative-writing benchmark that pairs focused and extended narrative generation with a theory-informed evaluation framework assessing event-level licensing, contextual fit, and functional gain alongside whole-text multilingual orchestration. It matters because code-switching in stories should be a deliberate narrative resource, not just surface multilingual decoration.
The paper makes a genuine conceptual contribution by treating code-switching as a situated, narrative language choice rather than mere token-level alternation. The distinction between genuine code-switching and surface multilingual material (Figure 1) is theoretically precise, and the dual-level framework is well motivated. That said, the experimental design is undermined by heavy dependence on a single LLM—GPT-5.6-Sol—for construction, annotation, and judging while also evaluating it. Although family-exclusion ablations preserve the overall ranking, the GPT-side judge rates GPT-family outputs about $0.5$ points (/20) higher relative to human ratings in Focused Narrative. Additionally, the Licensing dimension ($D_1$) is effectively a ceiling test: every model scores between $4.941$ and $4.999$ out of $5$, so it contributes almost no discriminative signal.
The theoretical scaffolding is unusually careful for a benchmark paper. The authors ground CSWEval in interactional sociolinguistics and literary multilingualism, asking not just whether a switch is grammatical, but why this language is used by this speaker here and now. Human validation of the automated pipeline is strong, and the sensitivity checks—counterfactual Chinese-only replacements and position-swap perturbations—lend credence that the framework responds to the constructs it claims to measure.
Three issues limit the benchmark's validity and generalizability. First, the data source is heavily synthetic: about 82% of final prompts derive from SwitchLingua, which raises questions about how well the task reflects spontaneous bilingual interaction. Second, $D_1$ Licensing suffers from a severe ceiling effect, with even the weakest model scoring $4.941/5$; this suggests the licensing checks are too permissive or that the model-generated prompts provide overly explicit scaffolding. Third, the score absolutes are fragile to upstream pipeline choices: changing the event annotator shifts Extended scores by a pooled mean of $+0.745$ points (/20), meaning independent reproduction could yield markedly different numbers even if rankings are stable.
The evidence supports the main empirical claims, but with caveats. The authors fairly note that prior benchmarks largely neglect generation, contextual choice, orchestration, and salience (Table 1). Their finding that multilingual literary integration shows the greatest variation across model means is the most compelling discriminator. The moderate cross-level correlations—Contextual Appropriateness and Functional Gain correlate with Whole-text performance at $r=.254$ and $r=.255$—support the view that event-level and whole-text quality are related yet distinct, though these values are derived from LLM judges rather than human raters and should be interpreted cautiously.
The authors release code (Apache-2.0) and prompts (CC BY-NC 4.0) and report decoding hyperparameters: temperature $0.7$, top-p $0.7$, and a $30{,}000$ token output budget. However, full reproduction is hindered by dependence on specific, sometimes proprietary LLM versions for Seed construction, event discovery, evidence construction, and judging. Because changing the event annotator shifts Extended scores by a pooled mean of $+0.745$ points (/20), exact score reproduction requires not only the same weights but the same model versions and API behavior. The benchmark would benefit from a fully human-labeled subset released as a stable reference for future models.
Large language models (LLMs) have demonstrated multilingual capabilities across diverse language generation and understanding tasks. Yet effective code-switching requires more than multilingual proficiency: models must decide when, where, and why to switch languages in ways that fit the surrounding context. This context-sensitive ability remains insufficiently evaluated. We introduce CSWBench, a benchmark for evaluating context-sensitive code-switching in creative writing through two complementary settings: Focused Narrative and Extended Narrative. We further propose CSWEval, a theory-informed framework with two complementary levels: event-level evaluation of Licensing, Contextual Appropriateness, and Functional Gain, and whole-text evaluation of task-specific quality---overall literary and narrative quality in Focused Narrative and quality emphasizing multilingual organization in Extended Narrative. Evaluating twelve LLMs, we find that models exhibit meaningful but uneven code-switching competence across narrative contexts. Model differences are particularly pronounced in multilingual literary integration, while multilingual-focused dimensions show stronger difficulty effects overall than the general narrative dimensions; narratively salient switches are associated with stronger contextual and functional realization; and event-level CS quality is moderately associated with whole-text performance.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.