KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?
KoAGE investigates whether large language models can know, learn from context, and coin Korean neologisms. The authors introduce NeoTrace, an automated pipeline that collects and dates 4,271 neologisms across six genres, and propose a cutoff-aware benchmark that separates lexical recall from contextual inference and generation. By comparing model outputs with attested human coinages under original materials and simulated community settings, the paper offers a temporally grounded view of how models adapt to evolving vocabulary.
KoAGE is a well-conceived benchmark whose greatest strength is its temporal design: every model is evaluated on pre- and post-cutoff splits of the same lexicon, cleanly separating memorization from adaptation. The addition of two generation tasks that compare model coinages to human attested forms is novel and valuable. However, the pervasive use of LLM-as-judge, the very small sample size in the multi-agent community simulation, and single-run evaluation temper the strength of the conclusions.
The paper finds that 'models define only 13–30% of cutoff-year words, yet can infer meaning from one usage,' which captures the central tension between memorization and adaptation that the benchmark is built to expose.
The temporal alignment of words and model cutoffs is the paper's most important methodological advance, and the three-tier acquisition design—definition from word alone, definition from one usage, and structural formation analysis—effectively disentangles recall from comprehension. The NeoTrace pipeline is carefully validated, achieving 'overall precision reaches 95.5%, against 19% for the raw top-2k ranking,' and the framework transfers to English with only three component replacements. The main findings are consistent across 21 proprietary and open-weight models, suggesting robustness of the core patterns.
The evaluation relies on LLM judges at four separate stages, and although small-sample human validation is reported, the agreement rates leave non-trivial room for drift across 5,930 instances. The community simulation (G2) is especially fragile: it runs on only 25 cards with one backbone per community, and the authors concede that 'a stricter harness with controlled personas and more cards is needed before agents' coinage habits can be set against people's.' Additionally, the generation protocol itself shapes outcomes; the neutral method sentence in G2 'steers the form of the words, with abbreviations at 44% under a shortening-only wording and 32% under the neutral one, against 18% of the human coinages in the same sessions.' Finally, all acquisition results are single runs at vendor-default temperature, and several open-weight models are quantized to 4-bit, which may affect generation fidelity.
The evidence in Table 2 and Figure 6 supports the central claim that contextual inference far outstrips lexical recall, and the '13–30%' cutoff-year recall band is robust across five temporally analyzed models. Comparisons to related work are fair: the authors correctly note that 'KoNeoBench is the only dedicated benchmark, but it leaves critical problems open,' and Table 1 convincingly positions KoAGE as the only benchmark with per-model cutoff splits and coinage evaluation. However, the paper does not directly address how much of the strong post-cutoff 'contextual inference' might be attributable to social or lexical contamination in pretraining data rather than genuine compositional reasoning.
The authors release their code, dated lexicon, and prompts, and Appendix B documents exact model identifiers, API versions, and decoding settings. A critical barrier to reproduction is that 'All results are single runs at vendor default temperature, since several current models accept no temperature argument,' and open-weight models use mixed precision, with 'the 32B/33B EXAONE models in 4-bit NF4 with bfloat16 compute.' While fixed seeds for option shuffling and fixed-length masking are helpful, independent reproduction would require substantial API costs and exact judge access, and the small G2 card set means community-coinage statistics carry wide Wilson intervals.
Korean neologisms arise through diverse mechanisms, a demanding test of language models beyond lexical memorization. To enable systematic evaluation, we introduce NeoTrace, an automated, rerunnable framework that collects and dates neologisms across genres. Its 4,271 entries support KoAGE, a cutoff-aware benchmark with three acquisition and two generation tasks---the latter, to our knowledge, the first to compare model coinages with attested neologisms under their original materials and community context. Across 21 models, lexical recall diverges from contextual inference: in a five-model temporal analysis, models define only 13-30% of cutoff-year words, yet can infer meaning from one usage. However, structural understanding remains limited: even correct definitions barely improve formation analysis of pre-cutoff neologisms. Given word-building materials, single models often follow human formation mechanisms, whereas multi-agent coinage favors abbreviation and depends on interaction framing. KoAGE thus separates what models know, learn from use, and coin.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.