Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models
The paper asks whether language models can transfer hierarchical syntactic generalizations across languages. It introduces a cross-lingual poverty-of-stimulus (CPOS) design where positive evidence for hierarchical subject-verb agreement is present only in the source language (L1), and tests transfer to an ambiguous target language (L2). The core finding is that transfer is weak and only reliably detectable when the two languages share vocabulary tokens, challenging the view that models abstract language-universal hierarchical rules independently of lexical overlap.
The paper finds limited cross-lingual transfer of hierarchical subject-verb agreement bias. Under full vocabulary overlap, English-to-English transfer yields a $\Delta\mathrm{acc}$ of +5.45 on CLAMS, but cross-lingual effects drop to +1.68 for German and below +1.1 for Russian and Japanese. Crucially, removing vocabulary overlap almost eliminates even within-language transfer, indicating that any observed generalization is tightly coupled to shared lexical embeddings rather than abstract structural rules. The overall assessment is that models do not robustly acquire a language-universal hierarchical bias that transfers independently of lexical form.
The CPOS experimental design is elegant and well-motivated: by removing positive hierarchical evidence from L2 and freezing the Transformer during continued pretraining, the authors create a clean causal test of transfer. The vocabulary-overlap ablation is carefully implemented following Kallini et al. (2025) and directly isolates a key confound. Statistical analysis with Bayesian mixed-effects models lends rigor to small-effect interpretation, and the authors are transparent about the limitations of their synthetic benchmarks.
Effect sizes are small and decline steeply with typological distance, with Japanese L1 producing only a $\Delta\mathrm{acc}$ of +0.53 on CLAMS. The two evaluation benchmarks disagree: MultiBLiMP shows a negative transfer effect for German L1 and lacks credible effects for non-English L1s, which the authors acknowledge may reflect low statistical power or benchmark-specific artifacts. In addition, freezing all Transformer layers during L2 training is a strong, ecologically invalid design choice that prevents the model from adapting its syntactic representations to L2, potentially artifactually suppressing transfer and limiting relevance to standard multilingual training.
The evidence supports the headline claim only under contrived conditions: shared vocabulary and a frozen Transformer. The comparison to prior work is fair—Kallini et al. (2025) and Škrjanec et al. (2026) are appropriately cited and their vocabulary-overlap methodology is replicated faithfully. However, the discrepancy between synthetic CLAMS and naturalistic MultiBLiMP leaves it unclear whether the observed transfer reflects genuine hierarchical bias or surface statistical regularities tuned by the minimal-pair design. The authors honestly list this ambiguity as a limitation but do not resolve it.
Code and data are publicly released and hyperparameters are fully documented in Appendix C, which supports independent reproduction. However, only three random seeds are used per condition, and the authors note that this low replication limits statistical power, especially for the smaller MultiBLiMP benchmark. The Stanza-based corpus parsing pipeline and the exact token-ID offsetting procedure for vocabulary overlap are also detailed, though reproducing the exact filtered Wikipedia corpora requires careful adherence to the described heuristics.
Language models (LMs) often exhibit successful cross-lingual transfer through continued (pre)training, still their underlying process remains unclear. One possibility is that LMs can leverage the evidence toward language-universal principles, namely, hierarchical structure, from a source language (L1) to a new language (L2), while many other non-linguistic explanations, e.g., improved initialization of position embeddings, can be potentially possible. In this study, to quantify the cross-linguistic transferability of linguistic evidence in cross-lingual LM training, we introduce a new experimental design, namely, a cross-lingual poverty-of-stimulus (CPOS) setting that ablates how LMs' hierarchical generalization in L2 can be led by structural evidence provided in L1. We contrast counterfactual L1 and L2 learning scenarios where the L2 corpus is ambiguous between linear and hierarchical grammar rules, while the amount of positive evidence for the hierarchical rule is controlled on the L1 side. Our experimental results show that LMs can partially, yet not perfectly, transfer L1 structural evidence to L2, yielding only a small but detectable hierarchical bias, while the strength of transfer depends on language pairs. We further ablate the conditions for successful transfer, motivated by recent reports on the substantial impact of vocabulary overlap. We indeed find that vocabulary overlap is crucial even for the cross-lingual transfer of hierarchical syntactic generalization. This suggests that this also explains the language-pair-dependent tendencies in the main experiments and suggests that the transfer of linguistic evidence in continued new-language learning is only conditionally achievable.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.