Tracing Subliminal Learning Through Pretraining
Large language models can pick up hidden behavioral preferences—termed subliminal learning—when trained on data generated by other models. This paper investigates whether such transfer occurs in pretrained base models under hard-label and on-policy soft-label distillation, testing across different initializations, data orders, corpora, and architectures. By tracing how the abilities to transmit and receive a trait develop separately during pretraining, the authors challenge the notion that shared model origins are a prerequisite for unwanted preference transfer.
The paper provides a timely and empirically rich expansion of subliminal learning from instruction-tuned settings to pretrained base models, leveraging controlled suites like PolyPythia and DataDecide OLMo. Its strongest contribution is the developmental analysis showing that teaching and learning signals emerge at different pretraining stages, which elegantly explains observed cross-model asymmetries. However, the mechanistic picture remains incomplete: while the authors isolate number frequencies as one carrier in hard-label distillation, the discrepancy with prior work is attributed to model dependence without a deeper causal account, and persistent transfer after animal-token removal in soft-label distillation is left unresolved.
The demonstration that subliminal learning does not require shared initialization or pretraining conditions is robust and well-supported by cross-model experiments using PolyPythia variants and OLMo. The cross-checkpoint grid analysis is a clear methodological advance: a separable teacher-student factor model explains roughly 90–98% of transfer variance and shows that sender and receiver abilities develop on distinct trajectories. Additionally, the replication of transfer under plain SGD and full fine-tuning refutes recent claims that this phenomenon is an artifact of LoRA or adaptive optimizers.
The hard-label mechanistic analysis is suggestive but potentially overstated. The synthetic-sequence manipulation retains 40–92% of transfer in Pythia, yet only 5–36% in OLMo, and the authors acknowledge this contradicts prior global-shuffling results in larger instruction-tuned models. Their explanation—that the signal resides in number frequencies for some models but sequence structure for others—is plausible but post-hoc and weakens the universality of the frequency mechanism.
In the soft-label top-$K$ setting, removing animal-name tokens reduces but does not eliminate transfer in same-model pairs, and the authors concede that residual semantic tokens may remain unfiltered, leaving the channel for this persistent transfer unexplained. The scope is also narrow: most detailed experiments use small base models and animal preferences, so claims about broader trait classes or instruction-tuned settings should be treated cautiously.
The empirical evidence strongly supports the central claim that subliminal transfer persists across diverse pretraining conditions, with careful controls and stratified permutation tests backing the statistical comparisons. Comparisons to related work are fair when citing model-specific differences, though the paper’s synthetic-sequence results are framed as challenging prior conclusions without fully reconciling the methodological gap between matching first-number frequencies and globally shuffling teacher outputs. The soft-label experiments, using exact reverse-KL computation rather than policy-gradient approximations, provide a cleaner signal than some recent on-policy studies and contextualize the findings within the rising practical use of GKD.
Experimental details are extensive, with full appendices specifying hyperparameters, prompts, filtering rules, and model configurations, and the authors pledge to release all code. Most training uses standard frameworks and publicly available checkpoints, and controls are carefully matched by seed and corpus properties. Reproduction is nevertheless hindered by the sheer scale of some sweeps—particularly the 13x13 cross-checkpoint hard-label grids and the retraining of PolyPythia weight-seed variants—and by dependence on Qwen3-32B for synthetic document generation, which introduces stochasticity unless generation seeds and system prompts are strictly controlled.
Large language models trained on data generated by other models can sometimes inherit traits unrelated to that data, a phenomenon called subliminal learning. We study both hard label distillation, where students train on numbers generated by a biased teacher, and on-policy soft label distillation, where students generate the data and the teacher's logits provide the learning signal. Using Pythia, PolyPythia, and DataDecide OLMo models, we find that transfer persists in both settings across differences in weight initialization, pretraining data order, pretraining corpus, model size, and architecture. Across pretraining checkpoints, the ability to transmit a preference and the ability to acquire one develop separately: some checkpoints are more effective teachers but weaker students, and vice versa. Consistent with this distinction, cross-model transfer can be strongly asymmetric in magnitude when teacher and student roles are reversed. To investigate what carries the preference signal, we construct synthetic number sequences that match the teacher's generated number frequencies but have no dependence on the prompt or preceding numbers and find that these still induce substantial transfer in the hard-label setting. In top-$K$ soft-label distillation, removing animal tokens from the loss reduces transfer, although not completely in the same-model setting. In our experiments, soft label distillation produces more consistent transfer with less extensive training than hard label distillation. The growing use of on-policy distillation in model post-training makes this a particularly relevant setting for assessing unintended preference transfer.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.