Asymptotic Behavior of Transformers

The strongest contribution is the rigorous derivation of limiting ODEs for increasingly realistic architectures. For single-layer models, the history collapses to an empirical token distribution that tracks the ODE $dP/dt = \pi^*(P) - P$. For deep transformers with RoPE, MLPs, local and global attention, the authors show the empirical KV-cache distribution $\eta_t$ tracks a measure-valued ODE $d\eta/ds = \Gamma(\eta) - \eta$ on the logarithmic time scale $s_t = \log t + O(1)$. The contraction argument using a layer-weighted norm to exploit feed-forward triangularity is technically inventive, and the explicit three-token cyclic construction yielding almost-sure non-convergence is clean and definitive.

A central weakness is the disconnect between the proved regimes and practical language modeling. The convergence guarantees require temperatures high enough to make the output softmax nearly uniform, which is far from typical sampling at $\tau = 1$. Conversely, the non-convergence result relies on an extremely stylized three-token, single-layer, NoPE, attention-only transformer without MLPs; whether realistic trained networks exhibit persistent limit cycles or chaos at standard temperatures is entirely open. The paper also leaves the low-temperature dynamics of deep architectures largely unexplored: the ODE may have multiple attractors, and the only general statement is that ICT sets need not be singletons. Finally, while the authors motivate the work by observations of LLMs locking into good or bad states, no empirical evidence links the theoretical ODE or its bifurcations to actual model behavior.

The evidence is exclusively theoretical, built on stochastic approximation and explicit small-scale constructions. Comparisons to related work are fair and well-scoped: the authors clearly distinguish their autoregressive, fixed-model, infinite-time setting from training-time mean-field analyses and from length-extrapolation research. However, the paper would benefit from even minimal simulation of the ODE for richer settings to illustrate whether attractor multiplicity appears beyond the three-token toy. The lack of any empirical connection to contemporary LLMs means the reader is left uncertain whether the derived ODE is a useful description of, say, repetitive degeneration or mode collapse in real transformers.

Reproducibility is partial. The small examples in Sections 2.3 and 4 are fully explicit: transition matrices, logits, and ODEs are given in closed form, so the bifurcation and non-convergence can be verified independently. The general multi-layer results, however, depend on unspecified architecture-dependent constants and existential thresholds. No code, data, or simulation scripts are mentioned, and the temperature bounds are not computable from the text alone. An independent reproduction of the main theorem would require deriving the model-specific Lipschitz constants and contraction coefficients from scratch.