The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention
The paper asks how language models retrieve relational facts from a context list. Its core claim, the ordinal addressing hypothesis, is that models navigate to a fact not by the entity name in the question, but by the fact's serial position — its order of mention. The authors formalize this as additive fact addresses $a_p$ in hidden states and test it via activation steering across fourteen models, reporting transfer across contexts, low-rank geometry, and a pretraining trajectory reminiscent of human serial-position effects.
The paper presents an elegant, falsifiable formalization of ordinal addressing in LLMs, backed by extensive activation patching. Its central claim that models universally address facts by order of mention, however, is only strongly supported in Qwen; Gemma and especially Llama show far weaker steering and template fits. The sizable gap between ordinal-vector transfer and the ceiling in Table 1, together with collapse to 14% in prose for Llama, suggests the phenomenon is partial and architecture-dependent rather than a universal geometric law.
The ordinal addressing hypothesis is operationalized with unusual clarity: Hypothesis 3.3 posits context offsets and shared fact addresses so that $h_\ell(C,q) = c_C + a_{p_C(q)} + \varepsilon_\ell(C,q)$, yielding falsifiable predictions about invariance, reversal, and transfer. The Qwen results are genuinely strong — query changes reverse cleanly when facts swap order (cosine $-0.93$), the three-dimensional address subspace holds 69% of held-out change variance, and the pretraining trajectory in OLMo-2-7B is compelling. 'Questions switch from the named person to order of mention: at 5B tokens the query change is the same in both arrangements ... and it reverses by 42B ... while the model answers only half the questions' (Section 7.2).
Despite the universal framing, the evidence is model-dependent and prompt-dependent. Transfer rates in Table 1 range from 84.1% in Qwen down to 36.2% in Llama, and in prose paragraphs they drop further to 44.5%, 42.4%, and 14.0%. This variability undermines the claim that order-of-mention addressing is a generic property of LLM reasoning geometry. The fact-address template also explains only 29% of Gemma's query-dependent variance versus 66% for Qwen (Section 4), suggesting the low-rank structure is not consistently dominant.
A second concern is the large gap between intervention and ceiling rates. In Qwen the ceiling is 96.9% while the ordinal vector reaches only 84.1%; in Llama the gap exceeds 55 percentage points. This implies that the steering vector captures only a fraction of the addressing signal, and alternative or complementary mechanisms — including lexical and reflexive retrieval as found by Gur-Arieh et al. — likely operate in parallel. The depth-window selection procedure introduces another caveat: main blocks were chosen because 'an earlier, related edit of the query state changed the most complete answers on separate word sets' (Appendix A), creating a risk of block-selection bias.
The paper usefully distinguishes itself from prior mechanistic work on binding. It correctly notes that prior studies focus on bound-token states within single contexts, whereas this work tracks query states across contexts. 'Closest to our question, studies of contextual binding find binding identifiers, propositional probes, and positional and lexical contributions to bound-entity retrieval ... Prakash et al. [29] find ordering IDs that a lookback mechanism uses to retrieve bound information, and Gur-Arieh et al. [12] find that this positional mechanism grows noisy in the middle of long lists, where lexical and reflexive mechanisms take over' (Section 2). Our external fetches confirm these characterizations are accurate.
The comparison is less complete when reconciling findings. Gur-Arieh et al. show that positional mechanisms become unreliable in longer lists and are supplemented by lexical and reflexive ones; Zhou's serial-position crowding and weaker mid-list transfer echo this, but the paper does not test whether its ordinal vectors fail in the same regimes or quantify the contribution of the alternative mechanisms it acknowledges.
The experimental protocol is documented in substantial detail: greedy decoding to 16 tokens, bootstrap intervals over word sets, and native chat templates. 'Prompts use each model's native chat template ... and computation is in FP32' (Appendix A). Exact reproduction may be hindered by the ad hoc main-block selection procedure — blocks were chosen because 'an earlier, related edit of the query state changed the most complete answers on separate word sets' (Appendix A) — which adds a researcher degree of freedom that is not cross-validated. The precise word-set pools are not described in the paper text, so replicators must rely on the GitHub repository for exact prompts and item lists; reproducing the OLMo-2 checkpoint sweep further requires substantial compute access.
Human reasoning depends on how objects are related within propositions. \textit{How do relations organize the language representations of contextual contents?} We give an LLM a list of facts in its context (e.g., \emph{Alice eats an apple. Bob eats a pear.}) and measure how its hidden state changes when the question switches from what Alice eats to what Bob eats. Averaged over many lists, this change is a steering vector, which we call the \emph{ordinal vector}. It points to a fact by its \emph{order of mention}, the order in which the facts were stated in the context. We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains. We state this as the \textit{ordinal addressing hypothesis}: each order of mention has a \emph{fact address} in the model's state, shared by all contexts, and a question moves the state to the fact address of the fact it asks about, while the context supplies what that fact says. Across Qwen, Gemma, and Llama, fact addresses are (1) \emph{ordered by mention}: query states are organized by the order of facts, not of names, even when one fact has multiple subjects; (2) \emph{steerable}: added to a question about the first fact of a new list, the ordinal vector makes the model answer with the second fact of that list; (3) \emph{low-rank}: they span a low-rank subspace in which the first-mentioned fact is the easiest to reach, surprisingly similar to human recall; and (4) \emph{emergent}: they are shared in late-middle layers, hold from 1.5B to 32B parameters, and form early in pretraining. Language models reach a stated fact by where it was mentioned, deepening our understanding of LLM reasoning.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.