Why don't Graph Transformers attend to the same token?
While large language models concentrate attention on initial tokens to prevent over-mixing, state-of-the-art graph transformers rarely exhibit this 'attention sink' behavior. This paper identifies the attention scoring function—not graph symmetry or locality—as the decisive factor, showing that replacing GRIT's gated logit with a standard dot product raises sink rates dramatically. The finding reframes attention sinks as an artifact of dot-product scoring rather than an inherent property of deep transformers.
The paper presents a focused and methodologically clean investigation into attention sinks in graph transformers, convincingly arguing that sink formation is driven by the scoring function. Its central ablation—swapping only the logit computation while holding all other components fixed—yields a stark difference in sink rates and is supported by both theoretical bounds and norm-intervention experiments. However, the evidence is narrowly scoped to GRIT on molecular graphs at modest depth, and the broader claim that sinks are a property of dot-product attention 'rather than of Transformers' outpaces the experiments, which test only one gated architecture.
The strongest evidence comes from the controlled scoring-function ablation and the norm-intervention analysis. By changing only $z_{uj}$ from GRIT's clipped, pair-dependent gated score to a simple dot product, the sink rate jumps from under 0.1 to over 0.68 on identical ZINC molecules, while GRIT's saturated nonlinear score keeps coordinate activations pinned at the clipping bounds and barely responds to scaled node norms. The three theoretical bounds on received attention—from automorphism orbits, local neighborhoods, and logit spread—also provide a clear diagnostic framework and correctly predict that symmetry, not locality, prevents sinks on regular graphs.
The title and closing claims generalize beyond the evidence, which covers only one gated-score architecture (GRIT), one graph domain (molecules and symmetric cycles), and models of only ten layers. The authors candidly acknowledge these limits, noting that 'our 10-layer models show little over-smoothing to prevent,' which leaves the functional motivation for sinks in graph transformers unresolved. Additionally, the accuracy-matched controls achieved at much higher error (test MAE ~0.54) are far from the state-of-the-art operating point, so it remains unclear whether the scoring-function effect dominates when all models are trained to convergence at competitive accuracy.
Within its scope, the evidence is persuasive. The ablation ratios in Table 1 separate functional importance from mere reception: removing the most-attended node in GRIT doubles error relative to a random node, whereas dot-product sinks are close to no-ops. The comparison to GraphGPS is fair but incomplete because several components differ at once; the paper rightly addresses this by isolating the scoring function. Comparisons to the LLM literature are accurate, and the concurrent work by Liu et al. is appropriately distinguished.
Reproduction is feasible in principle because the code is public, standard datasets are used, and hyperparameters are listed in Appendix B. Nevertheless, several implementation details could block exact reproduction. The GRIT-dotlogit arm requires swapping a specific coordinate-wise clipped score for a dot product while retaining pair representations and values; the paper notes that 'the official code' clips the scalar logit differently, creating ambiguity about which version was used as the base. Seed-dependent variation in some virtual-node quantities and the lack of released training curves or layer-wise checkpoint files are additional friction points.
In large language models (LLMs), attention concentrates heavily on the first token. This 'attention sink' is believed to protect deep models against over-mixing. Even though graph transformers face a related pressure, we find that a SoTA graph transformer, GRIT, rarely forms sinks on molecules, and that it sinks less when trained on larger graphs, the opposite of the trend in LLMs. In principle, a graph can prevent sinks: we prove and show empirically that its symmetry and locality limit the attention any node can receive. On molecules, however, with distinguishable atoms and global attention, neither limit prevents sinks; what stops them lies in the model. We find that the attention scoring function is the main cause. Replacing GRIT's gated logit with a dot product raises its sink rate from 0.09 to 0.68 on identical graphs. In Graph Transformers using dot-product attention, removing the sink gives about the same error as removing a random node, while GRIT's most-attended nodes matter to its predictions. Attention sinks are therefore a property of dot-product attention rather than of Transformers. In our experiments, they form when dot-product scoring has a distinguishable node to concentrate on.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.