From Plato to Aristotle: Toward a General Theory of Representation Hypotheses
Conflicting studies on whether neural networks converge to a shared representation have used different alignment metrics and reached opposite conclusions. This paper proposes the Lattice Representation Hypothesis (LRH), which formalizes alignment criteria into a lattice ordered by strength, and argues that disagreements arise because prior work evaluates criteria at different points on this lattice. It also proves in idealized models that the training procedure and objective determine which criteria are met reliably across independent runs, supported by experiments on dozens of pretrained vision and language models.
The paper offers a compelling conceptual advance by structuring the fragmented empirical debate over representational convergence into a formal lattice of alignment criteria, which clarifies why prior studies conflict. Its core theoretical results are rigorous and substantiated by controlled experiments, though its ambition to be a general theory overreaches in places because the guarantees apply only to stylized models and the empirical arguments about pretrained networks remain largely post-hoc and correlational.
The lattice framework is a genuinely useful synthesis of prior empirical disputes, and the identification of precisely why calibrating a maximum over layers can reverse CKA trends is one of the paper's sharpest contributions. The crossing-law analysis of neighbor sets and the spanning-tree composition results are elegant, and the controlled experiments on linear autoencoders and finite Gibbs models confirm their theoretical predictions with preregistered tests. The authors show that 'the largest CKA baseline belongs to the layers of largest effective dimension and can outgrow the aggregated score along a model family,' directly diagnosing a subtle protocol artifact behind ARH's conclusions, and that 'the nearest-neighbor forest makes up a median 0.875 and 0.889 of the shared MST edges at mid depth and the last layer.'
The paper's most significant limitation is the chasm between its stylized theoretical models and its claim of a general theory: for both the linear autoencoder and the finite Gibbs model, 'results on trained networks outside the premises of the theorems contradict several of these predictions.' Several preregistered tests also failed, including 'two secondary Monte Carlo predictions failed' in the Gibbs model and a preregistered prediction that CIFAR-5m neighbor-set agreement would be unreliable under gradient flow. While the critique of SOG is sharp, the bridge-overlap size trend that could carry global structure beyond nearest neighbors 'stays unresolved,' and the paper's framing of PRH sometimes conflates the original study's empirical evidence with the strongest possible claim.
The evidence supports the central explanatory framework well: the lattice clarifies why PRH's claim of distance-level convergence sits at the top while 'Its evidence comes from mKNN, a criterion near the bottom that compares neighbor sets,' and why ARH's calibrated CKA trends vanish when the calibration threshold tracks the growing random-pairing baseline. The BPC collapse is at least partly explained as a scale effect at fixed k, and the SOG spanning-tree overlap is shown to largely repeat nearest-neighbor overlap. Nevertheless, the global bridge-overlap trend in SOG remains unresolved, and some network experiments contradict the stylized predictions, suggesting the lattice framework describes idealized conditions more robustly than messy pretrained networks.
Reproducibility is partially strong but uneven. The theoretical models are fully specified, and the authors preregistered many predictions, which is exemplary. However, the paper only advertises a project page and does not explicitly state that code or data will be publicly released, so exact reproduction of the pretrained-model experiments depends on the availability of numerous large open models; hyperparameters and full protocols are relegated to appendices. As noted, 'Appendix F gives the full protocol,' and 'Our project page can be found at https://eethanshi.github.io/LRH.'
The Platonic Representation Hypothesis (PRH) holds that increasingly capable models converge toward a common representation. However, recent studies have shown different views on aligning representations via their neighborhoods, spanning trees, and distances. We show that these views can differ because the alignment criteria behind them sit at different points of one lattice, and we formalize this explanation as the Lattice Representation Hypothesis (LRH). Under the LRH, the convergence claimed by the PRH is agreement at the top of the lattice, whereas its evidence comes from neighbor overlap, a criterion near the bottom. The Aristotelian Representation Hypothesis (ARH) restricts convergence to local neighborhoods because calibration keeps the growth of neighbor overlap with model size but largely removes that of centered kernel alignment. We show that this removal can come from calibrating a maximum over layers, so global convergence remains open. Back into Plato's Cave (BPC) infers coarse shared structure from the collapse of neighbor overlap on larger evaluation sets, whereas we trace it at least partly to a fixed number of neighbors covering shorter distances. We show that the spanning-tree overlap used by Structure over Geometry (SOG) mostly repeats one-nearest-neighbor overlap. The LRH also extends to independently trained models, whose training procedure and objective decide the criteria met reliably across runs. We prove this claim for the training procedure in a linear autoencoder and for the objective in a finite contrastive Gibbs model. Experiments on dozens of pretrained vision and language models and thousands of runs of both theoretical models support these conclusions. We hope this work paves the way toward a general theory of representation hypotheses, with a framework to analyze them and compare their evidence. Project page: https://eethanshi.github.io/LRH
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.