Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models
This paper adapts Non-Linear Multi-Dimensional Concept Discovery (NLMCD) from vision to LLMs to model concepts as low-dimensional manifolds rather than linear directions. It introduces a fuzzy generalized Rand index called Concept-Based Alignment (CBA) that compares independently discovered concept spaces across layers, languages, and training stages without requiring shared tokenizers or architectures. The authors claim CBA is more sensitive than linear baselines (CKA, PCA), uses it to reveal cross-layer block structures, and applies it to study multilingual sharing, cross-model alignment, and post-training drift in Tulu-3.
The paper makes a useful methodological step by transferring manifold-based concept discovery to token-level LLM activations and showing that CBA reveals block-structured layer alignments that linear metrics obscure. However, its core claims are weakened by a major unresolved methodological issue: HDBSCAN assigns more than half of late-layer tokens to noise (median coverage ~0.45, dropping to ~0.4), and because noise tokens carry near-zero membership vectors, they contribute trivial agreement to the CBA score, potentially inflating apparent alignment. Additionally, the framing of multilingual results as a counterpoint to the Platonic Representation Hypothesis is overstated, since comparing English-only GPT-2 against multilingual Qwen is not a clean test of representational convergence. The citation of Vielhaben et al. as NeurIPS 38 in 2026 contains a clear bibliographic impossibility (volume 38 corresponds to 2025, not 2026).
The neighboring-layer sanity check is well designed: for every source layer the best-aligned other layer is almost always adjacent, and CBA achieves a Neighbor z of 2.515 versus 1.867 for PCA-512 and 1.714 for RBF CKA, supporting its finer-grained sensitivity. The finetuning-drift analysis on the fully exposed Tulu-3 pipeline is a smart use of publicly available checkpoints; the observation that DPO and RLVR remain closely aligned (CBA ~0.85–0.86) while SFT produces the largest representational shift from the base model is genuinely useful and is cross-referenced against independent parameter-level evidence. The cross-layer block structures are qualitatively consistent across Qwen scales (4B, 8B, 32B), suggesting the pipeline captures a real organizational pattern.
The central weakness is that CBA is structurally confounded by HDBSCAN noise. With median coverage of 0.4469 and late-layer coverage near 0.4, roughly 36% of all token pairs are noise–noise pairs whose near-zero membership vectors yield $d_{ms} \approx 1$ in both partitions, contributing zero to $d_{\mathrm{cross}}$ and thereby inflating alignment scores artificially. The paper never reports noise-adjusted scores or a sensitivity analysis excluding noise points, so it is impossible to know whether the 'block structures' reflect true concept correspondence or simply shared failure-to-cluster. Second, the qualitative annotations in Results 3 and 4 are generated by an off-the-shelf LLM (Mistral-Small-24B) with no reported inter-annotator agreement, human validation, or error rates; the authors concede these 'labels are hypotheses for inspection,' yet downstream categorical claims (syntax-dominated early layers, semantic peaks in multilingual sharing) depend directly on them. Third, the multilingual probe uses only English–Mandarin, so conclusions about universal multilingual sharing are extrapolated from a single language pair.
The CBA versus CKA comparison in Figure 3 provides empirical support that linear CKA near-saturates and misses finer substructures, which is a fair and visually convincing demonstration. However, the paper does not establish that the structures CBA reveals correspond to meaningful concepts rather than artifacts of the UMAP+HDBSCAN pipeline. For instance, coverage and DBCV both decline toward late layers, yet CBA still reports high alignment scores there; without a noise-adjusted ablation, readers cannot tell whether late-layer blocks are geometric reality or clustering artifact. Comparisons to related work are generally fair, though the positioning against sparse autoencoders is underdeveloped: SAEs also produce decompositions that can be compared across layers and models (e.g., via feature splitting or merging metrics), and the paper does not directly compare CBA against SAE-based alignment schemes.
The authors provide an anonymized code repository and report fixed hyperparameters (UMAP seed 42, HDBSCAN min cluster size 50, min samples 50), which is welcome. Nevertheless, independent reproduction faces several hurdles. First, HDBSCAN with explicit noise assignments is not fully deterministic across library versions, and fuzzy memberships depend on density estimates that could shift. Second, the activation reservoir contains only 100,000 token positions sampled from 5,000 sentences, which is tiny relative to the representational capacity of models with hidden dimensions up to 5,120; whether the manifold structure is stable at this sample size is untested. Third, cross-model comparisons require character-span alignment between different tokenizers, and the paper does not quantify how many tokens are discarded or merged in this step, making it hard to reproduce the English–Mandarin and Qwen–Llama alignments exactly. Compute budget and wall-clock times are also omitted, though the authors acknowledge HDBSCAN dominates complexity.
Understanding information processing in large language models (LLMs) requires dissecting the geometric organization of their internal token representations. While existing mechanistic interpretability (MI) methods seek to extract concepts, they are constrained by a strong linearity assumption challenged by evidence of non-linear feature manifolds. We move beyond linear concepts by adapting Non-Linear Multi-Dimensional Concept Discovery (NLMCD) from computer vision to token-level LLM activations, modeling concepts as low-dimensional manifolds. To compare concept manifolds across layers and models, we introduce a concept-based alignment (CBA) score, a generalized Rand index that measures geometric proximity without explicit feature matching. Our analysis yields six key findings: (i) a neighboring-layer sanity check shows CBA is more sensitive than PCA- or CKA-based linear baselines; (ii) layer-by-layer alignment matrices reveal two block structures in intermediate and late layers, consistent across models and obscured by linear metrics; (iii) concept composition remains syntax-dominated through most of the network before giving way to increasingly mixed syntactic-semantic concepts in later layers, with increasing output-orientation toward the final layers; (iv) multilingual concept sharing between English and Mandarin is training-dependent rather than universal, strongest in Qwen, weaker in Llama, and absent in GPT-2; (v) inter-model alignment mirrors this structure, with strong correspondence between same-family Qwen models of different scale but weak alignment across model families; and (vi) across Tulu-3 training stages, alignment is highest between adjacent stages, with the largest shift between the base model and SFT, while subsequent preference-alignment stages (DPO, RLVR) leave early layers largely unchanged and RLVR mostly preserves DPO's concepts in late layers.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.