CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations
Cross-architecture weight-space transformations map trained networks into different architectures (e.g., compression, upscaling), yet prior equivariant weight-space models only handle fixed architectures. CrossGMN addresses this by treating the task as refining a target initialization using a source network, making equivariance well-defined: invariant to source permutations and equivariant to target permutations. The authors prove universality for continuous cross-architecture operators and show strong empirical speedups for knowledge distillation, especially on implicit neural representations.
CrossGMN is a theoretically grounded and architecturally elegant solution to an important gap in weight-space learning. The core insight—encoding cross-architecture mappings as $F:\mathcal{V}_{\mathrm{src}}\times\mathcal{V}_{\mathrm{tgt}}\rightarrow\mathcal{V}_{\mathrm{tgt}}$ with source invariance and target equivariance—resolves a genuine conceptual obstacle, and the graph metanetwork design follows naturally. However, the empirical gains are highly uneven: impressive on small INRs but modest on classifiers, with several settings where speedup is near or below $1.0\times$. The work opens a promising direction but leaves questions about scalability and broad architectural coverage.
The problem formulation is the strongest contribution. By reformulating cross-architecture transformation as a refinement operator on a target initialization, the authors obtain a clean symmetry structure and a tractable learning target. The universality theorem is a solid theoretical guarantee, and the graph metanetwork construction—using layerwise joins for hidden neurons and positionwise joins for boundary neurons—preserves these symmetries in a principled way. On INR compression, the method delivers large and consistent distillation speedups.
The empirical benefits shrink rapidly outside the INR regime. On CIFAR-10 classifiers, median speedups hover near $1.0\times$ and some ViT reductions actually slow down distillation. Second, the universality guarantee is restricted to MLPs under a general-position assumption, and the evaluation tops out at roughly 200k parameters, leaving open whether the graph construction scales efficiently to modern network sizes. Third, cross-architecture generalization to unseen source architectures is weak: the pooled median speedup on heterogeneous MLPs is only $1.06\times$ with a 14% failure rate, suggesting the single-model claim is not yet practically compelling.
The authors benchmark against a strong anchor initialization—the best-performing structured-pruning baseline for each setting—which makes positive speedups meaningful when they occur. The ablation in Table 6 is carefully designed: removing cross edges or replacing them with FiLM conditioning both degrade performance, lending direct evidence that explicit equivariant communication between source and target neurons is superior to pooled conditioning. The lack of direct comparison to related architecture-change methods such as Net2Net or LiGO is a minor gap, though the paper clearly articulates the conceptual difference.
The paper provides extensive implementation and hyperparameter details across Appendices D.1–D.7, including exact model zoos, distillation budgets, and wall-clock timings. The reliance on public datasets and existing model zoos helps reproducibility. However, no code repository is mentioned, and the graph construction, cross-edge policies, and parameter-to-graph encoding are intricate enough that absence of released code would materially block independent reproduction.
Weight-space networks operate directly on parameters of other neural networks, enabling tasks such as predicting model properties, editing trained models, and generating weights. Weight-space symmetries such as neuron permutations make equivariance a key design principle. However, existing equivariant weight-space architectures have primarily been studied for transformations that preserve the network architecture. In contrast, many practical transformations, including model compression and upscaling, map a trained source network into a target network with a different architecture. In this setting, the source and target permutation symmetries act on different parameter spaces, making equivariance less straightforward to formulate. Our key idea for addressing this mismatch is to reformulate cross-architecture operators with two inputs: a trained source network and an initialization of the target network. This lets us define equivariant cross-architecture operators that refine the initialization of the target network using information from the source network, while being invariant to source-network permutations and equivariant to target-network permutations. Based on this formulation, we introduce CrossGMN, a graph metanetwork that jointly processes both networks through symmetry-preserving cross-network message passing. We prove CrossGMN is universal for continuous cross-architecture operators on compact sets under a general-position assumption. We evaluate CrossGMN for model compression, predicting a smaller network's parameters to accelerate subsequent knowledge distillation. Across 2-D and 3-D INRs and image classification with MLPs, CNNs, and Vision Transformers, CrossGMN speeds up distillation by up to 8.89x, transfers across datasets without retraining (3.78x), and a single model can accelerate compression from heterogeneous source architectures into a common target architecture.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.