SLIM: Simplex-Lattice Interpolation Merging
Model merging for LLMs typically requires expensive benchmark evaluations to tune combination coefficients. This paper proposes SLIM, which treats coefficient selection as a classical mixture experiment on the simplex. It builds a quadratic surrogate of aggregate performance using only $T(T+1)/2$ evaluations—individual experts and equal-weight pairs—then optimizes the surrogate without further measurements. The core finding is that this minimal structured design predicts unseen mixtures accurately and yields competitive merges at a fraction of the cost of evolutionary or surrogate baselines, though restricting coefficients to the simplex may limit absolute performance.
SLIM offers an elegant, statistically grounded approach to reducing evaluation cost in model merging by restricting coefficients to the simplex and using Scheffé's $\{T,2\}$ lattice design to interpolate a quadratic from exactly $T(T+1)/2$ measurements. The empirical results are strong for the constrained setting: Spearman $\rho$ of 0.936 and 0.959 on held-out AVG predictions, and SLIM beats MAP with 45 evaluations in all four tested architecture-distribution settings while remaining competitive with evolutionary merging using 150–200 evaluations in three of four. Yet the simplex restriction itself appears to be a binding constraint on Llama-3.1-8B ID, where MAP and evolutionary search achieve higher scores by selecting coefficients summing to well above one. Moreover, the cubic extension doubles the budget to 120 evaluations but yields lower OOD AVG, casting doubt on whether richer polynomial surrogates help on this task.
The methodological core is rigorous and novel for the LLM-merging context: the Scheffé simplex-lattice interpolation is derived correctly, the identification minimum is proven, and the surrogate can be optimized with an analytic gradient. The ablations convincingly show that evaluation placement matters: at 36 evaluations the lattice design dominates random designs and ridge regression in predicting AVG (Table 2), and the matched-budget head-to-head with MAP is executed carefully using the authors' adjusted re-implementation of the baseline.
Despite accurate aggregate predictions, domain-wise fidelity is highly uneven: Gemma math scores correlate negatively with predictions ($\rho=-0.24$) and safety MAE reaches 9.2 points, so the quadratic is only reliable for AVG, not individual domains. The simplex constraint is a hard ceiling: MAP and evolutionary merging find higher-performing coefficients outside the simplex on Llama ID (sums of 2.5–3.75 versus 1.0), so SLIM trades absolute performance for sample efficiency. The cubic extension (120 evaluations) hurts OOD AVG relative to the quadratic (36 evaluations), suggesting that higher-order polynomial surrogates do not improve generalization here. Finally, the study is limited to $T=8$ linear task-vector merges on two architectures, leaving scalability to larger expert pools and nonlinear operators unaddressed.
The paper's main claims about predictability and minimal-cost selection on the simplex are backed by clean experiments, but baseline comparisons carry caveats. SLIM maximizes AVG inside the simplex, whereas MAP is designed to approximate multi-objective Pareto fronts over $[0,1]^8$ using domain-wise quadratic surrogates and NSGA-III. The authors fairly adjust MAP's code for a single-objective comparison, but this is not MAP's intended use case. The evolutionary baseline is more directly comparable, and SLIM performs well against it at far lower cost in most settings. The fact that MAP and evolutionary search outperform SLIM when allowed to explore outside the simplex (on Llama ID) indicates that the evidence supports SLIM's efficiency within its constrained domain, not universal dominance.
Reproducibility is thorough on paper but not yet practically available. Appendices list exact Hugging Face checkpoints, benchmark prompts, shot counts, and optimizer hyperparameters, and Algorithm 1 gives a complete procedural description. The authors pledge to release code and measured scores upon acceptance, which would allow independent reproduction without re-evaluating LLMs. Until then, reproduction is blocked by the absence of public artifacts. The authors' disclosure that generative AI was used heavily for code, derivations, and drafting adds a cautionary note that warrants close scrutiny of the eventual released artifacts.
Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose \textbf{Simplex-Lattice Interpolation Merging (SLIM)}, which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.