Your Transformer Is Secretly Elastic
Scaling transformer deployments is hampered by the fact that model families are released in only a handful of fixed sizes. This paper tackles elastic inference: letting a single pre-trained transformer adapt its compute budget on-the-fly without retraining. The core idea is a differentiable soft-pruning formulation that jointly learns a global ranking of every individual MLP neuron and attention head across multiple sparsity targets, keeping the original weights frozen. It matters because it promises to replace rigid model selection with a smooth accuracy-compute trade-off, and the authors back this with strong vision results and measured GPU speedups.
The paper offers a timely, technically sound advance in retraining-free elastic inference. Its differentiable relaxation of the unit-ranking problem is well motivated, and the sweep across 27 vision and text encoders is extensive. The gains are real but asymmetric: the method shines under deep pruning and on self-supervised vision backbones, while decoder-based text embedders remain brittle and the edge over block-level rankings like SnapViT vanishes under shallow pruning. It is therefore a strong incremental contribution rather than a universal panacea.
The multi-target differentiable ranking and the breadth of evaluation are the paper's strongest assets. On large self-supervised vision transformers the method produces remarkably smooth degradation curves: DINOv3 ViT-H+/16 loses only 0.8% ImageNet-1k linear-probing accuracy at 50% sparsity without any weight correction, and the practical deployment story is bolstered by up to 1.80× A100 throughput and sub-millisecond budget-switching overhead.
First, the advantage over strong one-shot baselines largely evaporates under shallow pruning: SnapViT is within 0.9% on DeiT and DeiT-III up to 30% sparsity, so the method is clearly beneficial only when aggressively pruning. Second, the 'retraining-free' framing understates a hidden data dependency: ranking optimization requires a calibration set and a modality-specific proxy loss (DINO self-supervision for vision, InfoNCE for text), and sensitivity to calibration size or domain shift is not studied. Third, measured speedups are limited to a single vision model on one GPU architecture; text encoders and other hardware are not profiled. Finally, the ablation on score initialization shows that poor starting points recover substantially but still trail strong initializations by up to ~10 points at 60% sparsity, so the method is not fully self-contained.
The central claim—that joint per-unit ranking across sparsity targets outperforms single-target block-level rankings—is well supported by Table 3 and the broad encoder sweep. Holding out weight correction for all methods isolates ranking quality, but it also disadvantages approaches like SparseGPT that are designed around weight updates, and the headline numbers are selectively driven by vision backbones. Decoder-based text models tell a weaker story: Qwen3 0.6B loses 18.0 MTEB points at 60% sparsity. The appendix comparison to fine-tuned EffiR further shows that the retraining-free approach still trades accuracy for convenience, trailing by 5.4 nDCG@10 points at high sparsity.
Reproducibility is above average: the authors release code, evaluation scripts, and SLURM job definitions on GitHub, and they report key hyperparameters such as 14 sparsity targets and 300 optimization steps. Yet the repository is nascent (only one commit at time of writing) and lacks explicit documentation of learning rates, batch sizes, and per-model wall-clock times. Reproducing the full 27-model sweep up to 8B parameters will require downloading calibration datasets and substantial GPU resources; the paper asserts sub-linear scaling but does not quantify total construction time or energy cost.
Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once. Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to $1.80\times$ higher throughput and $1.26\times$ lower single-query latency on an A100 GPU. Elastic models are produced in minutes, with construction time scaling sub-linearly with parameter count.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.