Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models
The paper challenges the rapid shift toward Vision Transformer-based continual learning by demonstrating that lightweight, pruned Convolutional Networks can outperform existing foundation model approaches. The authors propose Pruned Adaptation Modules (PAM), which freeze early ResNet layers and introduce sparsely structured task-specific modules, yielding significant parameter reductions while improving accuracy. This work fills a critical methodological gap by establishing a strong, efficient baseline that questions whether recent advances reflect genuine progress or merely the absence of rigorous ConvNet comparisons.
PAM establishes itself as a compelling baseline for class-incremental learning, delivering on its promise of parameter efficiency and competitive performance. The core mechanism—structured pruning of task-specific ResNet layers combined with confidence-based inference—is well-motivated and empirically validated across diverse benchmarks. However, the comparison between ResNets trained solely on ImageNet-1K and ViT-B/16 models pre-trained on ImageNet-21K introduces a confounding factor regarding representational capacity that complicates the attribution of performance gains solely to the proposed architectural innovations.
The parameter efficiency claims are substantiated by clear evidence; the paper demonstrates a "~5× reduction in trainable parameters and a ~6× reduction in total parameters" compared to existing FM-based methods. The ablation studies demonstrating that pruning after the first epoch outperforms later pruning (Figure 4a), and that confidence-based module selection exceeds distance-based alternatives (Figure 4c), provide robust support for the design choices. The stability of PAM in long-horizon experiments on ImageNet-R (Figure 6) demonstrates superior resistance to catastrophic forgetting compared to prompt-based and adapter-based alternatives.
The evaluation protocol employs ResNets pre-trained on ImageNet-1K while comparing against ViT-B/16 models initialized from ImageNet-21K checkpoints, as noted in the implementation details: "existing methods utilize the pre-trained ViT-B/16-IN21K model which initially trained on ImageNet-21K... we employ pre-trained ResNet... models that are trained solely on ImageNet-1K." This mismatch in pre-training data scale and diversity obscures whether PAM's success stems from superior architectural design or from inherent differences in the base models' learned representations. Additionally, the confidence-based selection mechanism relies on maximum softmax probabilities (Equation 6: $\hat{b}=\arg\max_b \frac{1}{|\mathbf{x}_{test}|}\sum \max_{y} p_b(y|x_i)$) without calibration or theoretical analysis of failure modes when task distributions overlap significantly, and the method's reliance on ResNet-specific architectural constraints limits generalizability to other backbone types.
The experimental evidence strongly supports the efficiency claims and demonstrates that PAM outperforms state-of-the-art methods on CIFAR-100, CUB-200, and Cars-196, though average accuracy occasionally lags on certain benchmarks. Table 1 shows PAM (RN152) achieves 93.79% final accuracy on CIFAR B0 Inc5 versus 85.97% for EASE and 81.46% for CODA-Prompt. However, comparisons to related work would benefit from controlled experiments using identical pre-training regimes to disentangle the impact of architectural choices from pre-training data advantages. The saliency-based pruning using $L_1$-norm ($s_c = \sum|W_c^i|$ in Equation 2) is standard but effective, though the paper does not compare against unstructured pruning or alternative importance metrics.
The paper provides detailed implementation specifics including the Adam optimizer, learning rate $0.001$, batch size 48, and 96% pruning magnitude applied after the first epoch ($e=1$ in Algorithm 1). Experiments were conducted using PyTorch and the PILOT framework with five random seeds, enhancing reliability. Nevertheless, the absence of an explicit code repository link or data preparation scripts in the provided text raises concerns about practical reproducibility, particularly for the confidence-based inference algorithm and the exact structured pruning implementation. The pseudocode in Algorithm 1 clarifies the training loop but omits low-level details such as batch normalization handling during pruning or the specific random seed initialization for the classification head.
The continual learning literature has rapidly shifted from traditional class incremental learning (CIL) techniques to foundation model (FM)-based CIL methods without a clear understanding of how these newer approaches compare to strong, lightweight convolutional baselines. This abrupt transition has created a substantial methodological gap, making it difficult to assess whether recent FM-based CIL progress reflects genuine advances or merely the absence of rigorous baselines. To address this gap, we introduce Pruned Adaptation Modules (PAM), a simple yet effective method that freezes the vast majority of the pre-trained ResNet while enabling scalable continual adaptation through sparse task-specific layers. PAM yields up to a ~5x reduction in trainable parameters and a ~6x reduction in total parameters, significantly reducing the cost of continual updates. Across diverse benchmarks, PAM consistently mitigates catastrophic forgetting and outperforms state-of-the-art FM-based CIL approaches. Our findings position PAM as a strong and transparent baseline that helps bridge the gap between traditional and FM-based CIL, guiding future research for a more accurate assessment of true progress in continual adaptation. The code can be found at: https://github.com/ElifCerenGokYildirim/PAM.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.