Two Routes to the Middle: Placement Search and Brain Readouts Converge on Where Continual Learners Should Specialize
Continual learners that keep a task-specific adapter in every ViT block accumulate parameters linearly in the number of tasks. This paper asks which blocks should specialize, showing that exhaustive placement search traces an inverted-U curve peaking at intermediate depth, while cheap internal criteria systematically favor the deep but suboptimal layers. To sidestep expensive search, the authors propose LS-B, which observes the first few tasks through a frozen fMRI encoding model of human visual cortex and commits task-specific capacity to the blocks whose brain readouts vary most across tasks relative to their stable structure.
LS-B is a creative but incremental attempt to replace placement search with a neuroscience-derived structural prior. The authors are commendably honest that the brain readout lands in the right intermediate band yet fails to recover or rank the exact search optimum. At matched storage and observation, the selected blocks outperform both shallow and deep extremes, but the paper concedes that observation—not the readout—supplies most of the gain over narrower static baselines. The result is intriguing neuroscience-inspired engineering, but the brain signal acts more like a coarse regularizer than a precise optimizer.
The placement-search diagnostic is the clearest contribution. On Split ImageNet-R, sliding a four-block task-specific window along a ViT-B/16 yields an inverted U with up to $3.5$ pp accuracy swings, exposing that internal spectral and activation criteria systematically select the suboptimal deepest blocks. The neuroscience framing is executed with care: the fMRI readout produces stable, backbone-specific allocations that overlap the intermediate band found by search, and LS-B retains only $60\%$ of full BiLoRA storage while staying within $1.5$ pp of its final accuracy.
The central worry is that the readout is too weak to guide fine-grained architectural decisions and that observation, not brain alignment, drives the reported gains. The authors acknowledge that the readout approaches the search optimum without recovering the best window or ranking placements, and that its correlation with held-out placement accuracy is effectively zero.
The matched-observation decomposition shows that observing the first $K=3$ tasks supplies the bulk of LS-B's lift over static allocations, while the readout merely decides where the residual capacity goes. The evidence is also narrow—one benchmark, one fMRI subject, and a host adapter inherited without tuning—and supplementary runs on CIFAR-100 and CUB-200 show mixed results and dataset-dependent committed blocks.
The paper fairly demonstrates that cheap internal criteria miss the intermediate optimum, while the brain readout avoids the deep-layer bias. However, the evidence does not support treating the readout as a surrogate validation objective: the authors note that its ranking of placements does not track accuracy, with rank correlations of $+0.10$ and $+0.38$ ($p=0.84$ and $0.36$) across the eight search windows. Comparisons to related work are accurate, and the distinction between brain-inspired routing and LS-B's direct layer-allocation rule is drawn clearly. Still, the supplementary experiments reveal that the cortical signature and committed blocks shift across datasets and backbones, suggesting the prior is less transferable than the main narrative implies.
Experimental transparency is unusually high: training hyperparameters, slot budgets, seed protocols, and a registered analysis plan are all documented, and the authors report $374$ ten-task runs with $13$ bitwise reproductions. Code is released. Even so, reproduction faces friction: the fMRI encoding model relies on the Natural Scenes Dataset and an archived but non-reproducible validation split, and the placement-search baseline costs $27$ full continual-learning runs per backbone. The authors also disclose that several comparisons span uncommitted code changes and mixed software versions, and that one registered supplementary condition was not run.
Continual learners that keep a task-specific adapter in every block of a pre-trained vision transformer accumulate storage linearly with the number of tasks; keeping task-specific adapters in only a few blocks curbs this growth but raises the question of where to place them. We investigate this question from two perspectives. Algorithmically, training all contiguous four-block placements yields an inverted U: final accuracy peaks at intermediate depth and varies by up to 3.5 percentage points (pp), while inexpensive criteria based on weight spectra or activation statistics favor the deepest blocks. From neuroscience, the hierarchical organization and intermediate-stage plasticity of the visual cortex motivate us to ask whether a measurement taken outside the learner can guide layer specialization without placement search. LS-B observes the first tasks through a frozen fMRI encoding model of twelve human visual areas and commits task-specific capacity once to the blocks whose readouts vary most across tasks relative to their stable structure. Across three ViT-B/16 backbones, LS-B yields stable, backbone-specific allocations. On the two backbones with placement search, AugReg and iBOT, the selected blocks overlap the intermediate-depth region identified by search. Under matched storage and observation budgets, the selected blocks outperform the shallowest and deepest four-block configurations. On Split ImageNet-R, LS-B uses 60% of full-BiLoRA adapter storage while remaining within 1.5 pp of its final accuracy. The allocation requires no labels or backpropagation, adds under 0.6% runtime, and exhibits backbone-specific cortical signatures.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.