Continual pretraining Prostate MRI ViT Encoder for task alignment
The paper tackles misalignment between general MRI pretraining objectives and downstream clinical tasks by continually pretraining a 3D prostate MRI ViT encoder. Using LoRA on a prostate segmentation task, the authors evaluate six downstream tasks and use CKA to link representational geometry to performance. The motivation is strong—vision foundation models often need heavy task-specific fine-tuning—but the empirical gains are fractional and inconsistently positive.
The paper presents a limited but well-intentioned empirical study. Its central claim that continual pretraining improves task alignment is only weakly supported: segmentation tasks show sub-1% Dice improvements, while classification, regression, and super-resolution either degrade or remain flat. The conclusion states that results show 'improved and comparable performances in all 6 downstream tasks,' yet Tables 2–5 reveal clear regressions in zone classification (-2.3%), overall PIRADS AUC (-0.4%), regression MSE, and super-resolution SSIM. Without significance tests or confidence intervals, it is impossible to distinguish signal from noise.
The CKA analysis is novel for this setting, but the observed shifts are tiny (0.4% at layer 11) and only weakly correlated with the largest performance delta. The study would benefit from larger continual pretraining data, stronger baselines, and statistical rigor.
The use of Centered Kernel Alignment (CKA) to relate representational geometry to downstream performance is the most compelling contribution. The authors show that for lesion segmentation, CKA similarity between the continual-pretrained and fine-tuned encoders increases by roughly 0.4% at the deepest layer, tracking with the small segmentation gains. The experimental breadth across four task categories—classification, regression, segmentation, and super-resolution—is also a strength, and the choice of LoRA for parameter-efficient adaptation is sensible given the small dataset size.
The headline issue is that the paper overstates positive findings while downplaying regressions. The conclusion claims improved or comparable performance on all six tasks, but zone classification, overall PIRADS AUC, volume regression, and super-resolution all degrade relative to the pretrained baseline. No p-values, confidence intervals, or repeated runs are reported, so even the modest segmentation gains could be spurious.
Second, continual pretraining uses only 396 training scans—orders of magnitude smaller than the tens of thousands shown effective in comparable radiology work [8]. The CKA analysis is restricted to four hand-picked layers (3, 5, 7, 11), omitting the rest of the network, and no ablations examine LoRA rank, scaling factor $\alpha$, or alternative parameter-efficient methods. Finally, the paper lacks a full fine-tuning baseline during the continual pretraining stage, making it impossible to assess whether LoRA is simply underfitting.
The evidence partially supports the segmentation-alignment hypothesis, but the support is fragile because improvements are within likely experimental noise and no error analysis is presented. Comparisons to related work are thin: only Ilse et al. [8] is discussed at length, and the quoted scale '30k, 50k, 100k, 1M, and 1.3M samples' understates the maximum dataset size in that study, which uses up to 3.5 million chest X-rays. The paper does not benchmark against standard parameter-efficient transfer baselines (e.g., adapters, BitFit, prompt tuning) or against full fine-tuning of the encoder, leaving open the question of whether LoRA is the right tool or simply a convenient one.
Reproducing this work would be difficult. While the authors list datasets, loss functions, batch sizes, and training epochs, critical hyperparameters—learning rate, LoRA rank $r$, and scaling factor $\alpha$ in $W = W_0 + \alpha BA$—are omitted. The underlying ProFound ViT pretrained weights are not publicly available, and there is no mention of code release. CKA was computed on layers 3, 5, 7, and 11, but it is unclear whether the same data split or number of samples was used for all pairwise comparisons, and variance estimates for CKA are absent. Without these details, independent reproduction is infeasible.
Building foundation models for MRI has emerged as a promising direction for advancing medical image analysis and improving accessibility of task specific models. Existing approaches primarily focus on effective pretraining strategies and evaluating their performance across diverse downstream tasks. While these methods show performance gains, limited attention has been given to understanding task alignment between pretraining and downstream objectives. Unlike language models, where the two stages are naturally aligned, such alignment is less evident in vision based applications. Motivated by this gap, we explore continual pretraining. Specifically, continually pretraining a pretrained ViT encoder using Low-Rank Adaptation (LoRA) on a prostate segmentation task to enhance task alignment and downstream performance. We further perform layer-wise representational analysis using Centered Kernel Alignment (CKA) to assess representational shifts and their correlation with task outcomes. Our experiments show improved performance in 3 of 6 downstream tasks and comparable results in the others. Our Feature analysis reveals that, for segmentation tasks, continually pretrained encoder features exhibit greater similarity to those of the finetuned encoder than the original pretrained encoder, correlating with observed performance gains.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.