Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

The visual feature memory module is a well-motivated mechanism to mitigate the vision-language modality gap: by caching confident visual prototypes and using them as an auxiliary predictor, the model shifts part of the decision from cross-modal similarity to intra-modal similarity. The ablation studies (Table 4) support the additive value of each component, showing consistent gains when the memory module and self-training are stacked. The approach is also reasonably backbone-agnostic, giving improvements on both CLIP-ViT-B/16 and CLIP-ResNet-50.

The most immediate issue is the glaring mismatch between the claimed “55 semantic segmentation datasets” and the actual five benchmarks used (ADE20K, Cityscapes, COCO-Stuff164k, PASCAL Context, and VOC). This suggests either a typo or an attempt to inflate perceived scale. Second, the hyperparameters $K$, $\lambda$, and training iterations are tuned per dataset (Table 1) with no disclosed selection protocol, which invites questions about test-set overfitting. Third, the proposed v-v attention modification adds only about 1 mIoU in the training-free setting and lacks analytical justification—no attention-map visualizations or theoretical analysis are provided to explain why this mapping should preserve regional semantics better than the standard q-k formulation. Finally, the method relies on self-training with model-generated pseudo-labels, so it is not purely training-free, yet the paper blurs this distinction by repeatedly labeling the approach “annotation-free.” The comparison with prior work is also incomplete: stronger recent methods may be omitted, and disabling PAMR for all methods does not fully equalize differences in self-training recipes and backbone details.

Table 2 shows the method reaches the highest average mIoU (35.9%) among the listed competitors, leading on ADE20K, Cityscapes, and PASCAL Context. However, it trails GroupViT on VOC (75.7% vs. 79.7%) and is not best on COCO either, so the “state-of-the-art” framing is only partially true. The authors disable PAMR post-processing for all methods to improve fairness, but differences in self-training pipelines, backbone initialization, and prompt engineering are not controlled. Table 4 provides internal ablations, yet without statistical significance tests or confidence intervals it is unclear whether the ~1-point gain from the v-v attention is reliable rather than noise.

Reproducibility is weakened by the absence of released code, precomputed prototypes, or detailed selection criteria for the dataset-specific hyperparameters. Constructing the visual prototype memory requires processing the entire unlabeled target set to select top-$K$ features per class, which is expensive and sensitive to the choice of $K$ (which swings from 128 to 4,096 across datasets). The EMA strategy is mentioned for inference smoothing, but the exact update rule and whether it runs during training or only at test time are ambiguous; only $\beta=0.9999$ is reported. Because the self-training pseudo-label generation process is not fully specified, independent reproduction would involve substantial guesswork despite the use of the MMSegmentation codebase.