Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation

The core insight—that mask transformers suppress unseen categories because their void token is learned from limited training vocabularies—is well-articulated and empirically validated. The CLIP-conditioned objectness adjustment (COAT) elegantly leverages CLIP's unbiased, large-scale training to correct these probabilities without additional learnable parameters. The OVR refinement protocol is notably simpler than MAFT+'s representation-consistency losses yet achieves comparable or better alignment, and the ablation in Table 6 demonstrates that aggressive constraints (Gram matrix, RC loss) are unnecessary. Figure 4 provides compelling per-class evidence that COAT specifically rescues underrepresented categories like paintings (+192% relative improvement) without catastrophically degrading frequent classes.

The paper's primary limitation is a lack of clarity on training data overlap. COCO (used for training) and ADE20K/Cityscapes share many semantic categories, yet the paper reports 'unseen' class performance without precisely defining vocabulary overlap metrics or category-level splits. The trust factor $\gamma=0.5$ in COAT appears arbitrarily tuned despite claims of robustness—Figure 5 shows performance degrades 20% at $\gamma=1.0$, so the operating point matters significantly. The semantic segmentation evaluation (Table 3) shows OVRCOAT actually underperforms MAFT+sem by 1–9% mIoU, suggesting the method is strictly specialized for panoptic tasks and does not generalize well to pure semantic segmentation, a limitation the authors acknowledge but dismiss quickly.

The evidence supports the main claim of improved out-of-vocabulary panoptic performance. Table 1 shows consistent PQ gains across ADE20K, Mapillary, and Cityscapes, with ablations in Table 2 demonstrating that COAT and OVR are complementary (combining yields +1.8% PQ over OVR alone on ADE20K). The oracle experiment (Table 5) validates that OVR genuinely improves mask-level classification (+11% PQ with ground-truth masks). Comparisons to MAFT+ are fair in that both use ConvNeXt-L backbones, though the paper's memory efficiency claims rely on batch-size-1 measurements that may not linearly extrapolate. The attribution of COCO drops to COAT's design is post-hoc (Section 4.1), and the qualitative analysis (Figure 6), while illustrative, lacks failure cases showing false positives introduced by COAT's permissive objectness adjustment.

Reproducibility is moderately good. The paper specifies the ConvNeXt-Large backbone from OpenCLIP, the two-stage training protocol (stage 1: frozen CLIP at $10^{-4}$ lr; stage 2: unfrozen at $5\times 10^{-5}$), and loss weights ($\alpha=0.1$). The code link is provided in the abstract. However, critical details are sparse: the exact number of training iterations per stage is omitted, data augmentation pipelines are not described, and the Hungarian matching cost weights are not specified. The memory measurements (Table 7) lack standard deviation or hardware utilization metrics beyond 'three NVIDIA A100 GPUs'. The small batch size (9 across 3 GPUs) suggests training may be unstable or slow to converge without extensive hyperparameter tuning. The supplementary material promises extended qualitative analysis but the main paper should include model checkpoint sizes and inference time standards.