Towards Automatic Video Annotation with ASH: Zero-Shot Open-Vocabulary Multi-Object Tracking and Segmentation
SAM3-ASH tackles fully automated video tracking annotation without any dataset-specific training. Its Generalized Presence Token (GPT) enables SAM3 to process multiple text prompts simultaneously by virtual batching over a single shared image encoding, collapsing per-prompt cost from $\mathcal{O}(N)$ to $\mathcal{O}(1)$. The Annotation and Segmentation Handler (ASH) splits videos of arbitrary length into overlapping temporal chunks and reconciles identities across boundaries via IoU matching, bounding peak GPU memory below 25 GB. Evaluated on eight benchmarks, the pipeline sets a zero-shot HOTA record on MOTS20 and stays within a narrow gap of trained specialists on tasks where mask-based output aligns with the ground-truth format.
The paper delivers an elegant, training-free solution to two concrete engineering blockers in open-vocabulary video annotation: redundant image encoding for multi-prompt inference and unbounded memory growth in memory-attention trackers. The GPT derivation is mathematically trivial yet practically impactful, and the ASH chunking strategy is validated with impressive boundary consistency. Still, the real-world utility is tempered by large performance gaps on standard bounding-box benchmarks and a post-hoc classification step that bottlenecks throughput on large vocabularies.
The zero-weight generalization claim for GPT is convincingly verified: the authors show that virtual batching yields exactly the same detection counts and near-identical masks as sequential single-prompt inference, with mean mask IoU of 0.998. The ASH chunking mechanism is equally well substantiated—on DanceTrack test seams, IoU matching preserves every one of 18 active identities across chunk boundaries despite severe occlusion, and SAM3-ASH achieves the highest reported HOTA (68.1) on MOTS20, a benchmark SAM3 cannot even process without ASH.
Performance on box-annotated benchmarks lags trained specialists by a wide margin—HOTA drops from 68.1 on MOTS20 to 45.2 on MOT17 for the same underlying video, and gaps of 7–14 points remain on DanceTrack and SportsMOT against top methods. While the authors attribute much of the MOT17 shortfall to mask-versus-box annotation mismatch, the 9.2 HOTA deficit to SAM2MOT on DanceTrack reveals that occlusion handling and detection recall are genuine weaknesses inherited from SAM3's memory bank. Additionally, the post-hoc classification pass is a severe scalability bottleneck, collapsing throughput to 0.017 FPS on LV-VIS's 1,196-class vocabulary and consuming over 317 GPU-hours for that benchmark alone.
The internal evidence is strong and carefully measured: GPT equivalence is checked frame-by-frame, memory profiling shows SAM3 exhausting a 40 GB partition at frame 993 while ASH holds a flat 9.3 GiB, and the direct MOTS20-versus-MOT17 comparison illustrates the annotation-format issue vividly. Comparisons to prior work are generally fair and taxonomy-aware, though the juxtaposition of MOTS20 and MOT17 HOTA scores to argue that box benchmarks underestimate mask trackers is illustrative rather than a controlled experiment. The contrast with SAM2MOT on DanceTrack is also slightly understated: SAM2MOT is training-free on that benchmark too, yet outperforms SAM3-ASH by 9.2 HOTA, indicating that the gap is not merely the cost of zero-shot generalization but specifically of missing occlusion-memory correction.
Hyperparameters and hardware constraints are specified with unusual precision: chunk size $\Delta=50$, overlap $\omega=10$, prompt batch $B=5$, thresholds $\tau_{\text{overlap}}=0.2$ and $\tau_{\text{class}}=0.1$, and a 40 GB MIG H100 partition with peak usage capped at 25 GB. The checkpointing logic and cost model parameters are also disclosed. However, the paper relies on SAM3 weights, which are tied to an ICLR 2026 publication and not yet publicly available; no code repository URL is provided, and the 317 GPU-hour LV-VIS run—while reproducible in principle—places large-scale replication out of reach for many labs.
Memory-attention-based Video Instance Segmentation (VIS) methods have demonstrated strong zero-shot tracking capability, yet their substantial memory requirements confine them to short video clips and their single-prompt inference design makes multi-category open-vocabulary tracking computationally prohibitive. This work introduces two contributions toward fully automated tracking annotation of arbitrary video. The Generalized Presence Token (GPT) reformulates SAM3's inference pipeline to process N text prompts simultaneously via virtual prompt batching, reducing image encoding cost from O(N) to O(1) with no modifications to any learned component. The Annotation and Segmentation Handler (ASH) extends any memory-attention VIS tracker to sequences of arbitrary length through overlapping temporal chunks with IoU-based inter-chunk identity matching, requiring no dataset-specific training. Instantiated on SAM3, the resulting pipeline -- SAM3-ASH -- achieves state-of-the-art HOTA on MOTS20 under fully zero-shot conditions and remains competitive with trained specialists across seven additional benchmarks, while peak GPU memory consumption stays below 25 GB, establishing a practical baseline for scalable, training-free automated video annotation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.