Mamba-VMR: Multimodal Query Augmentation via Generated Videos for Precise Temporal Grounding
This paper addresses video moment retrieval (VMR) for complex multi-verb queries by proposing a two-stage framework that generates auxiliary short videos via text-to-video diffusion (CogVideoX) as temporal motion priors, then processes them through a linear-time Mamba network. The approach tackles the limitation of static image augmentations—which miss motion dynamics—while avoiding the quadratic complexity of Transformer-based methods on long untrimmed videos. The framework achieves state-of-the-art results on TVR with particular strength on multi-verb queries, though its effectiveness depends heavily on external video generation quality.
The paper presents a well-motivated approach combining LLM-guided query decomposition, subtitle-enhanced video generation, and efficient Mamba-based fusion for VMR. The experimental validation is thorough with comprehensive ablations demonstrating the value of motion priors over static images. However, the framework's practical deployment is constrained by reliance on CogVideoX generation quality (limited to 6 seconds at 8 FPS) and the computational overhead of LLaMA-3.1 inference, which are not fully accounted for in latency metrics. While the memory efficiency advantages are clear, the accuracy gains over strong baselines like ICQ are modest (+1.07% R@1@0.5), raising questions about the cost-benefit trade-off of video generation in production settings.
The ablation studies rigorously validate each component, showing that removing temporal prior generation causes a significant 6.44% drop in R@1@0.5 (45.20 to 38.76), confirming that motion-rich priors outperform static augmentations. The multi-verb analysis in Appendix A provides compelling evidence for the core hypothesis: on queries with $\geq 3$ verbs, the method achieves 35.94% R@1@0.5 versus 22.13% for EventFormer, demonstrating robust handling of sequential actions. The memory efficiency claims are substantiated with quantitative comparisons showing Mamba-VMR consumes 45-70% less GPU memory than Transformers and scales to sequence length 1024 where vanilla Transformers encounter OOM.
The framework exhibits a critical dependency on generated video quality from CogVideoX, which operates at limited resolution (720$\times$480) and frame rate (8 FPS), potentially introducing hallucinations or missing fine-grained motions that mislead retrieval. The LLM-based subtitle matching and query decomposition using LLaMA-3.1 (8B parameters) introduce substantial computational overhead not reflected in the 1.2-second inference latency claim, as this metric excludes offline video generation. The SOTA improvements over ICQ are marginal (+1.07% on TVR R@1@0.5), suggesting diminishing returns for the added complexity of video generation compared to static image augmentation. Additionally, the loss function $\mathcal{L} = \lambda_1 \mathcal{L}_{\text{bound}} + \lambda_2 \mathcal{L}_{\text{rel}} + \lambda_3 \mathcal{L}_{\text{cont}}$ relies on manually tuned weights ($\lambda_1=1.0, \lambda_2=0.5, \lambda_3=0.1$) without sensitivity analysis.
The evidence supports the claim that motion priors improve multi-verb retrieval, with Table 4 showing CogVideoX (45.20) outperforming static DALL-E images (39.45) by 5.75% R@1@0.5. However, the comparison between Mamba and Transformer architectures in Appendix C reveals minimal accuracy differences at shorter lengths (44.47 vs 45.20 at length 512), indicating the primary advantage is computational efficiency rather than retrieval precision. The comparison to ICQ is fair in modalities (static vs video) but understates the inference cost disparity—ICQ requires single-image generation while Mamba-VMR requires full video diffusion. The ablation on loss functions (Table 6) shows that removing $\mathcal{L}_{\text{cont}}$ causes a 4.12% drop, validating the importance of contrastive alignment between generated videos and target moments.
The paper provides detailed experimental protocols including hyperparameters (AdamW, lr=$1\text{e-4}$, batch size 32, 20 epochs), loss weights ($\lambda_1=1.0, \lambda_2=0.5, \lambda_3=0.1$), and architecture specifics (4 bidirectional SSM layers, $d=512$, $N=16$). The code and models are claimed to be available, though the provided GitHub URL contains a typo ("Manba" instead of "Mamba"). Reproduction requires substantial resources: LLaMA-3.1 8B for subtitle matching, CogVideoX for video generation, and 4$\times$RTX 4090 GPUs for training. The pre-computation of generated videos offline mitigates inference latency but requires significant storage for the 108K queries in TVR. The TVR and ActivityNet-Captions datasets are publicly available, though the paper evaluates on the hidden test set of TVR using validation set results only.
Text-driven video moment retrieval (VMR) remains challenging due to limited capture of hidden temporal dynamics in untrimmed videos, leading to imprecise grounding in long sequences. Traditional methods rely on natural language queries (NLQs) or static image augmentations, overlooking motion sequences and suffering from high computational costs in Transformer-based architectures. Existing approaches fail to integrate subtitle contexts and generated temporal priors effectively, we therefore propose a novel two-stage framework for enhanced temporal grounding. In the first stage, LLM-guided subtitle matching identifies relevant textual cues from video subtitles, fused with the query to generate auxiliary short videos via text-to-video models, capturing implicit motion information as temporal priors. In the second stage, augmented queries are processed through a multi-modal controlled Mamba network, extending text-controlled selection with video-guided gating for efficient fusion of generated priors and long sequences while filtering noise. Our framework is agnostic to base retrieval models and widely applicable for multimodal VMR. Experimental evaluations on the TVR benchmark demonstrate significant improvements over state-of-the-art methods, including reduced computational overhead and higher recall in long-sequence grounding.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.