Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering
Episodic memory video question answering asks models to reason over long egocentric recordings by compressing the visual stream into a queryable memory. This work isolates retrieval within a fixed online textual-memory pipeline, systematically comparing retrieval units, context budgets, and reranking strategies on QAEgo4D-Closed. The authors show that their best Parent strategy matches state-of-the-art accuracy with far fewer tokens, and reveal that higher temporal retrieval coverage can actually hurt accuracy by injecting hard distractors.
The paper provides a carefully controlled, training-free study that cleanly attributes gains to retrieval design rather than caption quality or reasoning capacity. Its central empirical result—that temporal retrieval metrics and answer accuracy diverge as the budget grows—is well documented. As the authors note, "Accuracy, however, follows a different trend. It peaks at 56.4% for $K=12$, then decreases as additional captions are included." (Section IV-B) Yet the study is confined to one dataset and one fixed memory generator, and its final call for "evidence-aware" memories outpaces the actual architectural advances presented.
The experimental isolation of retrieval is methodologically sound: holding the textual memory constant across all conditions lets the authors directly compare Flat, Subchunk, Parent, Hybrid, and Reranked variants. The Parent strategy is convincingly the best trade-off, and the paper articulates why: "Overall, the results show that the most effective strategy is not simply the one with the highest retrieval metrics, but the one that best couples retrieval precision with readable context." (Section IV-A) The retrieval-budget curve in Figure 4 is a standout finding that demonstrates more retrieved context is not always better.
The scope is narrow: every result comes from QAEgo4D-Closed with a single descriptor-reasoner pair (Qwen3-VL-8B and Gemini 2.5 Flash), so the claimed principles may not transfer to other memory backbones or open-ended settings. The paper's closing argument that caption-based memories must become "evidence-aware" is more aspirational than demonstrated, since none of the tested methods explicitly model uncertainty, temporal relations, or answer discriminativeness during indexing. The qualitative support is also thin; with only two examples shown, claims about spatial ambiguity rest on limited evidence. The failure case does illustrate the point: "the retrieved captions are semantically related to the query, but only provide coarse evidence about the knife being near the sink." (Section IV-D)
A second limitation is that the study remains entirely within a multiple-choice setting, which shields the reasoner from the additional challenge of free-form generation and may inflate the apparent value of retrieval relative to more open-ended episodic QA tasks.
The internal ablations in Table I and Figure 4 directly support the claim that retrieval granularity and budget affect reasoning non-monotonically. The comparison with prior work in Table II is transparent about measurement asymmetries, admitting that "these memory measurements are not directly equivalent across methods, since they represent different memory representations." (Section IV-C) The token-count reduction—from roughly 10,880 to 3,600 tokens per query—is the strongest comparative claim and is clearly substantiated. However, direct comparisons to visual-memory or KV-cache systems should be treated cautiously because the metrics are not directly comparable.
The paper names all core models and hyperparameters: Qwen3-VL-8B for captioning, Gemini 2.5 Flash with temperature 0 and structured JSON decoding for reasoning, llama-3.2-nemoretriever-300m-embed-v1 for embedding, and a $k_0=60$ reciprocal-rank fusion constant. The authors state: "All experiments use Qwen3-VL-8B as the descriptor model and Gemini 2.5 Flash as the reasoner model, with temperature set to 0 and structured JSON decoding." (Section IV) However, no code, prompts, or vector-database configuration details are released, and the captioning pipeline is inherited from prior work without full specification here. The use of proprietary API endpoints for Gemini and the NVIDIA reranker means exact reproduction depends on external services that may drift, and the absence of reported random seeds or runtime estimates poses further practical barriers.
Long egocentric videos pose a central challenge for multimodal language models: the visual stream must be compressed into a memory representation that can be searched efficiently while still preserving the evidence needed for question answering. Building on an online textual-memory framework in which first-person videos are converted into temporally grounded captions, this work systematically studies the role of retrieval in episodic-memory video question answering. We evaluate how different retrieval units, query formulations, context budgets, and reranking strategies affect multiple-choice answering on QAEgo4D-Closed, isolating retrieval as the main variable while keeping the underlying generated memory fixed. Our best configuration, Parent, matches fine-grained caption subchunks, then provides their full parent captions to the reasoner. It achieves 56.4 % accuracy, matching strong full-memory and retrieval baselines while exposing only a small fraction of the textual memory at question time. More importantly, our analysis shows that better temporal coverage does not necessarily translate into better answers: increasing the retrieval budget improves temporal Hit@K and Recall@K, but can reduce accuracy by introducing visually plausible distractors and uncertain caption evidence. These findings suggest that retrieval-augmented egocentric memory should not be optimized only for evidence recall, but also for the reliability, specificity, and answer-discriminative value of the retrieved context. The study motivates more structured episodic memories that move beyond raw caption retrieval toward evidence-aware representations for long-form egocentric reasoning.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.