StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

The technical design of the evaluation protocol is robust and carefully considered. The byte-level resource budgeting mechanism that converts a fixed memory budget $M$ into model-specific token caps $B_i = \lfloor M_{\text{bytes}} / (d_i s_{\text{emb}} + 2L_i h_i^{kv} s_{kv}) \rfloor$ (Appendix A.1) elegantly normalizes comparisons across models with different embedding dimensions and architectures. The three-process asynchronous pipeline (Frame Player, Encoder-and-Memory Updater, Responder) accurately emulates real-world streaming dynamics without synchronization-induced blocking. The empirical analysis is comprehensive, spanning 12 models and two benchmarks (OVO-Bench and StreamingBench), with sensitivity analyses across memory budgets {1.5G, 1.0G, 0.5G, 0.3G, 0.1G} and input resolutions (224×224 to 448×448).

The evaluation scope is constrained by computational limitations, restricting experiments to 7B–8B models and excluding closed-source systems (GPT-4V, Gemini), which may exhibit different scaling properties. The equal weighting in the default StreamingScore ($w_f = w_a = w_t = w_r = 0.25$) is acknowledged as arbitrary, though the appendix demonstrates robustness to weight variations (Spearman $\rho \in [0.972, 0.993]$). A more fundamental concern is that the FIFO eviction policy for offline models, while enabling fair comparison, may underestimate the potential of sophisticated compression or summarization mechanisms that real-world streaming systems might employ. The framework also assumes a fixed frame rate (1 FPS) which may not capture variable-rate sampling strategies that could optimize information density.

The evidence supports the central claim that offline models often outperform online models under streaming constraints when equipped with comparable memory budgets. For instance, Qwen3-VL-8B achieves 58.00 overall on OVO-Bench compared to Flash-VStream-7B's 33.15 (Table 1), demonstrating that native streaming architectures sacrifice accuracy for incremental update mechanisms. The comparison is methodologically sound: offline models use a standardized FIFO memory bank while online models use native mechanisms, with both constrained by the same byte-level memory budget. The paper fairly positions its work against prior benchmarks like OVO-Bench and StreamingBench, noting that these focus primarily on accuracy while StreamingEval adds system-level deployability metrics. The observation that "being 'online' does not necessarily translate into practical deployability" (Section 5) is well-substantiated by the throughput and latency measurements.

The paper provides strong reproducibility foundations with detailed hardware specifications (NVIDIA RTX 4090-48G, 40.32 TFLOPS peak throughput), inference settings (BF16, FlashAttention-2), and exact formulas for memory budget conversion. The memory calculation accounts for both visual token embeddings and KV cache storage across all transformer layers. The authors commit to releasing code at https://github.com/wwgTang-111/StreamingEval. However, the lack of released code at the time of writing prevents verification of the inter-process communication overhead claims. The multi-process emulator design using "inter-process queues and/or shared buffers" (Section 3.2) is described but not validated against real distributed serving systems. The limitation to a single GPU environment also means multi-GPU serving dynamics are not captured.