CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation
Joint audio-visual diffusion models repeatedly run both denoising branches and expensive cross-modal attention at every step, making inference costly. CAVC observes that audio-side drift caused by mid-stage caching can be repaired by a few terminal joint steps, whereas video-side degradation persists, and exploits this asymmetry with a training-free sandwich structure: full joint warm-up, video-only continuation with cached audio, and brief terminal recoupling. The result is a modest wall-clock speedup with improved visual dynamics, though the gains are bounded by the preserved video backbone.
CAVC is a carefully motivated, training-free caching strategy that delivers a favorable quality-efficiency trade-off for a specific dual-stream transformer, but its practical impact is limited by modest 1.10–1.35× wall-clock speedups and evaluation on a single architecture with a distilled LoRA. The theoretical perturbation analysis is honest about its assumptions, and the asymmetric-recoverability narrative is strongly supported by ablations; however, the reliance on unverified Lipschitz constants and no-reference metrics tempers the strength of the guarantees.
The central claim of modality-asymmetric recoverability is well supported. Branch-freezing ablations in Table 4 show that terminal recoupling closes 81.5–91.3% of the CLAP gap for audio-side caching, while video-side dynamics show 0% recovery even with five terminal steps. The complexity analysis in Section 3.4 correctly identifies that fixed decoding and Phase-C costs dilute branch-level savings, explaining why measured wall-clock gains (1.10–1.35×) are smaller than raw step-reduction ratios. The adaptive trigger avoids the pitfalls of fixed schedules, as shown in Table 3b.
The speedups are modest for an acceleration paper and only validated on one model pipeline (LTX-2.3-22B-dev with distilled LoRA), so cross-architecture generalization is unproven. The adaptive cache trigger relies on an empirical heuristic (θ=0.30, W=3) without a principled selection criterion for other models or prompt distributions. The theoretical bounds assume local Lipschitz conditions that are neither verified nor tight; the paper itself notes these do not guarantee perceptual quality and empirical behavior is what matters. Additionally, several metrics in Table 4 fluctuate non-monotonically with more terminal steps (e.g., AV Sync drops from 0.679 at 1 step to 0.502 at 3 steps), suggesting residual instability that the paper does not fully explain.
The evidence supports the asymmetric-recoverability premise better than broad acceleration claims. TeaCache and Cache-DiT baselines are fairly positioned as modality-agnostic strategies that introduce net overhead on this pipeline and degrade dynamics when applied uniformly. However, the evaluation leans heavily on no-reference metrics (VBench, DNSMOS, AV-F1) and MLLM evaluators whose correlation with human perception in audio-visual generation is asserted but not validated by the paper. The improvement in Dynamic Degree (+34.7% on General-80) is real under these metrics, yet the paper offers no human study to confirm perceptual gains.
Experimental configuration is documented in exceptional detail—including BF16 precision, exact LoRA snapshots, hardware, CPU offload, and method-specific cache parameters—so a well-equipped team could reimplement the method. Nevertheless, no public code repository, URL, or data link is provided in the main paper or appendices; the authors reference an anonymous code artifact and specific script paths without enabling access. This absence is a significant block to independent reproduction, calibration of the adaptive trigger on new prompts, and verification of wall-clock timings.
Diffusion-based audio-visual generation methods typically perform joint denoising of audio and video latents throughout the entire sampling trajectory, repeatedly executing modality-specific denoising branches and cross-modal attention at every timestep. However, we observe that such uniform joint computation is not always necessary. The key is a modality-asymmetric recoverability: audio-side drift introduced by mid-stage caching can be substantially reduced by only a few terminal joint denoising steps, whereas video-side degradation caused by caching leads to persistent loss of motion and temporal details. This asymmetry suggests that inference acceleration should selectively cache recoverable audio-side computation while preserving full video refinement. Motivated by this observation, we propose \textbf{Convergence-Adaptive Audio-Visual Cache (CAVC)}, a training-free modality-asymmetric caching framework. CAVC adopts a sandwich-structured computation allocation: after detecting that audio denoiser outputs enter a cache-tolerant regime, it caches and reuses the audio denoiser output to bypass audio denoising and cross-modal attention in the middle stage, while the video branch continues independent spatio-temporal refinement; a few joint steps are then restored at the end to reduce accumulated cross-modal drift and improve synchronization. Experiments across multiple public benchmarks demonstrate that CAVC achieves 1.10--1.35x wall-clock acceleration, improves several visual dynamics and temporal statistics, and maintains competitive audio fidelity and audio-visual alignment.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.