Adaptive Video Distillation: Mitigating Oversaturation and Temporal Collapse in Few-Step Generation
Video diffusion models suffer from prohibitive inference costs, but standard image distillation techniques like DMD cause severe oversaturation and temporal collapse when naively extended to video. This work introduces a video-specific distillation framework featuring an adaptive regression loss that dynamically reweights real-data supervision to prevent color artifacts, a temporal variance regularizer to combat static output, and an inference-time frame interpolation module that halves sequence length during high-noise steps to accelerate generation. Applied to Wan2.1, the method enables stable 4-step synthesis with state-of-the-art VBench scores.
The paper presents a technically sound and well-validated solution to video distillation artifacts. The adaptive regression mechanism is elegant—using a timestep-aware EMA cache to suppress gradient contributions from outlier samples effectively resolves the tearing and object-fusion artifacts that plague naive regression. However, the temporal regularization loss lacks intrinsic stability and requires manual clipping to prevent hallucinatory frame jumps, suggesting the objective is somewhat ad-hoc. Additionally, the reliance on a proprietary 150K video dataset for regression supervision and the delicate balancing of three competing loss terms may hinder reproducibility and practical adoption.
The central claim that naive regression on real data causes object fusion while adaptive weighting fixes it is convincingly demonstrated. In ablations, naive regression drops Instance Preservation to 83.04 versus the DMD baseline of 88.88, whereas the adaptive loss restores it to 92.39—matching the teacher. The frame interpolation strategy is also practical, yielding a 30\% speedup (7.8s vs 10.8s for DMD on the 1.3B model) with negligible quality loss by exploiting the observation that high-noise steps exhibit minimal temporal variance.
The temporal regularization loss $\mathcal{L}_{\text{temp}}=-\log(\mathbb{E}_{x\sim p_{\theta}}[\mathrm{Var}(x)]+\epsilon)$ lacks a convergence mechanism and must be clipped at 0.6 to prevent 'severe frame jumps,' indicating the loss can destabilize training if left unbounded. The paper also claims the method supports simultaneous fine-tuning to new domains (e.g., anime) during distillation, but provides only a single qualitative example without dataset details or quantitative validation. Furthermore, the distribution matching loss uses text-only conditions while the regression loss requires high-quality proprietary videos, creating a data imbalance that may limit generalization.
Quantitative evidence is robust across VBench1 and VBench2 benchmarks, with the method achieving the highest total scores against DMD, LCM, PCM, DCM, and rCM baselines on both 1.3B and 14B models. Table 2 effectively isolates components: temporal regularization boosts Dynamic Degree from 72.22 to 97.77, while adaptive regression fixes the Instance Preservation degradation caused by naive regression. However, comparisons to non-distillation acceleration methods (e.g., improved ODE solvers) are absent, and the user study relies on only 12 annotators with majority voting, offering limited statistical rigor.
The authors release source code and provide detailed hyperparameters ($\alpha=0.95$, $k=3.0$, $\omega_{\text{temp}}=0.05$), which aids reproduction. However, the regression loss depends on a curated proprietary dataset of 150K videos that is not publicly available, potentially blocking exact reproduction of the reported scores. Training requires 24 GPUs, which is resource-intensive but feasible. The frame interpolation module requires pre-training a separate UNet for 10,000 iterations on VAE latents, adding engineering complexity but is fully specified.
Video generation has recently emerged as a central task in the field of generative AI. However, the substantial computational cost inherent in video synthesis makes model distillation a critical technique for efficient deployment. Despite its significance, there is a scarcity of methods specifically designed for video diffusion models. Prevailing approaches often directly adapt image distillation techniques, which frequently lead to artifacts such as oversaturation, temporal inconsistency, and mode collapse. To address these challenges, we propose a novel distillation framework tailored specifically for video diffusion models. Its core innovations include: (1) an adaptive regression loss that dynamically adjusts spatial supervision weights to prevent artifacts arising from excessive distribution shifts; (2) a temporal regularization loss to counteract temporal collapse, promoting smooth and physically plausible sampling trajectories; and (3) an inference-time frame interpolation strategy that reduces sampling overhead while preserving perceptual quality. Extensive experiments and ablation studies on the VBench and VBench2 benchmarks demonstrate that our method achieves stable few-step video synthesis, significantly enhancing perceptual fidelity and motion realism. It consistently outperforms existing distillation baselines across multiple metrics.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.