Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
Cinematic video generation demands both inferring plausible action and camera trajectories from sparse text or reference inputs, and independently controlling character, scene, action, and camera factors without unwanted side effects. The authors argue that simply feeding MLLM features into a Video DiT fails because the DiT can bypass those latents via native text-encoder and VAE-reference pathways. Factor-Aware Predictive Learning (FAPL) fixes this with a three-stage curriculum—latent-only grounding, factor-span replacement, and fixed-generator predictive learning—to force the DiT to rely on MLLM representations and, in turn, teach the MLLM to output generation-compatible latents.
The paper correctly identifies latent bypass as a central bottleneck in MLLM-conditioned video generation and proposes a clean, architecture-agnostic training strategy with consistent quantitative gains. However, the evaluation rests solely on automatic metrics, the editing supervision relies on synthetic paired data of unclear provenance, and no code or data is released, which limits confidence in the reported improvements.
The three-stage design is principled and directly targets the identified bottleneck. Stage I removes alternate pathways so the DiT must decode MLLM latents alone; Stage II replaces factor-specific spans in the conditioning space to teach the generator what should change and preserve; and Stage III freezes the tuned DiT to adapt the MLLM toward generation-compatible prediction without decoder drift. Ablations in Table 2 support each stage: factor swapping raises $\cap$-Acc from 48% to 57%, and adding mixed-observation training in Stage III lifts Motion Order from 57% to 61% while cutting camera errors.
FAPL depends on 100K source–target editing pairs that are described only as constructed through factor-specific selection and generation, but the paper gives no protocol, model, or quality control for this synthesis, making it impossible to assess bias or artifact propagation into supervision. The evaluation lacks any human perceptual study, relying instead on coarse automatic proxies (Motion Order, camera trajectory error, and FiVE $\cap$-Acc) that may miss cinematic fidelity failures and subtle factor leakage. Additionally, the Stage III assumption—that a frozen DiT can gracefully decode a shifting MLLM distribution—is not diagnostically tested for representation drift or reconstruction degradation. The authors themselves note that current MLLMs and the pretrained DiT place hard ceilings on camera reasoning and acting realism.
Table 1 reports broad gains over UniVideo (13B) and Omni-Video 2 (14B): FAPL improves Motion Order by 13 points over UniVideo and $\cap$-Acc by 16 points, with consistent advantages in camera trajectory and multi-shot face consistency. Table 2 decomposes these gains stage-wise, showing each signal contributes measurably. The comparison within the MLLM-DiT family is fair, but the baseline set is narrow and the bypass claim is illustrated with cherry-picked examples rather than a causal ablation of pathway contributions. The central diagnostic quote remains apt: native reconstruction objectives do not guarantee effective MLLM latent use.
Hyperparameters are partially stated (AdamW, lr $1\times10^{-5}$, BF16, 50 sampling steps, CFG 5.0) and the backbone (UniVideo 13B) is public, but critical details are omitted: the exact prompt rewriting rules into four ordered fields, the span-matching heuristic during replacement, and the observation-mode sampling weights $\rho_m$ and $\pi_m$. The 100K paired editing set is summarized as quality-filtered selections from CoinVE-200K and OmniVBench, yet no selection code, generation model, or data manifest is provided. Stage III requires backpropagating gradients through a frozen 13B DiT into the MLLM and connector, which is feasible but compute-intensive; without released checkpoints or training scripts, independent reproduction is effectively blocked.
Cinematic video generation requires both temporal inference from limited inputs (i.e., sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, and factor-span replacement trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate our consistently better performance on three cinematic video generation tasks (i.e., sparse-to-video, factor-aware editing, and multi-shot generation) over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.