Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

The paper proposes Factor-Aware Predictive Learning (FAPL) to make multimodal large language model (MLLM) latents usable by a Video Diffusion Transformer (DiT) for cinematic video generation. It uses three stages: latent-only generator grounding with a frozen MLLM and connector, factor-span replacement on paired videos for factor-aware control, and a final stage optimizing the MLLM and connector through a frozen DiT for limited-input temporal inference. The authors report better performance on sparse-to-video, factor-aware editing, and multi-shot generation tasks over baselines.

Key takeaways

• A DiT may bypass MLLM latents through native conditioning pathways, so MLLM use is not automatically effective.
• Latent-only generator grounding trains the DiT with a frozen MLLM and connector to strengthen use of MLLM latents.
• Factor-span replacement on paired videos targets a specified factor change while preserving non-target factors.
• The paper reports consistently better results on three cinematic video generation tasks over baselines.

Why it matters

Cinematic generation needs both sparse-input temporal inference and control over characters, scenes, actions, and cameras. Showing how to make MLLM representations actionable for video diffusion could inform how multimodal priors are integrated into generative video models.

Read with care

Source claims not independently checked; the supplied text gives no quantitative results or baseline names.