PhyEvolve: Multi-Agent State-Evolution Planning for Physics-Aware Video Generation
PhyEvolve addresses a core failure mode in text-to-video generation: user prompts underspecify physical mechanisms and state dependencies, leading to incoherent dynamics. The paper proposes a training-free, three-agent framework—Director, Physical Evolver, and Composer—that plans explicit causal state trajectories, including during occlusions, and rewrites the prompt before feeding it to a frozen video generator. The work matters because it lifts physical consistency into an explicit planning stage rather than hoping the diffusion model hallucinates correct physics from vague text.
PhyEvolve achieves the highest reported scores on four physical-video benchmarks with Minimax-H3 held fixed, and it cuts end-to-end latency versus prior agentic systems by roughly 75%. However, the gains are measured on a single generator backend, and absolute success rates remain sobering: VideoPhy-2 Hard Joint reaches only 12.8%, and on StEvo-Bench the full three-agent system beats the ablation without a Director by just 0.4 percentage points. The paper is best read as a strong demonstration that explicit state planning improves physical continuity, not as a general solution to physics-aware generation.
The separation of physical actions from observation changes is conceptually clean, and the ablation confirms that the Physical Evolver is the dominant source of improvement. Removing it causes drops of 16.7 points in Evolution Success and 24.5 points in Physics on StEvo-Bench, far larger than removing either the Director or the Composer. The Composer’s sentence-level locking during fidelity refinement is also a thoughtful detail that prevents uncontrolled drift in the replacement prompt.
The Director and Composer contribute marginally on the out-of-sight benchmark: Table 6 shows that omitting the Director yields 45.3% Evolution Success versus 45.7% for the full system, and omitting the Composer yields 44.3%. This casts doubt on whether the full three-agent decomposition is necessary relative to a leaner Physical-Evolver-plus-prompt scheme.\nReliability is fully tethered to the LLM’s internal physics knowledge, with no calibrated dynamics or precise timings. The paper itself notes that evaluation is limited to short clips and that longer-horizon behavior is unproven. The user-study methodology is also thin: no sample sizes, confidence intervals, or blinding procedures are reported, and some per-benchmark preference margins fall near 60%.
The head-to-head against NEWTON is fair on the shared H3 backend, but the two frameworks serve different purposes: NEWTON trains a planner to call heterogeneous physics tools and runs a verifier loop across multiple generators, while PhyEvolve is a lightweight, training-free prompt rewriter. PhyEvolve’s large efficiency gains therefore reflect a simpler design rather than equivalent functionality delivered faster. Moreover, the margin over NEWTON is widest on StEvo-Bench (45.7% vs. 29.0% Evolution Success), which directly rewards PhyEvolve’s explicit state-trajectory focus, and narrower on PhyGenBench (79.6% vs. 77.5%). On VideoPhy-2, semantic adherence remains the binding bottleneck, indicating that stronger physics prompting alone cannot close the fidelity gap.
The experimental setup is documented with model names, inference configurations, and the full text of agent prompts in Appendix A.1. H3 generation parameters and agent hyperparameters are also specified. The main barrier to reproduction is the lack of a released codebase or parsing logic for the structured agent outputs; independent groups would need to reimplement the review-and-repair loop and secure access to an H3 inference stack. In addition, the evaluation depends on proprietary or hard-to-replicate benchmark protocols such as the VideoPhy-2 automatic evaluator.
Recent advances in text-to-video generation have substantially improved visual fidelity, yet existing models still struggle to understand the physical evolution implicit in user prompts, making it difficult to generate videos with continuous physical state evolution. Here, we propose PhyEvolve, a multi-agent framework that enhances natural-language prompts through structured state-evolution planning. PhyEvolve consists of three agents: a Director, a Physical Evolver, and a Composer. Given a user prompt and an optional first frame, the Director grounds the scene and disentangles physical events from observation changes. The Physical Evolver uses scene evidence to identify physical mechanisms then constructs causally linked state transitions that unfold throughout the video, whether visible or not. The Composer utilizes these transitions to enhance the original prompts. Without retraining or model modification, PhyEvolve works directly with existing generators through prompt replacement. Extensive experiments on four major benchmarks show that PhyEvolve achieves state-of-the-art performance, outperforming the cutting-edge prior agentic baseline by 7.9 percentage points on average. These results demonstrate that explicit causal state-evolution planning improves physical consistency and state continuity in generated videos through natural-language prompt enhancement.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.