AgentDreamer: Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation
AgentDreamer tackles text-controllable physics simulation by replacing ambiguous video diffusion priors with explicit 3D motion plans generated by an off-the-shelf multimodal agent. The agent reasons over scene geometry and text to produce dense parametric trajectories, which guide a differentiable MPM simulator toward physically feasible motion aligned with the instruction. A new metric, Motion-IG, isolates the information that simulated motion contributes to recovering masked text beyond the prompt alone. While the experimental scope is narrow, the framework offers a clean separation between semantic reasoning and physical grounding.
The paper introduces a compelling decomposition in which an LLM agent handles high-level 3D motion planning and a differentiable MPM simulator enforces physical feasibility. AgentDreamer wins in human evaluation (73.6% first-place rank) and Motion-IG scores across nine evaluators on 20 instructions, suggesting explicit 3D priors improve text alignment over video-guided baselines. Nevertheless, the evaluation is limited to eight curated scenes and leans heavily on a single proprietary agent, so the method's generalizability remains largely untested.
The motivation for abandoning video priors is sharply argued: a generated video of a rolling ball may depict sliding, so optimizing a simulator against it can yield physically plausible yet textually incorrect motion. The parametric-chunking strategy elegantly avoids structural distortions that plague direct per-point prediction, yielding coherent 3D target trajectories $\widehat{\mathbf{p}}_{k}(t)$ that the MPM matches via $\mathcal{L}_{\mathrm{motion}}(\bm{\theta})$. Motion-IG is a principled alternative to MLLM-as-a-judge that isolates the incremental evidence provided by motion, and its category-level trends match human preferences.
The experimental scale is very small: only 20 text instructions across eight scenes, with no train/test split or held-out generalization study, raising concerns about cherry-picking. The method is critically dependent on a single proprietary agent (GPT-5.6 Sol), and there is no ablation substituting open-weight models. Table 3's correspondence to classical mechanics is derived from only 20 plans and feels post-hoc; it does not demonstrate true reasoning over physical laws rather than prompt-conditioned pattern-matching. Finally, the paper avoids discussing failure modes where the agent proposes kinematically impossible trajectories and the MPM is forced to merely approximate them.
The ethics statement admits that pretrained multimodal models may reflect errors or biases, yet none of the reported results quantify how often the agent produces incorrect or infeasible motion plans.
The evidence supports the headline claim that explicit 3D planning improves alignment over PhysFlow and MotionPhysics on this benchmark: Motion-IG rises from an average of 0.52 to 0.71 with open-weight evaluators and from $-0.22$ to $+0.48$ with API evaluators, while human evaluators prefer AgentDreamer in over 80% of pairwise trials. However, the comparison pool is shallow—only two baselines—and the baselines are evaluated under the authors' chosen metrics rather than their own reported settings. The related-work framing is largely accurate: we verified that C-Drag uses trajectories for video-level motion control, and that Physics3D distills physical priors from video diffusion, though the latter is not explicitly text-conditional as implied by the grouping.
While the appendix provides motion-planning and evaluation prompts and identifies public scene sources (PAC-NeRF, GaussianVerse, PhysFlow), the authors do not commit to releasing code, MPM hyperparameters, or the exact 20 instructions. Reproduction is further blocked by proprietary components: GPT-5.6 Sol for motion planning and black-box API models for the metric. The use of teacher-forced likelihoods for open-weight evaluators and verbalized probabilities for API evaluators means the numerical Motion-IG scale is inconsistent across evaluator groups, though relative rankings are preserved.
We propose AgentDreamer, an agent-guided optimization framework that aligns text instructions with simulated motion through explicit 3D motion planning. Given a text instruction and scene geometry, an off-the-shelf agent translates the instruction into a structured semantic motion prior, specifying what should move, in which direction, and in what temporal order. The agent represents this prior using parametric functions over a series of temporal chunks, yielding dense 3D trajectories. We then optimize the parameters of a differentiable MPM simulator such that its rollout best matches the prior while remaining constrained by the scene dynamics. This allows the agent to provide semantic guidance from language, while the simulator grounds it in physically feasible motion. We further introduce Motion-IG, an information-theoretic metric for text-motion alignment based on masked text recovery. By comparing the likelihood of recovering masked motion-related content with and without the simulated motion, Motion-IG measures the information contributed by the motion beyond that available from the remaining text and scene context. Experiments show that AgentDreamer improves text alignment over video-based guidance, producing simulations that more closely follow the requested physical behavior.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.