Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Ego2Act introduces a benchmark and automated evaluator for goal-directed egocentric manipulation video generation. It tests whether models can plan and execute multi-step physical tasks from a single starting image and high-level goal, spanning 110 real-world cases. Experiments reveal severe limitations in current video generators, with even the best proprietary model scoring only 64.0 on human evaluation and frequently skipping prerequisite steps or violating physical continuity.
The paper presents a well-motivated benchmark addressing a clear gap in evaluating video models as embodied world simulators. The rubric-based Ego2ActJudge demonstrates credible alignment with human judgments, and the dataset construction protocol is thorough, with multiple human executions and controls per case. However, the reliance on closed, proprietary APIs for both generation and evaluation raises accessibility concerns, and the human study covers only a subset of the benchmark.
The task construction and verification rubric is admirably rigorous: each case requires multiple valid execution sequences, closed-world constraints, and explicit dependency verification. The multi-stage gating design of Ego2ActJudge is principled, and ablations show that sequential gating meaningfully reduces score inflation; removing the primary physics continuity gate or the task initiation gate degrades alignment with human ratings.
The human evaluation panel covers only 25 of 110 cases (600 videos), which may not fully support benchmark-wide claims about model rankings. Automated evaluators also exhibit an upward scoring bias relative to human raters, and the authors acknowledge that the VLM backbone largely misses physics violations that unfold between frames, making Physics scores optimistic. Additionally, the study relies entirely on black-box proprietary video generators and the Gemini-3.7-Flash API, limiting independent verification and long-term reproducibility.
The evidence supports the key claim that current video models fail at multi-step egocentric manipulation: the top model scores only 64.0 on human evaluation, while open-weight models drop as low as 3.9. Comparisons to baselines are generally fair: Ego2ActJudge outperforms VideoScore, PQSG, and RBench in human correlation, and the authors correctly note that perception-heavy metrics score positive and negative human controls almost identically. However, the mapping of baseline outputs to Task/Physics scores required non-trivial adaptation, and the geometric mean aggregation, while justified, remains a design choice that heavily penalizes single-axis failure.
Replication is hampered by dependence on six closed-source video generation APIs and the Gemini-3.7-Flash judge backbone, with exact prompts only partially disclosed in appendices. While the authors commit to releasing benchmark videos, judge code, and analysis scripts, the cost and API access requirements are non-negligible (about US$0.04 per video in judge costs). Hyperparameters such as temperature are left at provider defaults, and frame-sampling strategies (8 frames for Task, 24 for Physics) lack ablations, leaving sensitivity unclear.
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.