DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination
DuoCook evaluates whether a single vision-language model can centrally coordinate two cooking agents in Unreal Engine kitchens inspired by Overcooked. The benchmark spans 20 maps across five categories—basic, shared equipment, dynamic terrain, combined dynamic/shared, and separated workspaces—requiring the model to assign high-level actions from top-down visual observations while the engine handles low-level navigation. A memory ablation tests the value of a shared, self-authored belief notebook. Performance is far from saturated even with heavy scaffolding, suggesting current VLMs remain poor at real-time centralized coordination, though the task interface provides substantial structured assistance.
The paper introduces a sensible benchmark taxonomy for centralized coordination, but its experimental conclusions are fragile. The authors are admirably transparent about key weaknesses—stating that "a single run does not quantify variability in model responses" (Section 5) and that performance is "far from saturated" (Section 4.3)—yet they still draw comparative rankings among models whose best mean completion is only 26.2% and whose next-best is below 12%. The heavy scaffolding (grid labels, prop legend, live-state JSON, engine-computed legal actions, and NavMesh navigation) narrows the claimed scope from raw visual planning to assisted scheduling under generous structural support.
The five map categories provide a well-motivated taxonomy of coordination challenges, from independent workstations to forced handoffs across separated regions. The five reported metrics—total score, order completion rate, delivery precision, action acceptance rate, and API expenditure—are complementary and thoughtfully chosen. The memory ablation is cleanly designed: the notebook is the only difference between conditions, with six-response action history retained in both, so the contrast isolates shared self-authored memory from simple recency. The scoring formulas are explicit, e.g. $R_i = 20 + m_i b(w_i)$ for order rewards.
The most serious flaw is the absence of repeated runs: every reported result derives from a single episode per model, map, and condition. The authors admit that "results are descriptive and do not estimate variability across repeated model runs" (Section 4.1), which makes the reported model rankings and memory-effect signs unreliable. Bootstrap intervals in the memory ablation resample maps, not runs, so they cannot establish whether Claude's +4.8 point gain or Gemini's -8.1 point loss would replicate. Performance is extremely low—GPT-6-astra completes 26.2% of orders while the rest are below 12%—so differences among weaker models are especially likely to reflect noise. The task interface also provides heavy, largely unacknowledged scaffolding: top-down grid-labeled images, a prop legend, pot-ingredient count indicators, a ~4,000-word system prompt, live-state JSON, engine-computed legal operations, and NavMesh navigation. These affordances lower visual grounding and motor demands so much that the benchmark mainly tests structured scheduling under assistance, not the broader "visual planning" implied by the title. Finally, centralized single-model control of both players sidesteps adaptation to an independent partner, which is central to much of the cited multi-agent literature.
The evidence supports the modest claim that current VLMs struggle with centralized coordination under this interface, but it does not support bolder comparisons to raw-visual or multi-agent benchmarks. The authors correctly note that "scores are not directly interchangeable with those from different observation and control interfaces" (Section 2.2), acknowledging that DuoCook's structured inputs place it closer to assisted scheduling than to raw video-game play. The comparison to VideoGameBench is accurate: Zhang et al. (2025a) do introduce a paused "Lite" setting that separates inference latency from action selection, matching the description here.
Reproduction is severely hampered. The authors state that "supporting episode logs, per-map metric tables, and analysis scripts are not included in the current manuscript artifacts" (Section 5), and no repository or supplementary data is referenced. Sampling hyperparameters such as temperature or top-p are not reported for any model. API costs are measured in USD but depend on proprietary pricing tiers that may change. With only one run per condition, even perfect artifact release would not establish whether the reported 26.2% mean completion for GPT-6-astra or the mixed memory effects are stable. Independent reproduction would require rebuilding the 20 UE5 maps via SimWorld Studio, rerunning API calls at nontrivial expense, and reimplementing the notebook retention mechanics from the mathematical description $r_i^{(p)}(t) = 2^{-(t-t_i)/H_i^{(p)}}$ (Section 4.4) without released code.
Coordinating two players in a cooking task requires a vision-language agent to track changing scene states, sequence recipe steps, and allocate shared resources under a time budget. We introduce an Overcooked-inspired benchmark for evaluating these capabilities in Unreal Engine environments constructed using coding agents. A single vision-language model controls both players from top-down visual observations, selecting high-level actions while the engine executes navigation. The benchmark contains 20 maps in five categories. These categories support evaluation of centralized coordination under equipment contention, changing terrain, and handoff dependencies. A prop legend and pot ingredient-count indicators provide explicit visual cues, while grid-based destinations support spatial grounding. The evaluation compares four VLMs on all 20 maps, measuring task score, order completion, delivery precision, action acceptance rate, and model API expenditure. A memory ablation on the same 20 maps examines the effect of a shared, self-authored notebook while retaining six-response action history in both conditions. The strongest evaluated model, \texttt{gpt-6-astra}, achieves 26.2\% mean order completion. Notebook memory has mixed score effects across models, with positive mean differences for Claude and Qwen and negative differences for GPT and Gemini.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.