HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games
HiSTEVE tackles the challenge of building Minecraft agents that can perform long-horizon tasks using only native keyboard-and-mouse control, without relying on environment APIs or predefined skills. It proposes a hierarchical VLA agent that pairs a capability-aware harness—equipped with structured memory, a VLM planner, and a policy-coupled completion detector—with a single low-level VLA policy for visuomotor execution. The core idea is to close the loop between high-level planning and raw pixel-to-action control in a partially observable open world, which matters for developing generally capable game-playing agents.
HiSTEVE is a solid but uneven step forward. The paper demonstrates strong atomic-task execution and convincingly shows that harness-level orchestration drives long-horizon gains, yet the aggregate OpenHA improvement is razor-thin and GUI performance regresses. The most meaningful advance is the separation of planning and control timescales via a state-aware harness, not a uniformly stronger low-level policy.
The MineStudio Bench gains are large and well-supported by ablations, and the regime-specific architectural changes are empirically motivated. The gradient-conflict analysis cleanly justifies RS-Norm, while the policy-coupled completion detector is both novel and substantially more accurate than prompted general-purpose VLMs.
The OpenHA aggregate improvement is only 0.6 percentage points, and the GUI family falls from 33.3% to 27.2%, directly contradicting the framing of uniformly reliable native-interface control. More importantly, reinforcement learning on the VLA leaves LongBench goal success unchanged at 14.3%, which means all long-horizon gains come from the harness rather than improved low-level execution. Recovery and Open-ended tasks also stall at 14.3% final-goal success, indicating that robust failure recovery remains largely unsolved.
Evidence strongly supports the MineStudio and completion-detector claims, but comparisons elsewhere are weaker. The 0.6 point OpenHA margin lacks variance estimates or statistical tests and may not be significant. LongBench baselines are extremely weak—OpenHA reaches only 10.7% goal success—so HiSTEVE's 25.0% is relatively better while still indicating that long-horizon autonomous play is far from solved.
Reproducibility is mixed. The Qwen2.5-VL-7B backbone and many architecture details are documented in appendices, but the harness relies on GPT-4o, a proprietary model with no version pinning and unstable behavior over time. The paper references a project page but does not explicitly commit to releasing code, checkpoints, or harness prompts. Training compute, wall-clock time, and environment versions are omitted, and the RL reward construction requires simulator-side state that may be hard to replicate independently.
Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.