From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
Wedge addresses the inefficiency of generic environment generation for embodied navigation: pretrained agents fail for different environmental reasons, so simply adding more diverse or realistic training scenes may not address their actual shortcomings. The paper proposes a policy-conditioned generation framework that diagnoses an agent's specific weaknesses by contrasting its successful and failed rollouts, uses an LLM to propose executable environmental features (such as target visibility or blind-path ratio), validates them via success-rate gaps, and generates targeted training environments. This approach is tested across three heterogeneous agents—an RL policy, an imitation-learned model, and a black-box VLM—showing consistent gains in ProcTHOR and transfer to HM3D.
The paper makes a strong case for policy-specific environment generation and delivers a clean, end-to-end framework. Wedge consistently outperforms random and hard-mined baselines across all three agents, with particularly compelling cross-domain transfer to HM3D. The iterative re-diagnosis experiments showing that weaknesses persist or shift across rounds add empirical depth. The absolute gains for the in-domain DD-PPO agent are modest (+3.5 SR), but the direction of the results is consistent and the method is well-motivated.
The demonstration that agents solving the same ObjectNav task exhibit distinct weakness profiles is compelling and well-supported. The rollout calibration heuristic—retaining environments where $2 \leq n_{\mathrm{succ}}(x') \leq 3$—is a pragmatic filter that measurably improves adaptation over using all weakness-matched candidates. The cross-domain findings are especially strong: weakness-conditioned adaptation transfers from ProcTHOR to HM3D, whereas hard-example gains do not.
Several aspects weaken the generality and interpretability of the framework. First, weakness discovery relies on a proprietary LLM (Gemini-2.5-Pro) with limited reproducibility; while validation filters remove some bad proposals, the sensitivity of the overall loop to LLM hallucinations or prompt variations is not studied. Second, the validation thresholds—such as $\tau_\Delta = 10$ and $\tau'_\Delta = 5$ percentage points—are hand-tuned and the paper provides no ablation of their sensitivity. Third, by focusing on a single top-ranked weakness per round, Wedge ignores potential interactions between co-occurring failure modes, which the authors acknowledge as a limitation.
The evidence supports the core claim that policy-conditioned generation outperforms generic alternatives. The comparison against matched random and hard-mined baselines is fair and controlled for adaptation budget, and the Appendix Table 3 ablation shows that combining mined and generated environments ($54.5$ SR) outperforms either alone. However, the paper omits comparisons to established adaptive curricula such as PLR or regret-based environment design, and the main results lack statistical significance tests. The HM3D transfer evaluation uses small episode counts ($300$ for Gemini, $1000$ for Uni-NaVid), leaving open whether the transfer gains are robust.
While the authors release code and prompts, independent reproduction faces practical barriers. The weakness-discovery loop depends on a closed-weight LLM, so outputs may vary across runs and are not fully reproducible. The simulator primitives and prompts are clearly listed, which helps, but the stochastic LLM proposal step remains a moving part. Discovery itself is cheap—about ten LLM calls per agent—but one full adaptation round still costs roughly one GPU-day for DD-PPO, two for Uni-NaVid, and half a day plus API fees for Gemini.
Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation agents with different adaptation mechanisms, we find that policies exhibit distinct weakness profiles and benefit from different training environments. Weakness-conditioned adaptation consistently outperforms matched random and hard-example alternatives on ProcTHOR, while the resulting gains for Uni-NaVid and Gemini also extend to HM3D after ProcTHOR adaptation. Iterative re-diagnosis further shows that dominant weaknesses can persist or shift as the policy improves, providing updated guidance for subsequent environment construction. Together, these results show that policy failures are not only diagnostic signals, but actionable guidance for constructing training environments that target a policy's current weaknesses.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.