Model What Matters: Decision-Guided Selective World Modeling for Language Agents
Reinforcement learning for language agents suffers from sparse trajectory-level rewards that do not directly supervise environment dynamics. DGS-WM addresses this by introducing Decision-Guided Selective World Modeling: instead of forcing the policy to predict entire environment observations as auxiliary targets, it selects only observation units that support the subsequent policy response and discounts already-mastered targets via confidence-adaptive weighting. This selective co-training yields consistent gains over full-observation counterparts across QA and web-shopping benchmarks.
The paper proposes a clean and intuitive extension to policy-world modeling co-training that is well-motivated by the observation that full observations contain decision-irrelevant noise. Empirically, DGS-WM consistently outperforms both RL-only and full-observation ECHO baselines across model scales and RL algorithms. The gains are generally modest but reliable. The method is somewhat incremental—it builds directly on the ECHO/PaW co-training paradigm—and adds computational overhead due to per-unit scoring, yet the core insight and ablation support its validity.
The central thesis—that environmental observations do not possess equal supervision value—is convincing and well-articulated. The authors validate it with a careful ablation showing that removing either Decision-Guided Observation Selection or Confidence-Adaptive Weighting degrades performance. The selection mechanism itself, using relative filtering of response support scores within each observation, is pragmatically designed to adapt to varying score scales across interactions. The cross-model and cross-algorithm experiments further strengthen the claim that selective world modeling generalizes beyond a single training setup.
First, the response support score remains a heuristic correlation rather than a causal measure of decision relevance; the authors acknowledge this explicitly but do not offer any mitigations. Second, the method relies on hand-engineered splitting rules tailored to each environment (sentence boundaries for QA, UI blocks for WebShop), limiting applicability to unstructured feedback. Third, while the paper emphasizes stability, the absolute improvements are small and the experiments lack error bars or statistical significance tests, making it hard to judge whether the gains are robust or simply noise.
The evidence supports the main claim that selective supervision outperforms full-observation supervision, with direct ablations and cross-algorithm comparisons. However, the paper under-compares to PaW, another recent co-training method that also addresses noisy observations via action-entropy selection and clipped MAE loss; DGS-WM is framed against ECHO rather than against the broader state of the art. Moreover, metrics are reported as single-point averages without standard deviations or confidence intervals, weakening the empirical argument for consistent improvements.
Appendix A provides detailed hyperparameters, including LoRA rank 32, learning rate $5 \times 10^{-6}$, batch size 144, and DGS-WM coefficients $\kappa=0.5$ and $\gamma=1.0$. The authors describe the reward functions, task setups, and observation splitting heuristics. However, no code repository or model checkpoints are released, which would block independent reproduction. In addition, candidate scoring requires an extra forward pass per observation unit, yet the authors do not quantify this computational overhead or its interaction with large-batch training.
Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.