Iterative Policy Refinement through Semantic Rollout Analysis
Existing methods for structured imitation learning rely on extensive human input or static LLM knowledge that can conflict with expert demonstrations. This paper proposes a closed-loop framework that treats policy development like software debugging: an LLM writes diagnostic analysis code to test semantic latents from tabular rollouts, identifies structure-demonstration misalignment, and iteratively edits the symbolic policy structure before re-fitting continuous parameters through imitation learning. The work matters because it automates refinement of interpretable policies and shows on two control tasks that it can outperform zero-shot LLM structures while using far less subsequent RL compute.
The paper presents an intuitively appealing and well-motivated framework that leverages the interpretability of structured policies to automate debugging via LLM-generated diagnostics. The separation of symbolic structure and continuous parameters, combined with tabular rollout analysis, is a genuine conceptual advance. However, the empirical evaluation is narrow—limited to two low-dimensional tasks with a single demonstration each—and some baseline comparisons are potentially misleading. The comparison with MLES is structurally asymmetric because MLES performs fully LLM-driven evolution without any demonstrations, whereas the proposed method relies on one demonstration for parameter fitting between revisions. Additionally, the RL phase uses a finite-length variant of GSPO, an algorithm originally designed for language-model training, with limited justification for its use in continuous control. Nonetheless, the internal ablations and the concrete off-track recovery example provide credible evidence that the mechanism works as described.
The strongest part of the paper is the diagnostic mechanism itself: using semantic latents as an interface for automated, code-based inspection. The authors formalize structured policies as bipartite graphs with interpretable intermediate values, and the Car Racing off-track recovery example concretely shows how a generated test exposes a failure that parameter tuning alone cannot fix. The ablation in Figure 4 and Table 2 demonstrates that feeding generated analysis results back to the LLM is substantially more stable than providing raw trajectory tables directly. The appendices also provide unusually detailed prompts and model generation specifications, which greatly clarify the LLM interface.
The experimental scope is very limited: only Car Racing and Door Opening are evaluated, each with exactly one demonstration, and both tasks use low-dimensional semantic observations. The comparison with MLES is structurally unfair because MLES does not use demonstrations, while the proposed method uses both a demonstration and LLM refinement; presenting a 224x query reduction without controlling for this difference is misleading. Additionally, the RL phase raises concerns by adopting GSPO, an algorithm originally designed for language-model training, for continuous control with only hyperparameters listed and no justification for its suitability here.
The authors candidly note that because tests are generated from the existing policy structure, the LLM may not anticipate every failure mode or generate tests for it. This is a fundamental and unquantified ceiling on reliability. High-compute RL comparisons with the MLP baseline also lack statistical significance ($p \approx 0.136$), yet the prose highlights the highest mean without sufficient caution.
The evidence largely supports the core claim that iterative refinement improves over zero-shot LLM-generated structures and unstructured MLPs on the tested tasks. Table 1 shows a statistically significant 15% improvement in imitation learning ($p \approx 0.007$) and roughly 80% fewer RL resets to reach comparable performance. However, high-compute RL differences against the MLP are not statistically significant, and the absence of a standard differentiable architecture search baseline (e.g., DARTS) or direct comparison to KIM and GaP weakens the comparative story. The prose fairly describes the value of combining symbolic structure with continuous updates, though the MLES comparison conflates different learning paradigms.
Several factors impede independent reproduction. No code, data, or LLM interaction logs are released. The pipeline depends on GPT-5-mini with high reasoning effort, a proprietary and potentially non-stationary API, and the exact prompt chaining details are underspecified. The RL procedure uses a non-standard adaptation of GSPO with a 150-step likelihood window and group formation by starting position, whose sensitivity and suitability for continuous control are unexplored. Furthermore, with only one demonstration per task, results may be highly sensitive to demonstration quality, which is not characterized.
Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.