DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents
Long-horizon language agents frequently re-issue identical tool calls and receive redundant observations, a failure mode the authors call a loop. The paper shows that compressing the agent's history to fit a context window dramatically increases these loops, which in turn strongly predict task failure. DeLoop solves this by mining preference pairs directly from the agent's own rollouts—treating the looping action as a rejected example and the first later non-looping action as a chosen one—and training with direct preference optimization without any external rewards, gold answers, or LLM judges.
The paper offers a clever and practically useful source of self-supervision: the agent's own repeated actions. As shown, "DeLoop cuts the loop rate 9.4×, from 57.3% to 6.1%" (Section 5.2), and "Pair construction therefore requires no task reward, gold label, or LLM judge" (Section 4.1). The core claim is well supported by scaling experiments and cross-domain tests, though the gains are uneven—the 8B model sees only a +0.8 point success lift from base self-supervised pairs, and larger improvements rely on cross-domain data, blurring the causal picture.
The empirical characterization of loops is strong and well-measured. The definition is crisp: "A loop occurs when the agent repeats a previous tool call with identical arguments and receives the same observation" (Section 3.1). The paper rigorously shows compression exacerbates looping: "Summarization at 4,096 tokens raises the loop rate to 57.3%" (Section 3.3). The self-mined preference pairs are elegant, and the cross-model scaling (Qwen3-8B to 32B) and cross-domain generalization to ALFWorld and AppWorld substantially strengthen the claim that the signal transfers.
The loop detector is narrow: it only flags exact repeats with identical observations, potentially missing semantic loops. More importantly, "Qwen3-8B still loops in 29.0% of runs with the full history" (Section 3.3), suggesting a deeper policy flaw than compression alone. The 8B success gain from DeLoop itself is slim (+0.8), and Appendix F.1 notes that "Training thus suppresses the looping action rather than teaching one memorized recovery" (Appendix F.1), implying the model learns to avoid a specific bad action rather than robustly using its memory. The DeLoop-Mix improvement likely reflects broadened task exposure as much as loop-breaking.
The evidence cleanly supports the claim that loops are harmful and that DeLoop reduces them. Comparisons to related work are fair but incomplete. The authors correctly distinguish their contribution from context-management methods, showing that "Better context management does not close this gap: DeLoop-Mix lowers the loop rate under every memory policy we test" (Section 3.3). However, they omit stronger inference-time baselines such as per-step reflective prompting or hard-coded prompt penalties beyond the duplicate blocker. Section 5.2 notes that "the strongest labeled baseline, gold-label DPO, reaches 22.3%", while DeLoop reaches 6.1%, supporting the claim that self-mined pairs outperform outcome-supervised methods for loop reduction.
The paper provides substantial detail: the DPO hyperparameters ($\beta=0.05$, LoRA $r=32$) and greedy decoding are specified, and training set sizes are reported. However, the authors state that "Code and data will be publicly available" (Abstract), meaning reproduction is currently blocked pending release. The summarization prompt is only paraphrased, the exact task splits are not yet released, and some baselines rely on harness implementations that are not fully open-sourced. The structural loop detector itself is simple enough to reimplement, but end-to-end reproduction awaits the promised artifacts. As noted, "For Qwen3-8B, we train for one epoch with LoRA DPO ($\beta=0.05$) on 8,000 self-mined CodeGym pairs" (Section 4.2).
Language-model agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini agents and become far more common once the agent's history is compressed to fit its context window. To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges. On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.