AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
LLM agents face an expanded attack surface because malicious content can hide in user queries, tool descriptions, and environment observations. AgentSafeEvo casts agent safety as an attacker-defender co-evolutionary game over multi-turn tool interactions, where the attacker constructs full adversarial tasks and the defender learns to execute tasks safely. A novel learnability-guided task weighting mechanism uses within-task reward variance to shape updates for both players. Experiments on Qwen3-8B show strong safety gains and broad preservation of general capabilities.
The paper presents a well-motivated extension of adversarial co-evolution from language-level safety to the full agent task-execution loop. Safety gains over Static-AT are substantial, and the method preserves or improves general capability on ALFWorld and ToolHop. However, the entire training signal depends on an unvalidated Qwen3.8-27B judge for binary safety and utility labels, making judge bias a critical unaddressed risk. Additionally, the BFCL-V3 function-calling score declines slightly, suggesting that not all agent capabilities are fully preserved.
The strongest aspect is the shift from static or language-only adversarial training to co-evolving complete adversarial agent tasks spanning six attack categories. The learnability-guided task weighting is a concrete fix to a real problem with zero-sum GRPO: tasks on which the defender always fails yield zero variance and thus no learning signal. The ablation supports this, showing that removing weighting increases ASR on Agent-SafetyBench from 17.84% to 22.91% and degrades general capability on ALFWorld and ToolHop.
Several issues limit confidence in the conclusions. First, the entire training signal depends on a single reward model, Qwen3.8-27B, for binary safety and utility judgments; no validation against human annotators or alternative judges is reported, and any systematic bias in this judge would directly distort both attacker and defender optimization. Meanwhile, BFCL-V3 accuracy declines marginally, indicating potential fragility in core function-calling precision.
Second, while ALFWorld and ToolHop improve over Static-AT, the latter also gains on ToolHop, suggesting that the capability lift may partly stem from exposure to diverse synthetic tasks rather than co-evolution per se. The paper also omits direct quantitative comparison with ARLAS on shared benchmarks and does not evaluate against other recent training-based agent alignment baselines, leaving absolute safety margins relative to the broader state of the art unclear.
The evidence supports the core safety claim: the ablation shows that Static-AT underperforms on all safety and general-capability metrics, and the frozen-attacker analysis confirms that online task generation alone is insufficient. The qualitative comparison to ARLAS is accurate---ARLAS indeed focuses on indirect prompt injection inserted into observations of given web tasks, whereas AgentSafeEvo constructs adversarial tasks across six attack types. However, the paper lacks a direct quantitative head-to-head with ARLAS on AgentDojo or other shared benchmarks, and it does not compare against other recent training-based defenses, making it difficult to isolate the impact of co-evolution from dataset scale and diversity.
Hyperparameters, prompt templates, and the training algorithm are fully documented in the appendices, which aids reproduction. However, the red-team data-synthesis pipeline is explicitly withheld to mitigate misuse, meaning the exact 20K-task SFT and 12K-seed RL datasets cannot be reconstructed. The paper does not mention public code release, and the training stack uses internal frameworks (PettingLLMs, verl) without public links. Rollouts are performed with vLLM and evaluated partly on proprietary models such as GPT-5, which may not be accessible to other researchers. Reproducing the full pipeline would therefore require rebuilding the training data from scratch, a significant barrier.
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.