Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
Policy-Verified Self-Training asks whether a runtime policy gate's discarded verdicts can be recycled as training signals for the same agent. The paper proposes PVST: an agent rolls out trajectories, a deterministic gate compiled from written policy judges compliance, and the verdicts become acceptance filters and preference labels. On AgentDojo prompt-injection benchmarks, post-training Qwen3-8B with these verifier-derived rewards cuts violating trajectories from 37.2% to 10.3% and successful attacks to 0.5% without any stronger teacher model, human preference labels, or attack-specific supervision. The result suggests that compiled policy checkers can serve not only as runtime constraints but as a scalable, auditable source of self-training supervision.
The paper presents a creative and internally rigorous demonstration that deterministic policy verdicts can supervise post-training for tool-using agents, formalized by decomposing the trajectory reward as $r(\tau)=u(\tau)-\lambda v(\tau)$ and consuming only the sign of $v(\tau)$. The strongest evidence is a monotone ablation across negative-label sources and replication on Qwen3-4B. However, the work is held back by a severe utility collapse on travel tasks driven by excessive avoidance of guarded tools, by a teacher gate that itself refuses legitimate actions, and by the absence of empirical head-to-heads against prior prompt-injection defenses such as SecAlign or the instruction hierarchy. The conceptual framing—turning a runtime guardrail into a teacher—is valuable, but the practical promise depends on fixing the verifier and avoiding inaction traps.
The control experiments are exemplary: the share of violating negatives monotonically predicts final compliance ($0\% \to 31.9\%$, $55\% \to 20.2\%$, $100\% \to 10.3\%$), isolating the gate signal from preference optimization alone. The method also transfers measurably to an unseen attack template and to a held-out suite with its own compiled rule set, and a paired diagnostic shows that gate-trained agents drop forbidden calls to zero while preserving some permitted ones. These results are reinforced by a companion compiler (NOMOS) that deterministically translates policy text into executable gates without LLM calls at runtime.
A major flaw is that the trained agent learns avoidance, not conditional compliance: on held-out travel tasks, allowed create_calendar_event calls collapse from 16.7 to 0.0 per 100 trajectories and write-task utility drops to $0.0\%$ (seed 0), because the tool is over-penalized. The teacher gate is itself buggy—it refuses legitimate calendar events and banking payments—so tasks with refused legitimate writes are silently excluded from training, biasing the model toward inaction.
Only one model family is tested, and most secondary arms use a single seed, leaving key comparisons like DPO-oracle and cut-position ablations underpowered.
Within the paper's own design, the evidence cleanly supports the claim that gate-selected negatives outperform attack-oracle and violation-free negatives. Yet the paper does not empirically compare PVST against contemporary defenses such as SecAlign or the instruction hierarchy, so the purported advantage of needing no attack labels is a feature description, not a demonstrated empirical advantage over alternatives. The suite-transfer claim is also appropriately scoped—the authors admit that $16.7\%$ residual violations on workspace amount to measurable transfer, not general policy semantics—but the abstract and contributions could be read as implying broader robustness than the shared-tools/predicates transfer experiment justifies. Concurrent work AgentLTL indeed uses FO-LTL constraints from gold traces rather than compiled policy text, confirming the paper's characterization.
The protocol is documented with unusual precision—every evaluation number is traced to specific scripts, and the gate replay is deterministic—but full reproduction is blocked because LoRA weights, training rollouts, and $\tau^2$ evaluation trajectories are omitted for size, and the remaining artifacts are available only from the author on request rather than in a persistent repository. Hyperparameters are reported (LoRA rank 16, SFT lr $10^{-4}$, DPO $\beta=0.1$), but exact reproduction depends on replicating the authors' custom NOMOS compilation, which includes known departures from the written policy clauses that directly affect the reward signal.
Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward. We ask whether these decisions can instead become verifiable policy rewards for training the agents they guard. We introduce Policy-Verified Self-Training (PVST), a post-training paradigm in which an agent generates its own interaction trajectories and an executable policy verifier, here a deterministic gate compiled from a written policy, evaluates whether each action complies. Compliant trajectories are reinforced, and violations become negatives for preference optimization; beyond the gate and the benchmark's task-success signal, PVST needs no stronger teacher model, no human preference labels and no attack-specific supervision, although rollouts are collected in the benchmark's attacked environment. On held-out AgentDojo tasks under prompt injection, policy-verified preference training of Qwen3-8B reduces violating trajectories from 37.2% to 10.3 $\pm$ 0.9% (three seeds) and successful attacks from 12.4% to 0.5% at equal or higher pooled task utility, although utility falls on travel, compared with 22.0% when negatives are selected by an attack oracle and 31.9 $\pm$ 0.9% with negatives that contain no violation. The learned behavior holds under an attack template never used in training (37.7% to 11.0%) and on a suite excluded from training (34.8% to 16.7%), and the retained verifier's intervention rate falls from 39.2% to 8.8% (seed 0). Our analysis identifies two conditions under which the verdict is most informative: success must not already imply compliance, and guarded actions must appear in both permitted and forbidden contexts.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.