Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection

The core conceptual distinction between injection presence and behavioral compromise is important and increasingly relevant as internal-state defenses proliferate. The experimental design is careful: all methods share the same structured inputs, the same 1,000 template-matched benign controls, and scores are frozen before outcome labels are joined. The authors correctly cite concurrent work BASIS, which independently reaches a similar conclusion using separate existence and breach probes, strengthening the finding. The observation that Attention Tracker specifically measures attention diversion toward the untrusted span at the final query position gives a mechanistic interpretation for its stronger successful-attack signal.

The strongest limitation is the narrow model coverage: only two open-weight models of similar size (35B and 26B parameters). The authors state they could not evaluate larger or more varied models due to limited compute, so generalization across model families and scales is unverified. The structured prompt template makes the trusted/untrusted boundary unusually explicit, which may inflate detection scores relative to realistic agent trajectories where boundaries are ambiguous or absent. The attack corpus is outcome-conditioned: unsuccessful baseline attacks were replaced with adaptive variants designed to increase behavioral success, meaning the successful attacks are a stress-test selection rather than a natural sample. Furthermore, some methods are substantial adaptations of their original forms—PIShield receives a combined prompt rather than separately serialized data, Role Confusion uses an exploratory Systemness adaptation, and the Dong probe is trained on AgentDojo trajectories but evaluated on structured prompts—so the readouts may not reflect the original methods' intended semantics. The paper also lacks causal interventions or detector-adaptive attacks, so it cannot confirm whether the observed signals are merely correlated with success or mechanistically necessary for it.

The evidence within the benchmark supports the stratified claims, and the 95% bootstrap intervals show that the direction of the gaps is not due to noise. However, the evidence is correlational: outcome-stratified AUROC shows association, not causation. Comparisons to related work are generally fair, with the authors clearly distinguishing their setup from the original methods and noting that they compare adaptations rather than exact reproductions. The citation of BASIS as independent concurrent support is accurate based on the fetched abstract. A gap is the absence of comparison to strong input-text baselines (e.g., PromptGuard or simple length/syntax heuristics), which would help establish whether the internal signals add value beyond surface features that the authors warn about in citing Fomin and Li et al.

Reproduction would be partially but not fully feasible from the paper alone. The authors share evaluation data at an anonymous repository, specify exact model checkpoints and revisions, document chat-template rendering, and report hyperparameters such as Dong's 5 epochs, batch size 64, and learning rate $10^{-4}$. Layer and head selections are reported. However, no code repository is provided, and the five methods would need to be reimplemented from the descriptions in Appendix A. Several implementation details that could matter—such as the exact J-Lens probe feature aggregation, the synthetic ignore sentences used for Attention Tracker head selection, and the specific AgentDojo splits used to fit the Dong probe—are described but not released, which is a significant barrier to exact reproduction. Re-running the full pipeline also requires white-box access and substantial compute for two large models.