AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents
LLM agents with persistent external memory often leave derived artifacts—such as summaries—intact after a user deletes a source record, creating a gap between storage-level deletion and behavioral privacy. AfterDeleteBench introduces a paired behavioral audit that compares an exposed-then-deleted agent against a never-exposed twin to attribute post-deletion secret recovery to the original exposure. The paper shows that exact reconstruction can persist even when no verbatim copy of the secret survives, and that the response interface itself is part of the measurement system, making the benchmark a useful narrow probe into a broader agent-memory privacy problem.
The paper offers a methodologically careful benchmark that exposes a real and under-studied failure mode: fixed-weight LLM agents can behaviorally recover exact secrets from derived persistent state after source-record deletion. Its paired-world design, lifecycle decomposition, and interface diagnostics are genuine strengths. However, the scope is tightly bounded to synthetic exact-string targets, externally controlled extractive memory, and complete provenance metadata. Within those narrow guardrails the results are convincing, but they do not yet establish how production agents with autonomous, learned, or compressed memory would behave.
The paired clean/forget world design provides clean attribution of recovery to a specific prior exposure rather than model knowledge or chance. The finding that exact reconstruction persists without any surviving verbatim copy—relying solely on separately stored components under a public composition rule—is striking and well supported. Most importantly, the paper’s detailed analysis of the response contract shows that parser mismatches can hide real disclosures and even reverse the apparent effect of a write intervention, a subtle but critical point for the evaluation literature.
The evaluation is limited to synthetic exact-string secrets in an externally controlled, deterministic memory harness where the model never autonomously creates summaries or chooses retrieval rules. This makes external validity uncertain for natural language secrets or learned semantic memory, a limitation the authors acknowledge but do not overcome. The provenance-aware closure intervention is an idealized structural upper bound that assumes complete lineage metadata; it is not a practical deletion mechanism for realistic memory systems. Additionally, the learned adaptive auditors (PPO, contextual bandit, prompted LLM) fail to outperform a single direct query or a fixed expert script in this constrained task, raising questions about the benchmark’s value for driving more sophisticated auditing research.
The quantitative evidence strongly supports the claims within the benchmark’s narrow scope: $\Delta \mathrm{Rec} = 0.680$ ($[0.615, 0.745]$) for Qwen under exact deletion, and compositional reconstruction remains high across Qwen, Mistral, and Phi-3 when components survive under a public rule. The paper appropriately compares itself to concurrent work on agent unlearning and provenance, accurately noting that Wang et al. (2026) jointly suppress parameters and memory with dependency-based invalidation, that Raeesi and Roed (2026) hold a fixed model and vary the external database, and that Yao et al. (2026) pursue provenance-guided replay across plaintext, summaries, and cache.
Reproducibility artifacts are unusually thorough for a fixed-weight agent benchmark. The authors document pinned model revisions, exact decoding hyperparameters, seeds, JSON contracts, and prompt templates in Appendix I, and they retain raw prompts, replies, and matched-state hashes for validation. A transparent correction history is provided for the prompted-auditor implementation bug, including which cells were rerun and how validation was performed. The main current blocker is that the code repository is not yet public—the authors state it will be released only after acceptance—so independent reproduction remains impossible until then.
Many LLM agents maintain external persistent memory and derive additional state from past interactions, such as summaries. Deleting an original memory record therefore need not remove information carried by its descendants. We introduce AfterDeleteBench, a controlled behavioral audit for this failure mode in fixed-weight LLM agents. For each synthetic secret, we compare an exposed-then-deleted agent with an otherwise matched agent that was never exposed to the secret, allowing post-deletion recovery to be attributed to the original exposure. In our primary setting, a single query recovers the deleted secret from Qwen2.5-7B-Instruct in roughly two thirds of cases, while matched never-exposed controls have zero recovery. Crucially, recovery does not require a surviving verbatim copy: when the complete secret string is absent and only separately stored components remain under a public composition rule, exact reconstruction succeeds in 62-99% of cases across Qwen, Mistral, and Phi-3. We also show that the response interface is part of the measurement system: target-bearing replies can be scored as failures when the requested format and parser are misaligned. Finally, we separate reconstruction from write-back and fresh-session recall. Provenance-aware closure eliminates observed recovery in the tested positive-baseline settings, whereas an exact-literal write filter blocks reinsertion but cannot erase information already present. Together, these results show that source deletion should be evaluated behaviorally at the level of the agent and its memory system, not only as a storage operation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.