EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence
LLM agents can execute prohibited actions when tool responses omit safety-relevant facts, even if every returned statement is true and the user request is benign. EpiSEM-Bench introduces 300 matched unsafe/safe task pairs across six domains with controlled evidence suppression and an optional audit-based recovery tool. The work shows that robustness to missing evidence and useful recovery are distinct capabilities that single-score benchmarks conceal, making joint evaluation essential for safe agent deployment.
The paper offers a conceptually clean and rigorously structured benchmark. Its matched-world design and six-metric profile meaningfully advance agent safety evaluation by separating omission resistance from useful recovery. However, the idealized deterministic recovery tool and synthetic executable environments temper the ecological validity of the findings.
The benchmark's construction is methodologically strong. Pairing prohibited and permissible twins while holding requests and tools constant isolates permission-sensitive behavior, and the evidence-atom abstraction enables precise, persistent suppression without injecting falsehoods. The six core metrics are well-motivated and jointly prevent gaming by indiscriminate refusal or execution.
The ten-model evaluation is thorough and reveals distinct capability profiles. As noted, "High completion can coexist with prohibited execution, strong restraint can coexist with refusal after permission is established, and recovery access can be beneficial without being used consistently." This demonstrates that the multi-metric approach successfully surfaces failure modes hidden by single scores.
The recovery mechanism is unrealistically strong: a single valid query returns all omitted facts deterministically, which does not reflect costly or partial real-world verification. The benchmark also remains small, with 300 task pairs, and the 12-step action budget may artificially truncate complex rollouts. Additionally, the Safety Boundary Margin can be gamed by agents that refuse every action; the authors acknowledge that "An agent that always refuses can attain a maximal SBM without completing any permissible task," requiring readers to rely on joint interpretation rather than any standalone score.
Other limitations include the use of synthetic environments rather than live systems, and the evaluation of fast model variants whose behavior may not generalize to larger checkpoints in the same families.
The reported evidence supports the central thesis that models exhibit differing and sometimes orthogonal strengths in omission resistance, evidence acquisition, and permission-sensitive action. The matched-world comparisons are internally valid, and the behavioral partitions provide granular insight into failure modes. Comparisons to related work are fair: the authors correctly distinguish their intervention from adversarial injection and underspecified instructions. That said, the paper does not run prior benchmarks such as AbsenceBench or AgentNoiseBench on the same model suite, leaving open how much of the observed variation is benchmark-specific.
Specific trajectories strengthen the argument. For example, "Obtaining the evidence does not complete the decision process. MiMo-V2.5-Pro makes valid recovery calls substantially more often than it reaches strict unsafe recovery, illustrating failures after acquisition as well as failures to query." This kind of behavioral record backs the claim that evidence recovery and appropriate action are separable skills.
Reproducibility is partially addressed but not fully guaranteed. The authors document temperature-zero decoding, a 12-step budget, hash-based omission ordering, and checksums for data and software. They state that "The retained task snapshot, experiment metadata, and unrounded analysis records support reproduction without reconstructing values from rounded table entries." Yet there is no explicit statement that code, task snapshots, or rollout logs will be publicly released, and proprietary serving configurations for the evaluated models may drift.
Key procedural details are specified: "Temperature zero specifies the requested decoding setting. For omission combinations of the same size, the hash-based ordering is reproducible from the recorded ordering seed, scenario identifier, and masked atom identifiers." Still, an independent group would need both the executable environments and the exact model endpoints to reproduce the numerical profiles exactly.
Tool-using language model agents rely on environmental facts to decide whether an action is permitted. When a tool response omits a safety-relevant fact, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true. We introduce \bench{}, an executable benchmark for evaluating unsafe execution, safety boundaries, and evidence recovery in LLM agents under missing evidence. It contains 300 matched unsafe/safe task pairs across six domains. By systematically suppressing critical facts, the benchmark measures unsafe execution under critical evidence omission and determines the minimum observed failure cardinality, including failures with full evidence and cases with no observed failure after exhaustive testing. An independent audit tool tests whether agents seek missing facts and then withhold prohibited actions while completing permissible tasks. The benchmark reveals substantial variation across the ten models in omission robustness, evidence acquisition, and appropriate action. Resistance to missing evidence does not necessarily translate into useful recovery. Recovery access reduces unsafe execution for all ten models, but agents differ in whether they actively obtain the missing facts and use them to act appropriately in both worlds.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.