From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence
LLM entity extraction from full-length documents is widely deployed but its failures are poorly diagnosed. This paper runs a preregistered audit of two low-cost LLMs and a flagship control on 315 real-world cyber-threat-intelligence reports, finding that most false positives are background mentions rather than hallucinations, and that confidence signals can rank outputs for human review but cannot verify truth against relation-level counterfactuals. The results set a concrete deployment boundary for any high-stakes pipeline that triages extraction by confidence.
The paper is a careful, preregistered measurement study that successfully reframes LLM extraction failures from "hallucination" to "context misattribution" and draws a crisp deployment boundary: confidence is useful for triage, not truth. The 630 model–report runs and relation-level counterfactuals give strong internal support, but the flagship-tier control is incomplete on counterfactual and verification batches, and the reliance on LLM annotators for the three-tier study ($\kappa=0.79$) introduces unquantified bias relative to expert humans. The simulated review budgets are also not validated against actual human effort.
The preregistered protocol and transparent failure-mode decomposition are the paper's strongest elements. The full-scope mention census and three-tier annotation on 360 stratified candidates back-project cleanly to the false-positive mass, showing that 99.7–99.8% of false positives have an explicit textual source and that relation-level misbinding is a vanishingly small residual. The counterfactual audit is devastating and well-executed: even an unrelated background sentence binds a fake vulnerability at high confidence in a quarter of cases.
The flagship control (GPT-6.1-Sol) is absent from the counterfactual, verification, and generalization batches due to cost, so the claim that no signal verifies truth is scoped to the flash tier plus one third-party verifier. The three-tier annotation uses two LLM annotators rather than humans; while the authors disclose this fully and apply a conservative adjudication rule, the resulting failure-mode fractions could inherit LLM-specific blind spots. Repair experiments are uneven: "CoT flips sign on attack techniques" and gains do not transfer across models. Finally, the simulated review budgets are not backed by measured human effort, leaving the operational recommendation speculative.
The evidence inside the paper largely supports the headline claims. The reuse of Mezzi et al.'s public corpus and exact campaign-level split ensures direct comparability with the critical audit, and the authors scrupulously avoid placing differently-scoped numbers in one table. However, the comparison to the critical audit's higher CVE precision ($0.67$–$0.72$) is explained by scope mismatch (primary-only vs. full mention census), which weakens the ability to claim strict improvement. The newer-corpus test supports ranking-gain transfer but relies on annotation-derived gold with lower inter-annotator agreement ($\kappa=0.53$ borderline-excluded), so extrapolation of absolute precision values should be treated cautiously.
Reproducibility is a standout strength. The authors preregistered acceptance bars, freeze all prompts and batch outputs, record gateway and model identity per call, and commit to releasing all artifacts. Yet practical reproduction requires paid API access—the flagship control alone cost approximately 28.50 USD for 822 calls—and vendor content policies blocked 15 reports during ablations, which is a disclosed production constraint. Because serving gateways strip log-probabilities, the calibration analysis is limited to zero-logit proxies (verbalized confidence and self-consistency frequency), which is documented but narrows the signal space.
Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures actually consist of, and whether confidence signals can turn unreliable output into usable output. We run a preregistered, fully frozen evaluation of two new-generation low-cost LLMs (DeepSeek-V4-Flash, GLM-5.3-Flash) with a flagship-tier control (GPT-6.1-Sol) on a public corpus of 354 incident campaigns (315 unique reports, 630 model-report evaluations), scoring extraction quality against gold, confidence calibration, and robustness to single-sentence perturbations. Three findings emerge. (1) Unreliability persists across tiers: micro precision is 0.03-0.34 and verbalized calibration error (ECE) 0.53-0.94 across both splits; the gap does not close at the flagship tier. (2) The dominant failure is not hallucination: a full mention census plus a three-tier annotation of 360 sampled candidates (two independent LLM annotators from vendors outside the evaluated set, Cohen's kappa = 0.79 on the preregistered acceptance line, fully disclosed) back-projects to 65.6% background mentions and 19.9% gold-scope gaps, with only 1.2% unsupported candidates. (3) Confidence ranks but does not verify: zero-cost self-consistency-frequency tiering passes a preregistered top-30% precision-gain bar in all six test-split model-type cells (4/6 on train), the zero-cost text-frequency x SC frequency combination wins 3, ties 2, loses 1 across cells, yet relation-level counterfactuals on all 45 targets show that a single misattribution sentence flips high-confidence judgments in 95.6-100% of cases, and even a background sentence explicitly marked as unrelated still binds a fake vulnerability in up to 26.7% of them. We conclude with a deployment boundary: confidence tiering may set review priorities, never truth.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.