Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability
Time series anomaly detection benchmarks traditionally optimize for detection accuracy while neglecting explainability and interpretability. This paper introduces SHAD, a real-world benchmark comprising 215 high-dimensional multivariate time series collected from distributed cloud storage systems, complete with per-dimension labels and rich semantic annotations. The authors evaluate 18 detection algorithms, probe spatial attribution for explainability, and test frozen LLMs for interpretability, demonstrating that accurate detection does not guarantee diagnostic utility and that zero-shot semantic interpretation remains largely unresolved.
The paper makes a timely and practical contribution by providing the first benchmark that explicitly unifies detection, explainability, and interpretability evaluation for time series anomaly detection. SHAD's real-world provenance, expert-curated per-dimension annotations, and structured XML metadata represent genuine advances over prior resources. However, the experimental scope is uneven: the detection study is thorough, but explainability is only probed through naive dimension contribution rather than dedicated post-hoc methods, and interpretability is tested with a restrictive single-call zero-shot LLM protocol that yields near-random performance. The claim that SHAD enables complete pipeline evaluation is therefore partially realized rather than fully demonstrated.
The dataset construction is the clearest strength. The authors provide 215 series with 171 dimensions each, expert-verified per-dimension binary labels, and XML annotations that link anomalies to affected components, severity levels, and textual descriptions. The conceptual distinction between explainability as spatial attribution and interpretability as semantic understanding is crisply formalized and consistently used. The detection experiments are well-executed, covering 18 methods across semi-supervised and unsupervised paradigms with the robust VUS-PR metric, and the authors report results honestly, including the nuanced finding that unsupervised KMAD rivals top semi-supervised methods on system-wide events while struggling on subtle failures.
The explainability evaluation is surprisingly thin. The authors test only whether native anomaly-score contributions of three top detectors can rank anomalous dimensions, finding near-random NDCG, yet they do not benchmark established post-hoc explainers such as SHAP or LIME that they extensively discuss in related work. This conflates 'detectors are not natively explainable' with 'explainability on SHAD is hard,' which are different claims. The LLM interpretation experiments are similarly limited: a single zero-shot call with frozen models, even when primed with ground-truth anomaly presence or component labels, yields only marginal gains, but the paper over-interprets this as evidence that interpretability requires domain-specific solutions rather than improved prompting, tool use, or fine-tuning. Finally, all anomalies are injected in a controlled experimental framework on a single system topology, which may limit diversity relative to organically occurring production failures.
The evidence supports the claim that SHAD is a challenging detection benchmark: semi-supervised methods generally outperform unsupervised ones, yet no method dominates all anomaly families, and the clustering-based KMAD matches top deep learning methods on system-wide network events while faltering on subtle single-disk failures. The comparison to prior benchmarks in Table 1 is fair at a coarse level, though the binary checklist format oversimplifies nuances such as Exathlon's root-cause and effect-interval labels. The LLM results substantiate the intrinsic difficulty of interpreting 171-dimensional time series under zero-shot constraints, but they do not constitute strong evidence that domain-specific architectures are necessary, because the protocol excludes chain-of-thought, retrieval augmentation, or parameter-efficient tuning strategies known to improve reasoning.
The authors state that their public repository hosts the dataset and documentation, and the detection experiments reuse hyperparameters from the TSB-AD benchmark, which aids reproducibility. The evaluation metric VUS-PR with a 25-point buffer is precisely specified. Nevertheless, several details needed for independent reproduction are missing or relegated to appendices: the exact wording and temperature settings for the LLM interpretation questionnaire, the specific model checkpoint versions and API dates, and the full code for generating multi-panel images and Catch22 feature vectors. The split of the 83 nominal series used for semi-supervised training is mentioned but not formally characterized, leaving uncertainty about how training-set composition affects variance in the deep-learning results.
Time Series Anomaly Detection has received increasing attention, driven by the growing availability of complex time series data. This surge has led to the development of numerous detection methods, as well as a variety of benchmarks aimed at thoroughly evaluating their performance. However, most existing detectors remain largely agnostic to domain context, overlooking explainability and interpretability. One of the main reasons for this gap is that current benchmarks primarily focus on detection accuracy, and only few of them evaluate spatial explainability. Moreover, no benchmark currently provides sufficiently rich semantic annotations to support the generation of human-understandable interpretations of anomalies. To address these limitations, we introduce SHAD (Scality High-dimensional Anomaly Detection benchmark), a fully annotated benchmark composed of 215 multivariate, high-dimensional time series collected from real-world distributed cloud storage systems operated by Scality. The proposed dataset includes rich contextual information, covering three families of anomalies with varying degrees of severity. As further contribution, we provide a foundation for future work by evaluating baseline methods for Detection, Explainability, and Interpretability, covering all stages of a TSAD pipeline. For Detection, we benchmark a wide range of existing anomaly detectors, testing their effectiveness on the proposed real-world dataset. Then, we consider explainability by evaluating whether measuring the contribution of each dimension in the generated anomaly score can provide accurate anomaly attributions. Finally, for interpretability, we investigate the effectiveness of frozen LLM baselines in localizing and interpreting anomalies.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.