Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMs
Time series foundation models post strong scores on large benchmarks, yet their grasp of basic temporal logic remains unclear. This paper introduces SimpleTimeBench, a diagnostic suite of univariate and multivariate unit tests for primitives such as linear trends, periodic signals, and deterministic leading-indicator covariates. The authors show that prominent zero-shot TSFMs—including Chronos-2, Moirai, and Toto—fail on several trivial tasks, and that targeted fine-tuning on the full suite yields only marginal gains, suggesting that pre-training scale alone has not bought robust fundamental reasoning.
The paper makes a compelling case that leading time series foundation models suffer from systematic zero-shot blind spots, linking synthetic failures to real-world covariate underutilization. Its central observation that these models prioritize high-dimensional pattern matching over basic temporal reasoning is well supported by the zero-shot experiments and the Delaware River study. However, the leap from limited fine-tuning results to a fundamental architectural bottleneck is less convincing, and the NVIDIA experiment is more synthetic illustration than a natural real-world validation.
The diagnostic benchmark design is elegant and interpretable. By stripping away dataset noise and testing exact shifted-copy covariates, the authors isolate failure modes with surgical precision. The custom metrics—MAR, PVR, and MBR—go beyond standard CRPS to reveal precisely how models hedge or overshoot. The real-world Delaware River experiment is particularly compelling, showing that upstream water levels provide a 14-hour leading signal that a simple linear model exploits far better than Chronos-2.
First, the paper generalizes from three specific models to TSFMs as a class, which is a stretch given that only Chronos-2, Moirai, and Toto are evaluated. Second, the NVIDIA stock task is explicitly described as a synthetic construction where future prices are shifted backward to create a leading covariate; calling it a real-world validation is misleading. Third, the capacity-bottleneck conclusion rests on one fine-tuning protocol ($250$k steps, fixed learning rate) for a 28-distribution mixture. Marginal aggregate improvements could stem from optimization dynamics or dataset imbalance rather than an architectural ceiling, and the authors do not rule this out by varying model capacity or training length.
The covariate tests also assume an idealized setting—an exact time-shifted copy of the target—which, while useful for diagnosis, is rarely available in practice. The paper therefore shows convincingly that models fail to exploit perfect information, but does not establish how often such perfect information appears in applied forecasting.
Most quantitative claims are well supported. The GIFT-Eval multivariate-vs-univariate analysis is fair and shows that covariate benefits are nearly absent for Chronos-2 and Moirai. However, the paper repeatedly asserts that classical statistical models would solve SimpleTimeBench trivially but never actually evaluates them, weakening the comparison. Similarly, while DLinear is a useful lightweight baseline, the absence of other classical or deep baselines on SimpleTimeBench leaves open whether these failures are unique to foundation-scale pre-training or common to many global forecasters.
Reproducibility is reasonable but incomplete. The authors release the benchmark, evaluation code, and GIFT-Eval adapters. Fine-tuning hyperparameters are reported in detail: up to $250{,}000$ steps, learning rate $1e-4$, batch size $256$, and early stopping. What is missing are the exact zero-shot inference settings for the TSFMs—number of samples, temperature, specific model checkpoints, and context-length selection rules—without which independent reproduction of the zero-shot scores may vary. The real-world datasets are publicly sourced and clearly documented, which helps.
Despite the success of Time Series Foundation Models (TSFMs) on broad benchmarks, their ability to internalize basic temporal logic, especially in settings supported by exogenous covariates, remains under-examined. We introduce SimpleTimeBench, a diagnostic univariate and multivariate "unit test" suite for primitives such as monotonic trends, periodic signals and leading indicator covariates, scenarios where near-perfect forecasts should be trivial. Surprisingly, prominent multivariate TSFMs (Chronos-2, Moirai and Toto) frequently produce suboptimal zero-shot forecasts for these inputs. While fine-tuning Chronos-2 improves its behaviour on specific tasks, we show that this adaptation degrades performance on other fundamental patterns rather than enhancing its generalizable foundational capabilities. This reveals a gap between pre-training scale and basic temporal reasoning, suggesting that current TSFMs could potentially lack the inductive biases needed to capture simple predictable functions. We further demonstrate that these failures are not merely synthetic curiosities: they persist in real-world sensor forecasting, where TSFMs consistently underutilize leading indicators available in observed covariates. This inability to capture simple relationships limits the practical utility and reliability of current multivariate models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.