Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting
This paper tackles non-stationarity in long-term time series forecasting by asking how positional encoding shapes temporal dependencies in Transformer attention. The core idea is to reinterpret self-attention as a Bayesian posterior in which positional encoding acts as latent temporal evidence, and to replace static positional biases with an input-conditioned mixture-of-experts router that adaptively combines local and periodic kernels. The results matter because they offer a principled, lightweight, plug-and-play module that consistently improves forecasting across four diverse Transformer backbones.
The paper presents a compelling plug-and-play mechanism for adaptive positional encoding in time-series Transformers, supported by an elegant Bayesian reinterpretation and consistent empirical gains across multiple backbones and benchmarks. However, the central theoretical suboptimality result is essentially tautological under its own regime-conditioning assumption, and the main experiments omit variance estimates and explicit compute-cost comparisons that would strengthen the empirical claims.
BTA's strongest asset is practical generality. The authors integrate it into four architecturally distinct backbones and report averaged MSE reductions of roughly 6-10%, demonstrating that the module is truly plug-and-play. The case study is particularly persuasive: the MoE router assigns 0.75 weight to periodic kernels when the input window shows strong weekly seasonality, and 0.66 weight to local RBF kernels when the input is smoother, intuitively matching the temporal structure and reducing MSE by about 37% in both scenarios.
The theoretical argument is more interpretive than generative. Proposition 1 constructs latent temporal-evidence variables so that attention weights equal a Bayesian posterior, which amounts to a reparameterization rather than a derived model; the 'candidate-specific contribution' assumption in Eq. (21) is doing the heavy lifting. Consequently, Corollary 3's bound -- $\sup_{\phi\in\Phi_{\mathrm{static}}}\mathcal{L}_{\mathrm{reg}}(\phi)\leq\sum_{r=1}^{J}\pi_{r}\sup_{\phi_{r}\in\Phi}\mathcal{L}_{r}(\phi_{r})$ in Eq. (5) -- follows immediately from Jensen's inequality once regimes are postulated, and the paper does not establish conditions under which the inequality is strict.\n\nOn the empirical side, Table 1 reports single-run averages without standard errors, and Appendix Table 11 notes 'three-seed experiments' but keeps them out of the main results. Hyperparameter sensitivity is also high and dataset-dependent: for instance, ETTh1 prefers Top-$k=1$ while ETTm1 and Solar prefer $k=3$, and the best router hidden dimension varies from 96 to 192 across datasets, raising questions about robust transfer without a validation split.
The evidence supports the claim that adaptive temporal evidence improves forecasting, but the comparisons would be stronger with confidence intervals and a FLOPs-vs-accuracy tradeoff in the main tables. The ablation study is well-structured, isolating local kernels, periodic kernels, static weighting, TE-profile routing, and TE+Fourier routing; it shows monotonic improvement, with the Fourier component delivering the largest marginal gain of 5.65% MSE and 6.55% MAE. However, the paper does not explain why a discrete Fourier transform of lag-aggregated attention scores reliably guides kernel selection across domains, nor does it statistically test whether the backbone improvements are significant.
The authors specify PyTorch and an NVIDIA A100 GPU but provide no code repository URL in the manuscript. Full implementation details and hyperparameters are deferred to Appendix J, and the main-text reporting does not include random seeds or standard deviations, making it difficult to assess training stability and statistical significance. Exact search spaces for hyperparameters are not described in the main text, and the apparent reliance on per-dataset validation choices for router capacity and Top-$k$ selection raises a reproducibility barrier for anyone attempting to apply BTA to new datasets.
Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remains challenging. In this paper, we interpret self-attention as a Bayesian posterior over attended positions, combining a content-based prior induced by scaled dot-product similarity with a temporal-evidence likelihood induced by positional encoding. This formulation provides a probabilistic interpretation of how positional encoding shapes the temporal dependencies captured by self-attention. Building on this perspective, we derive a population-level evidence lower bound (ELBO) that characterizes the role of positional encoding in forecasting weakly stationary time series. Our analysis further highlights a limitation of static positional encoding: when temporal dependence structures vary across input windows, a fixed temporal bias must accommodate heterogeneous dynamics and can therefore be suboptimal for non-stationary forecasting. Motivated by this analysis, we propose Bayesian Time-aware Attention (BTA), which employs a mixture-of-experts (MoE) router to adaptively combine temporal kernels and generate input-conditioned, head-specific positional biases. Extensive experiments on standard LTSF benchmarks demonstrate that BTA consistently improves forecasting performance across multiple Transformer backbones and achieves competitive overall performance with modest computational overhead and stable training.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.