The Bayesian Learning Rule Beyond KL Geometry
The paper attacks a structural restriction in the Bayesian Learning Rule (BLR): its reliance on KL geometry ties it to fixed-support exponential families, ruling out posteriors with adaptive tails or bounded support. The authors show that mixed-space Bregman discrepancies—specifically Fenchel-Young discrepancies—generalize the BLR while preserving its efficient additive canonical-coordinate updates. This matters because it yields a principled framework for scalable variational optimizers, such as $q$-VON, that use heavy-tailed or bounded-support $q$-Gaussian posteriors and can be implemented by modifying only a few constants in existing code.
The paper presents a theoretically sound extension of the BLR beyond KL geometry. The identification of Fenchel-Young discrepancies as the structure that preserves additive updates is novel, and the recovery of the classical BLR as a special case is clean. However, the empirical evaluation is explicitly positioned as preliminary; the authors defer large-scale benchmarks and state-of-the-art comparisons to future work, which limits the practical impact of the contribution.
The convex-analytic derivation is rigorous. Theorem 1 shows that the canonical gradient under a mixed-space Bregman divergence retains the same additive form as the BLR, and Propositions 1 and 2 correctly establish the induced maxent family and its dual coordinates. The instantiation for Tsallis negentropy yielding bounded-support and heavy-tailed $q$-Gaussians is a compelling concrete payoff.
The largest weakness is the limited empirical evidence. The authors state that demonstrating state-of-the-art performance or scale is beyond the scope and that the CIFAR-100 experiment is not a large-scale benchmark, deferring transformer-scale training to future work. This undercuts the claim that the method readily scales to modern deep learning. A second concern is an unresolved uniqueness question: the authors note that whether mixed-space Bregman divergences are the only family that recovers BLR-style additive updates while preserving linear expectations remains open, leaving the theoretical picture incomplete.
The theoretical evidence supports the formal claims: the derivations of the generalized BLR update and the $q$-Gaussian Newton-like optimizer are complete. Comparisons to related work are fair but narrow. The paper correctly distinguishes its contribution from $f$-divergence approaches such as Alquier (2021), noting that only the KLD permits the necessary separation into linear expectations within that class, whereas the proposed Fenchel-Young framework avoids this restriction. Empirically, the method is shown to be competitive with IVON on CIFAR-10 and CIFAR-100, though the improvements are modest and the evaluation metrics standard.
Reproduction is plausible for an expert reader but not turnkey. The algorithms are specified in detail, and the authors note that only the sampling law, the constants relating $\mathbf{S}$ to the scale, and the corresponding diagonal Stein estimator change relative to IVON. However, no code repository is provided in the text, and key implementation details such as exact hyperparameter schedules for the image-classification experiments are relegated to appendices. The heavy reliance on appendices for derivations and experimental details means independent reproduction would require substantial manual transcription.
The Bayesian learning rule (BLR) exploits the relation between Bayesian updating and the Kullback-Leibler divergence (KLD) to derive efficient additive updates for exponential family variational posteriors. The KL geometry is powerful but restrictive: combined with posterior families with parameter-dependent support it can lead to undefined objectives. We ask which non-KL geometries preserve the additive BLR computation. Previous works that use $f$-divergences suggest that this is only possible with the KLD. Here, we show that this is not true. We show that additive information-geometric optimization algorithms can be derived for the class of mixed-space Bregman or Fenchel-Young discrepancies. These discrepancies induce generalized maximum-entropy families, dual coordinates, and generalized-BLR canonical gradient updates, recovering the classical BLR. The generalized BLR leads to new algorithms that \textit{cannot} be derived from the KL-based BLR. For the Tsallis negentropy with quadratic statistics, the generalized BLR yields bounded-support and heavy-tailed $q$-Gaussian posteriors and an easy Newton-like implementation. Experiments up to CIFAR-100 illustrate controllable support and tail behavior and competitive performance.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.