On Inherent Privacy of Posterior Sampling: A Unified R\'enyi-Divergence Framework
Bayesian posterior sampling offers a potentially 'free' source of differential privacy, but existing guarantees are fragmented and often require strong boundedness assumptions. This paper develops a unified Rényi-divergence framework that characterizes the inherent privacy loss of One-Posterior-Sampling (OPS) through exponential moments of the single-record log-likelihood ratio. The results yield exact, tighter bounds for a broader class of models and clarify how prior strength, sample size, and neighboring relations jointly shape privacy.
The paper presents a clean and useful theoretical advance. Its central identity — that the Rényi divergence between neighboring posteriors is governed by two exponential moments of $c_1(\boldsymbol{\theta})$ — is powerful and well exploited across bounded, sub-Gaussian, and sub-exponential regimes. The bounds often improve on prior work, and the analysis of Gaussian-Gaussian and linear-regression cases exposes subtle asymmetries under unbounded neighboring. As a purely theoretical contribution, however, it contains no simulations or empirical validation, leaving the practical tightness of its guarantees on real data unexplored.
The privacy-loss decomposition in Eq. (6) and the exact characterization in Theorem 3 are elegant and foundational. The authors write: '$D_{\alpha}(P\|Q) = c_{2} + (\alpha-1)^{-1}\log\mathbb{E}_{\boldsymbol{\theta}\sim P}\left[e^{(\alpha-1)c_{1}(\boldsymbol{\theta})}\right]$' (Theorem 3, Eq. 7). This formula cleanly reduces the problem to bounding exponential moments under the posterior, enabling the subsequent tiered analysis. The extension to sub-Gaussian GLMs and the discovery that unbounded neighboring can make one direction of the divergence finite only for $\alpha < \omega$ are concrete advances over the prior literature.
The analysis is entirely asymptotic and worst-case, and the sub-exponential tractability comes at the cost of looseness. The authors acknowledge that 'deriving these parameters for a specific model often introduces looseness' and that the resulting variance proxies are 'sub-optimal' (Section 6, after Corollary 13). For example, linear regression with covariate replacement yields an $n^2$-dependent variance proxy in Eq. (30), which is unlikely to be tight. More fundamentally, under unbounded neighboring the Gaussian-likelihood guarantees are restricted to $\alpha < \omega$, so pure DP is unattainable and even RDP guarantees have a bounded validity range: '$D_{\alpha}(Q\|P)$ (addition) diverges for $\alpha \geq \omega$' (Section 5.2), limiting the mechanism's applicability when records can be added or removed without bound.
The evidence is strongest where exact moment-generating functions are available — the Gaussian-Gaussian conjugate pair and Bayesian linear regression — and weaker where generic tail bounds are invoked. Comparisons to prior work are specific and quantitative; for Bayesian logistic regression the authors show that under label replacement, 'our bound $\alpha R^{2}\tau^{2}/2$ reduces theirs by 75%' (Example 10). This is a fair and verifiable claim within the paper's own derivations. However, the sub-Gaussian GLM bounds rely on a uniform bounded-score constant $L$, and the logistic-regression example assumes $\|\mathbf{x}_i\| \leq R$, so the practical margin over prior work depends on how conservatively these quantities are estimated in practice.
No source code, datasets, or experimental protocols are released, and the paper contains no empirical experiments. Theorems are accompanied by proofs deferred to the appendices, e.g., 'proof is provided in Appendix A.2' (Theorem 3), which makes the derivations reproducible in principle for a reader with sufficient mathematical background. Reproduction is therefore purely symbolic: an independent researcher would need to re-derive the bounds and, if numerical verification is desired, implement posterior samplers for each model from scratch. The absence of computational artifacts and the reliance on idealized assumptions (known variance, bounded covariates) are the main barriers to empirical reproduction.
Differential Privacy (DP) provides a rigorous framework for quantifying privacy guarantees. While most methods achieve DP by injecting calibrated random noise, the intrinsic randomness of certain procedures such as sampling can yield DP guarantees ``for free''. In this work, we develop a unified R\'enyi divergence framework to characterize the inherent DP guarantees of sampling from a Bayesian posterior distribution. We show that the privacy loss incurred by One-Posterior-Sampling (OPS) is governed by posterior exponential moments of the single-record log-likelihood ratio, and establish explicit DP guarantees under uniformly bounded, sub-Gaussian, or sub-exponential tail regimes. We apply the framework to representative Bayesian models, including categorical likelihoods with arbitrary priors, Gaussian-Gaussian conjugate pair, generalized linear models from the exponential family and linear regression with Gaussian likelihood, both with Gaussian priors on the regression coefficients. Our results elucidate sample size, model structure, neighboring relations in DP (bounded or unbounded), and how prior if applicable, jointly determine the inherent DP guarantees of OPS. Furthermore, our results recover the existing DP guarantees for OPS as special cases -- often with tighter privacy loss bounds -- and substantially broaden the class of Bayesian models for which the inherent DP guarantees of OPS can be rigorously characterized.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.