Generalized Discrete Diffusion from Snapshots

The theoretical framework is sound and significant. The Proposition in Section 3.1 showing that any rate matrix $Q_t$ can be represented via the interpolating form $K_t = \alpha_t I_m + (1-\alpha_t)\Pi_t$ is mathematically rigorous and genuinely unifies prior masked and uniform diffusion schemes. The uniformization-based Algorithm 1 for exact forward sampling is elegant and practical, requiring only column access to the rate matrix rather than expensive matrix exponentials. The snapshot ELBO derived in Section 4.3, optimizing $\mathcal{L}_{x_0}^{\text{snap}} = \int_0^1 \mathbb{E}[-\log \mu_\theta(x_t,t)_{[x_0]}] dt$, correctly aligns the training objective with the mean parametrization and standard transformer architectures.

The primary concern is the architectural mismatch in the autoregressive comparison. As stated in Section 5, the AR baseline uses causal self-attention while diffusion models use bidirectional attention. Bidirectional models inherently achieve lower perplexity than causal models with comparable parameters because they access future context during training, making the comparison in Table 2 (PPL 8.98 vs 20.49) not apples-to-apples. Additionally, while the semantic-informed kernel (Gauss) shows impressive results in Table 3, the computational overhead of KNN-based sampling with $k=64$ neighbors per token is not quantified against simpler uniform/mask baselines in terms of wall-clock training time or memory. Finally, the information-calibration decomposition in Section 4.3 assumes the snapshot objective minimizes the calibration gap $\mathrm{Cal}_\theta^s$, but does not empirically verify this holds across arbitrary noising processes.

The experimental evidence strongly supports superiority over existing discrete diffusion methods. Table 1 shows GDDS Absorb achieving 1.16 BPC on Text8 versus MDM's 1.58, and Table 2 reports OWT validation perplexity of 7.65 for GDDS Gauss versus 31.03 for MDM and 36.82 for UDLM. Zero-shot transfer results in Table 3 demonstrate that semantic noising provides consistent generalization benefits across seven downstream datasets. However, comparisons to prior work like GIDD (von2025generalized) and kinetic-optimal flow matching (Shaul et al., 2024) are discussed in Section 3.1, but the empirical comparisons focus mainly on MDM and UDLM. The lack of wall-clock time comparisons for the semantic kernel (which requires KNN lookups) versus standard kernels limits assessment of computational efficiency claims.

The paper provides detailed algorithms (1, 2, 3) and reports hyperparameters in Appendix C. The authors commit to releasing code and provide a project page. The main barrier to reproduction would be computational resources: 500k training steps on OpenWebText requires significant GPU time. The semantic kernel implementation using KNN with $k=64$ neighbors is specified, though the exact memory requirements for storing embeddings for vocabularies of size 50,257 are not detailed. The uniformization sampling method (Algorithm 1) is efficient and requires only Poisson sampling and column-wise transitions, making it reproducible without specialized hardware beyond standard deep learning infrastructure.