Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models
Continuous diffusion language models have historically lacked two key features for practical deployment—variable-length generation and KV caching—while also trailing autoregressive and discrete diffusion in quality. This paper introduces Clock Diffusion, a framework that reparameterizes global diffusion time into position-dependent local clocks to enable semi-autoregressive denoising over token embeddings. Together with vectorized training masks and Cache Grab samplers, ClockDLMs close much of the perplexity gap on OpenWebText and unlock efficient inference, though downstream gains remain uneven across tasks.
Clock Diffusion is a well-motivated and technically sound contribution that advances continuous diffusion language modeling. By deriving a principled variational bound for position-dependent noise schedules and introducing efficient vectorized training, the authors produce ClockDLMs that achieve a perplexity of 19.5 on OpenWebText, beating the continuous baseline RePlaid (21.3) and the discrete block baseline BD3LM (20.7 at $S=4$). However, the method does not beat autoregressive models (17.5 PPL), and its summarization ROUGE scores trail discrete diffusion by a wide margin. The Cache Grab samplers yield strong GSM8K results, but Cache Grab-SSD requires jointly training an AR head, complicating the training recipe. Overall, the paper advances the frontier for continuous DLMs, yet its advantages over discrete diffusion are narrower and more task-dependent than the abstract suggests.
The strongest aspects of the paper are its principled derivation and engineering efficiency. The variational objective in Eq. (10) cleanly decomposes into prior, reconstruction, and diffusion losses under local clocks, and the attention masks in Figure 2 enable training with a single forward pass per example. The OpenWebText results are convincing: ClockDLM ($S=4$) attains 19.5 PPL, establishing a new best result among continuous diffusion models and exceeding block discrete diffusion. The Cache Grab-Confidence sampler also achieves training-free speedups with minimal accuracy loss on reasoning tasks.
Despite strong perplexity results, the paper’s claims of broad superiority are unevenly supported. On CNN/DailyMail summarization, ClockDLM ($S=32$) scores 36.0/14.1/23.8 on ROUGE-1/2/L, which is substantially below BD3LM ($S=32$) at 39.4/17.5/27.0 and MDLM ($S=32$) at 40.0/18.0/27.1; the authors highlight the comparison to RePlaid but downplay the regression against discrete diffusion. Additionally, while Cache Grab-SSD exceeds AR accuracy on GSM8K with fewer NFEs, it relies on a jointly trained AR prediction head and a two-pass verification step, meaning it is not a drop-in sampler but a modified training objective.
The explanation for why sliding windows outperform blocks is post-hoc and lacks rigorous ablation. The authors suggest it reflects 'additional flexibility' and a 'more left-to-right decoding scheme even within active windows,' but provide no mechanistic evidence. Finally, the work is explicitly limited to small-scale models, leaving open whether the gains hold at scale.
The evidence generally supports the bounded claims about language modeling perplexity. Comparisons to RePlaid are fair because the authors adopt the same training recipe and embedding dimension. However, the GSM8K evaluation is performed on models trained on TinyGSM, a synthetic math dataset, which limits claims about general mathematical reasoning. The use of NFEs per sequence as the sole efficiency metric is standard in the diffusion literature but does not account for wall-clock time or memory bandwidth differences between continuous and discrete sampling. On summarization, the evidence actually contradicts the narrative of superiority over discrete diffusion: discrete baselines produce higher ROUGE across all configurations reported in Table 3.
The authors provide extensive appendices with full derivations, model definitions, and hyperparameters, which is commendable. However, the paper does not mention a public code release or open-source checkpoints, and the training recipe is intricate, combining learned noise schedules, self-conditioning, soft-capped logits, batch splitting, and an output logit prior. Reproducing these details without code would be labor-intensive and error-prone. The paper also discloses that generative AI was used to develop code and produce figures, which adds an additional layer of uncertainty for independent verification.
Recent works on continuous diffusion for discrete data have demonstrated performance on par with comparable discrete diffusion models. However, these continuous counterparts lack key features that are essential to practical use as language models, namely variable-length generation and support for a key-value cache, and they still lag behind the frontier of autoregressive and discrete diffusion quality. In this work, we address these limitations. We do so by introducing a model parameterization that uses position-dependent noise schedules to define semi-autoregressive (SAR) continuous diffusion language models (DLMs). Together with efficient training and sampling algorithms, we call this framework Clock Diffusion, and we present two special cases of our method: block and sliding window generation. We then define ClockDLMs, a family of Gaussian DLMs based on sliding window Clock Diffusion that attain state-of-the-art diffusion likelihood bounds on OpenWebText, even beating the performant block SAR discrete diffusion models. ClockDLMs trained on TinyGSM also substantially outperform continuous baselines on the GSM8K benchmark and match and exceed comparable SAR discrete diffusion models. Finally, building on our parameterization, we propose more efficient samplers that we dub Cache Grab, which adapt techniques from accelerated inference in discrete diffusion, such as committing tokens whose probabilities exceed a confidence threshold and self-speculative decoding, further improving our models' quality and efficiency.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.