FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
Speculative decoding drafters trained on short sequences collapse on long-context inputs because their position encodings extrapolate poorly. FOLD fixes this without retraining by keeping exact positions in a local window and compressing all distant context into a shared-position far memory. A concentration factor $\alpha$ in a decoupled softmax prevents attention dilution as the memory grows. Across six target–drafter pairs and contexts up to 256K tokens, the method consistently raises speedup, in the best case more than doubling it.
FOLD is a well-motivated, training-free fix for long-context speculative decoding that delivers robust speedups across models and drafter designs. The paper’s diagnostic experiments cleanly isolate distance as a cause of degraded draft acceptance and show that target-derived features already encode coarse position, justifying the design. Gains are largest on Llama-3.1-8B—raising average speedup from $2.11\times$ to $3.40\times$—but smaller on Gemma-4-12B, and the method cannot repair degradation that originates in the target model itself.
The empirical breadth is strong: evaluation spans three target families, two drafter architectures (DFlash and DSpark), two inference engines (vLLM and SGLang), and inputs up to 256K tokens. The ablations convincingly separate the contributions of the local window and the far memory (Table 4) and validate the concentration factor $\alpha$ (Table 5). The idea itself is elegant and inexpensive—applying a single virtual position to distant entries avoids retrieval overhead while preserving the learned local structure needed for drafting.
The memory overhead of storing an unrotated side cache, while acceptable on an H100, scales linearly with context and reaches up to 6.0 GiB for Qwen3.5-9B, which is material under memory pressure or batching. FOLD does not address target-model degradation; on Gemma-4-12B the gains are modest, and even StreamingLLM loses $\tau$ as inputs grow, suggesting positional extrapolation is only one of several failure modes.
The window size $W$ and concentration factor $\alpha$ are hand-tuned per drafter and fixed during decoding, and the paper does not ablate $W$, leaving some uncertainty about sensitivity to this choice.
The evidence supports the claim that FOLD improves over the original drafters and most baselines, though the comparison to ReRoPE is mixed: ReRoPE often comes second and sometimes ties, implying the decoupled normalization helps but is not always transformative. Retrieval-based baselines (CMR, InfLLM) sometimes achieve comparable acceptance length yet lower speedup because of selection and refresh overhead, which fairly highlights FOLD's advantage of reading the full memory without retrieval costs.
The paper releases code and uses publicly available checkpoints and benchmarks; the vLLM and SGLang implementations are described in detail. Reproduction is tractable on a single H100, but the fused Triton kernels and side-cache bookkeeping are non-trivial, so independent verification depends critically on the released codebase. The fixed hyperparameters and the lack of ablation over $W$ make it slightly unclear how sensitive results are to window size.
Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet most drafters are trained on short sequences and degrade on long inputs, while long-context training is costly. Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries. Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history attention affordable. We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position. To counter attention dilution as the memory grows, we sharpen attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window. Across six target-drafter pairs and diverse benchmarks of up to 256K tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08$\times$ to 4.22$\times$.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.