Adaptive Multi-Token Prediction: How far should we go?

The strongest contribution is the formalization of speculation as a survival problem. By defining acceptance survival $s_{t,j}=\mathbb{E}_{Q_t}[W_{t,j}]$ and rejection hazards $h_{t,j}$, the authors show that a single hazard vector determines the expected gain of every horizon simultaneously, giving the controller an $O(K)$ plug-in rule. The calibration identity (Lemma 1) tightly connects excess squared hazard error to throughput regret, and the exactness argument in Section 3.1 cleanly justifies why the horizon must be chosen before observing the draft. The partial-feedback algorithm—combining $\varepsilon$-greedy full-window exploration with inverse-probability-weighted OGD—is well-motivated and comes with a sublinear $\mathcal{O}(T^{4/5}(1+P_T)^{1/5})$ dynamic-regret guarantee.

The experimental protocol is too fragile to support strong general claims. Disabling KV caches makes verification artificially expensive and does not reflect production serving stacks; running a single unseeded trial per condition removes any estimate of variance; and several baselines are incomplete (plain decoding on GSM8K timed out after 1,166 of 1,319 problems, and Meta MTP GSM8K runs were capped at 100 problems). The one clear regression—Meta MTP 7B on AIME 2025, where AMTP is 15% slower than the fixed window—is attributed to a model-specific end-of-sequence artifact, raising questions about whether the controller is robust to tokenizer and stopping-rule quirks. In self-speculative pairings, where draft and target share a forward pass and longer horizons cost little extra, the gains over a fixed window are small (often within 1–3%), suggesting limited headroom when cost is not dominated by verification.

The paper fairly compares against the stronger benchmark of the best fixed window per model, not just plain decoding, and the 8-of-10 claim is directionally supported by Table 1. The largest throughput gains occur in cross-model pairings (Meta MTP 7B → Llama-2-13B and self-distilled 4B → Qwen3-14B), where the draft and target have very different latencies and adaptivity has room to amortize verification. In self-speculative settings the gains are marginal, which the authors acknowledge. The comparison to related work is thorough and correctly delineates why token-by-token stopping rules for autoregressive drafters do not apply to MTP. One gap is the absence of any direct comparison—empirical or theoretical—to the concurrent router of Chintha et al., 2026, which also selects span length before decoding.

Reproduction would be partially blocked by the missing implementation. No code repository is linked in the paper, and while the checkpoints are public HuggingFace models, the training-free controller involves non-trivial online update logic and cost-model estimation that must be reimplemented from the algorithm description. The experimental setup is under-documented for replication: hyperparameters live in Appendix Table 5, but the single-run protocol means there is no variance estimate, and disabling KV caching is a major departure from standard inference practice that limits external validity. Finally, some runs used timeouts or problem caps that complicate fair speedup comparisons across methods.