Low-Bit N-Gram Embeddings for Language Models
N-gram embedding tables give language models cheap capacity gains but can consume hundreds of billions of parameters. This paper asks whether those embeddings can be stored at low precision without losing their benefit. The authors systematically compare quantization schemes—uniform, normal-quantile, and Lloyd–Max codebooks with per-embedding scales—and find that aggressive bit reduction often preserves most of the tables’ gain, freeing memory to grow the tables themselves.
The paper offers a well-executed, practically motivated study showing that n-gram embedding tables tolerate aggressive quantization far better than expected. Across five small models trained from scratch, four-bit tables retain over 99% of the n-gram improvement while cutting table memory by roughly 75%, and similar savings are demonstrated on two large existing models. Still, the contribution is primarily empirical engineering: it provides no theoretical explanation for this robustness, and the compelling fine-tuning recovery experiments are limited to small models, leaving the large-model claims dependent on static post-training compression.
The experimental design is admirably thorough. The authors test multiple architectures, table sizes, and quantization families, cleanly isolating that scale selection dominates codebook choice—a useful guideline for practitioners. The most persuasive result is that reinvesting saved memory into larger tables can exceed original performance: at two bits, 1M-embedding tables deliver 122–126% of the baseline gain while using less than half the memory.
Several issues limit the strength of the conclusions. First, the large-model evaluation is narrow: downstream assessment covers only three tasks, and one of them shows a marked drop (WinoGrande falls 1.74 percentage points for DeepSeek, with a confidence interval that nearly includes zero). More importantly, the fine-tuning recovery shown on small models is not repeated on the large models, so the headline 47–55% memory savings may obscure harder-to-detect degradation that fine-tuning would be needed to fix. Second, optimal mixed-precision allocations differ between DeepSeek and Qwen, but the paper offers no rule for choosing them without an expensive validation sweep.
Finally, the work lacks any analysis of why hashed n-gram embeddings are so quantization-tolerant. Absent insight into the geometric structure or redundancy of these representations, the findings remain an unexplained empirical pattern.
The paper’s internal evidence supports its specific quantitative claims, but comparisons to alternative compression strategies are missing. The baseline comparisons—full-precision tables versus no tables—are fair and the use of improvement retention normalizes results across model sizes. However, the authors do not benchmark against other n-gram compression techniques they cite, such as tensorized or factorized embeddings, leaving open whether low-bit quantization is the best trade-off. They also omit a direct comparison to simpler post-training rounding or vector quantization, even though their own ablations show codebook family matters little next to scale selection.
Reproducing the small-model experiments appears straightforward. The authors disclose full configurations, training budgets, data pipelines, three random seeds, and optimizer hyperparameters in Appendix B, and they release code at a public GitHub repository. Reproducing the large-model results is more demanding: the paper specifies Lloyd–Max codebooks with error-fitted scales, but the exact extraction and re-quantization pipeline for the DeepSeek and Qwen checkpoints is described only at a high level, and rerunning them requires multi-GPU inference on proprietary weights.
Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.