Rethinking Sequence Compression with Test-time Regression
Sequence compression is a critical bottleneck in efficient language and multimodal models, yet most architectures rely on simple pooling or MLP projections that ignore inter-token correlations. This paper reframes compression as a test-time ridge regression problem: input tokens define a regression whose fitted coefficients become the compressed vector. The resulting TTR-Compressor solves exactly in closed form via a small Cholesky factorization, introduces only $2d$ learnable parameters, and yields strong gains on long-context retrieval and visual token merging.
TTR-Compressor is a clever and well-motivated re-framing of token compression, but its empirical advantages are narrower and more context-dependent than the abstract suggests. The gains on RULER retrieval tasks are substantial and consistent across scales (Table 1 and Appendix Table 7), and the visual-merging results show clear improvements at higher compression ratios (Table 3). However, the headline claim of matching MLP performance with 55.6% fewer tokens rests on a negligible average-gap (54.85 vs. 54.93 in Table 3), and the method’s sensitivity to key normalization (Table 5) reveals that the regression formulation alone is not sufficient. When the evaluation context is too short to invoke sparse attention, all compressors perform identically (Table 2), indicating that the operator does not improve general pretraining, only long-context retrieval quality.
The theoretical formulation is clean and computationally practical. Casting compression as a differentiable ridge regression with a dual Cholesky solve yields an exact, input-adaptive representation in $\mathcal{O}(T^2 d + T^3)$ time, which is efficient for the small $T$ regime (e.g., $T=64$ or $T=9$). The derivation of redundancy-aware weighting in Appendix B rigorously shows why correlated tokens receive diminishing influence relative to additive pooling. The end-to-end learning of both the scalar targets and the coordinate-wise regularizer is elegant, and the 1B-scale scaling experiments in Appendix E.1 confirm that the RULER improvements are not an artifact of the small 345M setup.
TTR-Compressor’s reported gains are tightly coupled to specific design choices rather than the core regression idea. Table 5 shows that removing key normalization collapses retrieval accuracy from 0.5092 to 0.2531, proving that the regression framework is fragile without careful feature preprocessing. Table 6 further reveals that a simple learnable scalar regularizer achieves nearly the same average accuracy (0.4971) as the full learnable diagonal (0.5092), suggesting that the coordinate-wise regularization and the costly dual solve contribute only marginally.
The comparisons to attention-based compressors are also somewhat unfair: the paper claims attention-weighted pooling "scores each token independently," yet standard self-attention computes normalized scores over the full set, which inherently couples tokens. In visual merging, the headline 55.6% token reduction corresponds to a mere 0.08 average-point difference (54.85 vs. 54.93 in Table 3), and the MLP baseline is under-specified, leaving open whether the gains come from added capacity rather than the regression inductive bias.
The evidence supports the claim that TTR-Compressor improves retrieval keys for block-wise sparse attention on RULER needle-in-a-haystack variants. However, the benchmark choice is narrow: RULER’s synthetic retrieval tasks reward precise key matching but do not assess whether the compressed representations preserve broader semantic or compositional structure. The comparison to HiLS-Attention’s attention-weighted pooling is fair insofar as the paper uses the identical backbone, but HiLS-Attention itself is a very recent baseline and the authors do not compare against simpler alternatives such as a standard MLP compression head in the language setting. In vision, the paper does not control for total parameter count between the two-stage TTR+MLP pipeline and the plain MLP merger.
The paper provides reasonable architectural details and hyperparameters in Appendix D, including optimizer settings, weight decay, and learning-rate schedules for both the language and vision experiments. The authors also share wall-clock timing measurements (Figure 3) and release a project page. Nevertheless, several items are missing: no explicit code repository link or GitHub URL in the manuscript, no random seeds, no exact data mixture proportions for the multimodal training stage, and no specification of the MLP baseline’s hidden dimensions or activation function in the vision experiments. The 1B-parameter scaling results in Appendix E.1 reuse the same hyperparameters but do not state whether the training recipe was otherwise identical. Without these details, independent reproduction would require significant guesswork, especially for the multimodal training pipeline.
Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.