Rethinking Sequence Compression with Test-time Regression

The paper proposes TTR-Compressor, which treats input tokens as a regression problem whose closed-form fitted coefficients form the compressed representation, learned end-to-end via backpropagation. It is evaluated as a compressor for block-wise sparse attention in language models, where it improves RULER performance over attention-based compressors by a large margin. As a visual token merger in multimodal LLMs, it outperforms a standard MLP merger and matches its performance with 55.6% fewer visual tokens.

Key takeaways

• TTR-Compressor computes compressed representations as closed-form regression coefficients during the forward pass.
• On block-wise sparse attention, it improves RULER performance over attention-based compressors by a large margin.
• As a visual token merger, it matches the standard MLP merger's performance using 55.6% fewer visual tokens.

Why it matters

Sequence compression is a key information bottleneck in efficient language and multimodal models. A learnable alternative to pooling and projections could reduce token and compute costs while preserving accuracy.

Read with care

Source claims not independently checked; the supplied text is an abstract and author comment, without full experimental details.