Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers
This paper tackles the underexplored problem of token sparsification in audio transformers, where self-attention cost grows quadratically with input length. The authors systematically compare pruning and merging strategies across model sizes and datasets, and propose frequency-aware heuristics—Spectral Flatness (SF) and Low-Frequency Energy (LFE)—to score token importance. Their core finding is that audio-specific spectral cues consistently outperform generic attention-only methods in post-training settings, offering a practical path to faster inference on spectrogram-based models.
The paper delivers a practical, empirical benchmark but is ultimately incremental: it applies well-known signal-processing metrics to token embeddings and tests them on a single family of models. The frequency-aware heuristics are lightweight and effective, yet the theoretical justification for treating learned embeddings as 1-D signals amenable to spectral flatness is thin. The scope is narrower than claimed—limited to AST classifiers on three small-to-medium datasets—so conclusions about 'audio transformers' are overgeneralized.
The unified experimental protocol is a genuine strength. By evaluating post-training and fine-tuning sparsification on AST-Tiny, Small, and Base across three datasets, the authors establish a clear comparative baseline. The controlled study isolating attention formulations from frequency metrics is well-designed, and the finding that intensity pruning gains an edge for clips longer than two seconds (Fig. 9) is a useful deployment heuristic. The observation that fine-tuning washes out differences between heuristics is also valuable: it implies post-training is the high-stakes scenario where scoring matters most.
The work suffers from severe backbone and task monoculture: every experiment uses DeiT-based AST variants, so claims about audio transformers ignore AudioMAE, HTS-AT, and modern self-supervised backbones. The ToMe method is silently omitted at a $25\%$ keep rate (dashes in Tables 2(a)-2(c)) with no explanation, raising questions about stability. Fine-tuning experiments are restricted to AST-B, undermining cross-model generalizability claims. The 'novel' spectral flatness heuristic lacks theoretical grounding—computing the geometric-to-arithmetic mean ratio of an FFT over learned embedding dimensions is ad hoc, and the paper never justifies why this should correlate with semantic importance beyond hand-waved analogies to tonal audio signals. Finally, the evaluation is limited to classification; detection, separation, and ASR workloads are untouched.
The primary evidence consists of accuracy and $mAP$ tables averaged over three runs, but standard deviations are omitted in the main results, making significance impossible to judge. Pareto fronts (Fig. 1, Fig. 6) show performance-efficiency trade-offs visually, yet rely on FLOPs rather than measured latency or memory. Comparisons to Lee and Lee (2025) are largely fair—Lee and Lee do focus on TopK and intensity pruning with limited model coverage—but the current paper under-cites competing audio efficiency work such as FastAST, and its claim of a 'first systematic comparison' is overstated. The external check confirms Lee and Lee investigated intensity and variation pruning on AST/AudioMAE, so the critique holds, but it also shows they analyzed retention patterns in more depth than acknowledged here.
Reproducibility is hampered by the absence of a code repository, random seed reporting, and standard deviations in the main tables. The authors state results are averaged over three runs, but variance is shown only in subsidiary figures (five or six runs). Hyperparameters beyond epoch counts are not tabulated, and the choice to disable SpecAugment during sparsified training—a decision that could alter regularization dynamics—is mentioned but not ablated. FLOP counts are reported via DeepSpeed profiler, which is transparent, but without wall-clock timings or memory footprints the practical impact of sparsification remains speculative. Together, these omissions would block an independent reproduction without significant guesswork.
Transformer-based models have shown strong performance across diverse tasks, but their reliance on self-attention mechanisms often leads to high computational costs. Token sparsification, which reduces the number of tokens processed in transformer layers, offers a promising solution to maintain performance while improving efficiency. However, although widely studied in the vision and language domains, token sparsification remains underexplored in audio, with existing work often limited to a single method, model size, or dataset. This work presents a comprehensive evaluation of token sparsification techniques for audio transformers. We assess a range of token pruning and merging strategies across multiple datasets and model sizes, under both post-training and fine-tuning settings. Furthermore, we introduce a novel sparsification approach that combines attention scores with spectral cues derived from the frequency content of token embeddings. Our experiments show that frequency-aware approaches, particularly those based on spectral flatness and low-frequency energy, consistently outperform attention-only methods. We also find that sparsification based on spectrogram patch intensity is especially effective for longer audio inputs. These results highlight the value of incorporating frequency-domain and input-specific information into sparsification strategies to enhance performance in audio tasks.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.