Sparse attention selector trained on the language modelling loss itself
single source · 1 articles · research · confidence: medium · first seen 2026-09-10 20:00 UTC
A new preprint targets the cost of long context windows, which grows with the square of sequence length. Sparse methods let each query look at a small subset of tokens or blocks. Trainable versions usually train the selector to copy the original model's dense attention weights, which the authors argue ranks context by attention weight rather than by effect on the prediction. SAS adds the selector's continuous scores to the attention logits during training, so the language modelling loss updates it directly. The abstract reports it beating other trainable sparse attention baselines on reasoning, long-context and agentic tasks, but gives no scores or evaluation date.
What this means for you
Nothing to act on yet. This is an abstract alone: no scores, no evaluation date, no code or weights mentioned, and the comparison is against other trainable sparse attention methods rather than against dense attention. If you are choosing how to serve long contexts this quarter, nothing here changes that choice.
Key facts
- ·SAS is described as a gated sparse attention mechanism that optimises context ranking end-to-end with the language modelling loss. source
- ·The method injects the selector's continuous scores into the attention logits during training, so the language modelling loss updates the selector by backpropagation. source
- ·A memory-efficient Triton kernel integrates SAS into FlashAttention-style computation for long-sequence training. source
- ·The abstract reports gains over trainable sparse attention baselines on reasoning, long-context understanding and agentic tasks, with the largest reported gains at tight attention budgets. source
- ·The abstract gives no absolute scores, no baseline model names and no evaluation dates. source
- ·The paper is posted as arXiv 2609.13141, dated 10 September 2026. source
What the sources say
- Hugging Face Daily Papers (research) — Trains the token selector with the model's own loss instead of mimicking dense attention weights.
Sources
The original reporting. Follow these — they did the work.
- Hugging Face Daily PapersSAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking2026-09-10