Pyramid key search brings attention cost down to log-linear

single source· 1 articles · confidence: medium · first seen 2026-09-24 20:00 UTC

What this means for you

Nothing to act on yet: this is a preprint with no released code, no weights and no serving numbers — no latency, throughput or cost figures, and the benchmark results come without evaluation dates or harness. If you pay for long-context inference, the number to watch is throughput at a fixed length, not the complexity claim.

A preprint proposes PISA, a cheaper way to run attention — the step where every token in a model is compared with every other. Sparse attention keeps only some key blocks per query, but choosing which still means scoring every query-block pair, so cost stays quadratic in sequence length. PISA pools keys into about log N levels and narrows candidates from coarse to fine using LogSumExp scoring, for an overall cost of N log N, where N is the sequence length. Custom GPU kernels avoid materialising the full score matrix. The authors report comparable commonsense-reasoning results and better retrieval than the baseline; no evaluation dates or harness are given.

Key facts

  • ·PISA pools keys into O(log N) levels and selects candidates from the coarsest level downward, giving an overall complexity of O(N log N), where N is the sequence length. source
  • ·Conventional block selection requires scoring all query-block pairs, so it remains quadratic in sequence length. source
  • ·LogSumExp scoring is applied at each level to a bounded candidate set to pick candidates for the next finer level. source
  • ·Hardware-aware Triton kernels are provided for both training and inference, fusing hierarchical routing and LogSumExp scoring without materialising the query-key score matrix. source
  • ·The authors report performance comparable to the baseline on commonsense reasoning and better results on retrieval tasks; no evaluation dates or harness are stated. source
  • ·The paper is arXiv 2609.31093, posted on 24 September 2026. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire