Fathom speeds decoding when the KV cache lives in host memory

single source· 1 articles · confidence: medium · first seen 2026-09-14 20:00 UTC

What this means for you

Nothing to adopt yet: this is a paper, with no code, weights or serving integration attached. The speed-up only appears when the key index lives in host memory, so it is irrelevant to short contexts served from one GPU. If you do offload million-token agent sessions, the 1.67x is measured against three named baselines, not your stack.

Fathom, an arXiv preprint, attacks the scan that bounds decoding when long agent sessions push the KV cache — the stored key/value tensors that let a model skip reprocessing its context — into host memory. It stores the 4-bit key cache as bit planes, letting each query choose how many bits per channel it reads. At one million tokens on Qwen3-8B a decode step is 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki and SparQ r=32; in SparQ's 68-bit GPU time it reads 18% fewer bytes, with lower attention error on six of seven settings. It is not faster when the index stays on the GPU.

Key facts

  • ·Fathom reports a decode step 1.67x faster in GPU time at one million tokens on Qwen3-8B than the 136-bit scans of Double Sparsity, Loki and SparQ r=32. source
  • ·At the same GPU time as SparQ's 68-bit read (r=16), Fathom reads 18% fewer bytes with lower attention error on six of seven model and context settings. source
  • ·On real coding-agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan using 92 bits. source
  • ·The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly that channel's t-bit quantizer and each query spends its bit budget by reverse water-filling. source
  • ·The method is not faster when the index is resident in GPU memory, and every per-token scan matched exact top-k decoding on RULER-style tasks. source
  • ·The paper was posted as arXiv 2609.17652 on 14 September 2026. source

What the sources say

  • Hugging Face Daily Papers — Varies how much of each cached key a query reads, and benchmarks that against three sparse baselines.

Sources

The original reporting. Follow these — they did the work.

← the wire