Widening Kimi Delta Attention's gate range lets it track rotations

single source· 1 articles · confidence: medium · first seen 2026-09-20 20:00 UTC

What this means for you

Nothing to act on yet. This is a preprint with open code and weights, not a released model family, and the language-modelling results are reported by the authors without evaluation dates or an independent harness. If you work on state-tracking or long-context architectures, the change is two parameter ranges and the code is public.

A preprint proposes Complex KDA, a variant of Kimi Delta Attention — a linear recurrent model with cheap per-token updates, "diagonal plus rank one", a fixed-size correction rather than a full matrix. The authors widen two parameter ranges: gates to [-1,1] and the delta coefficient β to [0,2]. That lets one layer represent a 2D rotation, and the paper proves one layer can track any finite group isomorphic to a subgroup of 3D rotations. Reported results show the best length extrapolation among tested KDA range settings, and language-modelling scores above Transformers and close to the KDA baseline. Code and weights are open; no evaluation dates are given.

Key facts

  • ·Complex KDA extends Kimi Delta Attention by allowing gates in [-1,1] and the delta-rule coefficient β in [0,2]. source
  • ·The authors prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. source
  • ·A single CKDA layer is claimed to track every finite group isomorphic to a subgroup of SO(3), the 3D rotation group. source
  • ·Reported strongest length extrapolation among tested KDA range settings on S_3, S_4 and periodic audio continuation. source
  • ·In language modelling, CKDA is reported to outperform Transformers and other linear RNNs and to obtain similar results to a KDA baseline. source
  • ·Code is released at github.com/OpenEuroLLM/ComplexKDA, with models on Hugging Face. source

What the sources say

  • Hugging Face Daily Papers (research) — Offers proofs that one layer of an existing linear-attention design can track group-structured state, with modest language-modelling gains.

Sources

The original reporting. Follow these — they did the work.

← the wire