Register tokens beat carrying earlier text in diffusion language models

single source· 1 articles · confidence: medium · first seen 2026-09-13 20:00 UTC

What this means for you

Nothing to act on. This is an abstract, not a release: no weights, no code, no evaluation date. If you work on diffusion language models, the claim worth testing on your own harness is that a few fixed tokens carry state better than the context window does.

Diffusion language models — which write by repeatedly filling in masked-out tokens rather than strictly left to right — can keep working across generation chunks with a small set of "register" tokens, according to a preprint posted to arXiv. Instead of holding earlier text in context, the model decodes a chunk, clears it, and keeps only the registers' hidden values. On LLaDA and Dream, registers beat carrying the earlier text along on every benchmark tested, gaining up to 8.5 points on maths and 19.5 points on code. The abstract gives no evaluation dates or harness details.

Key facts

  • ·The preprint was posted to arXiv on 13 September 2026 as 2609.16372. source
  • ·Register tokens are a small number of fixed-position tokens whose continuous hidden states carry reasoning progress between generation chunks, so earlier text can be cleared. source
  • ·In comparisons on LLaDA and Dream, registers outperformed carrying the discrete earlier text on every benchmark tested. source
  • ·Reported gains reach 8.5 points on maths benchmarks and 19.5 points on code. source
  • ·The register state can be further refined with reinforcement learning on long-horizon reasoning tasks. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire