Blending two prompts adds a model's next-token predictions, study finds

single source· 1 articles · confidence: medium · first seen 2026-09-23 20:00 UTC

What this means for you

Nothing to act on yet. The paper claims one forward pass can emit two continuations, but it ships no code, no weights and no description of the decoding procedure, so there is nothing to test. The linearity the trick relies on also decays during pretraining and has to be fine-tuned back in.

A preprint posted on 23 September reports that transformers behave more linearly than their non-linear parts suggest: blend two prompts into one input, and the next-token predictions come out as the same blend of the two answers the model would have given separately. The authors say this superposition is built into the architecture rather than learned. It fades as pretraining goes on, and light fine-tuning brings it back. A guided decoding method then separates the mixed output, producing two continuations from a single forward pass — one run of the network. One paper's claim, with no scores, no evaluation date and no code.

Key facts

  • ·The paper is arXiv 2609.29845, posted on 23 September 2026 and surfaced on Hugging Face Daily Papers. source
  • ·The authors report that linearly combining inputs from two distinct text streams yields outputs that are a superposition of the individual next-token distributions, a claim they call the Superposition Linearity Hypothesis. source
  • ·They report the effect is intrinsic to the Transformer architecture rather than an emergent consequence of training, and that it tends to diminish as pretraining progresses. source
  • ·Lightweight fine-tuning is reported to substantially restore the linearity, reducing divergence between the predicted distribution and the average of the individual next-token distributions. source
  • ·A guided decoding procedure is introduced that disentangles superposed outputs, enabling two coherent continuations from a single forward pass. source

What the sources say

  • Hugging Face Daily Papers — Single arXiv preprint proposing a linearity hypothesis for transformers, plus a decoding trick to split mixed outputs.

Sources

The original reporting. Follow these — they did the work.

← the wire