Preprint gives each token several stored entries, chosen by context

single source· 1 articles · confidence: medium · first seen 2026-09-13 20:00 UTC

What this means for you

Nothing to act on today. No code, no checkpoint and no absolute scores, so there is nothing to reproduce against. If you run sub-billion models with value-embedding or bigram memory baselines, this is a comparison to watch, not a swap to make.

Researchers posted a preprint describing Mixture of Memory Embeddings (MoME), which adds cheap parameters to a language model as token-indexed lookup tables. Earlier memory embeddings keep one row per token; MoME keeps several and uses a learned gate over the model's hidden state to choose which to read at each position, so one token can draw on different rows in different contexts. In pretraining runs on nanochat, Llama-3/MobileLLM and Qwen3 backbones, matched for parameter count and training compute, it beat Value Embedding, Bigram and STEM baselines, and scaled better with memory size below a billion parameters. No absolute scores, code or evaluation date are given.

Key facts

  • ·MoME replaces each token's single memory row with a mixture of M slots, selected by a learned gate over the hidden state. source
  • ·It was compared against Value Embedding, Bigram and STEM baselines in iso-parameter and iso-training-FLOP settings. source
  • ·Pretraining experiments used nanochat, Llama-3/MobileLLM and Qwen3 backbones. source
  • ·The paper reports a more favourable memory-size scaling trend at sub-billion scale. source
  • ·The abstract gives no absolute benchmark scores and no evaluation date for the routing analysis. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire