Four techniques train 667B mixture-of-experts models at 1M-token context

single source· 1 articles · confidence: medium · first seen 2026-09-12 20:00 UTC

What this means for you

Nothing to act on yet: this is a preprint, the abstract names no code release or supported framework, and every figure comes from the authors' own component tests rather than an outside run. If your training plan assumes device memory is the fixed wall at long context, this is the direction to track.

A preprint bounds the four memory peaks that cap long-context mixture-of-experts training (only part of the network runs per token). PipelinedLLEP caps the tokens each source sends into an expert-dispatch chunk, cutting that peak by up to 59.3% without throughput loss. Ring-DTP circulates activations or weight shards around the vocabulary projection, cutting that peak 86.6%. Selective checkpoint offload keeps one long-lived tensor per boundary in CPU memory, and OffloadStreamAdamW buckets the offloaded optimizer step, 2.05× faster. Loss and gradients stay exact. Together they train 120B–667B models at 1M-token context and up to 10.4× the throughput of a tuned FSDP2 baseline — 8–32× its context reach.

Key facts

  • ·The paper identifies four unbounded memory peaks in common parallelism plans: expert dispatch, the vocabulary projection, gradient checkpoint boundaries and optimizer state. source
  • ·PipelinedLLEP is reported to cut the MoE dispatch peak by up to 59.3% without losing throughput. source
  • ·Ring-DTP is reported to cut the vocabulary projection peak by 86.6%. source
  • ·OffloadStreamAdamW is reported to make the offloaded optimizer step 2.05× faster. source
  • ·Composed on MoE models from 120B to 667B parameters, the methods train at 1M context length and up to 10.4× the throughput of a tuned FSDP2 baseline, 8–32× its context reach. source
  • ·The paper is arXiv:2609.14306, posted 12 September 2026; the results are the authors' own component tests. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire