Router prediction lets a 35B model run on a single 24GB machine

single source· 1 articles · confidence: medium · first seen 2026-09-15 20:00 UTC

What this means for you

Nothing to deploy yet. The claim that matters if you run models on your own hardware is that a predicted routing can stand in for the real one without losing experts, and the authors have published no evaluation dates or per-benchmark deltas. Treat 20 tokens per second as a number to reproduce, not a spec.

An arXiv preprint describes Edge0, an engine that streams a 35B mixture-of-experts model (only the experts a token needs are computed, yet all the weights normally sit in memory) from SSD and serves it on one 24GB machine at 20 tokens per second, inside 3GiB of peak active memory. A per-layer prerouter predicts the next layer's expert selection one token ahead, and that prediction is used as the routing. A recovery LoRA (a small trained adapter) recovers quality lost to 4-bit quantisation. The framework is open source; results are reported within a few points of the fp16 teacher across five public benchmarks, with no evaluation dates.

Key facts

  • ·Edge0 serves a 35B-class mixture-of-experts model at 20 tokens per second on a single 24GB machine, with peak active memory under 3GiB. source
  • ·The model's 4-bit weights total 19.5GB, more than the host machine holds, so weights are streamed from SSD. source
  • ·A per-layer prerouter predicts the next layer's routing one token ahead, and that prediction is consumed as the routing itself. source
  • ·An unmerged recovery LoRA, trained on the student path, is used to recover quality lost to int4 quantisation and routing replacement. source
  • ·Average results are reported within a few points of the fp16 teacher across five public benchmarks; no evaluation dates are given. source
  • ·The framework, checkpoints and adapters are described as open source, and an 8B tier runs on the same framework. source

What the sources say

  • Hugging Face Daily Papers — Sets out the prerouter mechanism and reports memory, throughput and benchmark deltas for SSD-streamed inference.

Sources

The original reporting. Follow these — they did the work.

← the wire