Google open-sources TPU Sync for cache transfers between chips

single source· 1 articles · confidence: medium · first seen 2026-11-13 10:32 UTC

What this means for you

If you serve large mixture-of-experts models on TPUs, this targets your stack; on GPUs it does nothing. Google's 90% figure comes with no published benchmark, so test it against your own traffic before sizing capacity around it. Nothing to adopt on the strength of the post alone.

Google has released TPU Sync, an open-source C++ library that copies a model's KV cache — the stored state of earlier tokens, kept so the model does not recompute them on every step — along with model weights, between TPU HBM (the high-bandwidth memory stacked beside the chip), host DRAM, storage and the network. Google reports more than 90% lower KV transfer latency on mixture-of-experts models (only part of the network runs per request) of 235B to 480B parameters. No evaluation date, harness or baseline is given for that figure, and the library's version and licence are not stated.

Key facts

  • ·TPU Sync is an open-source C++ library released by Google. source
  • ·It moves KV caches and weights across TPU HBM, host DRAM, storage and the network. source
  • ·Google reports over 90% lower KV transfer latency on mixture-of-experts models of 235B to 480B parameters. source
  • ·No evaluation date, harness or baseline is given for the 90% latency figure. source

What the sources say

  • MarkTechPost — Short release write-up carrying Google's transfer-latency claim and the model size range it covers.

Sources

The original reporting. Follow these — they did the work.

← the wire