Buoyant open-sources a KV-cache-aware router for vLLM clusters
single source· 1 articles · confidence: medium · first seen 2026-11-07 08:16 UTC
What this means for you
If you already serve models on Kubernetes with vLLM, this is worth a look: Apache 2.0, and it changes routing rather than your model. The 95% figure is the vendor's own on an unpublished benchmark, so measure it against your own traffic before planning capacity around it.
Buoyant has released Kinference, an Apache 2.0 control plane — the layer that decides which server handles each request — for vLLM inference servers on Kubernetes. It adds declarative routing policies, load balancing aware of the KV-cache (the stored state from earlier tokens, so a request lands on a server that has already processed a shared prompt) and per-tenant metrics. Buoyant reports a 95% cut in median time to first token on its own benchmark, with no evaluation date or harness given.
Key facts
- ·Kinference is released under the Apache 2.0 licence as an inference control plane for vLLM on Kubernetes. source
- ·It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics. source
- ·Buoyant reports a 95% reduction in median time to first token, measured on its own benchmark. source
- ·The write-up gives no evaluation date, harness or workload details for that benchmark. source
- ·MarkTechPost published its account on 7 November 2026. source
What the sources say
- MarkTechPost — Three-sentence release write-up; repeats the vendor's 95% latency claim without describing the benchmark.
Sources
The original reporting. Follow these — they did the work.