Hugging Face says tokenizers v1 is often tens of times faster than v0.23

single source· 1 articles · confidence: high · first seen 2026-09-21 00:00 UTC

What this means for you

If tokenisation is your bottleneck — long inputs, many concurrent requests, GPUs idling while CPUs catch up — v1 is a drop-in, because it emits the same token IDs as v0.23 and nothing downstream changes. It is still a release candidate, so test before pinning it. The speed figures are Hugging Face's own, from its own harness.

Hugging Face has published a release candidate of tokenizers v1, a rewrite of its text-to-token library that it says is often tens of times faster than v0.23 while returning identical token IDs. It says the API, vocabulary and merge ranks are unchanged, and that v1 loads every tokenizer family v0.23 loaded rather than specialising in byte-pair encoding. Benchmarks run from its tokbench repository cover single- and multi-threaded throughput, scaling across threads, latency, decoding throughput, memory and crate size, with a command to rerun them on your own hardware. IBM, NVIDIA and the ExecuTorch team contributed patches.

Key facts

  • ·The release candidate of tokenizers v1 is claimed to be often tens of times faster than v0.23. source
  • ·v1 produces the same token IDs as v0.23, with the API, vocabulary and merge ranks preserved. source
  • ·v1 loads every tokenizer family v0.23 loaded, staying general rather than specialising in BPE. source
  • ·Benchmarks are run from the tokbench repository and can be rerun on other hardware by command. source
  • ·IBM, NVIDIA and the ExecuTorch team contributed patches and helped test across hardware. source
  • ·A tokenizer converts text into the integers a model reads, via normalization, pre-tokenization and mapping to vocabulary IDs. source

What the sources say

  • Hugging Face Blog — Explains the rewrite's performance work and the benchmark suite comparing it with other tokenizer libraries.

Sources

The original reporting. Follow these — they did the work.

← the wire