Pre-tokenizer scheme encodes casing and accents as operators, shrinking vocabularies

single source· 1 articles · confidence: high · first seen 2026-09-17 20:00 UTC

What this means for you

Nothing to act on yet: one paper, small 98M-parameter GPT-2 models, and the authors say production-scale validation is still needed. If you maintain a tokenizer for multilingual or code text, the idea — reversible prefixes instead of separate vocabulary entries — is worth measuring on your own corpus.

A paper posted to arXiv proposes a lossless "pre-tokenizer" — the step that splits raw text before it becomes tokens — that rewrites capitalisation, accents and repeated characters as short operator codes placed in front of the base word, using code points Unicode reserves for private use. The authors report the scheme cuts the vocabulary slots needed by up to 19.7% across natural-language and code corpora, and that in their own runs 98M-parameter GPT-2 models trained this way emitted syntactically valid Python 9.12% of the time against 7.70% for a baseline, with fewer duplicated n-grams in prose. They say production-scale validation is still needed.

Key facts

  • ·The Functionalizer reduces actual vocabulary slot requirements by up to 19.7% under unconstrained exhaustion conditions. source
  • ·Downstream evaluations used 98M-parameter GPT-2 models. source
  • ·Python code syntax validity was 9.12% with the Functionalizer versus 7.70% without it. source
  • ·Operators cover casing (CAPITALIZE), 13 dedicated diacritic opcodes, and repetition (REPEAT, MULTIREPEAT), and are fully reversible. source
  • ·Transformation operators are encoded in the Unicode Private Use Area. source
  • ·The paper is arXiv 2609.15991, posted 17 September 2026. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire