Public software / Tokenization
bytepair
Tokenization that can be embedded and examined.
bytepair is a dependency-free C library for byte-level BPE tokenization. It reproduces the reference tokenizer's output for the Qwen3 vocabulary and exposes a C interface for applications that need tokenization without a Python or Rust runtime.
The vocabulary is stored as a memory-mapped file and validated when opened. The repository includes build instructions, examples and an audit harness for checking correctness and measuring performance on specified corpora.
Checking the implementation
Tests compare token sequences with the HuggingFace reference in both SIMD and scalar operation. Separate checks exercise Unicode normalisation, malformed vocabulary files and deliberate source mutations. The published measurements identify their machine and inputs so that performance results can be read in context.
Examining the vocabulary
The accompanying census asks whether a valid input can produce each token in the vocabulary. Reachable entries have witness strings; entries classified as impossible have a stated reason; cases without either remain explicitly unresolved.
For the documented Qwen3 vocabulary, the census reports 149,734 witnessed entries, 1,932 impossible entries and three unresolved cases. These are results for that vocabulary and encoding configuration.
The census documentation records the method, classifications and reproduction commands.
Source and use
The MIT-licensed repository contains the library, command-line tools and supporting documentation. Its README is the starting point for the build and supported tokenizer profile.