Tokenization — BPE, WordPiece, SentencePiece & Production Artifacts

Tokenization sits at the boundary between raw text and model computation, and its design choices cascade through every downstream capability. The naive approach — split on whitespace — fails immediately for languages without word boundaries (Chinese, Japanese, Thai) and creates an explosion of vocabulary entries for morphologically rich languages like Finnish or Turkish. The opposite extreme — character-level tokens — forces the model to learn language structure from scratch over extremely long sequences. Subword tokenization is the compromise that dominates modern LLMs: split rare words into pieces, keep frequent words whole.

Three algorithms dominate production systems: BPE (Byte-Pair Encoding, used by GPT-2/3/4, Llama 2/3, Mistral), WordPiece (BERT, DistilBERT), and SentencePiece (T5, LLaMA 1, ALBERT, XLNet). Each makes a different tradeoff between training efficiency, vocabulary quality, and language coverage.

What most resources skip: the tokenizer is frozen after training and cannot be changed without full retraining. A poor tokenization decision made during data pipeline design will haunt the model for its entire production lifetime. GPT-2's original 50,257-token vocabulary has been a known bottleneck for multilingual tasks — the shift to 100,277 tokens in GPT-4 reduced average token length for non-English text by roughly 30%. Every token-count difference is a direct cost and context-window tradeoff.

BPE, originally a data-compression algorithm adapted by Sennrich et al. (2016), works as follows. Training phase: initialize the vocabulary with all individual characters (or bytes in tiktoken's implementation). Scan the training corpus, count the frequency of all adjacent symbol pairs, merge the most frequent pair into a new vocabulary entry, repeat until the target vocabulary size is reached (GPT-4: ~100,277; LLaMA-3: 128,256). The merge priority list is deterministic and becomes part of the tokenizer artifact.

Encoding phase (inference): given a new string, apply the learned merge rules in priority order. "unhappiness" might become ["un", "hap", "pi", "ness"] or ["unhappy", "ness"] depending on which merges were learned. The split is fully deterministic — the same string always produces the same tokens.

tiktoken (OpenAI's production tokenizer) implements BPE on byte sequences rather than Unicode codepoints. Every possible byte (0–255) is always in the base vocabulary. This eliminates the "unknown token" problem entirely — any input, any language, any byte sequence is encodable without OOV tokens. The tradeoff: rare Unicode characters might be represented as 2–4 separate byte tokens.

The critical non-obvious insight: BPE merge order depends heavily on training data distribution. A tokenizer trained on English-dominant data will have fewer merges for Korean or Arabic, meaning those languages consume 2–4× more tokens per word. This is not a bug — it's a direct artifact of the corpus composition, and it creates a hidden tax on multilingual inference budgets.

WordPiece (used by BERT, DistilBERT, MobileBERT) differs from BPE in its merge criterion. Instead of picking the most frequent pair, it picks the pair that maximizes the likelihood improvement of the training corpus: select the merge where P(AB) / (P(A) × P(B)) is highest. This favors pairs that are more informative as a unit than their parts independently — "##ing" gets merged because "ing" after any stem is highly predictable as a suffix, not just because it's frequent. WordPiece uses a ## prefix to mark continuation tokens (tokens that are part of a larger word), which BERT relies on for token-level tasks like NER.

SentencePiece (Kudo & Richardson, 2018; used by T5, LLaMA 1, ALBERT, XLNet, Mistral) treats whitespace as a character, representing it as ▁ (U+2581). This is the key architectural decision: SentencePiece does not require pre-tokenization on whitespace before running BPE or its Unigram model. The consequence: it handles Chinese (no spaces), Japanese (no spaces), Arabic (right-to-left script), and Thai (no spaces) identically to English — one unified pipeline. LLaMA 2's tokenizer uses SentencePiece BPE with a vocabulary of 32,000; LLaMA 3 switched to tiktoken BPE at 128,256 tokens, gaining significant multilingual efficiency.

Unigram Language Model (alternative to BPE within SentencePiece): start with a large candidate vocabulary (~100K), then iteratively prune tokens that contribute least to corpus likelihood. Produces probabilistic tokenization — the same word can tokenize differently on different runs (useful for data augmentation, poor for deterministic inference). Production models rarely use stochastic tokenization at inference; they pick the most-probable segmentation (Viterbi decoding).

The vocabulary size decision sits at the intersection of three competing pressures: sequence length (larger vocab → shorter sequences → better context utilization), embedding matrix size (each vocab entry needs a d_model-dimensional vector — at 100K × 4096 dims × 4 bytes = 1.6GB just for the embedding table), and data sparsity (large vocabularies have long tails of tokens the model sees rarely during training, making their embeddings noisy).

GPT-2 used 50,257 tokens — a deliberate choice to fit the embedding matrix in memory on 2018-era hardware while covering English adequately. At 50K vocabulary, English prose averages ~0.75 words/token, but code drops to ~0.4 words/token (operators, punctuation, brackets each get their own token), and Korean or Chinese can hit 1 character per token for rare characters.

GPT-4's ~100,277-token vocabulary was an intentional investment: more coverage of non-English subwords means shorter sequences, which directly reduces inference cost via attention's O(n²) scaling and directly increases how much content fits in the context window. For a fixed 128K-token context window, 30% shorter average token spans for multilingual text means 30% more content fits.

The production rule of thumb: 32K–50K is sufficient for English-only or English-dominant models; 100K+ is the sweet spot for multilingual models where you want token efficiency parity across languages. Beyond 200K, embedding matrix memory dominates and returns diminish.

The most counterintuitive production failures in LLMs trace back to tokenization, not model capacity. Understanding these artifacts is what separates senior ML engineers from people who treat the model as a black box.

The r-counting problem: "How many r's in strawberry?" — most models answer 2 instead of 3. Cause: "strawberry" tokenizes as ["straw", "berry"] or ["st", "raw", "berry"] depending on the tokenizer. The model sees tokens, not characters. It cannot count characters it never individually processes. The r in "straw" is inside a token, invisible to the character-counting reasoning the model tries to apply.

Arithmetic inconsistency: "1000 + 2000 = ?" is easy; "1457 + 2683 = ?" is hard. Reason: small, round numbers often appear as single tokens in training data (the model has memorized their sums). Large, irregular numbers get split into individual digit tokens or rarely-seen multi-digit tokens where the model must actually compute — and has seen fewer examples of.

String reversal failures: "Reverse the string 'hello'." Models often fail because "hello" is one token — reversing a token ID has no meaning. A model that succeeds does so by implicitly decomposing into characters, which requires having learned the internal character structure of tokens from training patterns, not from direct character access.

Code vs prose tokenization divergence: Python's indentation structure means that 4 spaces + a token = at least 2 tokens. Template strings with repeated patterns (HTML, JSON) can be 3–5× more tokens than the semantic content suggests. This is why code-specialized models (CodeLlama, DeepSeek-Coder) are trained with tokenizers that treat common programming patterns as single tokens.