Building Production Tokenizers: BPE,...

The Tokenizer Nobody Talks About

Every production NLP system starts with a tokenizer, and almost nobody gives it the attention it deserves. I've watched teams spend months fine-tuning model architectures while their tokenizer silently butchered domain-specific terminology. The vocabulary your tokenizer learns determines what your model can even represent — get this wrong, and no amount of training will fix the downstream problems.

Three algorithms dominate production tokenization: Byte Pair Encoding (BPE), WordPiece, and SentencePiece. They're often treated as interchangeable. They aren't. Each makes different tradeoffs around vocabulary efficiency, handling of unknown tokens, and training speed that matter enormously once you're processing millions of documents per day.

BPE: The Algorithm That Started It All

BPE works by iteratively merging the most frequent pair of consecutive tokens. You start with individual characters and build up. The original formulation came from data compression, which tells you something about its strengths — it's excellent at finding repeated patterns.

Here's a minimal BPE training implementation that shows the core loop:

import re
from collections import Counter

def get_pairs(word_freqs):
 pairs = Counter()
 for word, freq in word_freqs.items():
 symbols = word.split()
 for i in range(len(symbols) - 1):
 pairs[(symbols[i], symbols[i+1])] += freq
 return pairs

def merge_pair(pair, word_freqs):
 merged = {}
 bigram = ' '.join(pair)
 replacement = ''.join(pair)
 for word, freq in word_freqs.items():
 new_word = word.replace(bigram, replacement)
 merged[new_word] = freq
 return merged

def train_bpe(corpus, vocab_size):
 word_freqs = Counter()
 for line in corpus:
 for word in line.strip().split():
 spaced = ' '.join(list(word)) + ' </w>'
 word_freqs[spaced] += 1
 merges = []
 while len(merges) < vocab_size:
 pairs = get_pairs(word_freqs)
 if not pairs:
 break
 best_pair = pairs.most_common(1)[0][0]
 word_freqs = merge_pair(best_pair, word_freqs)
 merges.append(best_pair)
 return merges

The critical detail most tutorials skip: BPE's merge order IS the vocabulary. When you serialize a trained tokenizer, you're saving that ordered list of merges. During inference, you replay them in order. Change the order, and you've got a different tokenizer entirely.

Production BPE Gotchas

In my experience running BPE tokenizers on financial text, the biggest issue was vocabulary coverage for rare but important terms. Ticker symbols like "AAPL" or chemical formulas would get split into individual characters because they never appeared frequently enough during training to form merge pairs.

This connects to the ideas in Apache Spark MLlib vs Distributed PyTorch for Large-Scale Fe.

The workaround isn't pretty but it works: maintain a pre-tokenization dictionary that forces certain terms to remain intact. Hugging Face's tokenizers library calls these "added tokens" and handles them before the BPE algorithm runs.

WordPiece: Google's Refinement

WordPiece looks similar to BPE at first glance, but the merge criterion is fundamentally different. Instead of picking the most frequent pair, WordPiece picks the pair that maximizes the likelihood of the training corpus. That sounds like a small change. It isn't.

The likelihood-based criterion means WordPiece tends to produce more linguistically meaningful subwords. Where BPE might merge "th" and "e" early (because "the" is common), WordPiece considers whether that merge improves the overall model of the training data. In practice, this means WordPiece vocabularies handle morphologically rich languages better than BPE — German compound words and Turkish agglutination get split more sensibly.

def wordpiece_score(pair, word_freqs, token_freqs):
 # Score a merge by likelihood improvement, not raw frequency
 a, b = pair
 pair_freq = 0
 for word, freq in word_freqs.items():
 symbols = word.split()
 for i in range(len(symbols) - 1):
 if symbols[i] == a and symbols[i+1] == b:
 pair_freq += freq
 # Likelihood ratio
 score = pair_freq / (token_freqs[a] * token_freqs[b])
 return score

BERT uses WordPiece with a vocabulary of 30,522 tokens. That number wasn't arbitrary — the team experimented with sizes from 8k to 128k and found diminishing returns past 30k for English text. For multilingual BERT, they bumped it to 110k, which still isn't enough for languages with large character sets like Chinese and Japanese.

SentencePiece: When You Can't Assume Whitespace

Here's where things get interesting. Both BPE and WordPiece assume you can split text on whitespace first, then tokenize each word independently. That assumption breaks for Chinese, Japanese, Thai, and any other language that doesn't use spaces between words. SentencePiece treats the input as a raw byte stream and learns segmentation from scratch.

SentencePiece also brings a unigram language model approach as an alternative to BPE. The unigram method starts with a large vocabulary and prunes tokens that contribute least to the corpus likelihood — it's subtractive where BPE is additive. I've found unigram mode produces slightly better results for code tokenization because it handles the weird distribution of programming tokens more gracefully.

import sentencepiece as spm

spm.SentencePieceTrainer.train(
 input='training_corpus.txt',
 model_prefix='my_tokenizer',
 vocab_size=32000,
 model_type='unigram',
 character_coverage=0.9995,
 num_threads=16,
 max_sentence_length=4192,
 shuffle_input_sentence=True,
 byte_fallback=True,
 normalization_rule_name='nmt_nfkc_cf',
)

sp = spm.SentencePieceProcessor()
sp.load('my_tokenizer.model')

text = "The transformer architecture changed NLP"
tokens = sp.encode(text, out_type=str)
ids = sp.encode(text, out_type=int)
decoded = sp.decode(ids)
assert decoded == text # Lossless roundtrip

The character_coverage Parameter

This one catches people. Setting character_coverage to 1.0 means "include every character from the training data in the vocabulary." Sounds reasonable until you realize your training data contains emoji, zero-width joiners, and obscure Unicode codepoints that each eat a vocabulary slot. Setting it to 0.9995 drops the rarest characters and replaces them with a byte-fallback encoding, which is almost always what you want.

Choosing the Right Algorithm

After building tokenizers for three different production systems, here's my honest assessment. BPE wins on simplicity and predictability — you can reason about what it'll do. WordPiece wins when linguistic quality of subword splits matters, which is most classification and extraction tasks. SentencePiece with unigram wins when you're dealing with multilingual text or code, because it doesn't make assumptions about word boundaries.

The vocabulary size question is simpler than people think. For monolingual English models, 32k tokens covers most needs. Multilingual models need 64k to 128k depending on language count. Code models benefit from larger vocabularies (50k+) because programming tokens have a long tail that smaller vocabularies can't represent efficiently.

Testing Your Tokenizer Before Training Your Model

Don't skip this step. Before you commit to a tokenizer configuration, run three checks.

First, measure fertility — the average number of tokens per word. A fertility above 1.5 for your target language suggests the vocabulary is too small or the training corpus wasn't representative. For English, good tokenizers land between 1.2 and 1.4.

Second, check domain term integrity. Take your most important domain terms (product names, technical terms, entity names) and verify they tokenize sensibly. If "Kubernetes" becomes ["Ku", "ber", "netes"], your downstream NER model will struggle.

Third, roundtrip every test case. Encode then decode. If you don't get the original text back byte-for-byte, you've got a normalization bug that will corrupt your training data silently.

def audit_tokenizer(tokenizer, test_terms, corpus_sample):
 total_tokens = 0
 total_words = 0
 for line in corpus_sample:
 words = line.split()
 tokens = tokenizer.encode(line, out_type=str)
 total_words += len(words)
 total_tokens += len(tokens)
 fertility = total_tokens / total_words
 print(f"Fertility: {fertility:.2f}")

 for term in test_terms:
 pieces = tokenizer.encode(term, out_type=str)
 if len(pieces) > 3:
 print(f"WARNING: '{term}' split into {len(pieces)} pieces")

 failures = 0
 for line in corpus_sample:
 ids = tokenizer.encode(line)
 decoded = tokenizer.decode(ids)
 if decoded != line:
 failures += 1
 print(f"Roundtrip failures: {failures}/{len(corpus_sample)}")

Production tokenization isn't glamorous work, but it's foundational. The teams I've seen struggle most with model quality are almost always the ones who grabbed a pretrained tokenizer without checking whether it fit their domain. Spend the time here. Your model will thank you.

Building Production Tokenizers: BPE, WordPiece, and SentencePiece Implementation Details