Handling Multilingual Text in...

When English Isn't Enough

Building NLP for English is a luxury most teams don't realize they have. English has abundant training data, well-understood tokenization, and models optimized specifically for it. The moment you add a second language — let alone twenty — every assumption in your pipeline needs revisiting.

I spent two years building multilingual NLP infrastructure for a platform serving users in 14 languages across Latin, Cyrillic, Arabic, and CJK scripts. The technical challenges were real, but the most surprising problems were the ones that no NLP textbook covers: right-to-left text mixing, language detection failures on short text, and the politics of choosing which languages to prioritize when you can't do them all equally well.

Language Detection Is Harder Than You Think

Every multilingual pipeline starts with language detection, and most teams underestimate how hard it's for short text. On full paragraphs, even simple n-gram classifiers achieve 99%+ accuracy. On tweets, search queries, and chat messages — the text you actually need to process — accuracy drops to 80-85% without specialized handling.

import fasttext

class LanguageDetector:
 def __init__(self, model_path='lid.176.ftz'):
 self.model = fasttext.load_model(model_path)
 self.min_confidence = 0.5
 self.min_length = 10

 def detect(self, text):
 if len(text.strip()) < self.min_length:
 return self._short_text_detect(text)

 predictions = self.model.predict(text.replace('\n', ' '), k=3)
 labels, scores = predictions
 top_lang = labels[0].replace('__label__', '')
 top_score = scores[0]

 if top_score < self.min_confidence:
 return {'lang': 'unknown', 'confidence': top_score}
 return {'lang': top_lang, 'confidence': top_score}

 def _short_text_detect(self, text):
 scripts = self._detect_scripts(text)
 if 'CJK' in scripts:
 return self._cjk_detect(text)
 if 'Arabic' in scripts:
 return {'lang': 'ar', 'confidence': 0.7}
 return self.detect_with_context(text)

The Code-Switching Problem

Real multilingual text doesn't stay in one language. Users mix languages within a single message — "Can you help me fix this? J'ai un probleme avec le login" is a single support ticket that's half English, half French. Standard language detection picks one language for the whole text, which means half your processing uses the wrong language pipeline.

We handle this with sentence-level detection. Split the text into sentences, detect language per sentence, then route each sentence to the appropriate pipeline. It's more expensive, but it's the only approach that handles code-switching correctly. About 8% of our incoming text contains code-switching, and sentence-level detection improved our downstream task accuracy by 6 points on that subset.

Tokenization Across Scripts

Whitespace-based tokenization works for most European languages. It fails completely for Chinese (no spaces between words), Japanese (mixed scripts with no consistent delimiters), and Thai (spaces indicate phrases, not words). Each script family needs its own tokenization strategy.

TOKENIZER_MAP = {
 'zh': 'jieba',
 'ja': 'mecab',
 'th': 'pythainlp',
 'ko': 'mecab-ko',
 'default': 'whitespace',
}

def tokenize(text, lang):
 tokenizer_key = TOKENIZER_MAP.get(lang, 'default')
 if tokenizer_key == 'jieba':
 import jieba
 return list(jieba.cut(text))
 elif tokenizer_key == 'mecab':
 import MeCab
 tagger = MeCab.Tagger('-Owakati')
 return tagger.parse(text).strip().split()
 elif tokenizer_key == 'whitespace':
 return text.split()

One thing that bit us: tokenizer versions matter enormously. Jieba 0.42 and 0.39 produce different segmentations for the same input, which means your embeddings change when you upgrade the library. Pin your tokenizer versions and retrain your models when you upgrade — treating tokenizer updates as transparent dependency bumps is a recipe for silent quality degradation.

For a related perspective, see Streaming Feature Computation with Apache Flink for Real-Tim.

Model Selection: Multilingual vs Language-Specific

The tempting approach is a single multilingual model for everything. XLM-RoBERTa, mBERT, or a multilingual sentence transformer handles all languages in one model. The reality: multilingual models underperform language-specific ones by 3-8% depending on the task and language. For high-resource languages the gap is smaller. For low-resource languages the gap can be huge because the multilingual model's vocabulary allocation favors high-resource languages.

Our compromise: language-specific models for the top 5 languages by volume, and a multilingual model as the fallback for everything else. This gives us the best accuracy where it matters most while keeping the system manageable. Running 14 separate language-specific models would be operationally painful, but 5 is fine.

Unicode Normalization Nightmares

Unicode normalization is the most boring-sounding problem that will ruin your week. The same visible character can have multiple Unicode representations. The accented "e" can be a single codepoint or a combining sequence. If you don't normalize, identical strings won't match, your deduplication breaks, and your metrics lie to you.

We covered a related topic in Batch Normalization vs Layer Normalization in Production Tra.

import unicodedata

def normalize_text(text):
 text = unicodedata.normalize('NFC', text)
 text = text.replace(' ', ' ') # non-breaking space
 text = text.replace('', '') # zero-width space
 text = text.replace('‌', '') # zero-width non-joiner
 text = text.replace('', '') # byte order mark
 return text

Always normalize to NFC form as the first step in your pipeline. NFC composes characters into their canonical form, which is what most downstream tools expect. Run normalization before language detection, tokenization, or any other processing. We had a bug that lived in production for three months where duplicate detection was failing because some documents arrived in NFD form and others in NFC. The visible text was identical, but the byte representations differed.

Multilingual NLP isn't just NLP with more languages. It's a different engineering discipline that touches tokenization, normalization, model selection, and evaluation in ways that monolingual work doesn't prepare you for. Plan for this complexity upfront rather than discovering it one incident at a time in production.

Handling Multilingual Text in Production NLP Pipelines