Speech Recognition Pipeline...

Q: What is Transfer Learning Is Not Optional?

When you have got limited data, you cannot afford to learn acoustic features from scratch. Transfer learning from a multilingual pretrained model is the starting point, full stop. The question is which pretrained model and how to adapt it. We have had the best results starting from Whisper-large-v3 or wav2vec 2.0 XLSR-53, both of which were.

Q: What is Self-Supervised Pretraining for Low-Resource ASR?

When you do not have enough labeled speech data, you might still have unlabeled audio. Radio broadcasts, podcasts, YouTube videos in the target language, even if not transcribed, these can be used for self-supervised pretraining to learn acoustic representations. The wav2vec 2.0 pretraining approach works by masking portions of the audio input and.

The Low-Resource Language Bottleneck

Speech recognition for English, Mandarin, and Spanish works remarkably well. Models like Whisper and Conformer achieve word error rates under 5 percent on clean audio for these languages. But try the same models on Yoruba, Igbo, or Twi and the error rates jump to 30-50 percent. The gap isn't primarily algorithmic. It's data.

I've been working on ASR systems for West African languages for the past two years, and the core challenge is always the same: there isn't enough labeled speech data to train a good acoustic model from scratch. English ASR benefits from tens of thousands of hours of transcribed audio. Most low-resource languages have fewer than 100 hours. Some have fewer than 10.

Transfer Learning Is Not Optional

When you have got limited data, you can't afford to learn acoustic features from scratch. Transfer learning from a multilingual pretrained model is the starting point, full stop. The question is which pretrained model and how to adapt it.

We've had the best results starting from Whisper-large-v3 or wav2vec 2.0 XLSR-53, both of which were trained on diverse multilingual data.

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

model = WhisperForConditionalGeneration.from_pretrained(
 "openai/whisper-large-v3"
)
processor = WhisperProcessor.from_pretrained(
 "openai/whisper-large-v3"
)

# Freeze encoder for first training phase
for param in model.model.encoder.parameters():
 param.requires_grad = False

model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(
 language="yoruba", task="transcribe"
)
# Unfreeze encoder after 5 epochs for end-to-end fine-tuning

Data Augmentation for Speech

With only 50-100 hours of training data, every sample counts, and augmentation is critical. Standard audio augmentation, speed perturbation, volume changes, additive noise, helps, but for low-resource languages we need to be more creative.

SpecAugment (masking frequency bands and time steps in the spectrogram) is non-negotiable. It's free, it doesn't require external data, and it consistently improves WER by 5-10 percent relative. We also found that pitch shifting by plus or minus 10 percent provides good augmentation, especially for tonal languages where pitch carries linguistic meaning.

The most impactful augmentation technique we've used is noise mixing with real-world background audio from the target deployment environment. If the system will be used in market settings, we record 10 hours of market ambient noise and mix it with clean speech at random SNR levels between 5 and 20 dB. This domain-specific noise augmentation improved our real-world accuracy by 15 percent.

Language Model Integration

An external language model can compensate for acoustic model weaknesses by constraining the decoder output to likely word sequences. This is especially valuable for low-resource languages where the acoustic model character-level predictions are noisy.

We train a 5-gram KenLM language model on whatever text data we can gather: Bible translations, news websites, social media posts, government documents.

import kenlm
from pyctcdecode import build_ctcdecoder

vocab = list(processor.tokenizer.get_vocab().keys())
lm_path = "models/yoruba_5gram.arpa"

decoder = build_ctcdecoder(
 labels=vocab,
 kenlm_model_path=lm_path,
 alpha=0.5,
 beta=1.0,
)

logits = model.generate_logits(audio_features)
text = decoder.decode(logits.cpu().numpy())

Handling Tonal Languages

Many African and Asian languages are tonal. The pitch pattern of a syllable changes its meaning. Mandarin has 4 tones; Yoruba has 3; Igbo has 2 plus downstep. Standard speech recognition architectures don't explicitly model tone, and for tonal languages this is a significant source of errors.

We've experimented with two approaches. First, adding a parallel pitch estimation head to the encoder that predicts f0 contours alongside the acoustic features. Second, using the CTC loss with tone-marked characters so the model implicitly learns tone from the orthography. The second approach is simpler and works surprisingly well when you have consistent tone marking in your training data.

Evaluation Beyond WER

Word error rate is the standard metric, but for low-resource language ASR it doesn't tell the whole story. We track character error rate (CER) which captures partial word correctness that WER misses, and for tonal languages we separately compute tone accuracy by comparing predicted and reference tone patterns.

We also measure semantic accuracy, using a bilingual speaker to judge whether the recognition output preserves the intended meaning, even if the exact words differ. For morphologically rich languages where surface form variation is high, this pragmatic metric correlates better with actual usability than WER.

For a related perspective, see Attention Mechanism Variants Beyond Standard Self-Attention.

Our current best system for Yoruba achieves 18 percent WER on clean read speech and 28 percent WER on conversational speech. That isn't English-level performance, but it's usable for applications like voice-activated menus, transcription assistance, and voice search. Two years ago we were at 40 percent WER, so the trajectory is encouraging.

Self-Supervised Pretraining for Low-Resource ASR

When you don't have enough labeled speech data, you might still have unlabeled audio. Radio broadcasts, podcasts, YouTube videos in the target language, even if not transcribed, these can be used for self-supervised pretraining to learn acoustic representations.

The wav2vec 2.0 pretraining approach works by masking portions of the audio input and training the model to predict the masked segments from the surrounding context. This is analogous to masked language modeling in BERT, but applied to audio waveforms. The pretrained encoder learns general acoustic features that transfer well to downstream ASR tasks.

We collected about 500 hours of unlabeled Yoruba audio from radio stations and podcasts, and used it to continue pretraining the XLSR-53 encoder. This additional pretraining improved downstream WER by 4 percent absolute compared to using the off-the-shelf XLSR model. The improvement was particularly noticeable for conversational speech, which is more varied than the read speech used in the supervised fine-tuning data.

Deployment Considerations

Deploying ASR for low-resource languages has practical challenges beyond model accuracy. The users are often in environments with limited internet connectivity and older hardware. Our production system needs to work offline on a mid-range Android phone, which rules out large Whisper models.

We use knowledge distillation to compress the fine-tuned Whisper-large model into a much smaller student model. The student architecture is based on Conformer with about 30M parameters (compared to 1.5B for Whisper-large). The distilled model runs in real time on a Snapdragon 680 processor with about 250ms latency for a 5-second audio segment.

For a related perspective, see Experiment Tracking Infrastructure: MLflow vs Weights and Bi.

The accuracy trade-off from distillation is significant but manageable. The distilled model WER is about 5 percent absolute worse than the full model on clean speech, and 8 percent worse on noisy speech. For our target application (voice-based agricultural information service), this level of accuracy is acceptable, especially with a confirmation step where the system reads back its transcription and asks the user to confirm.

Community and Ethical Dimensions

Building ASR for low-resource languages isn't just a technical project. The communities who speak these languages have legitimate concerns about how their language data is collected, stored, and used. We work directly with language community organizations and follow several principles.

All data collection is opt-in with informed consent in the speaker own language. Speakers can withdraw their data at any time. The trained models are released with open-source licenses so the community benefits from the technology we build with their data. We don't collect speech data from children or from contexts where consent can't be freely given.

We also involve native speakers in the evaluation process, not just as annotators but as judges of whether the system is respectful of dialectal variation, tone patterns, and cultural context. A system that standardizes away dialectal variation in the name of accuracy is doing a disservice to the language community.

These considerations add time and cost to the project, but they're non-negotiable. The alternative, extracting language data without community engagement, creates tools that may not serve the community needs and erodes trust in future research collaborations.

Speech Recognition Pipeline Optimization for Low-Resource Languages