OCR on Clean Documents Is Solved. Yours Aren't Clean.
Tesseract 5 with LSTM mode handles crisp, high-resolution text at 95%+ character accuracy. Google's Document AI and AWS Textract push that above 99% for standard business documents. If your documents look like they came off a laser printer and through a flatbed scanner, you don't need a custom pipeline. Use an API.
But if your documents look like mine did — faded receipts from thermal printers, handwritten forms that were faxed twice, construction blueprints photographed at an angle with a phone camera — the off-the-shelf solutions fall apart. That's where pipeline design matters.
Preprocessing Is 80% of the Work
Most OCR accuracy improvements come from preprocessing, not from the recognition model. A well-preprocessed image fed into Tesseract often outperforms a raw image fed into a modern neural OCR model.
import cv2
import numpy as np
def preprocess_noisy_document(image):
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# Deskew using Hough transform
edges = cv2.Canny(gray, 50, 150, apertureSize=3)
lines = cv2.HoughLinesP(edges, 1, np.pi/180, 100,
minLineLength=100, maxLineGap=10)
if lines isn't None:
angles = [np.arctan2(l[0][3]-l[0][1], l[0][2]-l[0][0]) for l in lines]
median_angle = np.median(angles) * 180 / np.pi
if abs(median_angle) > 0.5:
h, w = gray.shape
center = (w // 2, h // 2)
M = cv2.getRotationMatrix2D(center, median_angle, 1.0)
gray = cv2.warpAffine(gray, M, (w, h), borderValue=255)
# Adaptive thresholding
binary = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY, 21, 10)
# Morphological cleanup
kernel = np.ones((2, 2), np.uint8)
cleaned = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, kernel)
cleaned = cv2.morphologyEx(cleaned, cv2.MORPH_OPEN, kernel)
return cleanedDeskewing Matters More Than You Think
A 2-degree rotation sounds negligible. It isn't. On a full page of text, 2 degrees of skew causes line segmentation to fail — the horizontal projection profile that separates text lines gets smeared, and words from adjacent lines blend together. Tesseract's internal deskew is mediocre. We always deskew as a preprocessing step.
For heavily skewed phone photos, perspective correction (homography estimation from document corners) is necessary before rotation correction. OpenCV's getPerspectiveTransform handles the math; the hard part is reliably detecting document corners when the background is cluttered.
Text Detection Before Recognition
Older OCR pipelines assume the input is a full page of text. Modern documents don't work that way. A construction blueprint has text scattered across the drawing at various angles and sizes. A retail receipt has columns that don't align. Running Tesseract on the full image gives garbage because it can't figure out the reading order.
We use a two-stage approach: text detection first, then recognition on each detected region. CRAFT works well for scene text and noisy documents. For structured documents, a layout analysis model like LayoutLMv3 segments the page into text regions, tables, figures, and headers before OCR runs on each region independently.
See also: Generative Adversarial Networks for Synthetic Training Data .
This separation also lets you apply different preprocessing to different regions. A table might need column-aligned binarization while a handwritten annotation needs different thresholding parameters.
Post-Processing and Confidence Calibration
Raw OCR output contains errors. Always. The question is how to catch and fix them automatically. We use a three-layer post-processing stack.
First, dictionary-based correction for domain-specific terms. If you're processing medical records, a custom medical dictionary catches corrections that generic spell-checkers miss. Second, rule-based validation for structured fields — dates, amounts, reference numbers. A regex that matches expected formats catches more errors than language-model correction for these fields. Third, a confidence threshold that routes low-confidence regions to human review.
That confidence threshold is critical. Tesseract provides per-character confidence scores. Characters below 70% confidence are wrong roughly 40% of the time. Routing these to human review catches most errors while keeping review volume manageable — typically 5-15% of extracted text fields need human verification.
The full pipeline — preprocessing, detection, recognition, post-processing, human review routing — brings noisy document accuracy from the 60-70% range (raw Tesseract on unprocessed images) to 95%+ on our production datasets. Most improvement comes from preprocessing and post-processing, not from a fancier recognition model.
Handling Tables and Structured Data
Table extraction is where most OCR pipelines break down completely. A table has both spatial structure (rows, columns, cells) and semantic structure (headers, data, totals). Extracting text cell-by-cell without understanding the table structure produces unusable garbage — you get a stream of numbers and labels with no way to associate them.
Related reading: Text Classification Pipeline Architecture for Multi-Label Pr.
We use a dedicated table detection model (TableTransformer or DETR-based table detector) to locate tables in the document, then a cell extraction model to identify rows and columns. The OCR runs on each cell individually, and the spatial relationships are reconstructed from the cell coordinates:
from transformers import TableTransformerForObjectDetection
import torch
def extract_table(image, model, feature_extractor):
inputs = feature_extractor(images=image, return_tensors="pt")
outputs = model(**inputs)
# Post-process to get table and cell bounding boxes
target_sizes = torch.tensor([image.size[::-1]])
results = feature_extractor.post_process_object_detection(
outputs, threshold=0.7, target_sizes=target_sizes
)[0]
cells = []
for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
label_name = model.config.id2label[label.item()]
if label_name in ("table column", "table row", "table cell"):
cells.append({"type": label_name, "box": box.tolist(), "score": score.item()})
return cellsMerging OCR results with table structure requires matching each text region to the correct cell based on overlap. Ambiguous cases — text that spans two cells, or cells with no text — need heuristic handling. We use a scoring function that considers both overlap area and distance to cell center.
Handwriting Recognition
Handwritten text recognition (HTR) is a separate problem from printed OCR. Tesseract can't read handwriting. You need a dedicated model: TrOCR (transformer-based) works well for line-level recognition. For full-page handwritten documents, you first need text line detection, then TrOCR on each detected line.
In my experience, handwriting recognition accuracy tops out around 85-90% even with modern models, compared to 97%+ for printed text. The variance between writers is enormous — neat handwriting gets 95% accuracy while messy handwriting might be 70%. For production systems, we always route handwritten fields through human review with the model's predictions as suggested values. This cuts human processing time by roughly 40% compared to typing from scratch while maintaining 99%+ accuracy.
Multi-Language Document Processing
Production OCR systems often need to handle multiple languages — an English document with a Japanese company name, or a multilingual form with labels in one language and responses in another. Tesseract supports this with language packs, but the accuracy drops when multiple languages appear on the same page.
Our approach is language detection before OCR. We run a lightweight text classification model on each detected text region to identify the language, then apply the appropriate OCR model and dictionary. This per-region language detection avoids the ambiguity of whole-document language classification and lets us tune OCR parameters per language — CJK characters need different binarization thresholds than Latin text.
For a related perspective, see Data Versioning for Reproducible ML Experiments: DVC and Lak.
Document Classification Before OCR
When processing a mixed stack of documents — invoices, contracts, receipts, forms — classifying each document before running OCR lets you apply document-type-specific preprocessing and post-processing. An invoice has different expected fields than a medical form, and knowing the type lets you validate OCR output against the expected structure.
We use a lightweight CNN classifier (EfficientNet-B0, fine-tuned on 10,000 document images across 15 categories) as the first pipeline stage. It processes a thumbnail of the full page in 5ms and routes the document to the appropriate OCR configuration. The classifier achieves 97% accuracy on our document types, and the few misclassifications are caught by downstream validation rules.
import torch
from torchvision import transforms, models
class DocumentClassifier:
def __init__(self, model_path, class_names):
self.model = models.efficientnet_b0(num_classes=len(class_names))
self.model.load_state_dict(torch.load(model_path))
self.model.eval()
self.class_names = class_names
self.transform = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],
[0.229, 0.224, 0.225]),
])
def classify(self, image):
tensor = self.transform(image).unsqueeze(0)
with torch.no_grad():
logits = self.model(tensor)
probs = torch.softmax(logits, dim=1)
top_prob, top_idx = probs.max(dim=1)
return self.class_names[top_idx.item()], top_prob.item()End-to-End Document Understanding
The newest generation of document AI models — LayoutLMv3, Donut, and Pix2Struct — combine OCR and understanding in a single model. Instead of a pipeline of text detection, OCR, and then NLP for extraction, these models take the document image directly and output structured data.
Donut in particular is interesting because it's entirely OCR-free — it reads the image directly and generates the structured output as text. For simple extraction tasks (invoice total, receipt date), it works surprisingly well. For complex documents with dense text, the traditional pipeline still outperforms because dedicated OCR models have higher character-level accuracy than the end-to-end visual models.
We use a hybrid approach: Donut for simple documents with clear visual structure (receipts, ID cards), and the traditional pipeline for complex documents (contracts, technical reports). The routing decision is made by the document classifier, so the full system is: classify the document, route to the appropriate extraction pipeline, validate the output, and queue low-confidence results for human review.
This hybrid architecture handles about 85% of documents fully automatically. The remaining 15% — documents with unusual layouts, heavy damage, or rare document types — go to human operators with the model's partial extraction as a starting point. The operators correct errors and complete missing fields in about 60% less time than processing from scratch, which is the real productivity gain of the system.