What is Metrics That Matter?

Accuracy is meaningless for inspection. If 0.1% of parts are defective, a model that always predicts good achieves 99.9% accuracy while catching zero defects. The metrics that actually matter: The tradeoff between escape rate and false reject rate is the core engineering decision. Lowering the detection threshold catches more defects but increases.

Image Quality Assessment Metrics

Automated Visual Inspection Without the Hype

Automated visual inspection is one of those fields where the gap between vendor claims and production reality is enormous. Vendors promise 99.99% defect detection while glossing over the conditions required. In practice, getting reliable inspection working in a factory requires careful engineering of the entire imaging system, not just the ML model.

Image Quality Is Your Input Signal

Before thinking about neural networks, think about image quality. The inspection camera produces your raw signal, and no model can recover information that isn't in the image. I wasted months trying to train models on bad images before learning this lesson.

Lighting is the single biggest factor. Diffuse lighting reveals surface texture defects. Structured lighting (stripes or patterns projected onto the surface) reveals geometric defects like dents and warps. Backlighting creates silhouettes for dimensional measurement. The lighting geometry should be designed for the specific defect types you're catching.

Resolution matters but not the way people think. You don't need the highest resolution camera — you need sufficient resolution for the smallest defect. A 50 micrometer scratch on a 100mm part requires at least 50 micrometer per pixel spatial resolution. A 5-megapixel camera with an appropriate lens handles this. A 20-megapixel camera gives 4x more data to process with no improvement for that defect size.

Reference-Based vs Reference-Free Assessment

Reference-based inspection compares a test image against a known good part image. The difference is the defect signal:

import cv2
import numpy as np
from skimage.metrics import structural_similarity

def reference_comparison(test_img, reference_img, threshold=0.85):
 gray_test = cv2.cvtColor(test_img, cv2.COLOR_BGR2GRAY)
 gray_ref = cv2.cvtColor(reference_img, cv2.COLOR_BGR2GRAY)
 score, diff_map = structural_similarity(gray_ref, gray_test, full=True, win_size=11)
 defect_mask = (diff_map < threshold).astype(np.uint8) * 255
 kernel = np.ones((5, 5), np.uint8)
 defect_mask = cv2.morphologyEx(defect_mask, cv2.MORPH_OPEN, kernel)
 return score, defect_mask

Reference-free inspection uses anomaly detection — the model learns what normal looks like and flags anything unusual. Autoencoders and normalizing flows work well here. No per-product reference images needed, but the model must see enough variation in good parts during training.

Metrics That Matter

Accuracy is meaningless for inspection. If 0.1% of parts are defective, a model that always predicts good achieves 99.9% accuracy while catching zero defects. The metrics that actually matter:

Escape rate — defective parts that pass inspection. For safety-critical applications, target below 0.01%.
False reject rate — good parts incorrectly flagged. Each false reject costs the handling process plus rework or scrapping. Typically target 1-5%.
Throughput — parts inspected per minute. Must match or exceed line speed.
Latency — time from capture to pass/reject decision. For inline inspection, 100-500ms depending on the reject mechanism.

The tradeoff between escape rate and false reject rate is the core engineering decision. Lowering the detection threshold catches more defects but increases false rejects. The optimal threshold depends on the cost of a missed defect versus the cost of a false reject.

Building the Pipeline

A production inspection system isn't a model — it's a pipeline. Image capture triggers on part detection (typically a sensor or encoder signal). The image gets preprocessed (background subtraction, ROI extraction). The inspection model runs. Results get logged. The reject mechanism fires if needed.

We use a modular async pipeline where each stage runs independently. The capture stage fills a ring buffer. Preprocessing pulls from the buffer and queues for inference. The model stage runs on a dedicated GPU thread. This decoupling means slow inference on one image doesn't block capture of the next — you just need enough buffer depth to absorb latency spikes.

Deployment in a factory means dealing with industrial Ethernet, PLC integration, vibration, dust, and temperature extremes. The ML engineering is maybe 30% of the effort. The other 70% is systems integration, reliability engineering, and working with the production team on edge cases that no training dataset covers.

Anomaly Detection Approaches for Inspection

For products with high variability in appearance (natural materials like wood, stone, fabric), reference-based comparison doesn't work. Each piece looks different. Anomaly detection models learn the distribution of normal appearance and flag anything outside it.

Autoencoders are the simplest approach: train the network to reconstruct images of good parts. When a defective part passes through, the reconstruction error is high in the defect region. The reconstruction error map becomes the defect map. We threshold this map and apply connected component analysis to identify individual defects.

For a related perspective, see Apache Spark MLlib vs Distributed PyTorch for Large-Scale Fe.

import torch
import torch.nn as nn

class InspectionAutoencoder(nn.Module):
 def __init__(self):
 super().__init__()
 self.encoder = nn.Sequential(
 nn.Conv2d(3, 32, 4, 2, 1), nn.ReLU(),
 nn.Conv2d(32, 64, 4, 2, 1), nn.ReLU(),
 nn.Conv2d(64, 128, 4, 2, 1), nn.ReLU(),
 nn.Conv2d(128, 256, 4, 2, 1), nn.ReLU(),
 )
 self.decoder = nn.Sequential(
 nn.ConvTranspose2d(256, 128, 4, 2, 1), nn.ReLU(),
 nn.ConvTranspose2d(128, 64, 4, 2, 1), nn.ReLU(),
 nn.ConvTranspose2d(64, 32, 4, 2, 1), nn.ReLU(),
 nn.ConvTranspose2d(32, 3, 4, 2, 1), nn.Sigmoid(),
 )

 def forward(self, x):
 z = self.encoder(x)
 return self.decoder(z)

def compute_anomaly_score(model, image):
 recon = model(image)
 error = torch.abs(image - recon)
 score_map = error.mean(dim=1, keepdim=True)
 return score_map

PatchCore and other memory-bank approaches often outperform autoencoders. They store feature embeddings of normal patches and detect anomalies by measuring distance to the nearest stored embedding. The accuracy improvement is significant — PatchCore achieves 99%+ AUROC on the MVTec anomaly detection benchmark — but the inference cost is higher due to the nearest-neighbor search over the memory bank.

Lighting and Camera Stability

The most common false positive source in inspection systems isn't the model — it's lighting variation. A 5% change in illumination intensity from LED aging or power supply fluctuation changes the image enough to trigger false detects. We address this at three levels: hardware (regulated LED drivers, reference patches in the camera's field of view), software (per-image illumination normalization before inference), and operational (quarterly lighting calibration checks).

Camera stability is equally important. A vibration-induced shift of 2 pixels in the camera position changes the alignment with the reference image, creating false edge defects everywhere. Anti-vibration mounts, mechanical locking mechanisms for the camera position, and software-based alignment correction all help. We've found that spending $500 more on a rigid camera mounting bracket saves thousands in false reject costs over the system's lifetime.

Integration with Manufacturing Execution Systems

An inspection system that doesn't communicate with the production line is just an expensive monitor. Real deployment requires integration with the manufacturing execution system (MES) and programmable logic controllers (PLCs). Pass/fail signals go out via industrial Ethernet (EtherNet/IP or PROFINET) to trigger reject mechanisms. Defect data feeds into quality dashboards and SPC (statistical process control) charts.

We use OPC-UA as the communication protocol between the inspection software and the MES. It handles the structured data well — defect type, location, severity, image reference — and supports authentication and encryption for regulated industries. The integration typically takes 2-3 weeks of engineering time and requires close collaboration with the plant's automation team.

Handling Product Variability

Not all products are identical. Machined metal parts have dimensional tolerances. Natural materials like wood have grain patterns that vary piece to piece. Food products have color variation. The inspection system needs to distinguish "acceptable variation" from "defect," and the boundary between them is often subjective.

We handle this by maintaining a golden sample library — a curated set of 50-100 images representing the full range of acceptable variation. During deployment, the system compares the current part against the golden library using both the anomaly score and a nearest-neighbor distance. A part that looks unusual compared to the model's training data but similar to a golden sample passes inspection; a part that's unusual and distant from all golden samples fails.

This golden library approach also handles gradual process drift. When a new mold or tooling variation is introduced, adding examples to the golden library updates the acceptance criteria without retraining the model. This is operationally important — retraining takes hours and requires ML engineering time, while updating the golden library takes minutes and can be done by the production team.

Statistical Process Control Integration

Inspection data feeds directly into SPC charts. Defect rates per shift, defect type distribution, and defect location heatmaps give the production team early warning of process degradation before it reaches the reject threshold. A sudden increase in edge defects might indicate tool wear. A shift in defect location might indicate fixture misalignment.

We compute these statistics in real-time and display them on shop floor monitors. The visual feedback loop — operators see the defect trend moving toward the control limit — creates a powerful incentive for proactive maintenance. In one deployment, this feedback reduced the defect rate by 35% in the first month, not because the inspection system caught more defects, but because operators fixed problems earlier.

The inspection system's value isn't just catching defects — it's the data it generates about the production process. That data, analyzed over weeks and months, reveals patterns that human inspectors can't detect: subtle correlations between defect types and time of day, machine parameters, material batches, or environmental conditions. This process insight is often more valuable than the defect catching itself.

Image Quality Assessment Metrics for Automated Visual Inspection Systems