Sensors Do Not Lie, But They Do Not Tell the Whole Truth
Industrial sensor data is a different beast from the datasets you see in anomaly detection tutorials. You aren't looking for obvious outliers in clean, low-dimensional data. You're dealing with thousands of correlated sensors, measurement noise that changes with operating conditions, and a ground truth label set that's woefully incomplete because nobody documented the exact timestamp of that bearing failure six months ago.
I've spent the last four years building anomaly detection systems for manufacturing environments, specifically semiconductor fabs and chemical processing plants. The technical challenges are real, and most off-the-shelf solutions fail in ways that aren't immediately obvious.
Dimensionality: The Real Challenge
A typical semiconductor fab tool has 200-500 sensors reporting at 1-10 Hz. That's potentially 5,000 readings per second per tool, and a fab might have 300 tools. The naive approach of feeding all sensor values into a single anomaly detection model doesn't work.
What works better is a hierarchical approach. First, group sensors by physical subsystem (vacuum, thermal, gas delivery, RF power) based on domain knowledge. Then build separate models for each subsystem. Finally, aggregate the subsystem-level anomaly scores into a tool-level score.
import numpy as np
from sklearn.preprocessing import StandardScaler
class SubsystemModel:
def __init__(self, sensor_ids, window_size=60):
self.sensor_ids = sensor_ids
self.window_size = window_size
self.scaler = StandardScaler()
self.pca = None
self.threshold = None
def fit(self, data):
windowed = self._create_windows(data)
scaled = self.scaler.fit_transform(windowed)
from sklearn.decomposition import PCA
n_components = min(len(self.sensor_ids), 10)
self.pca = PCA(n_components=n_components)
transformed = self.pca.fit_transform(scaled)
reconstructed = self.pca.inverse_transform(transformed)
errors = np.sum((scaled - reconstructed) ** 2, axis=1)
self.threshold = np.percentile(errors, 99.5)
return self
def score(self, data):
windowed = self._create_windows(data)
scaled = self.scaler.transform(windowed)
transformed = self.pca.transform(scaled)
reconstructed = self.pca.inverse_transform(transformed)
errors = np.sum((scaled - reconstructed) ** 2, axis=1)
return errors / self.threshold
def _create_windows(self, data):
windows = []
for i in range(self.window_size, len(data)):
window = data[i - self.window_size:i, :]
features = np.concatenate([
window.mean(axis=0),
window.std(axis=0),
window[-1] - window[0],
])
windows.append(features)
return np.array(windows)Correlation-Based Features
One pattern that's consistently useful in industrial data: monitoring the correlation structure between sensors, not just the individual sensor values. When a machine starts degrading, it's often the relationships between sensors that change before any individual sensor goes out of range.
For example, in a chemical vapor deposition tool, the chamber pressure and gas flow rate have a stable linear relationship during normal operation. When the pump starts degrading, the pressure-flow relationship shifts before either sensor individually triggers an alarm.
Handling Operating Regime Changes
This is the part that trips up most anomaly detection approaches in industrial settings. A semiconductor tool doesn't operate in one steady state. It runs through a sequence of process steps (purge, deposit, clean, idle), each with completely different sensor signatures. What's normal during deposition is anomalous during idle, and vice versa.
We covered a related topic in Data Quality Frameworks for Machine Learning: Great Expectat.
You need separate models for each operating regime, and you need a reliable regime classifier to route incoming data to the right model. We use a hidden Markov model for regime identification because the transitions between states are sequential and the sensor signatures are distinct enough for clean classification.
The tricky part is the transition periods between regimes. Sensor values are changing rapidly and don't match any stable regime profile. We handle this by defining explicit transition states in the HMM and using wider thresholds during transitions. A 30-second transition window typically captures the regime change without generating false alarms.
False Alarm Management
In an industrial context, a false alarm doesn't just annoy someone. It stops production, triggers an investigation, and costs real money. At one fab I worked with, each false alarm cost an estimated 15,000 dollars in lost production time. The acceptable false alarm rate was one per tool per month, which means your false positive rate needs to be extremely low.
We use a multi-gate approach: an anomaly needs to persist for at least 3 consecutive evaluation windows (about 3 minutes), it needs to exceed the threshold by at least 2x, and it needs to be confirmed by at least 2 out of 5 subsystem models. This dramatically reduces false positives while adding only a small delay to true detection.
The real secret, though, is working closely with the process engineers during development. They know which sensor patterns are weird but normal. Encoding this domain knowledge into the model via explicit exclusion rules or regime-specific thresholds eliminates more false alarms than any algorithmic improvement.
Deep Learning Approaches for Industrial Anomaly Detection
While PCA-based reconstruction error works well for many industrial use cases, there are situations where deep learning approaches provide meaningful improvements. The question is when the additional complexity is justified.
See also: Weight Initialization Schemes and Their Impact on Convergenc.
Autoencoders (specifically variational autoencoders) are useful when the sensor relationships are non-linear and the number of sensors per subsystem is large (50 plus). The VAE learns a compressed representation of normal operating patterns, and deviations in the latent space or reconstruction error signal anomalies. We've found that a relatively small VAE (3 layers, 32-dimensional latent space) outperforms PCA when the subsystem has more than 30 correlated sensors.
Temporal convolutional networks (TCN) are useful when the anomaly signature spans a longer time horizon. PCA with windowed features captures patterns within the window, but some degradation modes unfold over hours or days. A TCN with dilated convolutions can capture patterns across 1000 plus timesteps while keeping the model computationally tractable. We deploy TCNs for slow-degradation monitoring (pump wear, heater element aging) where the signal evolves gradually.
The decision tree is straightforward: if your anomalies are point anomalies (sudden spikes or drops), PCA reconstruction works fine. If they're contextual anomalies (normal values but wrong for the current context), you need the regime-aware approach. If they're collective anomalies (a long sequence that's individually normal but collectively abnormal), you need a temporal model like TCN.
Deployment and Monitoring Architecture
The production deployment architecture for industrial anomaly detection has specific requirements that differ from typical ML serving.
Latency tolerance is actually generous compared to web services. Industrial control systems make decisions on time scales of seconds to minutes, so a 500ms inference latency is perfectly fine. What matters more is throughput: processing 5,000 sensor readings per second continuously without falling behind the data stream.
We deploy the models on edge hardware co-located with the industrial equipment, typically NVIDIA Jetson devices or small industrial PCs with Intel CPUs. The edge deployment avoids network latency to a cloud inference endpoint and ensures the system keeps working during network outages, which are common in factory environments.
For a related perspective, see Image Segmentation Pipelines for Medical Imaging: U-Net Vari.
The anomaly scores are streamed to a central monitoring dashboard and also to the local equipment control system. The control system can trigger automated responses (slow down a process, switch to a backup pump) for high-confidence anomalies, while medium-confidence anomalies just generate alerts for human review.
Model updates are pushed to edge devices on a monthly schedule. Each update goes through a shadow mode period where the new model runs alongside the old one and its predictions are compared. If the new model produces significantly more false alarms than the old one, the update is rolled back automatically.
Data Collection and Labeling Challenges
The hardest part of building industrial anomaly detection isn't the modeling. It's getting labeled data. Anomalies are rare by definition, and historical records of equipment failures are incomplete, imprecise, and scattered across maintenance logs, operator notes, and email threads.
We've developed a semi-automated labeling workflow. First, we collect all available failure records and align them to sensor timestamps (this alone takes weeks of work for a new facility). Second, we identify time windows around each failure event and mark them as potentially anomalous. Third, we work with process engineers to refine the windows, identifying the earliest detectable anomaly signature for each failure mode.
This produces a small but high-quality labeled dataset that we use for model evaluation, not for model training. The models themselves are trained only on normal data (one-class learning), which sidesteps the labeling problem for training. The labeled anomalies are used to measure detection performance and set alert thresholds. It's a pragmatic compromise that works well in practice.