Catching Bad Data Before It Trains Bad Models
Data quality for ML is different from data quality for analytics. An analytics dashboard can tolerate a few missing values — the averages still look right. An ML model trained on data with systematic quality issues will learn those issues as patterns and reproduce them in production predictions. I've debugged enough "model performance degradation" incidents to know that the root cause is usually data quality, not model quality.
Great Expectations and Deequ are the two frameworks I've used most for ML data validation. They approach the problem differently — Great Expectations is Python-native with a rich expectation library, while Deequ is Spark-native with a statistical testing focus — but both integrate into ML pipelines as quality gates that catch problems before they reach the training step.
Great Expectations for Feature Validation
Great Expectations works by defining "expectations" — assertions about your data — and running them against datasets. For ML pipelines, the expectations typically cover feature distributions, null rates, value ranges, and cross-feature relationships.
import great_expectations as gx
context = gx.get_context()
suite = context.add_expectation_suite("fraud_features_v3")
# Schema expectations
suite.add_expectation(
gx.expectations.ExpectTableColumnsToMatchSet(
column_set=[
"user_id", "txn_count_1h", "txn_count_24h",
"avg_amount_1h", "max_amount_24h", "merchant_diversity",
"country_count_1h", "time_since_last_txn_seconds",
]
)
)
# Distribution expectations based on training data stats
suite.add_expectation(
gx.expectations.ExpectColumnMeanToBeBetween(
column="txn_count_1h", min_value=1.2, max_value=4.8
)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column="avg_amount_1h", min_value=0.01, max_value=50000,
mostly=0.999
)
)
# Null rate expectations
suite.add_expectation(
gx.expectations.ExpectColumnValuesToNotBeNull(
column="user_id"
)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToNotBeNull(
column="txn_count_1h", mostly=0.98
)
)
# Cross-feature consistency
suite.add_expectation(
gx.expectations.ExpectColumnPairValuesAToBeGreaterThanB(
column_A="txn_count_24h",
column_B="txn_count_1h",
or_equal=True
)
)The mostly parameter is critical for ML data. Strict validation (no nulls, no outliers) will fail on real production data that has legitimate edge cases. Setting mostly=0.98 means you tolerate up to 2% violations, which catches systematic issues while ignoring noise.
Deequ for Statistical Data Quality
Deequ runs on Spark and is built around the idea that data quality checks should be statistical tests, not just range checks. It computes data profiles and can detect anomalies by comparing current profiles against historical baselines.
# Deequ verification suite (PySpark)
from pydeequ.checks import Check, CheckLevel
from pydeequ.verification import VerificationSuite, VerificationResult
check = Check(spark, CheckLevel.Warning, "fraud_features_quality")
check = (check
.hasSize(lambda x: x > 100_000, "Dataset must have >100k rows")
.isComplete("user_id")
.isComplete("txn_count_1h")
.hasCompleteness("avg_amount_1h", lambda x: x >= 0.98)
.isNonNegative("txn_count_1h")
.isNonNegative("txn_count_24h")
.hasMax("avg_amount_1h", lambda x: x <= 100_000)
.hasMean("txn_count_1h", lambda x: 0.5 <= x <= 10.0)
.hasStandardDeviation("avg_amount_1h", lambda x: x > 0)
.satisfies(
"txn_count_24h >= txn_count_1h",
"24h count must be >= 1h count",
lambda x: x >= 0.99
)
)
result = (VerificationSuite(spark)
.onData(features_df)
.addCheck(check)
.run()
)
result_df = VerificationResult.checkResultsAsDataFrame(spark, result)
failures = result_df.filter("constraint_status = 'Failure'")
if failures.count() > 0:
failures.show(truncate=False)
raise DataQualityError(f"{failures.count()} quality checks failed")Deequ's anomaly detection is the feature that sets it apart. You define a metric repository that stores historical data profiles, and Deequ automatically flags metrics that deviate from historical patterns. This catches drift that static thresholds miss — like a feature whose mean slowly shifts over months in a way that doesn't violate any individual check but represents significant distribution change.
Related reading: Experiment Tracking Infrastructure: MLflow vs Weights and Bi.
Integration Into ML Pipelines
Data quality checks belong at three points in the ML pipeline. Before feature engineering (validate raw data), after feature engineering (validate computed features), and before model training (validate the final training dataset).
The pre-engineering check catches upstream data issues — missing columns, changed schemas, unexpected nulls from a broken ETL job. The post-engineering check catches feature computation bugs — a new feature that produces NaN for certain inputs, a join that silently drops rows. The pre-training check validates the final dataset against the expected schema and distribution that the model was designed for.
Each checkpoint has different severity levels. A pre-engineering failure is a warning — investigate but don't necessarily block the pipeline, because the issue might not affect the features you compute. A pre-training failure is critical — if the training data doesn't match expectations, the resulting model can't be trusted.
Building a Quality Baseline
Both frameworks need a baseline to compare against. For a new pipeline, generate the baseline from a known-good dataset — typically the data used to train the current production model. Store the baseline statistics (means, standard deviations, null rates, value distributions) alongside the model in the registry.
The baseline should be updated periodically, but not automatically. Automatic baseline updates mask gradual drift by treating each day's data as "normal." Instead, update the baseline when you retrain the model on fresh data and verify that the model's performance is acceptable. The new training data's statistics become the new baseline for subsequent pipeline runs.
For a related perspective, see Prompt Engineering as Software Engineering: Version Control .
Choosing Between Them
If your pipeline is already on Spark, Deequ integrates with zero friction and its anomaly detection is genuinely useful for catching slow drift. If your pipeline is Python-native (Pandas, Polars, or similar), Great Expectations is the better fit — it runs anywhere Python runs, has a larger community, and its data docs feature generates human-readable quality reports that non-engineers can understand.
For large-scale ML pipelines, we often use both: Deequ for the Spark-based feature engineering stage (where data volumes make Pandas impractical) and Great Expectations for the final training dataset validation (where the data fits in memory and the richer expectation library is useful). The quality gate logic is the same — check against baselines, fail on critical violations, warn on minor ones — just implemented in different frameworks at different pipeline stages.
Whatever you choose, the important thing is having data quality checks at all. I've seen more model failures caused by bad training data than by bad model architecture. A well-trained model on bad data produces confidently wrong predictions. Data quality frameworks aren't glamorous, but they're the difference between a model that works and a model that works until it doesn't.
Custom Expectations for ML-Specific Checks
The built-in expectations in both frameworks cover general data quality well, but ML pipelines have domain-specific quality requirements that need custom checks. Feature correlation stability, class balance validation, and temporal consistency are examples that don't map to standard data quality patterns.
# Custom Great Expectations for ML data
class ExpectFeatureCorrelationStable(BatchExpectation):
"""Check that feature correlations match baseline within tolerance."""
metric_dependencies = ("table.custom.feature_correlations",)
success_keys = ("baseline_correlations", "tolerance")
def _validate(self, metrics, runtime_configuration=None, **kwargs):
current_corr = metrics["table.custom.feature_correlations"]
baseline = self.configuration["kwargs"]["baseline_correlations"]
tolerance = self.configuration["kwargs"].get("tolerance", 0.15)
violations = []
for pair, baseline_val in baseline.items():
current_val = current_corr.get(pair, 0.0)
diff = abs(current_val - baseline_val)
if diff > tolerance:
violations.append({
"pair": pair,
"baseline": baseline_val,
"current": current_val,
"diff": diff,
})
return {
"success": len(violations) == 0,
"result": {"violations": violations}
}Feature correlation stability catches a subtle failure mode: upstream changes that don't affect individual feature distributions but alter the relationships between features. A feature engineering bug might swap two column positions, leaving individual statistics unchanged but completely breaking the model because it learned a different mapping between features and the target.
This connects to the ideas in Time Series Forecasting with Temporal Fusion Transformers.
Data Quality in Production Serving
Data quality validation isn't just for training pipelines. The features that arrive at inference time can also have quality issues — missing values from a failed feature store lookup, stale values from a cache that hasn't refreshed, or corrupted values from a buggy feature computation.
We run a lightweight validation step in the serving path that checks each feature vector against basic quality constraints before sending it to the model. Missing critical features trigger a fallback to a simpler model or a default prediction. Out-of-range values get clipped to the training distribution bounds. This adds about 1-2 milliseconds to the serving latency but prevents the model from producing wildly incorrect predictions when the input is garbage.
The serving-side quality checks also feed into our monitoring system. High rates of missing features or out-of-range values are early indicators of upstream data issues that will eventually affect model quality. We've caught data pipeline failures within minutes by monitoring the serving-side quality check pass rate, well before the downstream model metrics showed any degradation. It's the cheapest early warning system we've built.
Handling Schema Evolution
Feature schemas change over time. New features get added, old features get deprecated, value ranges shift as the product evolves. The data quality framework needs to handle schema evolution without requiring a complete rewrite of every expectation suite.
We version our expectation suites alongside the feature schema. When a feature is added, we add new expectations for it. When a feature is deprecated, we remove its expectations and add a check that verifies it's no longer present (to catch pipeline configurations that still produce the old feature). When value ranges shift, we update the baseline statistics and adjust the tolerance bounds.
The versioned expectation suites are stored in the model registry alongside the model they validate. This means every model knows which data quality checks apply to its training data, and the training pipeline automatically uses the correct expectation suite for the model version being trained. When we roll back to a previous model version, the data quality checks roll back too.