Registry Architecture That Does Not Collapse Under Real Teamwork
Model registries sound simple until three teams ship models on the same Tuesday afternoon and nobody can tell which version is running in production. I've watched this happen at two different companies, and the root cause was identical both times: the registry was an afterthought bolted onto an experiment tracker, not a first-class system designed for concurrent multi-team use.
A production model registry needs to answer four questions without ambiguity. Which model version is deployed where? Who approved it? What training data and code produced it? And can we roll back in under five minutes?
Storage Layer Design
The registry's storage backend determines almost everything about its scalability. Most teams start with a filesystem or S3 bucket. That works for ten models. At fifty models with five versions each, you'll want structured metadata separate from artifact storage.
Here's the pattern that has worked for our team:
-- Registry metadata schema (PostgreSQL)
CREATE TABLE model_versions (
model_name VARCHAR(128) NOT NULL,
version INTEGER NOT NULL,
stage VARCHAR(32) DEFAULT 'development',
artifact_uri TEXT NOT NULL,
run_id VARCHAR(64),
created_by VARCHAR(128),
created_at TIMESTAMP DEFAULT NOW(),
description TEXT,
tags JSONB DEFAULT '{}',
PRIMARY KEY (model_name, version)
);
CREATE TABLE model_deployments (
model_name VARCHAR(128) NOT NULL,
version INTEGER NOT NULL,
environment VARCHAR(64) NOT NULL,
deployed_at TIMESTAMP DEFAULT NOW(),
deployed_by VARCHAR(128),
config JSONB DEFAULT '{}',
UNIQUE (model_name, environment)
);Artifacts themselves go to object storage with content-addressable paths. We hash the model binary and use that as part of the path, so identical models never duplicate storage.
Versioning Strategies That Scale
Semantic versioning doesn't map cleanly to ML models. A major version bump in software means breaking API changes. For models, the breaking changes are subtle — a shift in feature importance, a different output distribution, or changed latency characteristics that downstream services aren't prepared for.
The approach I've seen work best combines auto-incrementing integers with mandatory metadata tags. Every version gets a sequential number, but promotion through stages (development to staging to production) requires human-readable tags that encode what changed.
# Promoting a model version with required metadata
import registry_client
client = registry_client.connect("postgres://registry:5432/models")
client.promote(
model_name="fraud-detector",
version=47,
target_stage="staging",
metadata={
"training_data_hash": "sha256:a1b2c3d4",
"feature_schema_version": "3.2",
"expected_latency_p99_ms": 45,
"approval_ticket": "ML-2847",
"champion_version": 43,
}
)That champion_version field matters more than people expect. It creates an explicit lineage — you always know what came before, which makes debugging regressions straightforward instead of archaeological.
For a related perspective, see Generative Adversarial Networks for Synthetic Training Data .
Multi-Team Access Patterns
When five teams share a registry, namespace collisions become a real problem within weeks. The fraud team's "classifier_v2" and the recommendations team's "classifier_v2" are completely different models that look identical in flat listings.
We settled on a three-level namespace: team/project/model_name. It mirrors how most organizations actually think about ownership. The team level controls who can write; the project level groups related models; the model name identifies what it does.
Access control sits at the team level. Each team owns their namespace and can grant read access to other teams. Cross-team model sharing happens through explicit imports, not by reaching into another team's namespace directly. That might sound bureaucratic, but it prevents the scenario where Team A refactors their model and Team B's pipeline breaks because they were reading from Team A's staging slot without anyone knowing.
Artifact Lineage and Reproducibility
Every model artifact needs to link back to three things: the training code commit, the data snapshot identifier, and the environment configuration (hardware, library versions, random seeds). Without all three, reproducibility is a fantasy.
# Lineage metadata captured at registration time
lineage = {
"code": {
"repo": "github.com/org/ml-models",
"commit": "8f3a2e1",
"entrypoint": "training/fraud/train.py",
},
"data": {
"training_set": "s3://data/fraud/train_2024q3.parquet",
"data_hash": "sha256:e5f6g7h8",
"row_count": 2847291,
"feature_count": 147,
},
"environment": {
"python": "3.11.5",
"pytorch": "2.1.0",
"cuda": "12.1",
"gpu_type": "A100-80GB",
"training_hours": 4.2,
},
"metrics": {
"auc_roc": 0.9847,
"precision_at_1pct_fpr": 0.72,
"latency_p50_ms": 12,
"latency_p99_ms": 38,
}
}
client.register(
model_name="fraud/transaction-scoring/detector",
artifact_path="./model_checkpoint.pt",
lineage=lineage
)The metrics block isn't just for tracking improvement over time. It's the contract between the model and the serving infrastructure. If p99 latency jumps from 38ms to 120ms between versions, the deployment pipeline should flag that before promotion.
Stage Gates and Promotion Workflows
Manual promotion is fine for two models. At twenty, it's unsustainable. Automated stage gates enforce quality bars without requiring a human to check dashboards.
Our gate configuration works like this: development to staging requires passing unit tests and a basic performance check against the champion. Staging to production requires a seven-day shadow deployment where the new model's predictions are logged alongside the champion's, with automated comparison. Production requires a rollback plan that's been tested at least once.
Related reading: Batch Normalization vs Layer Normalization in Production Tra.
The shadow deployment gate catches things that offline metrics miss. I've seen models with excellent AUC scores that produce wildly different prediction distributions in production because the training data didn't reflect recent population shifts. Seven days of shadow traffic catches that.
Registry Disaster Recovery
The registry is a single point of failure for your entire ML stack. If it goes down, nobody can deploy, roll back, or even verify what's running. Treat it with the same seriousness as a production database.
Metadata gets replicated to a standby instance with sub-minute lag. Model artifacts in object storage already have cross-region replication. The recovery playbook gets tested quarterly, and the RTO target is under fifteen minutes.
One thing we learned the hard way: back up the stage transition history, not just the current state. When you need to understand why a model was promoted three months ago, that audit trail is invaluable. Without it, you're reading Slack messages and guessing.
Practical Migration Path
If you're running MLflow today, you don't need to replace it wholesale. MLflow's model registry works well enough for single-team use. The problems emerge at boundaries — when teams need to share models, when you need cross-model lineage, or when governance requirements demand audit trails that MLflow doesn't natively provide.
Start by wrapping MLflow's registry API with a thin service that adds namespace management and cross-team access control. That lets you keep MLflow as the storage backend while fixing the organizational pain points.
The critical migration step is establishing canonical model names early. Renaming models after they're wired into production pipelines is painful enough that most teams simply don't, and you end up with naming conventions that make no sense to anyone who joined after 2023.
Related reading: Learning Rate Scheduling: Cosine Annealing, Warmup, and Cycl.
Model Metadata Search and Discovery
As the registry grows beyond a hundred model versions, finding what you need becomes its own problem. A flat list of model names sorted by date tells you nothing about which models solve which business problems, or which models share training data and might be affected by the same data quality issue.
We built a search layer on top of the registry metadata. Models are tagged with their business domain (fraud, recommendations, pricing), their input feature sets, their training data sources, and their downstream consumers. When a data pipeline breaks, we can query "which models use features from the transactions table?" and get an immediate impact assessment instead of manually checking each model's lineage.
# Registry search query
results = client.search(
filters={
"data_sources": {"contains": "transactions_v3"},
"stage": "production",
},
sort_by="last_deployed",
limit=20,
)
for model in results:
print(f"{model.name} v{model.version} "
f"deployed {model.last_deployed} "
f"by {model.deployed_by}")The search index also powers a deprecation workflow. When we retire a data source or feature, the registry identifies all models that depend on it and generates migration tickets for each team. Without this, deprecations are a coordination nightmare where somebody always gets missed and discovers their model is broken three weeks later in production.
Governance and Compliance
In regulated industries, model governance isn't optional. You need to demonstrate that every production model was reviewed, that its training data met quality standards, that it was tested for fairness across protected attributes, and that there's a clear chain of custody from development to production.
The registry enforces governance through stage gate requirements. A model can't be promoted to production without a completed fairness assessment, a signed approval from the model risk team, and a documented rollback procedure. These requirements are encoded as mandatory metadata fields — if the fields aren't populated, the promotion API rejects the request.
Audit logging captures every state change: who registered the model, who promoted it, who rolled it back, and when each action occurred. The logs are immutable and stored separately from the registry itself, so they survive even if the registry is compromised. During compliance audits, we export the relevant log entries and join them with the model metadata to produce a complete history of every model that touched production during the audit period. It takes about ten minutes to generate what used to require two days of manual documentation assembly.