ML Model Versioning with MLflow and DVC: Tracking Experiments at Scale

Tracking machine learning experiments with just a spreadsheet or a few logged metrics works until your second teammate joins the project and your dataset crosses 100GB. I learned this the hard way on a retail forecasting project where we had 47 notebooks all named model_final_v2.ipynb, no way to reproduce a 0.89 F1 score from three weeks earlier, and a dataset that had silently changed underneath us. Combining MLflow for experiment and model tracking with DVC for data and pipeline versioning solved that chaos, but only after we stopped treating them as competing tools and started using them as complementary layers. This article covers the architecture I now use by default for teams of 3-15 engineers who need reproducibility at scale without grinding iteration speed to a halt.

Why Experiment Tracking Collapses Without Data Lineage

MLflow is excellent at answering "what hyperparameters produced this metric?" It is not designed to answer "what exact bytes of data produced this metric?" In my experience, that second question is where most reproducibility efforts fail. You can log learning rate, batch size, and architecture in MLflow and still be unable to rebuild a model because the training CSV was overwritten, an augmentation script changed, or a join against a feature table produced different results.

DVC fills that gap by versioning data, models, and pipeline stages with Git as the control plane. It does not replace your Git repository; it extends it. Data files are replaced with small .dvc metafiles that contain an MD5 hash, while the actual content lives in a remote storage backend like S3, GCS, or Azure Blob. Git tracks the metafile, DVC tracks the content. This means a git checkout v0.4.2 plus dvc checkout restores code and data together.

I've found that teams understand the separation best when they think about responsibilities: MLflow tracks the dynamic, high-cardinality state of each run (metrics, params, artifacts, environment), while DVC tracks the deterministic, versioned state of inputs and outputs (datasets, intermediate features, serialized models). If you try to force MLflow to version 50GB datasets by logging them as artifacts for every run, you will flood your artifact store, inflate tracking server latency, and pay for duplicate storage. If you try to use DVC to compare 200 hyperparameter trials, you will be writing manual scripts instead of using a purpose-built UI.

The Failure Mode I See Most Often

The classic anti-pattern is logging the dataset path as an MLflow parameter: params: {"data_path": "s3://bucket/data.csv"}. That path is mutable. Two runs with the same path can train on different data if the file was overwritten. A better pattern is logging the DVC hash or Git commit that uniquely identifies the data snapshot. That ties the immutable pointer from DVC into the queryable interface of MLflow, which is the bridge we will build explicitly later.

What Happens at Scale

At 10+ experiments per day per engineer, you need queries like "show me all runs on data version ab34f2 where augmentation was enabled and val_f1 > 0.85." MLflow's search API handles that. DVC alone cannot. At the same time, at 500GB of image data or 20M rows of transaction logs, you need deduplicated, content-addressed storage with cache sharing. MLflow's artifact store is not designed for that. The right mental model is not MLflow versus DVC, it is MLflow for the experiment database and DVC for the data supply chain. When you need to orchestrate the entire flow end-to-end, you will eventually layer a scheduler like those compared in ML Pipeline Orchestration: Airflow, Kubeflow and Prefect Compared, but even without a heavy orchestrator, MLflow + DVC gives you a solid foundation.

Architecting an MLflow Tracking Server That Survives Team Growth

A local mlruns directory is fine for solo work. For a team, it creates silos and merge conflicts. The setup I recommend for intermediate teams is a hosted tracking server with a relational backend and object storage.

On AWS, that looks like an MLflow server running on ECS or EC2, a Postgres RDS instance as the backend store, and S3 as the artifact store. On GCP, it is Cloud Run + Cloud SQL + GCS. The critical detail is to separate the tracking server from your training jobs. Your training code should talk to the server via MLFLOW_TRACKING_URI, not write to a local filesystem that other engineers cannot see. I've seen teams waste weeks trying to sync mlruns folders over Dropbox or Git.

Configuration matters more than most tutorials suggest. Set --serve-artifacts only if you want the server to proxy artifact access, otherwise configure direct access via signed URLs. Enable database connection pooling and set MLFLOW_SQLALCHEMY_POOL_SIZE appropriately. In my experience, the default SQLite backend will lock up under concurrent writes from parallel hyperparameter sweeps. Postgres handles it without drama. Also, enforce experiment naming conventions early. I use /team/project/initiative hierarchy, for example /retail/forecasting/promo-aug-v2, and require a owner tag on every run. Without that, search becomes painful at 10,000 runs.

Authentication and Reproducibility Hooks

Do not leave the tracking server open. Put it behind SSO and create service accounts for CI runners. Each run should log three things that most teams skip: the Git commit SHA, the Docker image digest if you use containers, and the DVC data hash. You can automate the first two with an MLflow autologging wrapper or a simple helper function. That triples the time it takes to debug a production issue from days to minutes.

Refer to the PyTorch Documentation for environment reproducibility details when logging torch.__version__, CUDA version, and determinism flags alongside your MLflow params; driver mismatches have caused silent metric shifts in my projects more than once.

import mlflow
import subprocess
import dvc.api

mlflow.set_tracking_uri("https://mlflow.internal.company.com")
mlflow.set_experiment("/retail/forecasting/promo-aug-v2")

def get_git_sha():
    return subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()

def get_dvc_hash(path="data/processed/train.csv.dvc"):
    # reads the .dvc file to extract md5
    import yaml
    with open(path) as f:
        return yaml.safe_load(f)["outs"][0]["md5"]

with mlflow.start_run(run_name="xgboost-baseline"):
    dvc_hash = get_dvc_hash()
    mlflow.log_param("git_sha", get_git_sha())
    mlflow.log_param("dvc_data_hash", dvc_hash)
    mlflow.log_param("dvc_data_path", "data/processed/train.csv")
    mlflow.log_param("learning_rate", 0.05)
    mlflow.log_param("max_depth", 8)
    
    # training code...
    val_f1 = 0.872  # example
    mlflow.log_metric("val_f1", val_f1)
    mlflow.log_metric("val_pr_auc", 0.913)
    mlflow.set_tag("owner", "priya")
    mlflow.set_tag("data_version", dvc_hash[:8])

This pattern makes the MLflow UI searchable by data version. You can run mlflow.search_runs(filter_string="tags.data_version = 'ab34f2'") and instantly compare all models trained on the same snapshot, regardless of who trained them.

Building Reproducible DVC Pipelines on Top of Shared Storage

A DVC pipeline is defined in dvc.yaml and locked in dvc.lock. The YAML declares stages, dependencies, and outputs; the lock file records the exact hashes after execution. In my experience, teams should start with 3-5 stages and resist the urge to create a 20-stage graph on day one.

A typical pipeline for tabular data might be: ingest -> validate -> featurize -> train -> evaluate. For vision, it might add preprocess and augment. Each stage runs a command, often a Python script, and DVC knows to rerun downstream stages only if an upstream output changed. That saves hours on large datasets.

Storage configuration is where cost and speed are won or lost. Use a single shared remote for the team, for example s3://company-dvc-store/retail-forecasting, and enable a local cache with dvc cache dir on each developer machine and CI runner. Set core.autostage = true so dvc add stages automatically, and use dvc push --jobs 8 to parallelize uploads. I've found that without cache deduplication, a team of six can generate 3-4TB of duplicated parquet files in a month. With content-addressed caching, that drops to under 400GB because only changed blocks are stored.

Writing a Maintainable dvc.yaml

Keep params in a separate params.yaml and reference them with ${} syntax. This lets you change a hyperparameter without editing dvc.yaml and keeps Git diffs clean. Also, explicitly declare metrics and plots so dvc metrics diff and dvc plots diff work in code review.

# dvc.yaml
stages:
  featurize:
    cmd: python src/featurize.py --config params.yaml
    deps:
      - src/featurize.py
      - data/raw/transactions.csv.dvc
    params:
      - featurize.window_days
      - featurize.promo_flag
    outs:
      - data/processed/features.parquet
    metrics:
      - metrics/featurize_stats.json:
          cache: false

  train:
    cmd: python src/train.py --config params.yaml
    deps:
      - src/train.py
      - data/processed/features.parquet
    params:
      - train.learning_rate
      - train.max_depth
      - train.n_estimators
    outs:
      - models/xgb_model.pkl
    metrics:
      - metrics/train_metrics.json:
          cache: false
    plots:
      - plots/confusion_matrix.json:
          cache: false

# params.yaml
featurize:
  window_days: 28
  promo_flag: true
train:
  learning_rate: 0.05
  max_depth: 8
  n_estimators: 1200

Running dvc repro will now execute only what is needed. If you tweak train.learning_rate, it skips featurize and reruns train and evaluate. If you update the raw data, it reruns everything. That dependency graph becomes your living documentation, which is far more reliable than a wiki diagram that is outdated by week two.

Binding MLflow Runs to DVC Data Versions for End-to-End Reproducibility

This is the core integration that makes the stack powerful. The goal is bidirectional traceability: from an MLflow run you can find the exact DVC data, and from a DVC data version you can find all MLflow runs.

I implement this with three links. First, log the DVC hash into MLflow as both a param and a tag, as shown earlier. Second, write the MLflow run ID into a DVC metric file so the Git/DVC history knows which experiment produced which model artifact. Third, tag the Git commit with the MLflow run alias when a model is promoted.

In practice, your training script should write a small JSON file like metrics/mlflow_info.json containing {"run_id": "...", "experiment_id": "..."}. Declare that file as a metrics output with cache: false in dvc.yaml. When you git commit the updated dvc.lock and that JSON, you have a permanent pointer from code+data version to experiment. Later, git log --grep=mlflow or dvc metrics show --all-branches gives you the lineage.

Another pattern I use is storing the MLflow model URI in dvc.lock via a post-train hook. After mlflow.sklearn.log_model returns a runs:/<run_id>/model URI, append it to a file that DVC tracks. This is especially useful when you later need to audit a deployed model: you pull the production Git tag, run dvc checkout, and immediately know which MLflow artifacts to download, which avoids hunting through the UI for a run from six months ago.

Reproducing Any Past Result

With this binding, reproduction is a two-line process for any engineer: git checkout <sha> && dvc checkout restores code and data, then mlflow run . --experiment-id ... or simply dvc repro rebuilds the model deterministically if you pinned seeds and environments. I also log conda.yaml or requirements.txt hash as an MLflow artifact to catch environment drift. Papers catalogued in arXiv Machine Learning repeatedly show that small changes in preprocessing or data splits can outweigh architecture improvements, which matches what I have seen in production; without data lineage you will misattribute gains.

When you are ready to move a validated model to production, the lineage pays off again. The serving layer described in Model Serving: Triton, TorchServe and BentoML Deployment Patterns can pull the exact model artifact by run ID instead of a floating latest.pkl file, which eliminates the class of errors where staging and production diverge.

Handling Merge Conflicts, Cache Bloat, and Remote Storage Costs

DVC merge conflicts are different from code conflicts. The dvc.lock file often conflicts when two engineers modify different pipeline stages. Unlike code, you should rarely resolve this by hand-editing the hash values. The safe resolution is to accept one side, then run dvc repro or dvc checkout to regenerate the correct hashes. I add a Git merge driver in .gitattributes: dvc.lock merge=dvc requires installing the DVC merge driver via dvc install. It automates most cases.

Cache bloat is the second silent problem. By default, DVC keeps every version of every output forever in the remote. I schedule a monthly dvc gc --workspace --projects --remote s3remote on the main branch to remove objects not referenced by any Git commit reachable from main. Run it first with --dry and review. I've recovered over $800 per month in S3 costs on a single computer vision project by garbage collecting orphaned image caches that were created during hyperparameter sweeps. Also set lifecycle rules on your S3 bucket to transition old cache objects to Infrequent Access after 60 days.

For local cache, set cache.type = symlink or reflink if your filesystem supports it, to avoid copying large files on dvc checkout. On Mac APFS, reflink gives you instant checkouts for 10GB files. On Linux, symlink is reliable but requires that your tooling follows symlinks correctly. Test this early; I have debugged training jobs that failed because a custom data loader did not resolve symlinked paths.

This table is how I explain the split to new hires. It prevents the common mistake of storing large datasets as MLflow artifacts or trying to use DVC metrics for hyperparameter search analytics. Use each tool for what it was built to do, and invest in the thin glue layer that links them.

Automating Model Promotion From Experiment to Registry to Deployment

Manual promotion is error-prone. Someone copies model.pkl to a production bucket, forgets to update the config, and you serve a stale model for a week. I automate promotion with three gates: automated evaluation, human approval, and registry tagging.

The CI job runs on every push to main that changes dvc.lock or metrics/*.json. It does dvc repro --force on a clean runner, pulls data with dvc pull, retrains, and logs to MLflow. It then compares the new metrics against the current Production model in the MLflow Model Registry using mlflow.pyfunc.load_model and a held-out validation set that is itself DVC-versioned. If the new model beats the incumbent by a threshold, say +0.01 F1, it registers the model with mlflow.register_model and transitions it to Staging. A Slack notification asks for approval to promote to Production. This workflow keeps DVC as the source of truth for data and MLflow as the source of truth for models.

The registry is the handoff point to deployment. Whether you deploy via Triton, a batch job, or a REST service, the deployer should pull by model alias, not by file path. For example, models:/retail-forecast-xgb@production always resolves to the approved version. Rollback is then mlflow.transition_model_version_stage to demote, plus git revert + dvc checkout if the data was faulty. Document this in your runbook; an on-call engineer should not need to guess which artifact is live.

Finally, close the loop with monitoring. A promoted model is not done. Log its predictions and join them back to DVC-versioned evaluation sets. The patterns in Model Monitoring in Production: Data Drift, Concept Drift and Alert Systems pair naturally here: when drift is detected, your alert can include both the MLflow run ID of the serving model and the DVC hash of the data it was trained on, which shortens root-cause analysis from hours to minutes.

# ci/promote.py  (run in CI after dvc repro)
import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient(tracking_uri="https://mlflow.internal.company.com")
model_name = "retail-forecast-xgb"
candidate = client.get_latest_versions(name=model_name, stages=["Staging"])[0]
prod = client.get_latest_versions(name=model_name, stages=["Production"])

# compare on versioned holdout set pulled via DVC
candidate_f1 = float(open("metrics/train_metrics.json").read().split('"val_f1":')[1].split(",")[0])
prod_f1 = 0.0 if not prod else float(client.get_run(prod[0].run_id).data.metrics["val_f1"])

print(f"Candidate {candidate.version} f1={candidate_f1} vs prod f1={prod_f1}")

if candidate_f1 > prod_f1 + 0.01:
    client.transition_model_version_stage(
        name=model_name, version=candidate.version, stage="Production",
        archive_existing_versions=True
    )
    # Tag git commit for traceability
    import subprocess
    subprocess.run(["git", "tag", f"model-{model_name}-v{candidate.version}", "-m", f"run_id={candidate.run_id}"])
    subprocess.run(["git", "push", "--tags"])
else:
    print("No promotion: threshold not met.")

This script is intentionally simple. In production I add checks for data hash equality, test coverage via Testing ML Systems: Data Tests, Model Tests and Integration Testing style data tests, and latency benchmarks before promotion. The principle remains: every promotion is traceable to a Git SHA, a DVC hash, and an MLflow run ID, without manual file copying.

Frequently Asked Questions