The Reproducibility Problem Nobody Admits To
Ask any ML engineer if their experiments are reproducible and they'll say yes. Then ask them to reproduce last month best model from scratch, not from a checkpoint, from the raw data and code. Most can't do it, and the reason usually isn't the model code. It's the data.
Data versioning is the part of ML reproducibility that gets the least attention and causes the most pain. Your model code lives in git. Your hyperparameters are logged in MLflow or Weights and Biases. But your training data? That's sitting in an S3 bucket that someone overwrote last Tuesday, and nobody noticed because the filename did not change.
DVC Approach: Git for Data
DVC (Data Version Control) takes a familiar metaphor and extends it. It uses git to track lightweight pointer files (.dvc files) that reference data stored in remote storage: S3, GCS, Azure Blob, or even a shared NFS mount. The actual data never enters the git repository; only the hash-based pointers do.
# Initialize DVC in an existing git repo
dvc init
dvc remote add -d myremote s3://my-bucket/dvc-storage
# Track a large dataset
dvc add data/training_images/
git add data/training_images.dvc data/.gitignore
git commit -m "track training images v1"
dvc push
# Later: switch to a different data version
git checkout experiment-v2
dvc checkout # pulls the data matching this git commitThe design is elegant for individual practitioners. You commit a .dvc file alongside your training code, and anyone who clones the repo can run dvc pull to get the exact same data. It's git-native, which means branching, tagging, and diffing work as expected.
DVC Pipelines for Reproducible Workflows
Where DVC gets interesting is its pipeline feature. You define a DAG of stages, data preprocessing, feature extraction, training, evaluation, and DVC tracks the inputs and outputs of each stage. When you change a stage code or input data, DVC knows which downstream stages need to be re-run.
# dvc.yaml
stages:
preprocess:
cmd: python src/preprocess.py --input data/raw --output data/processed
deps:
- src/preprocess.py
- data/raw
outs:
- data/processed
train:
cmd: python src/train.py --data data/processed --model models/best.pt
deps:
- src/train.py
- data/processed
outs:
- models/best.pt
metrics:
- metrics/train_metrics.json:
cache: falseI've found this works well for projects with a small number of well-defined stages. Where it starts to creak is when you have complex DAGs with conditional execution, parameterized stages, or stages that need to run on different infrastructure.
We covered a related topic in Time Series Forecasting with Temporal Fusion Transformers.
LakeFS: Git Semantics for Object Storage
LakeFS takes a completely different approach. Instead of adding version control on top of your storage, it replaces your storage layer with one that has version control built in. You interact with it through an S3-compatible API, so your existing tools, Spark, pandas, PyTorch DataLoaders, work without modification. They just point to a LakeFS endpoint instead of S3 directly.
import lakefs_client
from lakefs_client.api import branches_api, commits_api
branch_api = branches_api.BranchesApi(client)
branch_api.create_branch(
repository="ml-datasets",
branch_creation={
"name": "experiment-42",
"source": "main"
}
)
# Your training code reads from the branch endpoint
# s3://ml-datasets/experiment-42/training_data/
# Any writes go to this branch only
merge_api = commits_api.CommitsApi(client)
merge_api.merge_into_branch(
repository="ml-datasets",
source_ref="experiment-42",
destination_branch="main"
)Zero-Copy Branching
LakeFS killer feature is zero-copy branching. When you create a branch, it doesn't duplicate any data. It creates a new metadata pointer that shares the same underlying objects. Writes to the branch create new objects only for the changed files. For teams that run many parallel experiments, this is a significant advantage.
With DVC, each experiment variation that changes the data needs to push the full modified dataset to remote storage. With LakeFS, only the diff is stored. On a 500GB dataset where experiments typically modify 5-10 percent of the data, this translates to roughly 10x storage savings.
Head-to-Head: When to Pick Which
DVC is the right choice when your team is small (under 10 people), your data fits a file-based model, and your workflow is primarily sequential. The git-native workflow feels natural and the learning curve is minimal.
LakeFS is the right choice when multiple people or pipelines are writing to the same data concurrently, when you need atomic multi-file commits, or when your data pipeline already speaks S3. The operational overhead is higher because you're running a server, but the data management capabilities are substantially more powerful.
See also: Text Classification Pipeline Architecture for Multi-Label Pr.
One thing I'll add: neither tool solves the metadata problem on its own. You still need to track which data version was used for which experiment, and that linkage lives in your experiment tracking system. Make sure that connection is automated, not manual. A model checkpoint without a pointer to its exact training data version is a reproducibility timebomb waiting to go off.
Practical Implementation Patterns
Regardless of which tool you choose, there are patterns that make data versioning work in practice and anti-patterns that make it fail.
First, version your data schema alongside your data. A Parquet file with columns renamed between versions looks identical to both DVC and LakeFS, but it'll break your training pipeline. We include a schema file (just a JSON document listing column names, types, and descriptions) in every data version. The training pipeline validates the schema before loading data, catching mismatches early.
Second, automate the link between data version and experiment. When you start a training run, the first thing the script should do is record the current data version (git commit hash for DVC, commit ID for LakeFS) in the experiment tracker. We use a decorator for this:
import dvc.api
import mlflow
def track_data_version(func):
def wrapper(*args, **kwargs):
# Record DVC data version
with dvc.api.open(data/training.dvc) as f:
data_hash = f.read().strip()
mlflow.log_param(data_version, data_hash)
mlflow.log_param(data_remote, dvc.api.get_url(data/training))
return func(*args, **kwargs)
return wrapper
@track_data_version
def train_model(config):
# Your training code here
passThird, don't version intermediate artifacts unless you need to reproduce the exact pipeline state. Feature caches, preprocessed files, and temporary outputs should be regenerable from the source data and code. Versioning them wastes storage and creates a false sense of reproducibility. If the preprocessing code changed but the cached output did not, you're training on stale features.
For a related perspective, see Model Monitoring in Production: Detecting Data Drift and Per.
Migration Strategy
If you're adding data versioning to an existing project (which is most projects), here is the migration path we recommend.
Start by inventorying your data dependencies. Where does your training data live? How many versions exist? Who updates it and how often? This audit usually reveals surprises. In one project, we found three different S3 paths that various team members were using for training data, with no consistent naming convention.
Next, pick one dataset (ideally the smallest and most critical one) and version it. Get the full workflow running: commit the version, train a model, reproduce the result from scratch. Don't try to version everything at once.
Finally, enforce versioning in CI. Add a check that every training job references a specific data version (not latest, not a mutable path). This is the enforcement mechanism that prevents regression back to unversioned data access.
The whole migration typically takes 2-4 weeks for a small team. The technical work isn't hard. The organizational work of changing habits and enforcing the new workflow is the actual challenge.