When Single-GPU Training Isn't Enough
Your model has 7 billion parameters. Each parameter in FP32 takes 4 bytes. That's 28GB just for weights. Adam optimizer keeps two additional states per parameter — another 56GB. Add activations and you're looking at 100GB+ for training. No single GPU handles this. DeepSpeed's ZeRO (Zero Redundancy Optimizer) is how most teams solve this problem without re-architecting their model.
The key insight behind ZeRO is that data parallelism wastes memory. In standard data parallel training, every GPU holds a complete copy of the model, optimizer states, and gradients. With 8 GPUs, you've got 8 copies of everything — but each GPU only updates 1/8th of the parameters per step. ZeRO partitions these redundant states across GPUs.
ZeRO Stage Breakdown
ZeRO Stage 1: Optimizer State Partitioning
Stage 1 shards only the optimizer states. Each GPU holds 1/N-th of the optimizer states (momentum and variance for Adam). After the backward pass, each GPU updates its assigned partition and broadcasts the results. Memory savings: roughly 4× for Adam (optimizer states dominate memory usage).
Stage 1 is the easiest to adopt and has minimal communication overhead. For models up to about 13B parameters with 8 A100 80GB GPUs, Stage 1 often provides enough memory relief.
ZeRO Stage 2: Gradient Partitioning
Stage 2 adds gradient sharding on top of optimizer partitioning. During backward, gradients are reduced and partitioned so each GPU only keeps the gradients it needs for its optimizer partition. This roughly doubles the memory savings of Stage 1.
We covered a related topic in CI/CD Pipelines for Machine Learning: Testing Models Before .
# deepspeed_config.json for Stage 2
{
"train_batch_size": 32,
"gradient_accumulation_steps": 4,
"fp16": {
"enabled": true,
"initial_scale_power": 16
},
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
"reduce_scatter": true,
"overlap_comm": true,
"contiguous_gradients": true
}
}
The overlap_comm flag is critical. When enabled, communication (gradient reduction) overlaps with backward computation. Without it, training throughput drops 15-25% due to communication stalls.
ZeRO Stage 3: Parameter Partitioning
Stage 3 partitions everything — parameters, gradients, and optimizer states. Each GPU holds only 1/N-th of the model. During forward and backward passes, parameters are gathered on-demand from other GPUs, used, and then released. This is the most aggressive optimization and enables training models that are N× larger than what fits on a single GPU.
The tradeoff is communication volume. Stage 3 needs an all-gather for parameters during both forward and backward passes. On clusters with fast interconnects (NVLink, InfiniBand), this overhead is manageable — I've measured 10-15% throughput reduction compared to Stage 2 on 8×A100 nodes. On slower networks (ethernet between nodes), it can be 30-40%.
Configuration That Actually Works
The DeepSpeed documentation has dozens of configuration options, and the defaults aren't always ideal. Here's the configuration template I start every project with, then tune from there.
See also: Label Management Systems for Large-Scale Annotation Projects.
{
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": 2,
"gradient_accumulation_steps": "auto",
"fp16": {
"enabled": true,
"loss_scale": 0,
"loss_scale_window": 500,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"zero_optimization": {
"stage": 2,
"overlap_comm": true,
"reduce_scatter": true,
"contiguous_gradients": true,
"reduce_bucket_size": 5e8,
"allgather_bucket_size": 5e8
},
"gradient_clipping": 1.0,
"wall_clock_breakdown": false
}
Those bucket sizes matter more than you'd think. Too small, and you get many small all-reduce operations with high latency overhead. Too large, and you can't overlap communication with computation effectively. 500M elements (5e8) is a solid starting point for most models on NVLink-connected GPUs. For slower interconnects, reduce to 2e8 to increase overlap opportunities.
Integration with HuggingFace Trainer
Most teams I work with use HuggingFace's Trainer, which has first-class DeepSpeed integration. The setup is minimal.
from transformers import TrainingArguments, Trainer
args = TrainingArguments(
output_dir="./output",
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=2e-5,
num_train_epochs=3,
fp16=True,
deepspeed="ds_config.json",
)
trainer = Trainer(
model=model,
args=args,
train_dataset=train_dataset,
)
trainer.train()
One thing that trips people up: DeepSpeed manages the optimizer internally. If you pass a custom optimizer to Trainer, it might conflict with ZeRO's optimizer partitioning. Let DeepSpeed create the optimizer through the config file unless you have a specific reason not to.
ZeRO-Offload and ZeRO-Infinity
When GPU memory is still too tight, DeepSpeed can offload optimizer states and parameters to CPU memory (ZeRO-Offload) or even NVMe storage (ZeRO-Infinity). I've trained 30B parameter models on 4 A100 40GB GPUs using Stage 3 with CPU offload. It's slower than pure GPU training — about 2-3× throughput reduction — but it beats not training at all.
This connects to the ideas in Canary Deployments for ML Models: Traffic Splitting and Roll.
{
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
}
}
}
Pin memory is essential. Without it, the CPU-GPU transfers use pageable memory and throughput drops by another 30-50%. Make sure your system has enough pinned memory capacity — each offloaded tensor reserves pinned memory equal to its size.
Common Issues and Fixes
The number one issue I see: OOM during the first forward pass despite having configured ZeRO Stage 3. This usually happens because the model initialization creates full-size parameter tensors on GPU before DeepSpeed has a chance to partition them. The fix is to initialize with DeepSpeed's zero.Init context manager, which creates parameters directly in partitioned form.
import deepspeed
with deepspeed.zero.Init(config_dict_or_path="ds_config.json"):
model = MyLargeModel(config)
The second most common issue: training hangs during all-gather. This typically means one GPU crashed silently and the others are waiting for it in a collective operation. Set NCCL_DEBUG=INFO and NCCL_TIMEOUT=1800 (30 minutes) to get better diagnostics. Also check that your NCCL version matches across all nodes — version mismatches cause silent hangs more often than you'd expect.
Third issue: checkpointing takes forever. ZeRO Stage 3 needs to gather all parameters to rank 0 before saving. For large models, this can take 10+ minutes. DeepSpeed's universal checkpointing feature saves sharded checkpoints without gathering, which is faster but produces checkpoint files that must be loaded with the same parallelism configuration.