Making GPUs Work for Everyone Without Making Them Work for No One
GPU clusters are expensive, and shared GPU clusters are expensive and frustrating. Every ML team I've worked with has the same complaint: "I submitted my training job an hour ago and it's still pending." Meanwhile, someone else's job is using 8 GPUs to run an experiment that won't make it past the first epoch because they misconfigured the learning rate.
Managing shared GPU resources is equal parts technical infrastructure and organizational policy. The technical part is scheduling and allocation. The policy part is convincing teams that fair sharing doesn't mean equal sharing — it means proportional to actual need and priority.
Cluster Architecture Decisions
Before getting into scheduling, you need to decide how your GPU cluster is organized. The two main patterns are a shared pool (all GPUs available to everyone, managed by a central scheduler) and partitioned allocations (each team gets a dedicated slice, with optional borrowing).
Shared pools maximize utilization. When one team isn't training, other teams can use those GPUs. Partitioned allocations provide predictability — teams know exactly what resources they can count on. We've found that a hybrid works best: teams get guaranteed minimums (partitions they can always count on) with access to a shared overflow pool for burst capacity.
# SLURM partition configuration for hybrid GPU sharing
PartitionName=team-fraud Nodes=gpu[01-04] Default=NO MaxTime=72:00:00 Priority=100
PartitionName=team-recs Nodes=gpu[05-08] Default=NO MaxTime=72:00:00 Priority=100
PartitionName=team-nlp Nodes=gpu[09-12] Default=NO MaxTime=72:00:00 Priority=100
PartitionName=shared Nodes=gpu[01-16] Default=YES MaxTime=24:00:00 Priority=50
PartitionName=preemptible Nodes=gpu[01-16] Default=NO MaxTime=4:00:00 Priority=10
PriorityType=priority/multifactor
PriorityWeightFairshare=10000
PriorityWeightJobSize=1000
PriorityWeightAge=500
PriorityDecayHalfLife=7-0The preemptible partition is the key insight. It lets teams submit low-priority jobs that can use any idle GPU in the cluster. These jobs get killed if a higher-priority job needs the resources, so they're appropriate for hyperparameter sweeps and exploratory experiments — work that can be checkpointed and restarted cheaply.
Job Scheduling and Priority
SLURM's fair-share scheduling considers historical usage. Teams that have been using fewer GPUs than their share get priority bumped; teams that have been heavy users get deprioritized. This self-balances over time without requiring manual intervention.
See also: Batch Normalization vs Layer Normalization in Production Tra.
But pure fair-share doesn't account for urgency. A production model retraining job triggered by a data quality incident should preempt exploratory research, regardless of which team submitted what in the past week. We overlay a priority tier system on top of fair-share.
Tier 1 (highest): production incidents requiring immediate retraining. Tier 2: scheduled production retraining on established cadences. Tier 3: experiment development and validation. Tier 4: exploratory research and hyperparameter sweeps. Tier 1 and 2 jobs get dedicated GPU reservations. Tier 3 and 4 jobs compete through fair-share scheduling.
Resource Right-Sizing
The biggest source of GPU waste isn't scheduling inefficiency — it's engineers requesting more resources than their jobs need. People request 4 GPUs "just in case" for a job that uses one. They request A100s for inference benchmarks that would run fine on T4s.
We added resource usage tracking and reporting. Each team gets a weekly utilization report showing their GPU hours consumed, average GPU utilization during those hours, and memory utilization. A job that runs for 8 hours but uses only 30% of the GPU's compute capacity is a clear signal that the engineer should be using fewer or smaller GPUs.
import subprocess
def get_gpu_utilization():
result = subprocess.run(
["nvidia-smi",
"--query-gpu=index,utilization.gpu,memory.used,memory.total",
"--format=csv,nounits,noheader"],
capture_output=True, text=True
)
gpus = []
for line in result.stdout.strip().split("\n"):
idx, util, mem_used, mem_total = line.split(", ")
gpus.append({
"index": int(idx),
"compute_util_pct": int(util),
"memory_used_mb": int(mem_used),
"memory_total_mb": int(mem_total),
"memory_util_pct": round(int(mem_used) / int(mem_total) * 100, 1),
})
return gpus
def report_underutilized(threshold_pct=40):
gpus = get_gpu_utilization()
for gpu in gpus:
if gpu["compute_util_pct"] < threshold_pct:
print(f"GPU {gpu['index']}: {gpu['compute_util_pct']}% compute, "
f"{gpu['memory_util_pct']}% memory — underutilized")Multi-GPU Training Coordination
Distributed training jobs add scheduling complexity because they need multiple GPUs to start simultaneously. A job requesting 8 GPUs on a cluster with fragmented availability can wait hours even when the total free GPU count exceeds 8, because those GPUs are spread across different nodes.
We covered a related topic in A/B Testing Machine Learning Models: Statistical Rigor and C.
Gang scheduling solves this — it allocates all resources for a distributed job atomically, waiting until all requested GPUs are available on compatible nodes before starting. Without gang scheduling, you get pathological cases where half the GPUs are allocated and waiting for the other half, wasting resources.
For multi-node training, network topology matters. Training jobs that communicate heavily between GPUs (which is most training jobs) should be placed on GPUs connected by NVLink or InfiniBand, not across slow network links. We annotate nodes with their interconnect capabilities and add placement constraints to distributed job submissions.
Cost Tracking and Chargeback
Without cost visibility, teams have no incentive to use resources efficiently. We implemented a chargeback system that bills each team for their actual GPU usage, not their allocation. The billing rate varies by GPU type (A100 hours cost more than T4 hours) and by priority tier (preemptible hours are discounted 70%).
The chargeback model changed behavior more than any technical optimization. When teams see that their monthly GPU bill is $47,000 and 60% of that's underutilized A100 time, they start right-sizing their job requests. One team cut their GPU costs by 40% just by moving hyperparameter sweeps to preemptible T4s instead of running them on dedicated A100s.
Operational Lessons
GPU clusters need the same operational discipline as production services. Health checks for GPU hardware (memory errors, thermal throttling, driver crashes), automated node draining when issues are detected, and a maintenance schedule that doesn't conflict with major training runs.
See also: Prompt Engineering as Software Engineering: Version Control .
We also found that documentation about the cluster's capabilities and policies saved more engineering time than any scheduling optimization. A page explaining how to request GPUs, what the priority tiers mean, when preemptible jobs get killed, and how to checkpoint distributed training jobs reduced support tickets by about 60%. Engineers weren't wasting resources because they didn't understand the system — they were wasting resources because the system was poorly documented.
Quota Management and Budgeting
Each team gets a monthly GPU-hour budget based on their planned training workloads. The budget isn't a hard cap — teams can exceed it using the shared overflow pool — but it's a planning tool that forces teams to prioritize their experiments.
We track budget consumption in real time through a dashboard that shows each team's usage against their allocation, broken down by job type (production retraining vs experimental). When a team approaches 80% of their budget, they get a notification. When they exceed 100%, their non-production jobs drop to preemptible priority, which means they might get preempted by other teams' production work.
# Quota checking at job submission
def check_quota(team: str, requested_gpu_hours: float) -> QuotaResult:
budget = get_team_budget(team)
used = get_current_usage(team)
remaining = budget.monthly_gpu_hours - used.total_gpu_hours
if remaining >= requested_gpu_hours:
return QuotaResult(allowed=True, priority="normal")
elif remaining > 0:
return QuotaResult(
allowed=True,
priority="preemptible",
warning=f"Only {remaining:.0f}h remaining in budget"
)
else:
return QuotaResult(
allowed=True,
priority="preemptible",
warning=f"Budget exceeded by {abs(remaining):.0f}h"
)The key insight is that we never block job submissions — we just adjust priority. Blocking creates an adversarial dynamic where teams hoard resources early in the month. Preemptible demotion creates a gentler incentive: teams still get their work done, but they learn to right-size their experiments because oversized jobs in the preemptible queue wait longer.
Spot Instances and Cloud Burst
On-premise GPU clusters have fixed capacity. When demand exceeds capacity during peak periods (model evaluation season, quarterly retraining cycles), we burst to cloud GPUs. Spot instances on AWS provide GPU capacity at 60-70% discount, and our checkpointing infrastructure means interrupted spot instances just resume from the last checkpoint.
The cloud burst policy is simple: if the on-premise job queue depth exceeds 50 GPU-hours and the average wait time exceeds 2 hours, we automatically provision spot instances. When the queue drains below 20 GPU-hours, we release the spot instances. This keeps the on-premise cluster as the primary resource (lower cost per GPU-hour for sustained use) while using cloud for overflow (better than buying more on-premise capacity that sits idle 80% of the year).
One subtlety: cloud GPUs and on-premise GPUs don't always produce identical results due to hardware differences (different GPU models, different interconnects, different driver versions). For production retraining, we pin to on-premise hardware to ensure reproducibility. Cloud burst is reserved for experimental workloads where small numerical differences are acceptable.