Model Serving Latency Optimization:...

The Milliseconds That Matter Between Training and Serving

You've trained a model that beats every benchmark. Congratulations. Now serve it at 50 milliseconds p99 latency to 10,000 requests per second without burning through your infrastructure budget. This is where most ML projects discover that training accuracy and serving performance live in different universes.

Model serving optimization isn't a single technique — it's a stack of decisions that interact in non-obvious ways. Batching helps throughput but hurts latency. Caching reduces compute but increases memory. Hardware selection affects everything but commits you for months or years. Getting the balance right requires understanding what your serving workload actually looks like.

Request Batching Architecture

Individual inference requests on a GPU are wasteful. The GPU spends most of its time waiting for data transfer, not computing. Batching multiple requests into a single forward pass amortizes the overhead and dramatically improves throughput.

import asyncio
from collections import deque
import torch

class DynamicBatcher:
 def __init__(self, model, max_batch_size=32, max_wait_ms=10):
 self.model = model
 self.max_batch_size = max_batch_size
 self.max_wait_ms = max_wait_ms / 1000
 self.queue = deque()
 self.lock = asyncio.Lock()

 async def predict(self, features):
 future = asyncio.get_event_loop().create_future()
 async with self.lock:
 self.queue.append((features, future))
 if len(self.queue) >= self.max_batch_size:
 await self._process_batch()
 return await future

 await asyncio.sleep(self.max_wait_ms)
 async with self.lock:
 if not future.done():
 await self._process_batch()
 return await future

 async def _process_batch(self):
 if not self.queue:
 return
 items = []
 while self.queue and len(items) < self.max_batch_size:
 items.append(self.queue.popleft())

 features_batch = torch.stack([item[0] for item in items])
 with torch.no_grad():
 predictions = self.model(features_batch)

 for i, (_, future) in enumerate(items):
 if not future.done():
 future.set_result(predictions[i])

The tradeoff is latency vs throughput. A max_wait_ms of 10 means any individual request waits at most 10 milliseconds for a batch to form. In high-traffic scenarios, batches fill before the timeout. In low-traffic scenarios, you pay the full 10ms penalty. We typically set the timeout to half the latency budget — if the SLA is 50ms total, the batching timeout is 5ms, leaving 45ms for compute and network overhead.

Caching Strategies

Not every inference request needs a full model forward pass. If the same input (or a similar enough input) was recently scored, a cached result is perfectly acceptable for many use cases.

Exact-match caching works when inputs are discrete or low-cardinality. Product recommendation scores for a catalog of 100,000 items can be precomputed and cached entirely. The cache refresh runs every few hours or when the model updates.

For a related perspective, see Fine-Tuning Language Models with LoRA: Rank Selection and La.

For continuous inputs, approximate caching uses locality-sensitive hashing to group similar inputs and serve the same prediction. This works surprisingly well for models where small input perturbations don't change the output significantly — which is most well-trained models. We've seen 40-60% hit rates on recommendation and pricing models.

Model Optimization Techniques

Before throwing hardware at the latency problem, optimize the model itself. The techniques stack: quantization alone reduces latency by 2-3x; combined with operator fusion and graph optimization, you can get 4-6x speedups without any accuracy loss that matters in practice.

INT8 quantization is the safe default. It reduces model size by 4x and improves throughput by 2-3x on hardware with INT8 acceleration. The accuracy degradation is typically less than 0.1% on classification tasks. INT4 quantization pushes further but can cause noticeable accuracy drops on sensitive tasks — always benchmark on your specific workload before deploying.

import tensorrt as trt

def optimize_model_for_serving(model_path, output_path, precision="fp16"):
 logger = trt.Logger(trt.Logger.WARNING)
 builder = trt.Builder(logger)
 network = builder.create_network(
 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
 )
 parser = trt.OnnxParser(network, logger)

 with open(model_path, "rb") as f:
 parser.parse(f.read())

 config = builder.create_builder_config()
 config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)

 if precision == "fp16":
 config.set_flag(trt.BuilderFlag.FP16)
 elif precision == "int8":
 config.set_flag(trt.BuilderFlag.INT8)

 profile = builder.create_optimization_profile()
 profile.set_shape("input", (1, 147), (16, 147), (64, 147))
 config.add_optimization_profile(profile)

 engine = builder.build_serialized_network(network, config)
 with open(output_path, "wb") as f:
 f.write(engine)

Hardware Selection Framework

The GPU you train on isn't necessarily the GPU you should serve on. Training rewards raw compute throughput. Serving rewards low-latency inference with efficient memory usage.

For small models (under 500M parameters), CPU serving with ONNX Runtime is often cheaper per query than GPU serving. The per-instance cost is 3-5x lower, and the latency difference is negligible when the model is small enough. GPUs make sense when the model is large enough that CPU inference exceeds your latency budget, or when batch throughput requirements justify the higher per-instance cost.

Among GPUs, the T4 is the workhorse for inference. It's cheap (about $0.35/hour on AWS), has 16GB memory (enough for most single models), and delivers good INT8 performance with Tensor Cores. The A10G offers 2x the memory and better FP16 performance for about $1/hour. A100s are overkill for most inference workloads unless you're serving very large models or need the multi-instance GPU (MIG) feature to run multiple models per card.

Autoscaling and Load Management

Serving infrastructure needs to scale with traffic, but GPU instances take minutes to start — much longer than CPU instances. Predictive autoscaling, where you scale based on historical traffic patterns rather than current load, gives you headroom for traffic spikes.

We keep a warm pool of instances that are ready but not actively serving. During predicted low-traffic periods, the warm pool shrinks. Before predicted traffic spikes (Monday mornings, marketing campaign launches), the warm pool grows. This adds about 15% to the infrastructure cost but eliminates the latency spikes that cold-start scaling creates.

Request queuing with backpressure prevents cascade failures during unexpected traffic surges. If all serving instances are at capacity, new requests queue with a timeout instead of being rejected immediately. If the queue exceeds the timeout, requests get a graceful degradation response — a fallback prediction from a simpler model or a cached result — rather than an error.

Monitoring Serving Performance

Serving latency has a long tail that averages don't capture. A model with 20ms mean latency and 500ms p99 latency has a very different user experience than one with 40ms mean and 45ms p99. We track p50, p90, p95, and p99 latency, with alerts on p99 because that's where the pain is felt.

GPU memory utilization and compute utilization during serving hours tell you whether you're overprovisioned (low utilization, wasting money) or underprovisioned (high utilization, risking latency spikes). The sweet spot for serving is 50-70% GPU utilization — enough headroom for traffic bursts without excess idle capacity.

Multi-Model Serving on Shared GPUs

Running one model per GPU wastes memory when the model doesn't fill the card. NVIDIA's Multi-Instance GPU (MIG) feature on A100s lets you partition a single GPU into up to seven independent instances, each with its own memory and compute. We serve three to four small models per A100 using MIG, which cuts our per-model infrastructure cost by roughly 3x compared to dedicated T4 instances.

The setup requires careful resource planning. Each MIG instance gets a fixed allocation of memory and compute slices. A model that needs 6GB of memory fits comfortably in a 10GB MIG instance but wastes half the allocation in a 20GB instance. We profile each model's memory and compute requirements during the quality gate phase and assign MIG instances accordingly.

# MIG instance configuration for multi-model serving
# A100-80GB split into 4 instances
mig_config:
 gpu_instance_profiles:
 - profile: "3g.40gb" # 3 compute slices, 40GB
 models:
 - name: "fraud-detector-v47"
 memory_requirement_gb: 8
 compute_requirement: "medium"
 - name: "transaction-scorer-v12"
 memory_requirement_gb: 6
 compute_requirement: "low"
 - profile: "2g.20gb" # 2 compute slices, 20GB
 models:
 - name: "merchant-classifier-v8"
 memory_requirement_gb: 4
 compute_requirement: "low"
 - profile: "2g.20gb"
 models:
 - name: "user-segment-v15"
 memory_requirement_gb: 5
 compute_requirement: "low"

The tradeoff is isolation vs utilization. MIG provides memory isolation (one model can't access another model's memory), but compute isolation is approximate — a bursty model can temporarily starve its neighbors. For latency-sensitive models, we keep them on dedicated MIG instances with headroom. For best-effort models, we pack them more densely and accept occasional latency variance.

Model Compression Pipeline

The optimization pipeline isn't a one-shot process. Each serving optimization technique (quantization, pruning, distillation, operator fusion) has different accuracy-latency tradeoffs, and the optimal combination depends on the model architecture and the serving constraints.

We run a standardized compression pipeline on every model before deployment. The pipeline tries INT8 quantization first (lowest risk, highest impact), then applies operator fusion through ONNX optimization, then benchmarks the result against the latency and accuracy requirements. If the result doesn't meet the SLA, the pipeline tries more aggressive techniques: structured pruning at 20% sparsity, then 40%, checking accuracy at each step.

The compression pipeline produces a report for each model showing the original size, the compressed size, the latency improvement, and the accuracy difference on the evaluation set. This report is stored in the model registry alongside the compressed artifact, so anyone can see what optimizations were applied and their impact.

Model Serving Latency Optimization: Batching, Caching, and Hardware Selection