Neural Network Quantization: INT8, FP16 and Deployment Optimization

Deploying a 7-billion parameter model that runs fine on an A100 during research becomes a different problem when you need to serve it at 50ms p95 latency on a T4 or an edge device. The gap between training in FP32 and production inference is almost entirely a precision and memory bandwidth problem. Neural network quantization closes that gap by reducing the bit-width of weights and activations, trading negligible accuracy loss for dramatic improvements in throughput, memory footprint, and cost. In my experience leading deployment for vision and language models, moving from FP32 to a well-calibrated INT8 pipeline has consistently delivered 2.5-4x speedup and 3-4x memory reduction, provided the quantization strategy respects the distribution characteristics of each layer. This article details the mechanics behind INT8 and FP16 quantization, the calibration and training techniques that preserve accuracy, and the hardware-aware optimizations that make the difference between a model that merely runs and one that scales.

Symmetric vs Asymmetric Quantization: Choosing Your Mapping Strategy

At its core, quantization maps a floating-point range [r_min, r_max] to an integer range, typically [-128, 127] for signed INT8. The mapping is defined by a scale factor (S) and a zero-point (Z). The dequantized value is approximated as r ≈ S * (q - Z), where q is the quantized integer. How you choose S and Z determines whether you use symmetric or asymmetric quantization, and that choice has direct consequences for both accuracy and inference efficiency.

Symmetric quantization constrains Z to 0, which simplifies the arithmetic: r ≈ S * q. This is computationally cheaper because it eliminates the zero-point offset during matrix multiplication. I've found that symmetric quantization works exceptionally well for weights, which tend to be normally distributed around zero after training with weight decay. For activations, however, the story is different. ReLU and GELU outputs are strictly non-negative or heavily skewed, making asymmetric quantization with a non-zero Z far more effective at utilizing the full INT8 dynamic range. Forcing a skewed activation distribution into a symmetric mapping wastes nearly half of your representable values.

Per-Tensor and Per-Channel Granularity

Granularity is the other critical decision. Per-tensor quantization uses a single S and Z for an entire weight tensor, which is simple to implement but brittle. A single outlier channel can inflate the scale and crush the resolution for all other channels. Per-channel quantization assigns independent scale factors to each output channel of a convolutional or linear layer. In my work quantizing EfficientNet and ResNet variants, switching from per-tensor to per-channel for weights recovered nearly 1.2% top-1 accuracy on ImageNet with zero runtime overhead on modern hardware, since the scale can be folded into the requantization step. For a deeper look at how these architectural differences affect quantization sensitivity, see Modern CNNs: ResNet, EfficientNet and Vision Transformers Compared.

import torch

def symmetric_quantize(tensor, num_bits=8):
    # Per-channel symmetric quantization for weights
    # tensor shape: [out_channels, in_channels, kH, kW]
    qmin, qmax = -2**(num_bits-1), 2**(num_bits-1) - 1
    # Calculate scale per output channel
    max_val = tensor.abs().max(dim=1, keepdim=True)[0].max(dim=2, keepdim=True)[0].max(dim=3, keepdim=True)[0]
    scale = max_val / qmax
    scale = torch.clamp(scale, min=1e-8)
    
    # Quantize and dequantize to simulate precision loss
    q_tensor = torch.round(tensor / scale).clamp(qmin, qmax).to(torch.int8)
    dequantized = q_tensor.float() * scale
    return q_tensor, scale, dequantized

# Example: quantizing a Conv2d weight
conv_weight = torch.randn(64, 32, 3, 3)
q_weight, scales, dq_weight = symmetric_quantize(conv_weight)
print(f"Scale shape: {scales.shape}, Quantized range: [{q_weight.min()}, {q_weight.max()}]")

Calibrating INT8 Ranges: From Post-Training Quantization to Calibration Datasets

Post-Training Quantization (PTQ) is the fastest path to INT8. You take a trained FP32 model, run a small calibration dataset through it, and determine optimal clipping ranges for activations. It requires no retraining and can be completed in minutes, which makes it ideal for rapid deployment cycles. The challenge is that PTQ is entirely dependent on calibration quality.

The naive min-max calibration simply records the observed min and max of each activation tensor. This is fragile; a single outlier activation from an unrepresentative batch can set a scale that degrades resolution for 99.9% of values. In production, I've moved almost exclusively to entropy-based calibration using Kullback-Leibler divergence, as implemented in TensorRT. Instead of preserving the full range, it finds a clipping threshold T that minimizes the KL divergence between the original FP32 activation distribution and the quantized distribution. This intentionally saturates outliers to preserve resolution where the density is highest.

Building a Representative Calibration Set

Your calibration set does not need to be large — 500 to 2000 samples is typically sufficient — but it must be representative. I have seen INT8 models lose 3-4% accuracy because the calibration data was drawn from a preprocessing pipeline that normalized images differently than production. For language models, this means using real prompts with typical sequence lengths and padding behavior, not just short synthetic examples. For vision, include the full range of lighting conditions, aspect ratios, and class distributions you expect. I also recommend running calibration with the exact batch size you intend to deploy with, as activation statistics can shift subtly with batch-dependent layers. Techniques discussed in Training Deep Networks: Learning Rate Schedules, Batch Normalization and Gradient Clipping directly influence these activation distributions, so understanding your training-time normalization is a prerequisite for accurate PTQ.

import torch
from pytorch_quantization import quant_modules
from pytorch_quantization.nn import TensorQuantizer

# Enable PyTorch quantization with histogram calibrator
quant_modules.initialize()

# Define model - quantizers will be inserted automatically
model = get_pretrained_resnet50().cuda().eval()

# Configure calibrator (Histogram + KL divergence)
for name, module in model.named_modules():
    if isinstance(module, TensorQuantizer):
        module._calibrator._torch_hist = True
        module._calibrator._num_bins = 2048
        module._calibrator._calib_method = "histogram"

# Calibration loop - no gradients needed
with torch.no_grad():
    for i, (images, _) in enumerate(calibration_loader):
        if i >= 500: break
        model(images.cuda())

# After calibration, compute optimal scales via KL divergence
for module in model.modules():
    if isinstance(module, TensorQuantizer):
        if module._calibrator is not None:
            module.load_calib_amax()
            module.disable_calib()
            module.enable_quant()

# Now model runs in INT8 for linear/conv layers
torch.save(model.state_dict(), "resnet50_int8_ptq.pth")

For detailed API specifics on calibrators and quantizer modules, the PyTorch Documentation provides up-to-date guidance on native quantization support, including the FX Graph Mode Quantization workflow introduced in PyTorch 2.x.

Quantization-Aware Training: Simulating Precision Loss During Forward Passes

When PTQ fails to meet your accuracy target — typically when you see more than a 1% drop for CNNs or a significant perplexity increase for transformers — Quantization-Aware Training (QAT) is the next step. QAT does not actually train in INT8. Instead, it inserts fake quantization nodes into the graph that simulate INT8 rounding and clipping during the forward pass while keeping computations in FP32. The backward pass uses a Straight-Through Estimator (STE) to approximate gradients through the non-differentiable rounding operation.

The key insight is that the model learns to adapt its weight distributions to be more quantization-friendly. During QAT, weights that would have landed near a quantization bin boundary are nudged toward the center of a bin, making the final rounding less destructive. In my experience, QAT is non-negotiable for models with depthwise separable convolutions, such as MobileNetV3 and EfficientNet-Lite, and for transformer attention projections where activation variance is extremely high. Those architectures often lose 2-5% accuracy with PTQ alone, but recover to within 0.3-0.5% of FP32 after 10-15% of the original training schedule with QAT fine-tuning.

Implementing QAT Without Destabilizing Training

Do not start QAT from scratch. Start from a fully converged FP32 checkpoint, lower the learning rate by 10-100x, and disable aggressive augmentation like MixUp that changes the activation statistics. I typically freeze BatchNorm statistics for the first few epochs of QAT and use a small learning rate of 1e-5 to 5e-5 for AdamW. Quantize only the compute-intensive layers — Conv2d, Linear, and MatMul — while leaving Softmax, LayerNorm, and the final classifier head in FP32. Attempting to quantize LayerNorm in transformers almost always leads to instability because its variance computation is highly sensitive to low precision. The patterns behind attention and normalization are covered in depth in Transformer Architecture from Scratch: Attention, Encoders and Decoders, which clarifies why certain operations should remain in higher precision.

import torch.quantization as quant

# Prepare model for QAT from FP32 checkpoint
model = get_pretrained_transformer()
model.train()

# Define QAT configuration: per-channel symmetric for weights, per-tensor asymmetric for activations
qconfig = quant.QConfig(
    activation=quant.FakeQuantize.with_args(
        observer=quant.MovingAverageMinMaxObserver,
        quant_min=0, quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine
    ),
    weight=quant.FakeQuantize.with_args(
        observer=quant.MovingAveragePerChannelMinMaxObserver,
        quant_min=-128, quant_max=127, dtype=torch.qint8, qscheme=torch.per_channel_symmetric
    )
)
model.qconfig = qconfig
quant.prepare_qat(model, inplace=True)

# Fine-tune for 5-10 epochs with very low LR
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
for epoch in range(5):
    for batch in train_loader:
        optimizer.zero_grad()
        outputs = model(batch["input_ids"], batch["attention_mask"])
        loss = loss_fn(outputs, batch["labels"])
        loss.backward()
        optimizer.step()

# Convert to true INT8 model for deployment
model.eval()
quantized_model = quant.convert(model, inplace=False)
torch.jit.save(torch.jit.script(quantized_model), "model_int8_qat.pt")

Mixed-Precision Deployment: Orchestrating FP16, INT8 and FP32 on Modern Accelerators

Rarely will you deploy a model in pure INT8 end-to-end. The most performant deployments use mixed precision, strategically assigning different precisions to different operations based on their sensitivity and hardware support. FP16 (half precision) is the ideal intermediate: it halves memory and doubles throughput compared to FP32 on any modern NVIDIA GPU, while preserving enough dynamic range that it almost never degrades accuracy. INT8 provides the next 2x gain but requires more care.

My typical production recipe for a transformer on an NVIDIA T4, A10, or L4 is as follows: keep embeddings, LayerNorm, Softmax, and the final logits in FP32 or FP16, run all GEMMs (General Matrix Multiplications) for QKV projections and feed-forward networks in INT8, and keep attention scores in FP16. For CNNs on the same hardware, I quantize all backbone convolutions to INT8 but leave the detection head or segmentation head in FP16. This hybrid approach yields 90% of the performance benefit of full INT8 with a fraction of the accuracy risk.

Automatic Mixed Precision (AMP) during inference is simpler than QAT but still needs explicit control. Frameworks like TensorRT and ONNX Runtime allow you to set precision constraints per layer. I strongly recommend profiling with a tool like Nsight Systems to verify that your INT8 kernels are actually being selected. I have diagnosed multiple cases where a transpose or reshape operation forced a fallback to FP32, silently breaking the performance model and creating unnecessary dequantize-quantize pairs that add latency.

Kernel-Level Optimization: Leveraging Tensor Cores and Quantized GEMM

Quantization only translates to speedup if the underlying kernels can execute low-precision math efficiently. On NVIDIA GPUs, this means Tensor Cores. An A100 Tensor Core delivers 312 TFLOPS for FP16 and 624 TOPS for INT8, compared to just 19.5 TFLOPS for FP32. That theoretical 16x-32x gap is only realized when your matrix dimensions are multiples of the hardware requirements — typically 8 for FP16 and 16 for INT8 on Ampere and newer.

This has direct implications for model design. If your hidden dimension is 768 (as in BERT-Base), you are already aligned for INT8 GEMM. If it is 700, you will see padding overhead and reduced utilization. When quantizing, I pad weight matrices to the nearest alignment boundary and ensure batch sizes are multiples of 8. For weight-only quantization used in LLM serving (e.g., INT8 or INT4 weights with FP16 activations via bitsandbytes or GPTQ), the bottleneck shifts from compute to memory bandwidth. In that regime, the win comes not from faster math but from loading fewer bytes from HBM for each token generated. On a 70B model, going from FP16 to INT4 weights reduces the memory transfer per forward pass from 140GB to 35GB, which directly translates to lower time-per-token in autoregressive decoding.

On CPUs and edge devices, the equivalent is VNNI (Vector Neural Network Instructions) on Intel or DOT products on ARM. The PyTorch and ONNX runtimes will fuse QuantizeLinear + MatMul + DequantizeLinear patterns into a single quantized GEMM kernel. If your graph fails to fuse, you are paying the quantization overhead without the performance benefit. Always inspect the optimized graph. Recent work cataloged on arXiv Machine Learning continues to refine these fusion and kernel selection strategies, particularly for sparse and outlier-aware quantization methods.

Frequently Asked Questions