Making Models Smaller and Faster for Inference
A 7B parameter model in FP16 weighs 14GB. In INT8, it's 7GB. In INT4, it's 3.5GB. That's the difference between needing an A100 and running on a consumer GPU, or between serving 10 concurrent requests and serving 40. Quantization is how you get there, and the two main approaches — quantization-aware training (QAT) and post-training quantization (PTQ) — differ dramatically in effort, quality, and when they make sense.
Post-Training Quantization (PTQ)
PTQ is the quick path. You take a trained FP16 model, run a calibration dataset through it to collect activation statistics, and use those statistics to determine quantization parameters (scale and zero-point) for each layer. No retraining needed. You can quantize a 70B model in under an hour.
Weight-Only Quantization
The simplest PTQ variant quantizes only weights, keeping activations in FP16. This works because weights are static — their range is fixed after training, so you can find optimal quantization parameters once and be done. Activations vary per input, making them harder to quantize well.
def quantize_weights_symmetric(weight, num_bits=8):
# Find the max absolute value per output channel
max_val = weight.abs().amax(dim=1, keepdim=True)
scale = max_val / (2 ** (num_bits - 1) - 1)
scale = scale.clamp(min=1e-8) # avoid division by zero
# Quantize and dequantize
weight_int = torch.round(weight / scale).clamp(
-(2 ** (num_bits - 1)), 2 ** (num_bits - 1) - 1
)
weight_deq = weight_int * scale
return weight_deq, scale
Weight-only INT8 quantization preserves 99.5%+ of model quality across every benchmark I've tested. It's the lowest-risk optimization you can make to a production model, and I apply it by default to any model that'll be served in production.
GPTQ: The Standard for 4-Bit
GPTQ is the go-to algorithm for 4-bit weight quantization. It quantizes weights one layer at a time, choosing quantization parameters that minimize the layer's output error on a calibration dataset. The key insight is using second-order information (the Hessian approximation) to decide which weights can tolerate more quantization error.
In practice, GPTQ with 4-bit quantization and 128-group-size loses about 0.5-1.5 perplexity points on language modeling tasks compared to FP16. For most production use cases (chatbots, summarization, classification), this quality drop is imperceptible.
We covered a related topic in Camera Calibration and Geometric Transforms for 3D Vision Ap.
# Using auto-gptq library
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
quantize_config = BaseQuantizeConfig(
bits=4,
group_size=128,
desc_act=True, # activation-order quantization
)
model = AutoGPTQForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantize_config=quantize_config,
)
model.quantize(calibration_dataset)
model.save_quantized("llama-2-7b-gptq-4bit")
The group_size parameter controls the granularity. Each group of 128 weights shares one scale factor. Smaller groups (64 or 32) improve quality but increase the metadata overhead and slow down dequantization during inference.
Quantization-Aware Training (QAT)
QAT inserts fake quantization operations into the training graph. During forward, weights and activations are quantized and immediately dequantized (simulating quantization noise). During backward, gradients flow through these fake quantization ops using the straight-through estimator (STE). The model learns to be robust to quantization noise during training.
class FakeQuantize(torch.autograd.Function):
@staticmethod
def forward(ctx, x, scale, zero_point, num_bits=8):
qmin = -(2 ** (num_bits - 1))
qmax = 2 ** (num_bits - 1) - 1
x_int = torch.round(x / scale + zero_point).clamp(qmin, qmax)
x_deq = (x_int - zero_point) * scale
return x_deq
@staticmethod
def backward(ctx, grad_output):
# Straight-through estimator: pass gradients unchanged
return grad_output, None, None, None
QAT consistently outperforms PTQ, especially at aggressive quantization levels (4-bit and below). At INT8, the difference is small — maybe 0.1% accuracy on classification tasks. At INT4, the difference is significant — 1-3% accuracy on classification, 0.5-1.0 perplexity points on language modeling. At INT2 (extreme quantization), QAT is practically the only approach that produces usable models.
INT4 Benchmarks: QAT vs PTQ
I ran systematic comparisons on a Llama-7B class model across three tasks. On WikiText-2 perplexity, FP16 scored 5.68, GPTQ INT4 scored 6.12, AWQ INT4 scored 5.98, and QAT INT4 scored 5.84. On MMLU accuracy, FP16 hit 46.3%, GPTQ hit 44.8%, AWQ hit 45.1%, and QAT hit 45.9%.
The QAT advantage is consistent but modest at INT4. The question is whether it's worth the training cost. QAT requires 5-10% of the original pre-training compute budget for the quantization-aware fine-tuning phase. For a 7B model, that's maybe 100 A100-hours. For a 70B model, it's 1000+ A100-hours. If you're deploying at scale and serving millions of requests, that training cost pays for itself quickly through the small quality improvement. For a model serving 100 requests per day, PTQ is perfectly fine.
Related reading: Prompt Engineering as Software Engineering: Version Control .
AWQ: Activation-Aware Weight Quantization
AWQ occupies an interesting middle ground between PTQ and QAT. It observes that some weights are more important than others based on the activation magnitudes they interact with. Weights that multiply large activations get higher precision treatment (they're scaled up before quantization and the activations are scaled down proportionally).
In practice, AWQ consistently beats GPTQ by 0.1-0.3 perplexity points on language modeling with the same 4-bit configuration. The implementation is also faster — quantization takes minutes instead of hours because it doesn't require running calibration data through the full model layer by layer.
Hardware Considerations
Quantized models only accelerate inference if the hardware supports the quantized data type. INT8 matrix multiplication is natively supported on A100, H100, and consumer GPUs from RTX 3000 series onward. INT4 support is less universal — the compute kernels (like ExLlama for GPTQ models or Marlin for AWQ) implement INT4 as packed INT4 operations that get unpacked during matrix multiplication.
On an A100, INT4 inference with the Marlin kernel is about 3.5× faster than FP16 for batch size 1 (memory-bandwidth bound) and about 1.8× faster for batch size 32 (compute bound). The speedup varies by batch size because the bottleneck shifts from memory bandwidth to compute throughput as batches grow.
Mixed-Precision Quantization
Not all layers need the same precision. Attention layers are generally more sensitive to quantization than FFN layers because the softmax amplifies small errors in the attention scores. A common approach is to keep attention Q/K projections in INT8 while quantizing FFN weights to INT4.
For a related perspective, see Knowledge Distillation from Large Language Models to Efficie.
I've also found that the first and last transformer layers are disproportionately sensitive. Keeping these in INT8 while aggressively quantizing middle layers to INT4 gives nearly the same quality as uniform INT8 at the size of uniform INT4. It's an easy win that costs only a tiny amount of extra memory.
Practical Recommendations
For production inference: start with INT8 weight-only PTQ. It's a 5-minute operation that halves memory usage with essentially no quality loss. If you need further compression, move to INT4 with AWQ. If even that isn't enough quality, invest in QAT.
For edge deployment (phones, embedded devices): QAT at INT4 or even INT3 is usually necessary because you need both the quality and the smallest possible model size. Budget the training time.
For serving large models on limited GPUs: GPTQ or AWQ at 4-bit lets you run 70B parameter models on a single A100 80GB with room for KV cache. This is often the most cost-effective serving setup — one A100 instead of four.