Shrinking Models Without Losing What They Know
You've got a 70B parameter language model that performs beautifully but costs $4 per 1000 requests to serve. Your budget allows $0.40. Knowledge distillation lets you train a smaller "student" model (say 1.3B or 7B parameters) that captures most of the larger "teacher" model's capabilities at a fraction of the serving cost. I've used this technique to cut inference costs by 10× while retaining 90-95% of the teacher's quality on targeted tasks.
The Core Mechanism
Standard training minimizes the cross-entropy between the student's predictions and hard labels (one-hot targets). Distillation adds a second objective: minimize the KL divergence between the student's output distribution and the teacher's output distribution. The teacher's "soft labels" contain richer information than hard labels — they encode which wrong answers are close to right, which helps the student learn the decision boundary more efficiently.
import torch.nn.functional as F
def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.5):
# Hard label loss (standard cross-entropy)
hard_loss = F.cross_entropy(student_logits, labels)
# Soft label loss (KL divergence with temperature scaling)
student_soft = F.log_softmax(student_logits / temperature, dim=-1)
teacher_soft = F.softmax(teacher_logits / temperature, dim=-1)
soft_loss = F.kl_div(student_soft, teacher_soft, reduction='batchmean')
soft_loss = soft_loss * (temperature ** 2) # scale by T² to match gradient magnitudes
return alpha * soft_loss + (1 - alpha) * hard_loss
The temperature parameter T controls how much the distribution is smoothed. Higher T makes the teacher's output distribution more uniform, emphasizing the relative rankings of classes rather than the absolute probabilities. In my experience, T=4 works well for classification tasks and T=2 for language modeling.
Distilling Language Models
For LLM distillation, the setup is different from classification. You're training the student to match the teacher's next-token probability distribution at every position in the sequence. The vocabulary is typically 32K-128K tokens, so the teacher's output distribution is high-dimensional and sparse.
def lm_distillation_step(student, teacher, input_ids, temperature=2.0, alpha=0.7):
with torch.no_grad():
teacher_out = teacher(input_ids)
teacher_logits = teacher_out.logits
student_out = student(input_ids)
student_logits = student_out.logits
# Shift for causal LM
shift_student = student_logits[:, :-1, :].contiguous()
shift_teacher = teacher_logits[:, :-1, :].contiguous()
shift_labels = input_ids[:, 1:].contiguous()
# Token-level distillation loss
B, S, V = shift_student.shape
student_soft = F.log_softmax(shift_student.view(-1, V) / temperature, dim=-1)
teacher_soft = F.softmax(shift_teacher.view(-1, V) / temperature, dim=-1)
kl_loss = F.kl_div(student_soft, teacher_soft, reduction='batchmean') * (temperature ** 2)
ce_loss = F.cross_entropy(shift_student.view(-1, V), shift_labels.view(-1))
return alpha * kl_loss + (1 - alpha) * ce_loss
Top-K Distillation
Computing KL divergence over the full 128K vocabulary is expensive. Most of the probability mass is concentrated in the top 50-100 tokens. Top-K distillation only computes the soft loss over the teacher's top-K predicted tokens, setting the rest to zero. With K=50, this reduces computation by 2500× while losing negligible information about the teacher's preferences.
Related reading: Model Registry Architecture and Versioning for Multi-Team Or.
Intermediate Layer Distillation
Output-only distillation misses the intermediate representations that the teacher builds. Matching hidden states or attention patterns from internal layers gives the student more supervisory signal. The tricky part is that teacher and student usually have different hidden dimensions and different numbers of layers.
The standard approach maps student layers to teacher layers (e.g., student layer 6 matches teacher layer 24) and adds a linear projection to align dimensions.
class LayerDistillationLoss(torch.nn.Module):
def __init__(self, student_dim, teacher_dim, layer_mapping):
super().__init__()
self.projections = torch.nn.ModuleDict()
self.layer_mapping = layer_mapping
for s_layer, t_layer in layer_mapping.items():
self.projections[str(s_layer)] = torch.nn.Linear(student_dim, teacher_dim)
def forward(self, student_hiddens, teacher_hiddens):
total_loss = 0.0
for s_idx, t_idx in self.layer_mapping.items():
s_hidden = self.projections[str(s_idx)](student_hiddens[s_idx])
t_hidden = teacher_hiddens[t_idx].detach()
total_loss += F.mse_loss(s_hidden, t_hidden)
return total_loss / len(self.layer_mapping)
In my experiments, intermediate distillation adds 2-5% quality on top of output-only distillation. The improvement is more pronounced for tasks requiring complex reasoning, where the teacher's intermediate representations encode reasoning steps that output distributions don't fully capture.
Data Selection for Distillation
You don't need (and shouldn't use) the same data for distillation that you used for training the teacher. The distillation data should be diverse and representative of the student's deployment domain. I've found that using 10-20% of the teacher's training data is usually sufficient — the soft labels provide so much more signal per example than hard labels that you need fewer examples to reach the same quality.
For a related perspective, see Time Series Forecasting with Temporal Fusion Transformers.
An underappreciated trick: generate synthetic data by sampling from the teacher model, then distill on that. The teacher's own generations are naturally well-calibrated with its output distribution, making them ideal distillation targets. This is essentially self-distillation with data augmentation, and it works surprisingly well when you don't have access to the original training data.
Progressive Distillation
For large capability gaps (distilling 70B → 1.3B), direct distillation often struggles. The student simply can't replicate the teacher's behavior — it lacks the capacity. Progressive distillation bridges this gap by distilling through intermediate-sized models: 70B → 13B → 3B → 1.3B.
Each stage reduces model size by roughly 4×, which is a compression ratio the student can handle. The intermediate models also serve as fallback options — if the 1.3B student isn't good enough, you can deploy the 3B version at moderate extra cost.
We used progressive distillation for a domain-specific chatbot. The 70B teacher scored 85% on our quality benchmark. Direct distillation to 1.3B scored 71%. Progressive distillation (70B → 7B → 1.3B) scored 78%. Seven percentage points for free is hard to argue with.
This connects to the ideas in Object Detection Model Selection: YOLO vs DETR vs EfficientD.
Task-Specific vs General Distillation
If you know exactly what task the student will handle, distilling on that specific task's data produces the best results. But if the student needs to handle multiple tasks or the task distribution is uncertain, general distillation on a broad text corpus gives more transferable capabilities.
A hybrid approach works well: general distillation first (on a broad corpus, matching the teacher's language modeling distribution), then task-specific distillation (on your evaluation domain). The general phase gives the student broad linguistic capabilities; the task-specific phase sharpens performance where it matters for your application.
Measuring Distillation Quality
Don't just measure task accuracy. The whole point of distillation is that the student learns a good probability distribution, not just the argmax. Check calibration (are the student's confidence scores meaningful?), check coverage (does the student produce diverse outputs?), and check failure modes (where does the student disagree with the teacher, and why?).
I track the fraction of top-1 predictions where student and teacher agree, the average KL divergence on held-out data, and task-specific metrics. If the student agrees with the teacher's top-1 prediction 90%+ of the time but has high KL divergence, it's likely overconfident — the right answers are right but the uncertainty estimates are wrong, which matters for any downstream decision-making.