Your Model's First Step Determines Its Path
Weight initialization is the kind of topic that gets a paragraph in most tutorials and then never comes up again until training fails spectacularly. I've seen models that refused to converge, produced NaN after 50 steps, or trained 3× slower than expected — all traced back to initialization. It's not glamorous, but getting it wrong costs real GPU-hours.
The Variance Problem
Here's the fundamental issue. If you initialize weights too large, activations grow exponentially through layers. After 50 layers, a signal that starts at magnitude 1.0 could be at 1e15 — overflow territory. If weights are too small, the signal shrinks to zero and gradients vanish. The goal of every initialization scheme is to keep activation variance approximately constant across layers.
For a linear layer y = Wx with input dimension n_in, if each weight element is drawn from a distribution with variance σ², then Var(y) = n_in × σ² × Var(x). To maintain Var(y) = Var(x), we need σ² = 1/n_in. That's the core math behind most initialization methods.
Xavier/Glorot Initialization
Xavier initialization sets σ² = 2/(n_in + n_out), balancing the forward pass variance preservation with backward pass gradient preservation. It works well for sigmoid and tanh activations where the linear region near zero means the activation's derivative is close to 1.
# Xavier uniform
torch.nn.init.xavier_uniform_(layer.weight)
# Xavier normal
torch.nn.init.xavier_normal_(layer.weight)
Kaiming/He Initialization
ReLU changes the math. Since ReLU zeros out negative values, it halves the variance of activations. Kaiming initialization accounts for this by using σ² = 2/n_in (for the forward pass) or σ² = 2/n_out (for the backward pass). In practice, most teams use the fan_in mode.
For a related perspective, see Image Segmentation Pipelines for Medical Imaging: U-Net Vari.
# Kaiming for ReLU layers
torch.nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu')
# For GELU or other activations, the gain factor differs
torch.nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='leaky_relu', a=0.01)
I've found that for GELU activations (standard in modern transformers), using the leaky_relu nonlinearity with a=0.01 gives better initial training dynamics than the relu option. GELU's shape is closer to leaky ReLU than to hard ReLU, so the variance estimate is more accurate.
Transformer-Specific Initialization
Standard feedforward initialization formulas don't account for the residual connections in transformers. Each transformer block adds its output to the residual stream: x = x + sublayer(x). After L layers, the residual stream variance grows linearly with L if each sublayer preserves variance. For a 96-layer model, the final layer's activations are roughly 96× larger than the input.
GPT-2 and subsequent models handle this by scaling the output projection of each sublayer by 1/√(2L). This ensures that the cumulative effect of all residual additions maintains reasonable activation magnitudes.
class TransformerBlock(torch.nn.Module):
def __init__(self, hidden_dim, num_heads, num_layers):
super().__init__()
self.attn = MultiHeadAttention(hidden_dim, num_heads)
self.ffn = FeedForward(hidden_dim)
# Scale output projections by 1/sqrt(2*num_layers)
scale = (2 * num_layers) ** -0.5
torch.nn.init.normal_(self.attn.out_proj.weight, std=0.02 * scale)
torch.nn.init.normal_(self.ffn.fc2.weight, std=0.02 * scale)
The Small Init Controversy
There's been a trend toward very small initialization (σ=0.001 or even 0.0001) combined with proper learning rate warmup. The theory is that starting very small forces the model into a "kernel regime" where early training is essentially linear, making optimization easier. Once warmup is complete and the learning rate increases, the model transitions to the non-linear regime with well-conditioned gradients.
We covered a related topic in Sparse Attention Patterns for Long-Sequence Transformers.
In my experiments, small initialization with warmup converges faster in the first 10% of training but reaches the same final performance as properly scaled initialization. The real benefit is stability — I've never seen a NaN divergence with small initialization, while standard Kaiming initialization occasionally produces instabilities in deep models (60+ layers). The small init approach is more forgiving of architectural variations.
Initialization for Specific Components
Embedding Layers
Embedding tables are typically initialized with N(0, 0.02) in language models. This isn't theoretically motivated — it's empirical. The embedding dimension is usually large enough (768-8192) that the law of large numbers keeps downstream activations well-behaved regardless of the exact initialization variance. What does matter is that the embedding scale matches the rest of the network. If your hidden layers produce activations around magnitude 1.0, embeddings should too.
Layer Normalization
LayerNorm parameters (gamma and beta) should initialize to gamma=1, beta=0 so the initial behavior is identity normalization. I've seen codebases where gamma was initialized randomly, which defeats the purpose of careful weight initialization by immediately rescaling activations to random magnitudes.
Attention QKV Projections
A common practice is to initialize Q and K projections with smaller variance than V projections. The reasoning: Q^T K produces dot products that should start small (to produce a near-uniform attention distribution), while V projections determine the magnitude of the output. I've seen this improve convergence speed by 5-10% in practice, though it's not universally applied.
Related reading: Graph Neural Networks for Fraud Detection: Architecture and .
Diagnosing Initialization Problems
Before training starts, run a single forward pass and check these three things. First, activation magnitudes across layers should stay within an order of magnitude — if layer 1 produces activations around 1.0 and layer 50 produces activations around 100.0, your initialization has a variance growth problem.
Second, gradient magnitudes across layers should be roughly similar. Wildly different gradient scales mean either vanishing or exploding gradients during actual training.
Third, loss at initialization should be close to the theoretically expected value. For a classification task with C classes, the initial loss should be near -log(1/C) = log(C). If it's much higher, your model is producing overconfident wrong predictions, which usually means the final layer's weights are too large.
def check_initialization(model, sample_input):
activations = {}
hooks = []
for name, module in model.named_modules():
if isinstance(module, (torch.nn.Linear, torch.nn.LayerNorm)):
hook = module.register_forward_hook(
lambda m, i, o, n=name: activations.__setitem__(n, o.detach())
)
hooks.append(hook)
with torch.no_grad():
model(sample_input)
for name, act in activations.items():
print(f"{name}: mean={act.mean():.4f}, std={act.std():.4f}, "
f"max={act.abs().max():.4f}")
for h in hooks:
h.remove()
Run this diagnostic before committing to a long training run. It takes seconds and can save you days.