Fine-Tuning Language Models with LoRA:...

LoRA Isn't Magic, But It's Close Enough

Low-Rank Adaptation changed how we think about fine-tuning large models. Instead of updating every parameter — which requires storing optimizer states for billions of weights — LoRA freezes the pretrained model and injects small trainable matrices into specific layers. The memory savings are dramatic: fine-tuning a 7B parameter model drops from needing 80GB of GPU memory to under 16GB.

But "just add LoRA" isn't a strategy. The rank you choose, the layers you target, the learning rate — all of these interact in ways that the original paper only scratched the surface of. After fine-tuning dozens of models with LoRA across different tasks, I've developed strong opinions about what works and what's a waste of GPU cycles.

Rank Selection: The Parameter That Matters Most

LoRA decomposes weight updates into two low-rank matrices. The rank r controls the expressiveness of the adaptation. Higher rank means more trainable parameters and more capacity to learn task-specific representations. Lower rank means faster training and less risk of overfitting.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
 def __init__(self, original, rank, alpha=1.0):
 super().__init__()
 self.original = original
 self.original.weight.requires_grad_(False)
 d_in = original.in_features
 d_out = original.out_features
 self.lora_A = nn.Parameter(torch.randn(rank, d_in) * 0.01)
 self.lora_B = nn.Parameter(torch.zeros(d_out, rank))
 self.scaling = alpha / rank

 def forward(self, x):
 base_out = self.original(x)
 lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
 return base_out + lora_out

The common advice says "start with rank 8 or 16." That's fine as a starting point, but here's what the advice misses: the optimal rank depends heavily on the distance between your pretrained model's knowledge and your target task.

For tasks close to the pretraining distribution — like fine-tuning a general language model for a specific writing style — rank 4 often suffices. The model already knows the language; you're just steering it. For tasks that require new knowledge — like adapting a general model to understand medical terminology — you'll need rank 32 or higher because the model needs to learn new representations, not just adjust existing ones.

The Alpha Scaling Factor

The scaling factor alpha divided by rank controls how much the LoRA weights influence the output relative to the frozen base model. Setting alpha equal to rank gives the LoRA path equal weight to the original weights. Setting alpha to twice the rank doubles LoRA's influence.

I've found that alpha = 2x rank works well as a default for most tasks. It lets the LoRA adaptation have meaningful impact without overwhelming the pretrained knowledge. If your training loss plateaus early, try increasing alpha. If it oscillates, decrease it.

Layer Targeting Strategy

Not all layers benefit equally from LoRA. The original paper applied LoRA to the query and value projection matrices in self-attention. Later work showed that including key projections and MLP layers helps for more complex tasks. But targeting everything is wasteful — some layers don't need adaptation for your specific task.

My rule of thumb: start with attention projections only (q_proj, v_proj). If performance is insufficient, add k_proj and o_proj. If it's still not enough, add the MLP layers. Each addition roughly doubles your trainable parameter count.

from peft import LoraConfig, get_peft_model

# Conservative: attention only
config_minimal = LoraConfig(
 r=16, lora_alpha=32,
 target_modules=["q_proj", "v_proj"],
 lora_dropout=0.05, bias="none",
 task_type="CAUSAL_LM",
)

# Moderate: full attention
config_moderate = LoraConfig(
 r=16, lora_alpha=32,
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
 lora_dropout=0.05, bias="none",
 task_type="CAUSAL_LM",
)

# Aggressive: attention + MLP
config_full = LoraConfig(
 r=32, lora_alpha=64,
 target_modules=[
 "q_proj", "k_proj", "v_proj", "o_proj",
 "gate_proj", "up_proj", "down_proj"
 ],
 lora_dropout=0.1, bias="none",
 task_type="CAUSAL_LM",
)

Training Hyperparameters That Actually Matter

Learning rate is the single most important hyperparameter for LoRA fine-tuning, and it should be much higher than you'd use for full fine-tuning. Where full fine-tuning might use 1e-5 or 2e-5, LoRA works best with 1e-4 to 3e-4. The reason: you're training far fewer parameters, so each parameter needs to move more per step to have visible effect.

For a related perspective, see Mixed Precision Training with PyTorch: When FP16 Breaks and .

Dropout on the LoRA matrices prevents overfitting, especially at higher ranks. For datasets under 10k examples, I use 0.1 dropout. For larger datasets, 0.05 or even 0.0 works fine. Watch your validation loss — if it starts climbing while training loss keeps dropping, increase dropout or reduce rank.

Batch size interacts with LoRA in a non-obvious way. Because the LoRA path has fewer parameters, gradient noise is relatively higher. Larger batch sizes help stabilize training. If you can't increase the physical batch size due to memory constraints, use gradient accumulation — 4 to 8 accumulation steps is usually enough to smooth things out.

Merging and Serving

Once training is done, you have two options for serving: keep the LoRA weights separate and add them at inference time, or merge them into the base model permanently. Merging is simpler and has zero inference overhead, but you lose the ability to swap or combine multiple LoRA adapters.

# Merge LoRA weights into base model
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged_model_dir")

# Or keep separate for multi-adapter serving
model.save_pretrained("lora_adapter_dir")

For serving multiple tasks from one base model, keeping adapters separate is powerful. You load the base model once into GPU memory and swap adapters per request. With quantized base models, this means you can serve dozens of specialized tasks from a single GPU. That's the real production value of LoRA — not just cheaper training, but flexible multi-task serving with minimal hardware.

Fine-Tuning Language Models with LoRA: Rank Selection and Layer Targeting