Beyond BERT Embeddings
Sentence embeddings have come a long way since averaging BERT token outputs and hoping for the best. That approach produced embeddings that were surprisingly poor at capturing sentence-level meaning — two sentences with completely different meanings but overlapping vocabulary would end up with similar representations. The breakthrough came from contrastive learning, and understanding why it works tells you a lot about how to train and evaluate embeddings for your own use case.
How Contrastive Learning Fixes the Embedding Problem
The core idea is straightforward. Take a pair of semantically similar sentences (positive pair), and a pair of dissimilar ones (negative pair). Train the model to produce embeddings that are close for positive pairs and far apart for negative pairs. The training objective is usually some form of InfoNCE loss:
import torch
import torch.nn.functional as F
def info_nce_loss(query_embs, positive_embs, temperature=0.05):
query_embs = F.normalize(query_embs, dim=1)
positive_embs = F.normalize(positive_embs, dim=1)
similarity = torch.matmul(query_embs, positive_embs.T) / temperature
labels = torch.arange(similarity.size(0), device=similarity.device)
loss = F.cross_entropy(similarity, labels)
return lossThe temperature parameter controls how sharp the similarity distribution is. Lower temperature (0.01-0.05) makes the model more discriminative — it pushes negatives farther apart and pulls positives closer together. Higher temperature (0.1-0.5) is more forgiving. In my experience, 0.05 works well for most retrieval tasks, but you should tune it on your validation set since the optimal value depends on your negative difficulty.
In-Batch Negatives: The Free Lunch
The clever part of modern contrastive training is using other examples in the same batch as negative pairs. In a batch of 256 sentence pairs, each query has 1 positive and 255 negatives. This scales your effective negative count with batch size, which is why contrastive embedding training benefits enormously from large batches — we've seen consistent improvements up to batch size 2048.
For a related perspective, see Anomaly Detection in High-Dimensional Industrial Sensor Data.
There's a catch, though. In-batch negatives assume that any two random sentences are dissimilar. For general-purpose embeddings, this holds most of the time. For domain-specific training where many sentences are semantically related, random negatives aren't negative enough. You'll need hard negative mining — selecting negatives that are close in embedding space but semantically different.
Hard Negative Mining Strategies
Hard negatives are the single biggest lever for improving embedding quality after basic contrastive training. A hard negative is a sentence that's superficially similar to the query (high BM25 score, overlapping vocabulary) but answers a different question or describes a different concept.
import numpy as np
def mine_hard_negatives(queries, corpus, model, bm25_index, k=10):
hard_negs = []
for query, positive_id in queries:
bm25_candidates = bm25_index.search(query, k=50)
candidates = [c for c in bm25_candidates if c.id != positive_id]
query_emb = model.encode(query)
cand_embs = model.encode([c.text for c in candidates])
sims = np.dot(query_emb, cand_embs.T) / (
np.linalg.norm(query_emb) * np.linalg.norm(cand_embs, axis=1)
)
sorted_idx = np.argsort(-sims)
hard_negs.append([candidates[i] for i in sorted_idx[:k]])
return hard_negsTraining with hard negatives is unstable if you're not careful. Start with easy negatives (random) for a few epochs to get the model into a reasonable state, then switch to a mix of 50% random and 50% hard negatives. Pure hard negative training often causes the model to oscillate between states or collapse entirely. We learned this the hard way when a training run that looked great on the loss curve produced embeddings where every document had a cosine similarity above 0.95 with every other document.
This connects to the ideas in Knowledge Distillation from Large Language Models to Efficie.
Evaluation Beyond Standard Benchmarks
The MTEB leaderboard is the standard evaluation suite, and it's genuinely useful for comparing general-purpose models. But it doesn't tell you how a model will perform on your data. Here's what I actually evaluate when selecting an embedding model for a production system:
Retrieval accuracy on your domain is first. Build a test set of (query, relevant document) pairs from your actual use case — 200-500 pairs is usually enough to get stable metrics. Measure Recall@1, Recall@5, and Mean Reciprocal Rank. The model that tops MTEB might rank fifth on your domain.
Semantic textual similarity on your domain comes next. Take pairs of sentences from your data and have humans rate their similarity on a 1-5 scale. Compute Spearman correlation between model similarities and human ratings. This catches cases where the model's notion of "similar" doesn't match what your users mean by similar.
Related reading: Canary Deployments for ML Models: Traffic Splitting and Roll.
Speed and memory on your hardware matters more than people think. MTEB doesn't measure inference speed, which is critical in production. A model that's 2 points better on the benchmark but 3x slower might not be worth it when you're encoding 100k documents per day.
Practical Model Selection
For most production use cases today, the best cost-efficiency sits in the 384-to-768 dimension range. Larger dimensions (1024+) offer marginal quality improvements but double your storage and search costs. Unless you've measured a meaningful quality difference on your data, stick with smaller dimensions.
Multilingual models consistently underperform monolingual ones by 3-5 points on same-language tasks. If you know your data is English-only, use an English-specific model. The multilingual tax is real, even on the best models available today.
Don't sleep on matryoshka embeddings — models trained to work at multiple dimension sizes. You can store 768-dimensional embeddings but search at 256 dimensions for speed, then re-rank the top results at full dimensionality. This gives you the retrieval speed of a small model with the re-ranking quality of a large one, and the storage cost is paid once. It's one of the few genuinely free optimizations in this space.