Text Embedding Models: Sentence-BERT, E5 and Cohere Embed Compared

Choosing a text embedding model feels like a solved problem until you put one into production and watch your retrieval quality collapse on real user queries. Over the last three years I have shipped vector search for support ticket routing, semantic deduplication, and retrieval-augmented generation, and the lesson that keeps repeating is that no single embedding model is best for every workload. Sentence-BERT established the pattern for fast, reusable sentence vectors, E5 showed how contrastive pre-training with instructions can close the gap between academic benchmarks and asymmetric search, and Cohere Embed v3 demonstrated what a commercially tuned, API-first embedding service looks like when you control the entire training and serving stack. This article compares the three approaches on architecture, training data, practical retrieval behavior, and operational cost, with code patterns I actually use to evaluate and deploy them.

Sentence-BERT's Siamese Roots: Why Mean Pooling Still Beats CLS for Similarity

Sentence-BERT (SBERT) was introduced to fix a very specific pain point with BERT and RoBERTa: they produce great token representations but terrible sentence representations out of the box. If you take the [CLS] token from a vanilla BERT model and compute cosine similarity, you get correlations with human judgments that are barely better than averaging GloVe vectors. The SBERT paper proposed a siamese and triplet network structure on top of a pretrained transformer, fine-tuned with a natural language inference objective, so that the resulting embeddings can be compared with cosine similarity directly.

In my experience, the architectural simplicity is the reason SBERT variants like all-MiniLM-L6-v2 and all-mpnet-base-v2 are still everywhere in production. The pipeline is: tokenize, run through transformer, apply pooling, normalize. Most SBERT checkpoints use mean pooling over token embeddings with attention mask weighting, not the CLS token. I've found that mean pooling is more robust to truncation and domain shift, especially for short, noisy text like search queries or chat messages. A CLS vector tends to overfit to the NLI training distribution, while mean pooling preserves token-level signal.

The training objective matters more than the base model size. The original SBERT was fine-tuned on SNLI and MultiNLI with a classification objective, then refined with triplet loss on semantic textual similarity data. Modern checkpoints from the sentence-transformers library add curated retrieval pairs, paraphrase data, and community QA pairs. That makes them surprisingly strong for symmetric tasks like duplicate detection, clustering, and semantic textual similarity where query and document are interchangeable.

The mean-pooling trick and why normalization is non-negotiable

If you look at the implementation in Hugging Face Documentation, SBERT does not add a projection head by default. The embedding dimension is the hidden size of the backbone: 384 for MiniLM-L6, 768 for MPNet-base. After pooling, the vector is L2-normalized if you use normalize_embeddings=True. I always normalize. Without normalization, cosine similarity and dot product diverge, and your vector database will rank incorrectly if it assumes cosine. In FAISS, for example, an inner product index on normalized vectors is equivalent to cosine similarity, which lets you use fast IndexFlatIP or IndexHNSWFlat without a separate cosine computation.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")

# In my experience, batch size 64-128 is the sweet spot on an A10
queries = ["how to reset my password", "refund policy for annual plan"]
docs = ["To reset your password, go to account settings.", 
        "Annual subscriptions are refundable within 30 days."]

# Always normalize for cosine search
q_emb = model.encode(queries, normalize_embeddings=True, batch_size=64)
d_emb = model.encode(docs, normalize_embeddings=True, batch_size=64)

# cosine similarity = dot product on normalized vectors
scores = np.dot(q_emb, d_emb.T)
print(scores)

Where SBERT starts to break in asymmetric retrieval

The failure mode I see most often is asymmetric retrieval, where the query is short and the document is long. SBERT was largely trained on symmetric pairs, so a query like "cohere embed pricing" and a 400-token documentation page end up in different parts of the embedding space. You can mitigate this by chunking documents to 150-250 tokens and by fine-tuning with contrastive loss on your own query-document pairs, but at that point you are re-implementing what E5 and Cohere have already done at scale. For pure semantic search where query and passage lengths differ, I now treat vanilla SBERT as a strong baseline, not the final choice. It shines for clustering, classification via embeddings, and internal deduplication pipelines like those described in Text Classification Pipelines for Production: From Training to Serving.

E5 and the Instruction Prefix Revolution in Contrastive Pre-Training

E5 (EmbEddings from bidirEctional Encoder Representations) from Microsoft took the SBERT recipe and fixed two things: the scale and quality of contrastive pre-training, and the handling of asymmetric tasks. The key idea is embarrassingly simple: prefix every input with an instruction that tells the model what kind of embedding you want.

E5 was trained in two stages. First, a massive weakly supervised contrastive pre-training on 270M+ text pairs mined from Common Crawl, StackExchange, Reddit, and other sources, using InfoNCE loss with in-batch negatives. Second, supervised fine-tuning on high-quality labeled datasets including MS MARCO, NQ, and NLI. The instruction prefix was introduced during this stage: queries get query: and passages get passage: . The newer E5-Mistral variant extends this to decoder-only models with prompts like Instruct: Given a query, retrieve relevant passages\nQuery: .

That prefix is not a gimmick. In my benchmarks on an internal support KB of ~200k passages, removing the query: prefix dropped Recall@10 by 6-9 points on E5-base-v2. The model actually learns distinct subspaces for queries and documents. If you embed both sides with the same prefix, you undo the asymmetric tuning and fall back to SBERT-like symmetric behavior.

Query vs. passage prefixes in practice

You have to be disciplined in your ingestion pipeline. I keep two separate embedding functions and I never cache without storing which prefix was used. For E5-small/base/large, the tokenizer does not have special tokens for these prefixes; they are just plain text. That means truncation matters. The models have a 512 token limit, and the prefix consumes tokens. I truncate to 508 tokens for passages to leave room, and I always truncate queries on the right.

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

model_id = "intfloat/e5-base-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

def e5_embed(texts, prefix, normalize=True):
    # prefix is "query: " or "passage: "
    inputs = tokenizer([prefix + t for t in texts], 
                       padding=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs)
        # attention-weighted mean pooling
        mask = inputs['attention_mask'].unsqueeze(-1).expand(out.last_hidden_state.size()).float()
        summed = torch.sum(out.last_hidden_state * mask, 1)
        counts = torch.clamp(mask.sum(1), min=1e-9)
        emb = summed / counts
        if normalize:
            emb = F.normalize(emb, p=2, dim=1)
        return emb

queries = e5_embed(["how to upgrade plan"], prefix="query: ")
passages = e5_embed(["To upgrade your plan, visit billing settings..."], prefix="passage: ")

The code above follows the pooling approach recommended in the E5 repository and matches the SBERT mean-pooling logic. Check the PyTorch Documentation for torch.nn.functional.normalize if you want to implement max pooling or CLS pooling alternatives, but I have not seen them outperform mean pooling for retrieval.

E5-Mistral and the decoder shift

E5-Mistral-7B-instruct is a different beast. It uses a Mistral-7B backbone with last-token pooling and requires an instruction style prompt. It is substantially better on BEIR and MTEB, often beating Cohere Embed v3 on academic benchmarks, but at 7B parameters the inference cost is 15-20x higher than E5-base. I've found it useful when I need a high-quality teacher model to distill from or to re-rank top-100 results offline, not as a real-time encoder for vector search. If you are building RAG Architecture: Building Retrieval-Augmented Generation Systems, a common pattern is E5-base for first-stage retrieval and E5-Mistral or a cross-encoder for second-stage ranking.

Cohere Embed v3 Under the Hood: Input Types, Matryoshka and API-First Design

Cohere Embed v3 is not an open-weight model. You call an API, send text, and get vectors back. That sounds trivial, but the design choices behind the API reveal how much product thinking went into the training.

The biggest difference from SBERT and E5 is the explicit input_type parameter: search_query, search_document, classification, and clustering. Each type triggers a different internal projection. In practice, this is Cohere's version of E5's prefixes, but enforced server-side so you cannot forget. I have seen teams accidentally embed queries and documents with the same SBERT model and wonder why recall is poor; Cohere makes that mistake impossible. The second difference is Matryoshka Representation Learning (MRL). V3 embeddings are trained so that the first k dimensions are still useful. You can request 1024, 512, 256, or even 384 dimensions and get a truncated but meaningful vector without re-embedding. For cost-sensitive indexes, this is a real advantage.

Dimensionality is 1024 for Embed English v3 and Embed Multilingual v3. Compare that to 384 or 768 for SBERT/E5-base. A 1024-dim float32 vector is 4KB. At 10 million documents, that is ~40GB of raw vectors before HNSW overhead. Cohere lets you store 512-dim or 384-dim vectors at indexing time to cut memory in half while keeping most of the quality, which I have used to keep an index on a single 64GB instance instead of sharding.

Production behavior I have observed with Cohere Embed

Cohere is tuned aggressively for retrieval on noisy, real-world corpora: web pages, tickets, PDFs with OCR errors. On my internal evaluation set that mixes clean documentation and messy Zendesk tickets, Cohere Embed v3 + 512-dim truncation outperformed E5-base-v2 by 4 points on nDCG@10 and outperformed all-mpnet-base-v2 by 11 points. On clean, symmetric STS tasks, however, the gap narrows and E5-large occasionally wins. Cohere also handles long inputs more gracefully by truncating and compressing internally, while E5 will hard truncate at 512 tokens. If your documents are longer than ~400 tokens, chunk first regardless of provider.

import cohere

co = cohere.Client(api_key="COHERE_API_KEY")

# Cohere requires input_type - this is not optional for good results
query_emb = co.embed(
    texts=["how to reset password on mobile app"],
    model="embed-english-v3.0",
    input_type="search_query",
    embedding_types=["float"]
).embeddings.float[0]

doc_emb = co.embed(
    texts=["To reset your password on mobile, open Settings > Security > Reset Password."],
    model="embed-english-v3.0",
    input_type="search_document",
    embedding_types=["float"]
).embeddings.float[0]

# For storage efficiency, request truncated dimensions
compressed = co.embed(
    texts=["long documentation passage ..."],
    model="embed-english-v3.0",
    input_type="search_document",
    embedding_types=["float"]
).embeddings.float
# If using MRL truncation, you can slice to 512 dims client-side
# with minimal loss - see Cohere docs for truncated evaluation

The latency profile is also different. Open-weight models depend on your GPU availability and batching. Cohere's API is typically 80-150ms p50 for a single embedding, but supports batching up to 96 texts per call. For high QPS search, I batch queries and use a local E5 model to avoid network hop; for low QPS or batch ingestion pipelines where throughput matters more than p95, Cohere simplifies operations dramatically because there is no model server to maintain.

Direct Comparison: Retrieval Quality, Dimensionality and Inference Economics

Benchmarks like MTEB and BEIR are useful but they hide trade-offs that matter in production. I evaluate on three axes: retrieval quality on my own data, vector size and its impact on index cost, and inference cost at expected QPS. The table below reflects numbers I have reproduced or measured on a dataset of 50k queries over 1.2M passages, plus public MTEB averages for reference. Your numbers will vary, but the relative ordering has been stable across three different domains I have tested.

On pure MTEB retrieval average, the ordering I see is Cohere v3 ~ E5-large-v2 > E5-base-v2 > all-mpnet-base-v2 > all-MiniLM-L6-v2. On clustering and classification via k-NN, SBERT MPNet is closer to E5-base because those tasks are symmetric. On cost, the ordering flips: MiniLM is ~3x cheaper than E5-base and ~15x cheaper than Cohere at high QPS if you own GPUs, because you pay per million tokens to the API provider. I typically see Cohere at $0.10 per million tokens for Embed v3, while self-hosted E5-base on an A10g costs roughly $0.02-0.04 per million tokens at sustained utilization.

One nuance: all of these models are trained with cosine similarity. If you switch to dot product or Euclidean distance in your vector database without normalizing, you will silently degrade ranking. I enforce normalization at write time and validate with a nightly job that samples vectors and checks abs(norm - 1.0) < 1e-4.

Deploying Embedding Models for Vector Search Without Regretting It Later

The model is only half the system. In my experience, most vector search failures come from indexing and serving decisions, not from picking the wrong checkpoint. The three decisions that matter most are chunking, distance metric, and index parameters.

For chunking, I default to 256 tokens with 32 tokens overlap for retrieval corpora. For SBERT and E5, which are capped at 512 tokens, this keeps most chunks within the model's effective context without excessive truncation. For Cohere, you can send longer text but I still chunk, because a 1024-token document vector averages away the specific answer your query needs. Each chunk gets its own vector and metadata pointing to the parent document. At query time I retrieve chunks and then deduplicate by parent ID.

For indexing, I use HNSW if I need low latency and can fit the graph in memory, and IVF-PQ if I need to scale beyond RAM. With normalized embeddings, I use inner product distance. HNSW parameters M=16, efConstruction=200, efSearch=128 work well for up to a few million vectors; beyond that I increase M to 24. I also run a separate evaluation that measures Recall@k at different efSearch values, because the default in many vector databases is tuned for speed, not quality.

Batching and caching are the other levers. Embedding models are far more efficient with batches of 32-128. I front the embedding service with a cache keyed by hash(model_id + prefix + text) with a 7-day TTL, which cuts redundant compute by 30-40% on support search where the same queries repeat. For RAG pipelines, I pre-embed documents offline and only embed the query at request time, keeping p95 latency under 200ms even with E5-base on a single GPU. If you are considering fine-tuning an encoder for your domain, see Fine-Tuning LLMs with LoRA and QLoRA: Memory-Efficient Adaptation - the same principles of hard negatives and in-batch contrastive loss apply to embedding fine-tuning.

A practical evaluation harness before you switch models

Before migrating an index from one model to another, I run a shadow evaluation. I embed a 5k query set with both models, retrieve top-20 from two separate indexes, and compute nDCG, Recall@10, and a manual error analysis on 50 queries where the rankings disagree. I also track the fraction of queries where the correct document moves from rank >10 to rank <=3, because that is what users actually feel. Only if the new model wins on my data do I cut over. Public leaderboards do not replace this step, a point also covered in recent discussions on arXiv Machine Learning about retrieval evaluation bias.

Finally, version your embeddings. When you re-embed a corpus with a new model, the vector spaces are incompatible. I store embedding_model and embedding_version with each vector and keep the old index readable during migration. A blue-green index swap with an alias avoids downtime.

A Practical Decision Framework for Choosing Your Embedding Backbone

If you asked me to pick a default today, I would ask three questions. First, do you need to run offline or on-prem? If yes, Cohere is out and you are choosing between SBERT and E5. Second, is your workload symmetric or asymmetric? For symmetric tasks like clustering, deduplication, or semantic similarity where both sides look similar, all-mpnet-base-v2 is often enough and faster. For asymmetric retrieval where queries are short and documents are long, E5-base-v2 is the best open option without fine-tuning. Third, what is your QPS and team size? If you are a small team with low QPS and want the best retrieval without managing GPUs, Cohere Embed v3 with 512-dim truncation is hard to beat.

I also weigh fine-tuning cost. E5 and SBERT are easy to fine-tune with sentence-transformers and a few thousand labeled pairs using MultipleNegativesRankingLoss. Cohere supports fine-tuning through a separate endpoint but you lose portability. For one client with highly specialized medical terminology, fine-tuning E5-base on 8k query-passage pairs improved Recall@10 by 13 points over the base checkpoint, which no off-the-shelf model matched. That investment only makes sense if you have the data and the evaluation harness to prove it.

My current stack for a typical SaaS knowledge base is: E5-base-v2 for first-stage retrieval, chunk size 256, HNSW with normalized vectors, cached query embeddings, and a cross-encoder re-ranker for the top-50. When the product needs multilingual search, I swap to Cohere Multilingual v3 or E5-mistral for the re-ranking stage. The pattern is consistent, and it keeps the system explainable and debuggable without chasing the latest leaderboard.

Frequently Asked Questions