Vector Databases for AI: Embedding Storage, Similarity Search and Indexing

When I first moved a semantic search prototype to production in 2021, I tried to store embeddings in PostgreSQL with a custom extension and run brute-force cosine similarity in Python. It worked for 10,000 documents. At 2 million, query latency went from 40ms to over 4 seconds, recall became unpredictable, and nightly re-indexing jobs started failing silently. That experience forced me to understand what vector databases actually do differently from traditional storage. If you are building retrieval-augmented generation, recommendation, or multimodal search today, you will inevitably need to store and query high-dimensional vectors efficiently. This article covers how embeddings are created and stored, how similarity search works under the hood, and how modern indexing algorithms make sub-100ms search possible at scale.

Why Keyword Search Fails and Dense Vectors Take Over

In traditional search, an inverted index maps tokens to document IDs. It is excellent for exact matching but brittle for meaning. A query for "affordable electric car with long range" will not match a document containing "budget-friendly EV offering 320 miles per charge" unless you maintain exhaustive synonym lists and query expansion rules. Dense embeddings solve this by encoding semantic meaning into a continuous vector space.

An embedding is a fixed-length array of floats, typically 384 to 3072 dimensions, where semantically similar items cluster together. Models like sentence-transformers, OpenAI's text-embedding-3-large, or Cohere embed-v3 are trained with contrastive objectives to place paraphrases close together and unrelated concepts far apart. In my experience, switching from BM25 to embeddings improved recall on our internal support knowledge base by 34% for natural language queries, especially for long-tail phrasing and typos, without any manual synonym curation.

The trade-off is storage and compute. A single 1536-dimensional float32 vector consumes 6KB. One million vectors require ~6GB of raw vector data before indexes and metadata. You cannot scan that linearly for every query if you need low latency. That is the core problem vector databases are built to solve: storing billions of vectors with associated payloads and returning the approximate nearest neighbors in milliseconds, not seconds.

From Sparse to Dense: What Changes in Your Data Model

With sparse retrieval, your schema centers on text fields and term frequencies. With vector search, your primary key is often supplemented by a vector column and a payload. The payload holds the original text, source URL, timestamp, access controls, and any filterable attributes. Queries become two-stage: a vector similarity search followed by optional metadata filtering. I've found that teams who treat vectors as the only truth and discard the source text quickly regret it when they need to debug relevance or re-embed with a new model.

The Role of Embeddings in Generative AI Pipelines

For most Generative AI applications, vectors are not the end product but the retrieval layer. In a RAG pipeline, the language model never sees your entire document corpus; it sees the top-k chunks returned by vector search. The quality of those chunks determines whether the model hallucinates or answers accurately. If you are architecting agents that need to recall prior tool outputs or conversation history, the patterns described in Building AI Agents: Tool Use, Planning and Memory Architecture Patterns show why long-term memory almost always resolves to a vector store with temporal filtering. Similarly, when you move to image or audio search, the same vector primitives apply, as covered in Multimodal AI: Vision-Language Models and Cross-Modal Understanding.

Anatomy of an Embedding Pipeline: Turning Raw Content Into Queryable Vectors

Creating vectors that actually perform well requires more than calling an embedding API. The pipeline has four stages that each influence retrieval quality: chunking, normalization, embedding, and ingestion.

Chunking is the most underestimated step. Embedding models have context limits, often 512 or 8192 tokens, and they produce a single vector per input. If you embed a 10-page PDF as one vector, the semantic signal dilutes and specific facts become unretrievable. In practice, I chunk text into 300-600 tokens with 40-60 tokens of overlap for documentation, and smaller 150-250 token chunks for Q&A or chat logs where precision matters. Overlap preserves context across boundaries, so a sentence split between chunks is still searchable. For code or tables, use structure-aware splitters rather than naive character splits.

Before embedding, normalize your text consistently. Lowercasing is often handled by the tokenizer, but you should still strip boilerplate, normalize whitespace, and decide how to handle metadata like titles or breadcrumbs. I prepend the document title to each chunk with a separator, which in my tests consistently improved retrieval for title-specific queries without any model change.

For the embedding step itself, you have two families of models: bi-encoders that produce a single vector per text and late-interaction models like ColBERT. For 95% of initial builds, a bi-encoder is the right choice. Use the Hugging Face Documentation to load open models locally if you need data privacy, or use a hosted API for convenience. Always store the model identifier and version alongside your vectors. You cannot mix vectors from different models in the same index and expect meaningful distances.

# Production embedding pipeline with chunking and metadata
from sentence_transformers import SentenceTransformer
import tiktoken

model = SentenceTransformer('BAAI/bge-large-en-v1.5')
# bge models require query/passage prefixes for best results
def chunk_text(text, chunk_size=512, overlap=50):
    tokens = text.split()  # simplified; use tiktoken or model tokenizer in production
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk = " ".join(tokens[i:i + chunk_size])
        if len(chunk.split()) > 50:  # skip tiny chunks
            chunks.append(chunk)
        if i + chunk_size >= len(tokens):
            break
    return chunks

documents = [{"id": "doc_101", "title": "Refund Policy", "text": "...long text..."}]
records = []
for doc in documents:
    for idx, chunk in enumerate(chunk_text(doc["text"])):
        # Prepend title for context carry-over
        input_text = f"{doc['title']} | {chunk}"
        vector = model.encode(input_text, normalize_embeddings=True)
        records.append({
            "id": f"{doc['id']}_{idx}",
            "vector": vector.tolist(),
            "payload": {"title": doc["title"], "chunk_index": idx, "text": chunk}
        })
# records are now ready for upsert into your vector database

Ingestion is where operational concerns emerge. Most vector databases support upserts by ID, batched writes, and payload indexing. I've found that batch sizes of 100-500 vectors with parallel workers saturate embedding throughput without overwhelming the database's WAL. Always embed and write idempotently; if your job fails midway, you should be able to rerun without creating duplicates. For large backfills, disable index optimization during bulk load and rebuild after, which can cut ingestion time by 40% on IVF-based indexes.

Inside Approximate Nearest Neighbor Indexes: HNSW, IVF and Product Quantization

Exact nearest neighbor search, or brute force, computes the distance between your query vector and every stored vector and sorts the results. It guarantees perfect recall but is O(n*d) per query. At 10 million vectors of dimension 1024, that is untenable for interactive latency. All production vector databases therefore use Approximate Nearest Neighbor (ANN) algorithms that trade a small amount of recall for orders-of-magnitude speedups.

The three building blocks you will encounter are Hierarchical Navigable Small World (HNSW), Inverted File Index (IVF), and Product Quantization (PQ). Most systems combine them.

HNSW builds a multi-layer graph where each vector is a node connected to its nearest neighbors. The top layers are sparse and allow fast long jumps across the graph, while the bottom layer is dense and enables precise convergence. Search starts at an entry point on the top layer and greedily moves to the closest neighbor until it can no longer improve, then descends to the next layer. In my experience, HNSW offers the best latency-recall trade-off for datasets under ~50 million vectors and for workloads with frequent reads. Its weakness is memory: the graph structure adds ~30-50% overhead beyond the raw vectors and build times are longer.

IVF takes a different approach inspired by clustering. During build, it runs k-means to partition vectors into Voronoi cells, typically 4096 to 16384 cells for a million-scale dataset. Each vector is assigned to its nearest centroid. At query time, only the closest nprobe cells are scanned. If nprobe=10 out of 4096, you scan ~0.24% of the data. IVF is excellent for large, mostly static datasets where you can afford to tune nprobe for your recall target and where memory is constrained.

Product Quantization compresses vectors by splitting them into sub-vectors and quantizing each subspace separately. An 1536-dim float32 vector (6144 bytes) can be compressed to 192 bytes with PQ, a 32x reduction, with a typical recall loss of 2-5%. This is critical when your working set exceeds RAM. Systems like Qdrant and Milvus allow you to store quantized vectors on disk or in memory while keeping centroids in RAM.

Tuning Recall Versus Latency With Search Parameters

Every ANN index exposes runtime knobs. For HNSW, efSearch controls the size of the dynamic candidate list; increasing it from 64 to 256 will improve recall by 2-3 points but roughly double latency. For IVF, nprobe controls how many cells you scan. I always benchmark these parameters on a held-out query set with ground-truth neighbors computed via brute force on a sample. Plotting recall@k against P95 latency for different settings makes the trade-off explicit and prevents the common mistake of over-tuning for recall while missing SLOs.

Recent Indexing Research and Practical Impact

The ANN field moves quickly. Recent work on DiskANN and related graph-based methods demonstrates how to hold the graph on SSD and keep only quantized vectors in memory while still achieving <10ms latency at billion scale. If you follow research on arXiv Machine Learning, you will see a trend toward learned quantization and hybrid sparse-dense indexes that combine BM25 signals with vectors in a single index structure. For now, HNSW remains the default recommendation for most teams, but understanding these alternatives helps when you outgrow single-node deployments.

Choosing Distance Metrics: Cosine, Dot Product and Euclidean Under the Hood

Your choice of distance metric must match how your embedding model was trained, otherwise rankings will be incorrect even with a perfect index.

Cosine similarity measures the angle between two vectors, ignoring magnitude: cos(a,b) = (a·b) / (||a|| * ||b||). Most text embedding models, including bge and e5 families, are trained with cosine loss and expect cosine or inner product on L2-normalized vectors. If you normalize vectors to unit length at ingestion, cosine and dot product become equivalent and dot product is slightly cheaper to compute. Euclidean distance (L2) measures straight-line distance and correlates with cosine when vectors are normalized but diverges otherwise.

I have seen teams use Euclidean distance with normalized OpenAI embeddings and wonder why results felt slightly worse than expected. The issue was subtle score inversion for ties and incompatibility with the model's training objective. Always check the model card. OpenAI text-embedding-3 models and most sentence-transformers explicitly recommend cosine similarity or dot product with normalized vectors.

For maximum inner product search (MIPS) used by models trained without normalization, dot product is correct because vector magnitude encodes confidence or term importance. If you normalize MIPS vectors, you destroy that signal.

import numpy as np

def cosine_similarity(a, b):
    # Assumes a, b are numpy arrays
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def dot_product(a, b):
    return np.dot(a, b)

# Correct workflow for cosine-trained models: normalize once at write time
def normalize(v):
    norm = np.linalg.norm(v)
    return v / norm if norm > 0 else v

query_vec = normalize(model.encode("refund for delayed shipment"))
stored_vec = normalize(model.encode("compensation for late delivery"))

# After normalization, dot product == cosine similarity
score = dot_product(query_vec, stored_vec)  # ~0.89 for these paraphrases
# For querying a vector DB, ensure your DB is configured for Cosine or IP
# e.g., in Pinecone: metric="cosine", in Qdrant: distance="Cosine", in pgvector: vector_cosine_ops

Some databases store cosine distance as 1 - cosine similarity, so lower is better, while others return similarity where higher is better. Consistency matters when you set score thresholds for RAG. I recommend standardizing on similarity (higher is better) in your application layer and converting at the adapter boundary if your database returns distance. Log the distribution of top-1 scores for real queries; a sudden shift often indicates you changed normalization or metric inconsistently after a re-embedding.

Production Patterns for Incremental Updates, Sharding and Filtered Search

A vector database in production is not a static dump. Documents are added, updated, and deleted, permissions change, and queries almost always require filtering.

For incremental updates, understand whether your index supports true deletions and updates. HNSW graphs do not physically delete nodes immediately; they mark them as deleted and filter them at query time, reclaiming space during a compaction or rebuild. If you have a high churn rate, say updating 10% of vectors daily, schedule periodic index optimization during low traffic. In my experience, skipping this causes steadily degrading recall and growing memory fragmentation that only shows up after weeks.

Filtering is where vector databases differentiate from pure ANN libraries like FAISS. Real queries look like "find semantically similar support tickets where product = 'Widget Pro' and created_at > 30 days ago and tenant_id = 123". This requires combining vector similarity with boolean predicates. There are two execution strategies: pre-filtering, where the database intersects the filter first and then searches only the matching subset, and post-filtering, where it searches vector space first and then filters results. Pre-filtering is more accurate when selectivity is high (few documents match the filter), but it can be slower if the filter matches millions. Modern systems use metadata indexes and hybrid plans that estimate selectivity.

Sharding and replication follow standard distributed patterns. Vectors are typically sharded by ID hash or by tenant. For multi-tenant SaaS, I strongly recommend an isolated collection or partition per large tenant rather than a single shared collection with a tenant_id filter, because it isolates noisy neighbor effects and allows per-tenant scaling and TTL policies. Replication provides read scaling and availability; unlike traditional OLTP, vector search is read-heavy, so 2-3 replicas with eventual consistency are usually sufficient.

# Filtered search patterns: Qdrant and pgvector examples

# Qdrant: pre-filtered vector search with payload indexes
from qdrant_client import QdrantClient, models

client = QdrantClient(host="localhost", port=6333)
hits = client.search(
    collection_name="support_tickets",
    query_vector=query_vec.tolist(),
    query_filter=models.Filter(
        must=[
            models.FieldCondition(key="tenant_id", match=models.MatchValue(value=123)),
            models.FieldCondition(key="product", match=models.MatchValue(value="Widget Pro")),
            models.FieldCondition(key="created_at", range=models.Range(gte="2024-01-01"))
        ]
    ),
    limit=10,
    with_payload=True,
    search_params=models.SearchParams(hnsw_ef=128)  # tune recall/latency
)

# pgvector (PostgreSQL): IVFFlat with cosine distance and WHERE filter
# CREATE INDEX ON docs USING ivfflat (embedding vector_cosine_ops) WITH (lists = 1000);
# SELECT id, payload, 1 - (embedding <=> query_vec) AS similarity
# FROM docs
# WHERE tenant_id = 123 AND product = 'Widget Pro'
# ORDER BY embedding <=> query_vec LIMIT 10;

Monitoring deserves explicit attention. Track P95 and P99 search latency, recall@k on a sampled evaluation set, index size and segment count, and cache hit rates for payloads. When latency spikes, the cause is usually not vector search itself but payload fetching or overly permissive filters that force scanning. If you are optimizing the broader LLM stack that consumes these results, the trade-offs in LLM Inference Optimization: Continuous Batching, KV Cache and Speculative Decoding directly interact with retrieval latency budgets; a 50ms saving in vector search can be reinvested in a larger generation window.

When to Use a Dedicated Vector Database Versus pgvector or In-Memory Libraries

Choosing storage is less about hype and more about scale, team expertise, and operational constraints. I evaluate three tiers.

Tier one is an in-memory library: FAISS, Annoy, ScaNN, or hnswlib running in your application process. This is ideal for prototypes, offline batch jobs, or datasets under 500k vectors where persistence and concurrency are not critical. You get maximum flexibility and zero operational overhead, but you must handle serialization, backup, and scaling yourself. I've used FAISS Flat and HNSW inside Python microservices for evaluation harnesses where reproducibility mattered more than availability.

Tier two is extending your existing relational database with pgvector, or using SQLite with VSS. If your dataset is under 5-10 million vectors, your team already operates PostgreSQL, and your query load is moderate (tens of QPS), pgvector with HNSW or IVFFlat is often the pragmatic choice. It gives you transactions, joins, and familiar tooling. The limitations appear around horizontal scaling, billion-vector performance, and advanced filtering. Build times for HNSW indexes on large tables are also significantly longer and require increased maintenance work mem.

Tier three is a dedicated vector database: Pinecone, Qdrant, Weaviate, Milvus, or Chroma. Choose this when you need horizontal sharding, replication, hybrid sparse-dense search, role-based access, and ingestion rates beyond what a single Postgres instance sustains. The operational cost is higher, but for our team it eliminated an entire class of custom sharding and compaction logic we were maintaining. If you anticipate growing beyond 20 million vectors or need sub-50ms P95 at hundreds of QPS, start here rather than migrating later.

My decision heuristic is practical: start with pgvector if you already run Postgres and your scale is bounded. Abstract vector operations behind a repository interface so you can swap the backend without touching RAG logic. If load tests project sustained growth or you need native hybrid search with BM25, adopt a dedicated service early. Do not build a custom vector store on top of Redis or Elasticsearch unless you have a specialized reason; you will reimplement compaction, consistency, and recall testing that existing systems already solve.

Finally, plan for re-embedding. Models improve yearly, and 768-dimensional vectors from 2022 are not comparable to 3072-dimensional vectors from today. Version your collections, keep raw text immutable, and design your ingestion pipeline to replay from source. A blue-green collection pattern, where you build a new collection alongside the live one and atomically switch the alias, prevents downtime and lets you A/B test retrieval quality with real user queries before cutting over.

Frequently Asked Questions