Retrieval-Augmented Generation Architecture

When Your Vector Database Isn't Enough

Retrieval-Augmented Generation looked straightforward on paper: embed your documents, store them in a vector database, retrieve the top-k matches, feed them to an LLM, get an answer. Then we tried building one for a 50-million-document enterprise corpus and discovered that every assumption in that simple pipeline breaks at scale.

The architecture that actually works for enterprise RAG systems looks nothing like the tutorials. It's messier, has more moving parts, and requires careful attention to retrieval quality metrics that most teams don't even measure. Let me walk through what we built and where we went wrong along the way.

The Two-Stage Retrieval Pipeline

Single-stage vector search tops out at useful recall around 10 million documents, in my experience. Past that point, the embedding space gets too crowded — semantically distinct documents end up with nearly identical vectors, and your top-k results become a lottery. The fix is a two-stage pipeline: sparse retrieval for broad recall, followed by dense retrieval for precision re-ranking.

from dataclasses import dataclass
from typing import List

@dataclass
class SearchResult:
 doc_id: str
 content: str
 score: float
 source: str

class TwoStageRetriever:
 def __init__(self, sparse_index, dense_index, reranker):
 self.sparse = sparse_index
 self.dense = dense_index
 self.reranker = reranker

 def retrieve(self, query, top_k=5):
 sparse_hits = self.sparse.search(query, k=100)
 dense_hits = self.dense.search(query, k=100)
 candidates = self._rrf(sparse_hits, dense_hits, k=60)
 reranked = self.reranker.rerank(query, candidates, top_k=top_k)
 return reranked

 def _rrf(self, *result_lists, k=60):
 scores = {}
 for results in result_lists:
 for rank, result in enumerate(results):
 if result.doc_id not in scores:
 scores[result.doc_id] = {"result": result, "score": 0}
 scores[result.doc_id]["score"] += 1.0 / (k + rank + 1)
 fused = sorted(scores.values(), key=lambda x: -x["score"])
 return [item["result"] for item in fused[:k]]

Reciprocal Rank Fusion for merging the two retrieval stages isn't the only option, but it's the most forgiving. It doesn't require score calibration between the sparse and dense retrievers, which matters because BM25 scores and cosine similarities live on completely different scales.

The Cross-Encoder Re-Ranker

The re-ranking stage is where you spend your latency budget. A cross-encoder processes the query and each candidate document together, which means it captures fine-grained token interactions that bi-encoder embeddings miss. The tradeoff: a cross-encoder over 100 candidates takes 200-500ms, depending on document length and hardware. That's acceptable for enterprise search where users expect 1-2 second response times.

Document Chunking That Doesn't Destroy Context

Here's where most RAG tutorials lead you astray. They'll tell you to split documents into 512-token chunks with some overlap. That works for blog posts. It fails spectacularly for contracts, technical manuals, and financial reports where a single table or section header changes the meaning of everything that follows.

We switched to hierarchical chunking: each document gets chunked at multiple granularities, and the chunk metadata preserves the hierarchy.

def hierarchical_chunk(document, section_parser):
 chunks = []
 sections = section_parser.extract_sections(document)

 for section in sections:
 chunks.append({
 "content": section.full_text,
 "level": "section",
 "title": section.heading,
 "doc_id": document.id,
 "parent": None,
 })
 for i, para in enumerate(section.paragraphs):
 if len(para.split()) < 20:
 continue
 chunks.append({
 "content": para,
 "level": "paragraph",
 "title": section.heading,
 "doc_id": document.id,
 "parent": section.heading,
 "position": i,
 })
 return chunks

When a paragraph-level chunk gets retrieved, we pull its parent section as additional context for the LLM. This costs extra tokens but dramatically reduces hallucination — the model can see the section heading and surrounding context instead of guessing from an isolated fragment.

Measuring Retrieval Quality Independently

The biggest mistake we made early on was evaluating the entire RAG system end-to-end without measuring retrieval independently. When the LLM gave a wrong answer, we couldn't tell whether the retriever failed to find the right document or the LLM failed to extract the answer from the right document.

We covered a related topic in Anomaly Detection in High-Dimensional Industrial Sensor Data.

We now track three retrieval metrics separately. Recall@k measures whether the correct document appears anywhere in the top-k results. For our enterprise system, we target Recall@10 above 0.85. Below that, no amount of prompt engineering will compensate.

Mean Reciprocal Rank measures where the correct document ranks. An MRR of 0.5 means the correct document is typically second in the list. We target MRR above 0.6.

Context relevance is the tricky one — we use an LLM-as-judge to score whether each retrieved chunk actually contains information needed to answer the query. This catches the failure mode where the retriever finds topically related but not actually useful documents.

The Prompt Assembly Stage

Once you have your retrieved chunks, assembling them into a prompt isn't trivial. The order of chunks matters — LLMs exhibit a "lost in the middle" effect where information in the center of a long context gets less attention than information at the beginning or end. We sort retrieved chunks by relevance score and place the most relevant ones first and last.

def assemble_prompt(query, chunks, max_tokens=3000):
 sorted_chunks = sorted(chunks, key=lambda c: -c.score)
 if len(sorted_chunks) > 2:
 reordered = [sorted_chunks[0]]
 middle = sorted_chunks[2:]
 reordered.extend(middle)
 reordered.append(sorted_chunks[1])
 else:
 reordered = sorted_chunks

 context_parts = []
 token_count = 0
 for chunk in reordered:
 chunk_tokens = len(chunk.content.split()) * 1.3
 if token_count + chunk_tokens > max_tokens:
 break
 context_parts.append(chunk.content)
 token_count += chunk_tokens

 return "\n\n---\n\n".join(context_parts)

Caching and Performance at Scale

In production, we cache at two levels. The embedding cache stores pre-computed query embeddings, which saves about 50ms per query. The result cache stores full retrieval results for repeated queries, keyed on query text with a 15-minute TTL. That TTL is intentionally short — our document corpus updates hourly, and stale results are worse than slow results for enterprise users who need current information.

The total pipeline latency breaks down roughly as: embedding (20ms), sparse search (30ms), dense search (50ms), merge and dedup (5ms), re-ranking (300ms), prompt assembly (10ms), LLM generation (1-3s). The re-ranker dominates the retrieval portion, and the LLM dominates overall. Optimizing anywhere else first is usually wasted effort.

One thing I wish we'd done earlier: instrument the pipeline end-to-end from day one. We added tracing six months in and immediately found that 15% of queries were hitting a slow path in the sparse retriever caused by a regex-based query parser. That kind of issue is invisible without per-stage latency tracking.

Retrieval-Augmented Generation Architecture for Enterprise Search Systems