RAG Architecture: Building Retrieval-Augmented Generation Systems

Retrieval-Augmented Generation has moved from a research novelty to the default architecture for grounding large language models in private or rapidly changing knowledge. The core idea is simple: instead of relying solely on parametric memory, you retrieve relevant documents at query time and condition generation on that evidence. In practice, building a RAG system that is accurate, low-latency, and maintainable is a systems engineering problem that touches chunking, embedding, indexing, ranking, prompt construction, and evaluation. In my experience leading RAG deployments for technical documentation and financial research use cases, the difference between a demo that looks good on five queries and a system that survives production traffic is almost entirely in the architectural details around retrieval quality and failure handling. This article breaks down those details from ingestion to evaluation, with concrete implementation patterns you can adapt.

Anatomy of a Production RAG Pipeline: Retriever, Augmenter, and Generator

A naive RAG loop — embed query, vector search top-k, concatenate into prompt — will answer simple factoid questions, but it collapses on multi-hop reasoning, ambiguous queries, and noisy corpora. A production pipeline separates concerns into three subsystems that can be developed, measured, and scaled independently. The retriever is responsible for recall, the augmenter for precision and context shaping, and the generator for grounded synthesis.

The retriever itself is rarely a single vector search call. I've found that treating it as a cascade works far better: an initial high-recall fetch of 50-100 candidates using a hybrid dense-sparse index, followed by a lightweight filter, then a more expensive re-ranking stage. This mirrors classic information retrieval architectures and gives you knobs for latency versus quality. The augmenter then takes the re-ranked evidence and performs the unglamorous but critical work of deduplication, ordering, truncation, and citation alignment. How you order and format context has a measurable effect on answer faithfulness due to positional bias in long-context LLMs.

The generator is where many teams over-focus, but prompt design and decoding controls matter more than model size once retrieval is good. For grounded generation, I enforce a strict instruction set: answer only from provided context, cite source IDs for every factual claim, and explicitly state when information is missing. This reduces hallucination far more effectively than lowering temperature alone. The full loop should be observable end-to-end, logging the query, expanded queries, retrieved IDs with scores, final assembled context, and the raw completion, so you can debug a bad answer without guessing which stage failed. For more on shaping generator behavior, see Systematic Prompt Engineering: Chain-of-Thought, Few-Shot and Tool Use.

Modular vs. Agentic RAG Control Flow

Modular RAG executes a fixed DAG: query rewrite → retrieve → re-rank → generate. Agentic RAG gives the LLM control to call retrieval tools iteratively, reflect, and branch. In my experience, modular pipelines are easier to evaluate and cheaper to serve, and I recommend starting there. Agentic loops shine for research-style tasks where the answer requires 3-5 dependent searches, but they add latency, cost variance, and complexity in tracing. A common middle ground is a two-pass modular system: generate a draft answer, use it to issue a second retrieval for self-correction, then produce the final response.

Chunking and Ingestion Strategies That Preserve Semantic Coherence

Chunking is the most under-invested part of RAG and the fastest way to degrade retrieval. If you split a technical document arbitrarily every 512 tokens, you will sever tables from their captions, procedures from their preconditions, and definitions from their examples. The embedding for that chunk then represents a semantically incoherent fragment, which no retriever can recover from.

I have converged on a hierarchy of strategies depending on document type. For well-structured documentation and markdown, recursive character splitting with awareness of headers, code fences, and tables preserves logical units. You split on double newlines and section boundaries first, then sentences, then tokens, with overlap to retain local context. For unstructured PDFs, legal contracts, or scanned reports, semantic chunking using embedding similarity between sequential sentences yields more coherent boundaries than fixed windows, though it is more expensive to build. In both cases, each chunk should carry rich metadata: document ID, title, section path, page number, creation date, access control tags, and a chunk hash for deduplication during re-indexing.

Overlap is not free. A 10-15% overlap improves recall for answers that span a boundary, but 25%+ overlap bloats your Hugging Face Documentation token count and index size without measurable gains and can cause the same content to dominate top-k results. I typically use 100-150 token overlap for 600-800 token chunks. Another practical detail: prepend the document title and section hierarchy to each chunk before embedding. This cheap contextualization dramatically helps retrieval for chunks like "He will need to restart the service" that are meaningless without their parent heading.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from sentence_transformers import SentenceTransformer

splitter = RecursiveCharacterTextSplitter(
    chunk_size=750,
    chunk_overlap=120,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
    keep_separator=True
)

def make_chunks(doc_text: str, doc_meta: dict):
    # Prepend hierarchical context for better embedding
    header = f"{doc_meta['title']} | {' > '.join(doc_meta['section_path'])}"
    raw_chunks = splitter.split_text(doc_text)
    enriched = []
    for i, c in enumerate(raw_chunks):
        enriched.append({
            "text": f"{header}\n\n{c}",
            "metadata": {**doc_meta, "chunk_index": i, "hash": hash(c)}
        })
    return enriched

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
vectors = model.encode([c["text"] for c in enriched], normalize_embeddings=True)

Late Chunking and Contextual Retrieval

An emerging alternative is late chunking: embed the entire document, then pool token embeddings into chunk representations. This retains cross-chunk attention and produces more accurate vectors for short chunks. Similarly, Anthropic's contextual retrieval approach generates a 2-3 sentence summary that situates each chunk within the full document, then embeds the chunk plus its context. In our ablation on a 12,000-page API docs corpus, contextual retrieval lifted Recall@10 by 8 points with minimal increase in storage, at the cost of an LLM call per chunk during ingestion. For high-value, slowly changing corpora, that trade-off is justified.

Configuring the Retriever: Dense, Sparse, and Hybrid Search in Practice

The choice of retriever determines your ceiling. Pure sparse retrieval with BM25 is strong for keyword-heavy, entity-specific queries and remains the best baseline for out-of-vocabulary terms, version numbers, and error codes. Dense retrieval with a bi-encoder handles paraphrase and semantic intent far better but can underperform on exact matches unless trained with hard negatives. In production, you almost always want hybrid.

Embedding model selection matters more than most teams expect. General-purpose embeddings trained on web data do poorly on niche domains like biomedical text or code without domain adaptation. I evaluate embedding models on my own labeled retrieval set before committing to an index, measuring Recall@k and nDCG, not just generic MTEB leaderboards. Dimensionality also matters for cost: 768-dim vectors are often sufficient, while 1536+ improves separation only marginally but doubles storage and slows HNSW search. For a detailed model-by-model breakdown, see Text Embedding Models: Sentence-BERT, E5 and Cohere Embed Compared.

Query-side improvements are often cheaper than re-indexing. Query rewriting that expands acronyms, resolves anaphora, and generates 2-3 alternative phrasings improves recall without touching the index. HyDE, where you generate a hypothetical answer and embed it for search, helps for vague informational queries but hurts for precise lookups, so I gate it behind a classifier that detects underspecified intent.

from rank_bm25 import BM25Okapi
import numpy as np

# Hybrid search with score fusion
def hybrid_retrieve(query: str, dense_scores, sparse_scores, alpha=0.65, top_k=10):
    # Normalize both score distributions to [0,1]
    d_norm = (dense_scores - dense_scores.min()) / (dense_scores.ptp() + 1e-9)
    s_norm = (sparse_scores - sparse_scores.min()) / (sparse_scores.ptp() + 1e-9)
    fused = alpha * d_norm + (1 - alpha) * s_norm
    top_indices = np.argsort(fused)[::-1][:top_k]
    return top_indices, fused[top_indices]

# Example: BM25 for sparse
corpus_tokens = [c["text"].split() for c in chunks]
bm25 = BM25Okapi(corpus_tokens)
sparse_scores = np.array(bm25.get_scores(query.split()))

# Dense scores come from vector database cosine similarity
# dense_scores = vector_db.query(query_embedding, top_k=len(chunks))

Tuning Fusion and Filtering

Reciprocal rank fusion and convex combination are both effective for merging dense and sparse ranks; I prefer convex combination with a tuned alpha because it is calibrated and easier to reason about. Alpha around 0.6-0.7 favors dense for semantic corpora, lower for code and logs. Always apply metadata filtering before vector search when possible. Pushing filters like tenant_id, doc_type, or date range into the index reduces the search space and prevents cross-tenant leakage, which is a hard requirement for multi-tenant SaaS. Pre-filtering is more precise than post-filtering and, on most engines, faster.

Vector Database Selection and Index Operations at Scale

The vector database is the stateful heart of retrieval augmented generation. While you can prototype with in-memory FAISS, production demands persistence, incremental updates, filtering, and replication. On Hugging Face forums and in the PyTorch Documentation, you will see frequent discussion of index types, but the core trade-off is between graph-based and partition-based indexes.

HNSW offers excellent query latency and recall at the cost of higher memory and longer build times. IVF-PQ compresses vectors dramatically and scales to billions but requires tuning nprobe and retraining when distributions shift. For most enterprise RAG workloads under 10 million chunks, HNSW with quantized scalar or product quantization is the sweet spot. I quantize to int8 after verifying less than 2% recall drop on my evaluation set, which halves memory with negligible quality loss.

Operational details that bite teams later include versioning, deletion, and mutability. Upserts by chunk hash with a write-ahead log prevent stale fragments from lingering after a document update. I maintain a separate metadata store in Postgres that is source of truth for document lineage, and the vector index is derived and rebuildable. Periodic compaction and re-indexing are necessary as HNSW graphs degrade with heavy deletes. Monitor p95 search latency, recall drift via held-out queries, and index size per tenant to catch regressions before users do.

Context Assembly, Re-Ranking, and Mitigating Lost-in-the-Middle Effects

Retrieving the right documents is only half the battle; you must assemble them into a context window that the generator can actually use. Large context windows have not eliminated positional bias. Multiple studies and our own experiments show that LLMs attend most strongly to the beginning and end of the context, with a dip in the middle where key evidence can be ignored. Simply concatenating top-k in score order is therefore suboptimal.

I use a two-stage re-ranking and ordering pipeline. First, a cross-encoder re-ranker scores each candidate pair of query and chunk with full cross-attention, which is far more precise than bi-encoder cosine but too expensive for first-stage retrieval. A model like bge-reranker-large or Cohere rerank improves Precision@5 by 12-18 points in our benchmarks and is worth the added 80-150ms latency. Second, I order the surviving context by placing the highest-scoring chunks at the beginning and end, with supporting but lower-ranked chunks in the center. I also de-duplicate near-identical chunks using maximal marginal relevance to avoid wasting budget on redundant evidence.

Context compression is necessary when top chunks exceed token limits or cost budgets. Extractive compression keeps only the sentences within each chunk that are relevant to the query, judged by a lightweight model or even regex for tables. Abstractive compression summarizes, but risks adding unsupported content, so I avoid it for grounding. Each piece of context is presented with a stable delimiter and ID like [DOC-3] and I instruct the generator to cite those IDs. This makes answer verification trivial and enables automated faithfulness checks. In my experience, moving from raw concatenation to re-ranked, ordered, and compressed context reduces hallucination rate more than switching to a larger generator model.

Prompt Construction for Grounded Generation

The final prompt should separate system instructions, evidence, and user query with clear boundaries. I set temperature low (0.1-0.2) and include a refusal instruction: if the context does not contain the answer, say so rather than speculate. For complex queries, I apply a brief chain-of-thought that is instructed not to reveal reasoning but to plan citations first, which improves grounding without exposing internal traces. This is one area where Fine-Tuning LLMs with LoRA and QLoRA: Memory-Efficient Adaptation can help as an alternative to few-shot prompting if your format compliance is weak — a small LoRA adapter trained on 500 grounded examples consistently beats longer system prompts for citation discipline.

Evaluating and Hardening RAG Systems Against Hallucination and Drift

Without rigorous evaluation, RAG quality degrades silently as documents are updated, embeddings drift, and query distributions shift. I evaluate on three layers: retrieval quality, grounding faithfulness, and final answer utility. Retrieval is measured with Recall@k, nDCG, and hit rate on a curated set of 200-500 question-evidence pairs covering typical, edge, and adversarial queries. Grounding is measured with faithfulness classifiers that check whether each claim is entailed by the cited context. Final answers need human or LLM-as-judge ratings for completeness and correctness, but those are noisy without grounding checks beneath them.

Frameworks like RAGAS, TruLens, and custom scripts that implement context relevance, faithfulness, and answer relevance are helpfulStarting points, but you need to adapt them to your domain. In my projects, I maintain a golden set where each question is annotated with the exact chunk IDs that support the answer and the expected refusal cases where no evidence exists. I run this set on every index rebuild and prompt change in CI. A regression that drops Recall@10 by 5% often shows up as a subtle increase in partial answers weeks before users complain.

Hardening includes more than metrics. I apply strict access control by filtering at retrieval time, not generation time, so the model never sees unauthorized context. I version embeddings alongside model checkpoints and use staged rollouts for new embedding models with shadow traffic comparison. For high-stakes domains, I add a post-generation verifier that performs natural language inference between answer sentences and retrieved evidence, flagging unsupported claims for fallback or human review. Latency and cost guardrails matter too: circuit breakers that fall back to sparse-only search when vector search latency spikes, and token budgeting that truncates long tables rather than failing the request.

from ragas.metrics import faithfulness, context_recall, answer_relevancy
from datasets import Dataset

# Build evaluation set with question, ground_truth, retrieved contexts
eval_data = {
    "question": ["What is the retry policy for the ingestion API?"],
    "answer": ["The ingestion API uses exponential backoff with max 3 retries. [DOC-2]"],
    "contexts": [["The ingestion API retries failed requests with exponential backoff (max 3 attempts). [DOC-2]"]],
    "ground_truth": ["Exponential backoff, max 3 retries"]
}
dataset = Dataset.from_dict(eval_data)

# Run RAGAS - requires an LLM and embeddings for judging
from ragas import evaluate
results = evaluate(
    dataset,
    metrics=[faithfulness, context_recall, answer_relevancy]
)
print(results)
# In CI, assert faithfulness > 0.85 and context_recall > 0.80

The goal is not to chase a single leaderboard number but to build a system where each layer is instrumented. Retrieval augmented generation is a compound AI system: improvements to any single component compound only if the others are stable. When done well, a well-instrumented RAG pipeline delivers answers that are current, attributable, and far more trustworthy than parametric generation alone, while remaining cheaper than repeatedly fine-tuning for知识 updates.

Frequently Asked Questions