RAG Architecture: How to Build a Production RAG Pipeline

Retrieval-augmented generation (RAG) answers a question by first retrieving relevant passages from your own documents, then asking a language model to answer using only those passages, with citations. A production RAG architecture is two pipelines: an offline one that parses, chunks, embeds and indexes documents, and an online one that rewrites the query, runs hybrid keyword and vector retrieval, reranks the candidates and generates a grounded answer. When a RAG system gives bad answers, the cause is usually parsing, chunking or retrieval rather than the model, so measure retrieval on its own and fix it first.

Key takeaways

• Parsing and chunking decide your ceiling. A model cannot cite a table that your PDF extractor turned into word soup.
• Use hybrid retrieval (BM25 plus vectors, merged with Reciprocal Rank Fusion) by default. Pure vector search misses exact identifiers.
• Retrieve wide, then rerank narrow: a few dozen candidates into a cross-encoder, the best handful into the prompt.
• Build a golden set of real questions before tuning anything. Track retrieval metrics (recall at k, MRR) separately from answer metrics (faithfulness, relevance).
• Enforce permissions and deletions in the retrieval layer, not in the prompt. The model will repeat whatever you put in its context.

How a RAG pipeline works

Every stage has a characteristic failure, and each one looks like "the model hallucinated" from the outside.

When RAG is the right tool

RAG is the right tool when the model needs knowledge it does not have, that knowledge changes, and answers must point to a source. It is not the only option.

The last row matters more than it looks. "What is the status of order 4471?" is not a semantic search problem. Give the model a tool that queries the order system instead of embedding order exports.

Parsing comes before chunking

Most teams start tuning chunk size while their parser is quietly destroying the content. Look at the extracted text of twenty representative documents before you write any retrieval code.

What to check and fix:

• Repeated page furniture. Headers, footers, page numbers and legal boilerplate repeated on every page pollute both embeddings and BM25 scores. Strip them.
• Tables. Convert tables to Markdown or to one line per row with column names ("Plan: Pro, Seats: 25, SSO: yes"). A flattened table is unanswerable.
• Scanned PDFs. No text layer means no content. Run OCR and keep the OCR confidence as metadata so you can find bad pages later.
• Structure. Keep headings. They become your chunk boundaries and your contextual headers, and they are the cheapest quality win in the whole pipeline.
• Identity and version. Store a stable document ID, source URL, last-modified time, content hash and access control list with every document. You will need all of them for updates, deletions and permissions.

Chunking strategies

A chunk should be the smallest unit that answers a question on its own. That is the whole rule. The strategies differ in how well they find those units.

Sizing rules that hold up

• Start structure-aware: split on headings first, then pack paragraphs up to a limit. For prose, something in the 300 to 800 token range is a sensible first setting. FAQ entries and API parameters can be much smaller.
• Never split a table row, a code block or a numbered procedure across chunks if you can avoid it. Procedures answer "how do I" questions, and half a procedure is worse than none.
• Prepend a contextual header to the text you embed and index: document title plus section path. A chunk that says "Set the timeout to 30 seconds" is useless without "Payments API, Webhooks, Retry behavior" in front of it.
• Overlap matters less once you split on paragraph boundaries and add headers. Add it only if your evaluation shows answers straddling boundaries.
• Measure, do not guess. Change one chunking parameter, rerun the retrieval evaluation, keep the change only if recall moves.

A stronger variant of the contextual header is to have an LLM write one or two sentences situating each chunk within its document, then prepend that to the chunk before embedding and keyword indexing. It costs one cheap model call per chunk at index time and helps most on documents where chunks lean heavily on context established earlier.

Embeddings and the index

The embedding model maps queries and chunks into the same vector space. Rules that save time:

• Shortlist from public leaderboards, decide on your own data. MTEB is a fine way to find candidates. Your golden set decides. Domain vocabulary (legal, medical, internal product names) moves rankings a lot.
• Same model, same preprocessing, for queries and documents. Some models expect a query prefix or instruction. Read the model card.
• Normalize vectors and use cosine or inner product consistently across indexing and search.
• Version the index by embedding model. Changing the model means re-embedding everything. Build the new index alongside the old one and switch reads when the evaluation passes.
• Dimensions cost money. Larger vectors mean more memory and slower search. Some models support truncated dimensions; test the smaller setting before paying for the larger one.

For the vector store itself, if you already run Postgres, pgvector with an HNSW index is a pragmatic default that keeps your chunks, metadata, permissions and keyword index in one transactional system. Our vector database guide covers HNSW and IVF index internals and when a dedicated vector database earns its place, so we will not repeat it here.

Hybrid retrieval: keyword plus vector

Embeddings are good at meaning and bad at exact strings. Ask a pure vector index about ERR_CERT_DATE_INVALID or SKU AB-2210-X and it returns passages about certificates or products in general. BM25 finds the exact token instantly but misses "how do I cancel my plan" when the article says "subscription termination". Run both and merge.

Merge with Reciprocal Rank Fusion

Do not add raw scores together. BM25 scores and cosine similarities live on different scales, and a weighted sum quietly lets one retriever dominate. Reciprocal Rank Fusion (RRF) uses only ranks: each document scores the sum of 1 / (k + rank) across the lists it appears in, with k = 60 as the conventional constant from the original paper. It needs no tuning to work reasonably and rewards documents that both retrievers agree on.

Hybrid search in Postgres

With pgvector and Postgres full-text search, the whole thing is one query.

CREATE EXTENSION IF NOT EXISTS vector;
 
CREATE TABLE chunks (
  id          bigserial PRIMARY KEY,
  doc_id      text NOT NULL,
  tenant_id   text NOT NULL,
  content     text NOT NULL,
  embedding   vector(384) NOT NULL,
  tsv         tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
);
 
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
CREATE INDEX ON chunks (tenant_id);
 
-- $1 = query embedding, $2 = query text, $3 = tenant
WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks
  WHERE tenant_id = $3
  ORDER BY embedding <=> $1
  LIMIT 40
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rank
  FROM chunks, websearch_to_tsquery('english', $2) AS q
  WHERE tenant_id = $3 AND tsv @@ q
  ORDER BY ts_rank_cd(tsv, q) DESC
  LIMIT 40
)
SELECT c.id, c.doc_id, c.content,
       coalesce(1.0 / (60 + s.rank), 0) + coalesce(1.0 / (60 + k.rank), 0) AS rrf
FROM semantic s
FULL OUTER JOIN keyword k ON s.id = k.id
JOIN chunks c ON c.id = coalesce(s.id, k.id)
ORDER BY rrf DESC
LIMIT 40;

Note that the 'english' text search configuration stems words and drops stop words, which is right for prose and wrong for identifiers. If your users search for codes, index a second tsvector with the simple configuration and query both.

Filters, permissions and the HNSW trap

Metadata filters (tenant, product, document type, access group) belong in the retrieval query. They are also where approximate indexes surprise people. An HNSW scan in pgvector collects a fixed number of candidates (hnsw.ef_search, default 40) and applies your WHERE clause afterwards. If only a small share of rows match the filter, you can get back far fewer results than your LIMIT, sometimes none. Options, in order of preference:

• Partition or create partial indexes per large tenant so the filter is the index.
• In pgvector 0.8 and later, enable iterative index scans with SET hnsw.iterative_scan = relaxed_order; so the scan continues until enough rows pass the filter or it reaches the hnsw.max_scan_tuples limit.
• Raise hnsw.ef_search for filtered queries, accepting the latency cost.

Permissions deserve their own sentence: the retrieval layer must only ever return chunks the asking user can read. Do not retrieve everything and ask the model to ignore what the user should not see. It will not.

Query transformation

The user's message is often a poor search query.

• Condense conversations. In a chat, "and what about for annual plans?" retrieves nothing useful. Have a small model rewrite the latest turn into a standalone question using the conversation history, and retrieve on that.
• Multi-query. Generate two or three phrasings and fuse their results with RRF. It helps vague questions and costs one extra model call plus extra searches.
• HyDE. Generate a hypothetical answer and embed it instead of the question. It can help when questions and documents use very different language. Test it; it can also pull retrieval toward the model's own guesses.
• Routing. If you have several indexes (docs, tickets, policies), classify the question first and search the right one, or search all and fuse.

Every transformation adds a model call to your latency budget. Add them one at a time, and only when the evaluation shows a gain.

Reranking

An embedding compresses a whole chunk into one vector before it has seen the question. A cross-encoder reranker reads the question and the chunk together, token by token, so it can tell that a passage mentions your error code in passing versus one that explains how to fix it. That makes it far more accurate and far more expensive, which is why it runs on a shortlist.

The pattern that works:

1. Hybrid retrieval returns 30 to 50 candidates.
2. The reranker scores each question and candidate pair.
3. The top 4 to 8 go into the prompt.

Options range from small open cross-encoders you host yourself, to hosted rerank APIs, to using an LLM as a listwise reranker for low-volume, high-value queries. Whatever you pick, calibrate a minimum score on your evaluation set. When the best candidate scores below it, the honest answer is "I could not find that", and that response is a feature.

Generation: grounded prompts and citations

The generation step has three jobs: answer from the context, cite it, and decline when the context does not contain the answer.

• Number the sources in the prompt and require inline citations like [2].
• Give an explicit decline phrase and treat it as a valid, tested outcome.
• Validate citations in code. If the answer cites a source number that does not exist, or makes claims with no citation, do not show it.
• Put the strongest passages first and last. Models use information at the edges of a long context more reliably than the middle (the "lost in the middle" effect), which is another reason to send a few good chunks rather than many mediocre ones.
• Treat retrieved text as data, not instructions. Documents can contain text that reads like instructions to the model. That is indirect prompt injection, and it is covered in depth in our guide to securing AI agents in production.

A worked RAG pipeline in Python

Here is a compact pipeline that implements everything above except the database: structure-aware chunking with contextual headers, hybrid retrieval with RRF, cross-encoder reranking, a grounded prompt, a no-answer path and citation validation. It runs in memory, which is fine for a few thousand chunks and for evaluation; swap the index for the Postgres schema above in production.

"""rag_pipeline.py: pip install sentence-transformers rank-bm25 numpy"""
import hashlib
import re
from dataclasses import dataclass
 
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder, SentenceTransformer
 
EMBED_MODEL = "BAAI/bge-small-en-v1.5"  # examples: pick whatever wins on your golden set
RERANK_MODEL = "cross-encoder/ms-marco-MiniLM-L6-v2"
MAX_CHUNK_WORDS = 350
MIN_RERANK_SCORE = 0.0  # this reranker emits logits; calibrate on your eval set
NO_ANSWER = "I could not find this in the documentation."
 
 
@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    title: str
    section: str
    text: str
 
    def indexed_text(self) -> str:
        # Contextual header: every chunk carries its document and section.
        return f"{self.title} / {self.section}\n{self.text}"
 
 
def make_chunk(doc_id: str, title: str, section: str, paragraphs: list[str]) -> Chunk:
    text = "\n\n".join(paragraphs)
    # Content-addressed IDs stay stable when unrelated parts of a doc change.
    digest = hashlib.sha1(text.encode("utf-8")).hexdigest()[:12]
    return Chunk(f"{doc_id}:{digest}", doc_id, title, section, text)
 
 
def split_markdown(doc_id: str, title: str, markdown: str) -> list[Chunk]:
    """Split on headings, then pack whole paragraphs up to a size limit."""
    chunks: list[Chunk] = []
    section = title
    for part in re.split(r"(?m)^(#{1,3} .+)$", markdown):
        if re.match(r"^#{1,3} ", part):
            section = part.lstrip("#").strip()
            continue
        buffer: list[str] = []
        size = 0
        for para in (p.strip() for p in part.split("\n\n")):
            if not para:
                continue
            words = len(para.split())
            if buffer and size + words > MAX_CHUNK_WORDS:
                chunks.append(make_chunk(doc_id, title, section, buffer))
                buffer, size = [], 0
            buffer.append(para)
            size += words
        if buffer:
            chunks.append(make_chunk(doc_id, title, section, buffer))
    return chunks
 
 
def tokenize(text: str) -> list[str]:
    # Keep identifiers like err_cert_date_invalid and v2.3.1 intact.
    return re.findall(r"[a-z0-9][a-z0-9._-]*", text.lower())
 
 
def rrf(rankings: list[list[int]], k: int = 60) -> list[int]:
    scores: dict[int, float] = {}
    for ranking in rankings:
        for rank, idx in enumerate(ranking, start=1):
            scores[idx] = scores.get(idx, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)
 
 
class HybridIndex:
    def __init__(self, chunks: list[Chunk]):
        self.chunks = chunks
        self.embedder = SentenceTransformer(EMBED_MODEL)
        self.reranker = CrossEncoder(RERANK_MODEL)
        texts = [c.indexed_text() for c in chunks]
        self.vectors = self.embedder.encode(texts, normalize_embeddings=True)
        self.bm25 = BM25Okapi([tokenize(t) for t in texts])
 
    def dense(self, query: str, k: int) -> list[int]:
        q = self.embedder.encode(query, normalize_embeddings=True)
        return list(np.argsort(-(self.vectors @ q))[:k])
 
    def sparse(self, query: str, k: int) -> list[int]:
        scores = self.bm25.get_scores(tokenize(query))
        return [i for i in np.argsort(-scores)[:k] if scores[i] > 0]
 
    def search(self, query: str, candidates: int = 40, final: int = 6):
        fused = rrf([self.dense(query, candidates), self.sparse(query, candidates)])
        pool = [self.chunks[i] for i in fused[:candidates]]
        if not pool:
            return []
        scores = self.reranker.predict([(query, c.indexed_text()) for c in pool])
        ranked = sorted(zip(pool, scores), key=lambda pair: pair[1], reverse=True)
        return ranked[:final]
 
 
PROMPT = """Answer the question using only the numbered sources below.
Cite every claim inline like [2]. If the sources do not contain the answer,
reply with exactly this sentence and nothing else: {no_answer}
 
Sources:
{sources}
 
Question: {question}
Answer:"""
 
 
def answer(index: HybridIndex, question: str, llm) -> dict:
    """llm is any callable that takes a prompt string and returns text."""
    hits = index.search(question)
    if not hits or hits[0][1] < MIN_RERANK_SCORE:
        return {"answer": NO_ANSWER, "sources": [], "reason": "low_retrieval_score"}
 
    sources = "\n\n".join(
        f"[{n}] {c.title} / {c.section}\n{c.text}" for n, (c, _) in enumerate(hits, start=1)
    )
    text = llm(PROMPT.format(no_answer=NO_ANSWER, sources=sources, question=question)).strip()
    if text == NO_ANSWER:
        return {"answer": NO_ANSWER, "sources": [], "reason": "model_declined"}
 
    cited = sorted({int(n) for n in re.findall(r"\[(\d+)\]", text)})
    if not cited or any(n < 1 or n > len(hits) for n in cited):
        # Uncited or mis-cited answers are not shown. Log them for review.
        return {"answer": NO_ANSWER, "sources": [], "reason": "citation_check_failed"}
    return {"answer": text, "sources": [hits[n - 1][0].chunk_id for n in cited]}

Wiring it up takes a few lines. load_docs and call_model are yours: the first returns (doc_id, title, markdown) tuples, the second wraps whichever LLM provider you use.

docs = load_docs()
chunks = [chunk for doc in docs for chunk in split_markdown(*doc)]
index = HybridIndex(chunks)
 
result = answer(index, "How do I rotate the webhook signing secret?", llm=call_model)
print(result["answer"], result["sources"])

Three details in that code carry most of the production value. The reason field on every declined answer tells you whether retrieval or generation failed, which is the first question in every incident. Content-hash chunk IDs make re-indexing idempotent. And the citation check means a confident answer with no source never reaches a user.

Evaluating RAG

You cannot tune a RAG system by reading a few answers and nodding. Build a golden set first.

Build the golden set

• Collect 50 to 100 real questions to start: search logs, support tickets, Slack questions to the team that owns the docs. Synthetic questions generated from your documents are useful for coverage but are easier than real ones; do not let them be the whole set.
• For each question, record which documents (or sections) answer it. Label at document or section level, not chunk level, so labels survive re-chunking.
• Include questions with no answer in the corpus. The correct output for those is the decline phrase.
• Version the set in Git next to the pipeline code and grow it from production failures every week.

Measure retrieval and generation separately

If recall at k is low, no prompt will save you: fix parsing, chunking or retrieval. If recall is high but faithfulness is low, work on the prompt, the context ordering or the model. This split is the single most useful diagnostic in RAG work.

A retrieval evaluation is short enough to run on every commit:

import json
 
def evaluate_retrieval(index: HybridIndex, path: str, k: int = 6) -> dict:
    """golden.jsonl lines look like:
    {"question": "...", "relevant_doc_ids": ["kb-112", "kb-087"]}
    """
    recall_sum, rr_sum, n = 0.0, 0.0, 0
    with open(path, encoding="utf-8") as f:
        for line in f:
            case = json.loads(line)
            relevant = set(case["relevant_doc_ids"])
            if not relevant:
                continue  # no-answer cases are scored by the generation eval
            ranked = [c.doc_id for c, _ in index.search(case["question"], final=k)]
            n += 1
            recall_sum += len(relevant & set(ranked)) / len(relevant)
            rr_sum += next((1 / r for r, d in enumerate(ranked, 1) if d in relevant), 0.0)
    return {f"recall@{k}": recall_sum / n, "mrr": rr_sum / n, "questions": n}

For generation metrics, open-source frameworks such as RAGAS and DeepEval implement faithfulness and relevance with an LLM as judge. Use them, but check the judge against 50 or so answers that a human has graded before you trust its numbers. A judge that disagrees with your domain experts is measuring something else.

Watch production too

Offline evaluation tells you whether a change is safe to ship. Production tells you what to add to the golden set. Log the question, rewritten query, retrieved chunk IDs and scores, reranker scores, the final prompt size, the answer, the decline reason and any user feedback. A rising decline rate on one topic usually means missing or stale documents, not a model problem.

What goes wrong in production

These are the failures we see when we are the ones on the pager for a RAG system.

• Deleted documents keep answering. Someone removes a policy page; the chunks stay in the index for months. Treat deletion as a first-class event: sync on source deletes, and run a periodic reconciliation that removes chunks whose document no longer exists.
• Stale answers from partial re-indexing. A document changes, new chunks are added, old ones are not removed, and retrieval returns both versions. Re-index per document atomically: delete by doc_id, then insert, in one transaction.
• An embedding model upgrade breaks everything quietly. Query vectors from the new model searched against document vectors from the old one return plausible garbage, not errors. Store the model name with every vector and refuse mixed searches.
• Filters starve the vector index. The tenant filter plus HNSW returns three results instead of forty, and answer quality drops for small tenants only. See the filtering section above.
• Permission leaks through retrieval. A shared index without per-user filtering will eventually put another team's or another customer's content in someone's context. Filter in the query, and test it with a user who should see nothing.
• Near-duplicate documents crowd the top results. Five versions of the same onboarding guide take every slot. Deduplicate at index time by content hash and near-duplicate detection, or collapse results by doc_id before reranking.
• Prompt injection through documents. Any document a user can edit is an input channel to your model. Retrieved text must never be able to trigger tools or change instructions.
• Latency budget blown by stacking. Query rewrite, multi-query, two retrievers, a reranker and a large model each add a little. Write the budget down per stage and measure each one.
• Caches serve answers after the source changed. If you cache answers, key the cache on the IDs and hashes of the retrieved chunks, not just the question text.

Production checklist

• Parser output reviewed by a human on a representative sample, tables and scans included
• Structure-aware chunks with contextual headers and content-hash IDs
• Hybrid retrieval with RRF, identifier-friendly keyword index
• Permission and tenant filters in the retrieval query, tested
• Reranker with a calibrated minimum score and a tested no-answer path
• Citation validation in code
• Golden set in Git, retrieval and generation metrics in CI
• Deletion sync, per-document atomic re-indexing, model version stored with vectors
• Per-stage latency and cost logged for every query

If your RAG system needs to take actions as well as answer questions, the design questions shift to tools, state and limits, which we cover in our guide to building AI agents for production.

Frequently Asked Questions

What is retrieval-augmented generation (RAG)?

RAG is a pattern where an application first retrieves relevant passages from your own documents and then asks a language model to answer using only those passages, with citations. It lets a general model answer questions about private or fast-changing data without retraining. The quality of the answer depends mostly on whether retrieval found the right passages.

What is the best chunk size for RAG?

There is no universal best size. Start with chunks that follow the document's own structure (headings, then paragraphs) and land roughly in the 300 to 800 token range for prose, then tune against a labeled evaluation set. Smaller chunks give precise matches but lose context; larger chunks keep context but dilute the embedding and waste prompt space.

What is hybrid search in RAG?

Hybrid search runs keyword search (usually BM25) and vector search on the same query and merges the two ranked lists, typically with Reciprocal Rank Fusion. Keyword search catches exact tokens such as error codes, SKUs and names that embeddings blur, while vector search catches paraphrases. Most production RAG systems should use it by default.

Do I need a reranker in a RAG pipeline?

Usually yes, once you have an evaluation set that shows the right passage is retrieved but not ranked near the top. A cross-encoder reranker reads the query and each candidate together, so it scores relevance more accurately than embedding similarity. The cost is extra latency per query, so rerank a few dozen candidates, not hundreds.

How do you evaluate a RAG system?

Evaluate retrieval and generation separately. For retrieval, use a golden set of real questions labeled with the documents that answer them and track recall at k and MRR. For generation, check faithfulness to the retrieved context and answer relevance, using an LLM judge that you have calibrated against human labels.

Should I use RAG or fine-tuning?

Use RAG when the model needs knowledge it does not have, especially knowledge that changes or must be cited. Use fine-tuning when you need to change behavior, such as output format, tone or a narrow classification task. Many production systems use both: a tuned model for behavior and RAG for facts.

Need help taking RAG to production?

We build and run retrieval systems for teams whose demo works on ten documents and falls over on ten thousand. Senior engineers fix the parsing, retrieval and evaluation, wire in the permissions, and stay on the pager after launch. If we built it, we're on the pager.

• AI and machine learning services
• POC-to-production engineering
• Talk to an engineer