Building Production-Grade RAG Pipelines
Lessons learned from shipping retrieval-augmented generation systems that actually hold up under real query load.
Retrieval-Augmented Generation sounds straightforward on paper: embed your documents, store them in a vector database, retrieve the top-k chunks at query time, stuff them into the prompt. The gap between that mental model and a system that survives contact with production traffic is where most of my last eighteen months has gone.
The naive version works until it doesn't
Our first RAG deployment used a single embedding model, cosine similarity search over one million vectors, and a fixed 4k-token context window. It worked beautifully on our test set. Within a week of launch, we were seeing three failure modes we hadn't anticipated.
First, semantic drift — our queries came in the language of the customer, not the language of our documentation. The embeddings clustered around domain vocabulary that our training data had never seen. Second, context pollution — top-k retrieval returned chunks that were individually relevant but collectively contradictory, and the LLM dutifully synthesized a confident, wrong answer. Third, latency tail — p99 retrieval time was over 800ms because our index wasn't partitioned and every query scanned the full corpus.
What we changed
The fix for semantic drift was a hybrid search layer. We kept the dense vector index for broad semantic matching but layered BM25 sparse retrieval on top, then reranked the merged candidate set with a cross-encoder. The reranker was the expensive part, but because it only scored the top-50 candidates from the fast retrieval stage, we kept latency budget while reclaiming about fourteen percentage points on recall.
Context pollution we addressed at two levels. At the retrieval stage, we added metadata filtering so results came from consistent document sections. At the prompt stage, we switched from stuffing raw chunks to extracting structured claims from each chunk and letting the model reason over claims rather than prose. This cut hallucination rates roughly in half on our evaluation set.
The indexing pipeline matters more than the retrieval pipeline
The mistake I see most teams make is over-investing in retrieval algorithms and under-investing in indexing. A mediocre retrieval algorithm on well-chunked, well-embedded, well-metadata-tagged data will beat a sophisticated retrieval algorithm on badly-prepared data every time.
We moved to a chunking strategy that respected document structure rather than fixed-size windows. Headings, tables, and list boundaries became chunk boundaries. We embedded chunk metadata alongside chunk text. We re-indexed when source documents changed, not on a schedule. The retrieval algorithm barely changed — but the data feeding it became dramatically cleaner.
What I would do differently
Start with an evaluation harness before you write a single line of retrieval code. We built ours three months in, which meant the first three months of architecture decisions were made without a feedback loop. The harness became the single most valuable component of the system — every change to chunking, embedding, retrieval, or prompting ran through it before it touched production.
RAG is not a solved problem. But it is a problem with well-understood failure modes and incremental fixes. The key is measuring each fix against a fixed evaluation set rather than trusting your gut about what "feels" better.