Production RAG Systems: Architecture Patterns & Pitfalls

# Hybrid retrieval with reciprocal rank fusion
def hybrid_retrieve(query, filters, top_k=10):
    # Dense retrieval
    vector_results = vector_db.search(
        embed(query), 
        filter=filters,
        top_k=top_k * 2
    )
    
    # Sparse retrieval
    bm25_results = bm25_index.search(query, top_k=top_k * 2)
    
    # Reciprocal Rank Fusion
    combined = reciprocal_rank_fusion([vector_results, bm25_results])
    return combined[:top_k]

Pattern 2: Chunking Strategy

How you chunk documents determines retrieval quality more than your embedding model choice.

Pro tip: Store overlapping chunks (10-20% overlap) to prevent boundary effects. Also store the parent document for context expansion.

Pattern 3: Reranking

Two-stage retrieval (retrieve-then-rerank) consistently outperforms single-stage:

1. Stage 1: Fast retrieval returns 50-100 candidates
2. Stage 2: Cross-encoder reranker scores each candidate against the query
3. Return: Top 5-10 reranked results to the generator

# Reranking with a cross-encoder
from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query, candidates):
    pairs = [(query, doc.text) for doc in candidates]
    scores = reranker.predict(pairs)
    return sorted(zip(candidates, scores), key=lambda x: -x[1])

Pitfall 1: The „Lost in the Middle“ Problem

LLMs perform worse when relevant information is in the middle of a long context window. Solution: Place the most relevant chunks at the beginning and end of the context.

Pitfall 2: Stale Indexes

Your RAG system is only as fresh as your index. Build an incremental update pipeline:

# Incremental index update pattern
class RAGIndexManager:
    def update_document(self, doc_id, new_content):
        # 1. Remove old chunks
        old_chunks = self.get_chunks_by_doc(doc_id)
        self.vector_db.delete([c.id for c in old_chunks])
        
        # 2. Chunk new content
        new_chunks = self.chunker.chunk(new_content)
        
        # 3. Embed and store
        embeddings = self.embedder.embed([c.text for c in new_chunks])
        self.vector_db.upsert(new_chunks, embeddings)
        
        # 4. Update BM25 index
        self.bm25_index.update(doc_id, new_content)

Pitfall 3: Hallucination Despite Retrieval

RAG reduces hallucination but doesn’t eliminate it. Mitigations:

• Explicit grounding instructions: „Only use information from the provided sources“
• Citation requirements: Force the model to cite source chunks
• Confidence scoring: Flag low-confidence responses for human review
• Post-generation verification: Use a separate model to check claims against sources

Pattern 4: Multi-Step RAG (Agentic RAG)

For complex queries, a single retrieve-then-generate pass isn’t enough. Agentic RAG uses the LLM to iteratively:

1. Decompose the query into sub-questions
2. Retrieve for each sub-question
3. Synthesize partial answers
4. Decide if more retrieval is needed
5. Generate the final answer

Scaling Considerations

Evaluation Framework

Measure your RAG system across these dimensions:

• Retrieval quality: Precision@K, Recall@K, MRR, NDCG
• Generation quality: Faithfulness, answer relevance, completeness
• End-to-end: Human preference, task completion rate
• Operational: Latency, cost per query, index freshness

Conclusion

Production RAG is an engineering discipline, not a prompt. Invest in your retrieval pipeline, build robust chunking and indexing, implement reranking, and monitor continuously. The teams that treat RAG as a first-class system — not a quick hack — will build AI products that actually deliver reliable knowledge.

Related: AI Agent Evaluation & Testing Handbook