RAG Architecture Patterns

A RAG pipeline has three stages: indexing (processing and storing documents), retrieval (finding relevant chunks), and generation (creating responses with context).

Each stage has design decisions that significantly impact quality and performance.

The goal is high-precision retrieval: return exactly the information needed to answer the query.

Chunking strategy is crucial. Too small: lose context. Too large: reduce precision and increase costs.

Common strategies: fixed-size chunks with overlap, semantic chunking (by paragraph/section), and recursive chunking.

Consider document structure: headers, tables, and code blocks need special handling.

Embeddings convert text to numerical vectors that capture semantic meaning. Similar content has similar vectors.

Choice of embedding model matters: OpenAI ada-002, Cohere embed, or open-source alternatives like BGE.

Consider dimensionality, speed, and quality tradeoffs based on your scale.

Vector databases store and search embeddings efficiently. Popular options: Pinecone, Weaviate, Qdrant, Chroma, and pgvector.

Key features: approximate nearest neighbor search, metadata filtering, and hybrid search capabilities.

For many use cases, pgvector (PostgreSQL extension) provides sufficient performance with simpler operations.

Combine semantic search (embeddings) with keyword search (BM25) for better results.

Semantic search understands meaning but can miss exact matches. Keyword search catches specific terms.

Reciprocal Rank Fusion (RRF) is a common technique for combining rankings from multiple search methods.

Retrieved chunks must fit in the LLM's context window along with the conversation history and system prompt.

Strategies: rank and truncate, summarize older context, or use hierarchical retrieval.

Balance recall (finding all relevant info) with precision (not overwhelming the model).

Measure retrieval quality: precision, recall, and mean reciprocal rank (MRR).

Measure generation quality: accuracy, relevance, and groundedness (does the answer match retrieved content?).

Build evaluation datasets from real user queries for ongoing improvement.