A client once showed me a RAG system that 'worked' — the demo answered beautifully. In production, it hallucinated a refund policy that didn't exist, because the retrieval step pulled the wrong document and the model dutifully summarized it. The model was fine. Retrieval was the product problem.

Retrieval quality is data quality

Embeddings don't understand meaning. They capture statistical proximity in a vector space — a great match for fuzzy recall and a poor match for precision. When the stakes are a support answer or a compliance lookup, you need retrieval that can say 'nothing matches', and most pipelines can't.

What a production pipeline looks like

  • Chunk with structure, not fixed sizes. Split on semantic boundaries — headings, sections, tables — and keep metadata (source, date, version) attached to every chunk. Fixed 500-token slices destroy the evidence you need later.
  • Search with hybrid recall. Semantic search for concepts, keyword/BM25 for exact terms like part numbers and policy names. Merge and dedupe before re-ranking.
  • Re-rank before you generate. The top-5 by cosine similarity is rarely the top-5 for answering. A small cross-encoder re-ranker is the single highest-leverage upgrade most systems are missing.
  • Return the receipts. The generator should see the source and the answer should cite it, so a human can verify — and so the model is rewarded for grounded answers.
sql
SELECT id, source, embedding <=> $1 AS distance
FROM chunks
WHERE (embedding <=> $1) < $threshold        -- semantic
   OR plainto_tsquery('english', $query) @@ tsv  -- keyword
ORDER BY distance
LIMIT 50;   -- then re-rank, then answer

Evals for retrieval: the missing test suite

Retrieval evals are embarrassingly simple to build and almost nobody has them. Take fifty real questions, mark the chunk that contains the answer, and measure recall@k on every change. When you swap an embedding model or re-chunk, you'll know in minutes whether you've quietly made the system worse. That's the test suite most teams skip — and it's why their RAG 'sometimes works'.

Quality ceiling

The model is rarely the ceiling. The ceiling is whether the right chunk is in the context. Fix retrieval first, re-rank second, and only touch the prompt last.

When to say no to RAG

Some questions shouldn't be answered by retrieval at all. High-stakes lookups — a balance, a dosage, a policy clause — should go straight to a structured query against the source of truth, with the model only formatting the result. Knowing which questions need a database answer and which need a generative answer is the actual design work.

A RAG system that answers is a pipeline that knows its own data: hybrid recall, re-ranking, receipts, and evals that gate every change. Build those, and 'based on our documentation' stops being a disclaimer and starts being true.