RAG That Actually Works: Lessons from Production

Every team we meet has built a RAG prototype. Almost none have shipped one that users trust. The gap isn't the model — it's everything around it.

Start with an evaluation set, not a vector database

Before writing a line of pipeline code, collect 50–100 real questions your users would ask, with reference answers written by a domain expert. This becomes your ground truth. Without it, every architecture debate is a matter of opinion.

Retrieval quality beats model quality

A frontier model with bad context loses to a mid-tier model with great context. Spend your time on:

• Chunking that respects structure — split on headings and semantic boundaries, not fixed token counts
• Hybrid search — combine dense embeddings with BM25 keyword matching; each catches what the other misses
• Reranking — a cheap cross-encoder pass over the top 20 candidates dramatically improves the top 5

Grounding is a product feature

Show citations. Let users click through to sources. When confidence is low, say so and escalate to a human. The fastest way to lose trust is one confidently wrong answer — the fastest way to build it is an honest "I'm not sure, here's a human."

Measure before and after every change

Our pipelines run the eval set on every PR. Retrieval recall, answer faithfulness, and latency are tracked like any other regression. When a prompt tweak drops faithfulness 4%, we know before users do.

The teams that win with AI aren't the ones with the fanciest architecture — they're the ones who can prove their system works.