What shipping RAG to 500+ enterprise users actually taught me
Chunking strategies and vector databases are the easy 20%. The hard 80% is access control, stale documents, and users who ask questions no document answers.
I’ve now been running RAG in production for long enough that the demo-phase memories feel quaint. Here’s what the tutorials didn’t cover, roughly in the order it hurt.
1. Permissions are the feature
The first serious incident wasn’t hallucination. It was retrieval working too well: a regional salary band document surfaced in an answer to someone who had no business seeing it. Embeddings don’t know about RBAC.
Now every chunk carries access metadata, filtering happens at query time inside the retriever (never post-hoc in the prompt), and we log which documents fed every answer. If you’re building RAG for an org with more than one permission level, build this first. Not second. First.
2. Hybrid search isn’t optional past a few thousand documents
Pure vector search kept missing exact matches — policy codes, employee IDs, project names. Things people actually search for. Semantic similarity thinks “P-2214” and “P-2241” are basically the same document. BM25 does not.
Hybrid retrieval plus a re-ranker fixed a whole category of complaints in one release. The re-ranker earns its latency: we cast a wide net (top 40 hybrid results), then let it pick the 6 that go in the context. Answer quality moved more from that change than from any prompt work we ever did.
3. Stale beats wrong, but fresh beats both
Documents change. The HR policy that was true in January is wrong in March, and the old version is still in your index unless you built re-ingestion from day one. We didn’t. For a while, the bot confidently quoted a leave policy that had been superseded twice.
The fix was boring engineering: source-of-truth timestamps, nightly diff-based re-indexing, and a “last verified” date shown in the answer. That last one mattered most — it converted “the bot is wrong” tickets into “this document needs updating” tickets, which land on the right team.
4. Some questions have no document
Maybe a fifth of real user questions had no answer anywhere in the corpus. Early on, the model bridged those gaps with plausible fiction — the worst possible behavior in an enterprise setting.
We attacked it from both ends. Retrieval confidence gating on one side: if nothing scores above threshold, say so, and say it plainly. On the other side, those “no answer” queries became a report we hand to content owners monthly. RAG turned out to be a fantastic tool for discovering what your documentation doesn’t cover.
5. Nobody cares about your architecture
Not once has a user asked which vector database we use. (pgvector, then a dedicated store when scale demanded it. It mattered less than I expected.) They ask: is it right, is it fast, can I trust it, why did it say that.
Citations answered “why did it say that.” Latency work answered “is it fast.” Evals — accuracy, safety, groundedness, run in CI before any prompt or retrieval change ships — answer “can I trust it,” slowly, release by release.
The stack is the easy part. Trust is the product.