Six months into the project, the team still cannot explain why the model confidently cites a policy that was updated two years ago. The retrieval system is returning documents. The model is generating responses. The hallucinations persist. The usual diagnosis: “we need a better model.” The actual diagnosis: the retrieval layer was built as a search engine, and search engines are not evidence collection systems.
This distinction is not semantic. It changes what you build.
The RAG Misconception That Costs Six Months
Retrieval-Augmented Generation — RAG — stands for Retrieve, Augment, Generate. Most enterprise implementations focus heavily on the Generate step (which model, which prompt) and treat Retrieve as a solved problem because semantic search libraries exist and tutorials make chunking look trivial.
The architectural failure is in the Augment step. Augmentation is not concatenation. It is the process of assembling a context window that gives the model the specific evidence it needs to answer the question without gaps. When augmentation fails, the model fills gaps with training knowledge. That is the hallucination mechanism.
The mental model that predicts failure: if you think of RAG as “semantic search with a summarization layer on top,” you will build exactly that and wonder why the model still invents things.
The mental model that predicts success: RAG is an evidence collection system. The retrieval layer is an investigator that must return the right documents, the right sections, with the right metadata. The model is a reasoning layer applied to that evidence. The model cannot reason well over bad evidence.
The Naïve RAG Failure Modes
Four failure modes account for the majority of hallucination problems in enterprise RAG deployments:
Fixed-size chunking. The default in most tutorials: split text every 512 tokens, overlap by 50 tokens, store. A procedure described across two pages gets split at a chunk boundary. The model receives half a procedure and fills in the rest from training. Fix: semantic chunking or hierarchical chunking that preserves document structure.
No reranking. Top-K by cosine similarity is not top-K by relevance to the actual question. A document with high embedding similarity to the query terms may be less relevant to the user’s intent than a document with lower similarity but direct procedural coverage. Fix: cross-encoder reranking as a second-pass after initial retrieval.
Missing metadata filtering. The retrieval layer returns documents from the wrong department, wrong language, or superseded versions. The model cannot distinguish a current policy from a deprecated one without explicit temporal and classification metadata. Fix: every chunk carries source, date, owner, jurisdiction, and classification; filtering happens before vector search, not after.
No faithfulness check. The model’s response is not verified against the context it was given. A response can be fluent, confident, and factually inconsistent with the retrieved documents. Fix: faithfulness scoring as a gate, not a post-hoc log.
Each failure mode is independent. Fixing chunking without fixing reranking produces a different class of hallucination, not zero hallucination.
The Advanced Retrieval Stack That Actually Works
The retrieval techniques below are not a checklist to apply universally. Each addresses a specific failure mode. Deploying all of them without a target failure mode wastes engineering time.
Hybrid search with RRF fusion. BM25 (keyword matching) combined with dense vector search, results fused using Reciprocal Rank Fusion. BM25 captures exact-match terms, equipment codes, names, reference numbers that dense vectors handle poorly. Dense search captures semantic intent that BM25 misses. The combination outperforms either alone on enterprise document corpora. This is now the baseline, not an advanced option.
HyDE (Hypothetical Document Embedding). Instead of embedding the user’s question and searching for similar documents, the model generates a hypothetical ideal answer to the question, and that answer is embedded and used for retrieval. Documents retrieved this way match the structure and specificity of a correct answer rather than the structure of a question. Particularly effective for technical document retrieval where question phrasing diverges sharply from answer phrasing.
RAG-Fusion. Generate multiple reformulations of the original query, run parallel retrievals against each, fuse results. Improves recall for queries where the user’s phrasing is not the most direct path to the relevant document.
Cross-encoder reranking. After initial retrieval, a cross-encoder model scores each (query, document) pair jointly, rather than comparing independent embeddings. Slower than vector search, runs on a small candidate set (top-20 to top-50), substantially improves precision. Libraries like FlashRank provide self-hostable cross-encoder reranking.
Hierarchical chunking. Parent chunks hold document summaries; child chunks hold granular content. Retrieval operates at the parent level to identify relevant sections, then fetches the child chunks for context. Preserves document structure while enabling granular retrieval. Appropriate for long regulatory documents, technical manuals, and contract repositories.
The retrieval layer is not a detail. It is the constraint that determines how much of the model’s capability you can actually use.
The Evaluation Framework That Prevents Deployment Regret
Enterprise teams that deploy RAG without an evaluation harness operate on opinions. The opinions are often wrong, and the discovery is expensive.
The RAGAS framework provides four metrics that together measure RAG quality without requiring human annotation for every response:
- Faithfulness: is the response grounded in the retrieved context, or does it introduce claims not supported by the documents?
- Answer Relevancy: does the response address the user’s question?
- Context Precision: are the retrieved chunks relevant to the question?
- Context Recall: were all relevant chunks retrieved, or is material missing?
Low faithfulness with high context precision means the model is ignoring the evidence. Low context recall means the retrieval layer is missing relevant documents. Each metric identifies a different layer of the system to fix.
Build the evaluation dataset before deployment, not after: 50 to 100 representative questions with expected answers and source citations, drawn from the actual user population. Every prompt change, model upgrade, chunking modification, and reranking parameter change gets tested against this baseline. This is regression testing applied to a probabilistic system.
The production feedback loop extends the eval set continuously: user corrections, rejected answers, and escalated queries become new test cases. The eval harness is not a one-time gate, it is the mechanism for maintaining quality as the system evolves.
Security and Permissions in Enterprise RAG
The retrieval layer inherits data permissions. A user querying a RAG system should not retrieve documents they cannot access in the source system. Most enterprise RAG deployments skip this until the security review.
Row-level security in vector databases requires that every chunk carries user group, classification, and jurisdiction metadata, and that metadata is enforced at retrieval time, not filtered after retrieval. Filtering after retrieval means the document was retrieved and handled by the system, even if not shown to the user. Depending on jurisdiction, that matters.
Private-perimeter RAG keeps data within the organization’s infrastructure. Model inference runs on-premise or in a private cloud environment. No document content, no chunk, and no embedding leaves the perimeter. For regulated industries and GDPR-sensitive data, this is architecture, not preference.
The audit trail requirement deserves explicit treatment: for EU AI Act compliance and ISO 42001 alignment, the system must be able to answer “which documents grounded this response, for which user, at which time?” A RAG system without an audit trail is not compliant regardless of how accurate its responses are.
When RAG Is Not the Answer
RAG is the right architecture for a specific class of enterprise knowledge problem: question answering over a heterogeneous document corpus where the answer can be grounded in specific passages.
It is not the right architecture for every knowledge retrieval problem.
If the question requires synthesis across hundreds of documents simultaneously, tracking relationships between entities across a corpus, or connecting regulatory requirements across multiple jurisdictions, GraphRAG is more appropriate. Graph-based retrieval traverses entity relationships; vector retrieval finds similar passages. These are different operations.
If the knowledge domain is highly structured, tabular, or transactional, a deterministic query layer is more reliable than vector retrieval. The model generating SQL and executing it against a structured database will outperform vector search applied to documents that describe structured data.
If the knowledge is static, well-bounded, and stable, a compiled context layer or fine-tuned model may outperform retrieval. RAG adds latency and complexity for a benefit that only materializes when the knowledge domain is large, dynamic, or heterogeneous.
The right framing: RAG is an evidence retrieval architecture. Match the architecture to the epistemic structure of the problem. Start with that question, not with the framework that is easiest to deploy.
The ingestion pipeline is where most RAG quality problems are created before the model ever sees a query. That architecture is covered here.
If your team is evaluating which retrieval architecture fits your specific use case, the AI Opportunity Sprint is how we start.