RAG — retrieval-augmented generation — is the architecture where your model looks something up in your data before it answers. Instead of training the model on your knowledge or stuffing every document into the prompt, you index your corpus, retrieve the few passages most relevant to the user’s question at query time, and pass them to the model alongside the question. The model writes an answer grounded in the retrieved text and, ideally, cites it. The technique was named in a 2020 paper by Patrick Lewis and collaborators (arXiv:2005.11401) and has become the default any time an AI product needs to answer from a corpus it did not train on.
This founder’s explainer sits within the eval-first build playbook, one of the core guides in the idea-to-product manifesto. By the end you will know what RAG is, the four ingredients you are paying a vendor to build, three signals you need it, three signals you don’t, and the eval shape to grade it against.
What RAG is, in one paragraph
A bare-LLM application sends the question straight to a model like GPT-5, Claude Opus 4.8, or Gemini 2.5 Pro and lets it answer from training. A RAG application runs a retrieval step against your indexed corpus first, picks the top-k passages most likely to contain the answer, and packages them into the prompt alongside the question. The model answers using your text as substrate, often with citations. The benefit is grounded, current, source-attributable answers; the cost is new infrastructure (vector index, chunker, retriever) and new failure modes — the model can be misled by the wrong passage, miss the right one, or hallucinate around partial retrieval. The eval-first frame exists because retrieval systems fail in ways a feature-level user story cannot detect.
The four ingredients of a RAG system
Every RAG system decomposes into four ingredients. When a proposal says “we will implement RAG”, these are the four line items that should be visible.
1. The corpus and the chunker
The corpus is the body of text the system retrieves from — policies, support articles, contracts, product docs, customer transcripts. The chunker splits each document into the units the index will hold. Naïve chunking splits by character count; structural chunking respects headings and sections; contextual chunking prepends a one-sentence summary so the embedding captures what each chunk is about, not just what it says. Anthropic’s Contextual Retrieval post reports up to a 49% reduction in retrieval failure when contextual embeddings, contextual BM25, and a reranker are combined.
A vendor silent on chunking strategy is implicitly committing to the framework default — usually naïve fixed-length splitting — which is the single most common reason early-stage RAG systems retrieve the wrong passage.
2. The embedding model and the vector index
An embedding model converts each chunk into a vector that represents its meaning; the vector index stores those vectors and supports fast similarity search. Mainstream 2026 embeddings: OpenAI text-embedding-3, Cohere Embed v3, Voyage AI’s domain-tuned embeddings. Mainstream indexes: Pinecone, Weaviate, Chroma, Qdrant, Milvus, MongoDB Atlas Vector Search, and pgvector inside an existing Postgres deployment.
For most early-stage products these choices are not load-bearing. They become load-bearing with heavy domain jargon (generic embeddings mis-cluster), tight latency budgets, or index cost large enough to be an operating line item.
3. The retriever
The retriever embeds the question and pulls the top-k chunks back. The simplest is dense similarity. The 2024–2026 best practice is hybrid: dense similarity for semantic recall plus sparse BM25 for keyword precision, merged and reranked with a reranker model (Cohere Rerank or Voyage rerank). Reranking is the highest-ROI add-on in mainstream pipelines; a proposal that omits it is leaving recall on the table.
4. The generator
The generator is the LLM that writes the final answer using the retrieved passages. Its job is narrower than in a bare-LLM application: stay faithful to the retrieved text, decline when retrieval is weak, cite sources. The instruction “only answer from the provided passages, otherwise refuse” is what turns a generative model into a retrieval-grounded one; skipping it produces a system that hallucinates around weak retrieval.
Three signals you need RAG
Most founders arrive at “should I use RAG?” without a decision frame and over-scope. The three signals below justify the infrastructure.
Signal 1. Answers must cite your data
If the product’s value depends on the user trusting that the answer came from a specific source — a contract clause, a policy paragraph, a customer record, a regulation — the system must retrieve and attribute. A bare LLM cannot satisfy this because it does not have your corpus in its weights and cannot point at the document. Anything where the user response is “show me where you got that” is a RAG candidate.
Signal 2. Knowledge changes faster than the model retrains
If the corpus changes weekly — policies, pricing, contracts, support tickets — training or fine-tuning on that data is a losing race. RAG lets you re-index in minutes and the next query sees the new content. Rule of thumb: if knowledge changes more often than monthly, RAG is the right axis. The fine-tuning vs prompting vs RAG explainer walks the comparison.
Signal 3. Domain jargon won’t fit in context
Models in 2026 have large context windows — hundreds of thousands of tokens — but “fits in context” is not the same as “reasons well over the whole window.” Retrieval accuracy degrades inside long context well before the window fills, and cost grows with input tokens. If most queries need only a few paragraphs out of thousands, retrieval is correct even when the corpus would technically fit. RAG is the way to give the model the right 4,000 tokens out of 400,000.
Three signals you don’t
Every vendor explainer skips this section because each vendor sells the infrastructure RAG requires. It is the section a founder needs most.
Signal A. The answer is in a single document
If every question is answered by a single PDF, contract, or dashboard, load that document into the model’s context and ask the question. There is no retrieval problem to solve. Building a vector index and a reranker for a one-document use case pays for plumbing the product does not need and adds three failure modes to a feature that required none.
Signal B. The corpus is a static FAQ
If your support content is 40 well-curated entries that change quarterly, the answer is not RAG — it is a fine-tuned classifier or a prompt that lists the FAQ inline. Both are cheaper, more predictable, and easier to evaluate. Using RAG on a small static corpus is the AI-stack version of using Kubernetes to run a single container.
Signal C. There is no source-of-truth corpus
If the founder cannot point at the text the system should retrieve from, the problem is not retrieval — it is content. Vendors build retrieval well and assemble corpora poorly. A founder who scopes a RAG project before assembling the source-of-truth content watches the team build elaborate retrieval over thin material and never understands why the demo is weak.
How to grade a RAG system: the eval shape
A RAG system has two halves and is graded on both. Retrieval is graded on whether the right passages came back; generation is graded on whether the answer faithfully used them. The eval-first playbook treats this as a canonical multi-capability surface; the what-are-ai-evals explainer covers the basics. The minimum eval contract has four metrics.
Recall@k. On what fraction of test queries did at least one passage among the top-k retrieved chunks contain the answer? Starting threshold: 0.85 at k=5 for non-critical domains, 0.95+ for high-stakes ones (legal, medical, financial). Below threshold, no amount of generator polish rescues the system — the right passages are not getting through.
Mean reciprocal rank (MRR). When the answer is retrieved, how high in the ranked list does it appear? Higher MRR means the reranker is putting the right passage at the top and the generator has a shorter, cleaner context to reason over.
Faithfulness. Does every factual claim in the generated answer trace back to the retrieved passages? Graded by an LLM-as-judge with anchored language: “fully grounded”, “partially grounded with one unsupported claim”, “ungrounded”. Below 0.9, the system is fluently hallucinating — the worst failure mode for a retrieval-grounded product.
Citation accuracy. When the system says “see policy §4.2”, does §4.2 actually contain the cited fact? Graded by exact-match or fuzzy-match against the retrieved chunk’s source identifier. Below 0.95 is a credibility-destroying defect, especially in regulated contexts.
A proposal that does not specify thresholds across all four metrics is a proposal you cannot grade. SFAI Labs’ framing on decoding production-ready in agency proposals walks the same logic from the procurement side.
A worked example: a legal-research assistant
Imagine a non-engineer founder building an assistant for solo legal practitioners that answers questions about a state’s commercial-leasing statutes. The corpus is roughly 1,800 statute sections and 12,000 case-law summaries. Statutes update annually; case law updates weekly.
- Signal check. Yes on cite-able answers, yes on weekly-changing knowledge, yes on domain jargon (statute references like “RCW 59.18.060” are dense and abbreviated). All three fire. RAG is the right architecture.
- Chunker. Structural by statute section and by case-summary headnote. Contextual prepend of “[Section title] — [parent chapter]” on each chunk.
- Embedding and index. OpenAI
text-embedding-3-largefor first pass; Voyage’s legal-domain embedding as a fast follow if domain recall is weak. Pinecone or pgvector — not load-bearing at this scale. - Retriever. Hybrid: BM25 for statute-citation matches plus dense similarity for natural-language queries, merged and reranked with Cohere Rerank. Top-k = 8 for retrieval, top-3 after rerank into the generator.
- Generator. Claude Opus 4.8 with a refuse-when-retrieval-weak system prompt and a “cite the section or headnote ID after every claim” instruction.
- Eval contract. A 200-query test set sampled from real practitioner questions, graded on recall@5 ≥ 0.95, MRR ≥ 0.80, faithfulness ≥ 0.92, citation accuracy ≥ 0.98. The suite is the migration test on every model update and runs in CI on every code change.
The four ingredients and the eval contract are the founder’s defense against scope drift. With them, no agency can ship “RAG is working” as a vibes claim.
What changes if your vendor picks a wrong ingredient
A single bad ingredient pick degrades the whole system. Worth naming the patterns so a founder reading a proposal knows what to ask.
- Naïve chunking — the right answer lives in chunk 7 but its embedding is dominated by an irrelevant paragraph. Recall@k craters; the founder discovers it in production, not CI.
- No reranker — recall@k looks decent, but the top-1 passage is wrong half the time. The generator anchors on it. Faithfulness stays high (the model is faithfully grounding on wrong text); only an end-to-end eval catches it.
- Generic embeddings on a jargon-heavy corpus — recall is fine on plain queries and collapses on abbreviation-heavy ones. Fix: a domain-tuned embedding or hybrid retrieval with BM25 doing the abbreviation work.
- Generator prompt without a refusal clause — the model hallucinates around weak retrieval. The answer reads fluent and confident and is wrong, often citing passages that don’t exist.
A founder who scoped the four ingredients and held the vendor to threshold-level evals catches each of these in week 2 instead of month 4.
Frequently Asked Questions
What does RAG stand for in AI?
RAG stands for retrieval-augmented generation. A language model retrieves passages from your indexed corpus at query time and writes an answer grounded in those passages. The term was introduced in the 2020 Lewis et al. paper (arXiv:2005.11401).
Is RAG the same as a chatbot with my documents uploaded?
No. Uploading puts documents in the context window for one conversation. RAG indexes documents into a persistent vector store and retrieves on every query. Upload-style flows work for tens of pages; RAG scales to hundreds of thousands.
Do I need a dedicated vector database for RAG?
You need a vector index, not necessarily a dedicated database. pgvector inside an existing Postgres works for small to mid-sized corpora. Dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus) become the right choice when corpus size, query rate, or hybrid features outgrow Postgres.
How is RAG different from fine-tuning?
Fine-tuning changes the model’s weights so it has learned your data; RAG leaves the weights alone and looks the data up at query time. Fine-tuning is the right axis when behaviour needs to change (tone, format, refusal pattern); RAG is the right axis when knowledge needs to change. The fine-tuning vs prompting vs RAG explainer walks the comparison in detail.
How long does it take to build a basic RAG system?
A working prototype on a small corpus is a few engineer-days using LangChain or LlamaIndex against a default vector store and embedding. A production-ready system with hybrid retrieval, a reranker, a tested chunker, a refusal-clause generator, and a graded eval suite is closer to a 6 to 10 week engagement.
Will long context windows make RAG obsolete?
No. Retrieval accuracy degrades inside long context well before the window fills, and putting 400,000 tokens into every query is dramatically more expensive than retrieving the right 4,000. Long context complements RAG, not replaces it.
Who owns the eval suite — me or the vendor?
You own the rubric and the test inputs; the vendor formats them for the harness and runs them in CI. The rubric is a domain artifact — what correct means in your domain is your knowledge. The eval-first playbook makes this an explicit clause in the handoff contract.
How do I know if my RAG system is hallucinating?
Grade faithfulness on every release. An LLM-as-judge compares each factual claim against the retrieved passages and labels it grounded, partially grounded, or ungrounded. Below 0.9, the system is fluently hallucinating — fix is usually upstream (wrong passages retrieved, or missing refusal clause in the generator prompt).
What is “agentic RAG”?
Agentic RAG is the variant where the model decides whether to retrieve, what to retrieve, and whether to retrieve again. It raises ceiling quality on hard queries and expands failure surface. For a first AI MVP, ship the fixed-step architecture and add agentic decisions only when the eval suite shows fixed retrieval is the binding constraint.
Closing
RAG is the right architecture when your product needs to answer from a corpus you own, that changes faster than the model can be retrained, and is large enough that retrieval beats stuffing. It is the wrong architecture when there is no corpus, the corpus is one document, or it is a static FAQ. The four ingredients — chunker, embeddings plus index, retriever, generator — are what a vendor proposal should specify line by line. The four-metric eval contract (recall@k, MRR, faithfulness, citation accuracy) is what a founder uses to grade the system on every release.
A founder who can read a “we’ll build RAG for you” proposal against those eight items keeps the upper hand. The eval-first stance is what turns RAG from an architectural buzzword into a graded asset.
Dirk Jan van Veen, PhD
