The three-letter problem shows up in week two of every AI MVP. A founder sends the PRD to two agencies, and the proposals come back recommending three different things — one says fine-tune, one says RAG, one says prompt-only on a frontier model. The founder cannot tell which is right, because the agencies answer the engineer’s question (what is technically possible?) while the founder is asking the business question (which do I pay for first, and which ships at quality on time?). This guide is the founder’s version of the answer.
This guide extends the Eval-First Build Playbook and the broader Idea-to-Product Manifesto. Where the playbook teaches you to scope AI features in evals, this one teaches you to pick the technique that passes those evals at the lowest cost. The deliverable is a rule you can apply in a 15-minute meeting without an engineer in the room.
The three techniques, defined plain
Every AI MVP customization conversation collapses to one of three techniques. Define them once and the rest is mechanical.
Prompting (instructions only). You customize behavior using only the words you send the model — a system prompt, a few examples, any reference material that fits in the context window. Nothing about the model changes. Prompting is the default. Every AI feature starts here. Most stay here.
RAG (retrieve, then prompt). Short for retrieval-augmented generation. Reference material lives in a search index (typically a vector database, often hybrid with keyword search). When a user asks a question, the system retrieves the most relevant chunks, composes a prompt that includes them, and sends it to the model. The weights do not change — you are still prompting, just dynamically.
Fine-tuning (train the model weights). You take an existing model and continue its training on your labeled data, so the weights shift toward your domain. You ship a new checkpoint that you host yourself or pay a provider to serve. Querying costs about the same per token as the base, but the operational burden — labeled-data pipeline, training pipeline, model versioning, re-tuning on every base update — runs for the life of the product.
The four-axis comparison matrix
Choose across the four axes that matter to a founder: cost, time-to-ship, flexibility, and depth.
| Axis | Prompting | RAG | Fine-tuning |
|---|---|---|---|
| Cost (build) | Low — engineer-days of prompt + eval work | Medium — 2–6 weeks for pipeline, store, eval rig | High — labeled dataset, training infra, hosting |
| Cost (run) | Per-token inference; reducible with caching | Inference plus storage and retrieval calls | Inference (similar to base) plus dedicated hosting |
| Time-to-ship | Days to ~2 weeks | 4–8 weeks for first eval-passing version | 8–16 weeks once data prep is counted |
| Flexibility | Highest — change prompt, redeploy in minutes | High for retrieval; medium for behavior | Lowest — behavior changes require retraining |
| Depth of customization | Shallow — bounded by prompt and base capability | Medium — adds private knowledge; reasoning is still the base | Deepest — can shift rare formats, jargon, refusal patterns |
| Maintenance | Low — occasional prompt tuning | Medium — re-index, watch retrieval drift | High — re-train on base updates, version everything |
| Eval cost | One suite (generation) | Two (retrieval and generation) | Two (held-out and base-comparison) |
Prompting trades depth for speed. Fine-tuning trades speed for depth. RAG sits in the middle but adds a new failure surface (retrieval) that has to be evaluated separately. The right choice is the cheapest, fastest technique that clears your eval bar.
Three decision rules
Run these in order. The first rule that says yes is the answer. The discipline of running them in order — resisting the urge to skip to RAG or fine-tuning — keeps the MVP cheap and on time.
Rule 1: when prompting suffices
Use prompting alone when all of the following hold:
- Reference material fits in a frontier model’s context window. As of mid-2026 that runs from 200,000 tokens (Claude Sonnet 4.6 standard) to several million (Gemini 2.5 Pro, Claude long-context).
- The material does not change inside a single user session.
- You will update the prompt manually (or via config) when knowledge changes — a few times a month, not several a day.
- A strong base model already does the task acceptably on a few smoke-test inputs.
- Prompt caching makes per-query cost acceptable on the static portion.
If those five hold, prompting is the right starting technique. The instinct to graduate immediately is almost always wrong here.
Rule 2: when RAG is right
Graduate to RAG when any of these becomes true:
- The corpus is too large for a single prompt — for the frontier model with the longest context, not for the model you picked first.
- The material is dynamic — updates many times a day, comes from a database other systems write to, or arrives as a stream.
- The user requires citations to specific source documents, retrieved programmatically rather than generated.
- A prompt-only baseline has been evaluated and has a specific failure mode (hallucinated long-tail facts, or wrong answers on prompts too long for the model to attend to evenly).
RAG is the second technique, not the first. Building it before a prompt-only baseline is a common, expensive mistake — you cannot tell whether RAG improved the system because you never measured the system without it.
Rule 3: when fine-tuning earns its weight
Fine-tuning is right when all of the following hold:
- Both a prompt-only baseline and a RAG baseline have been evaluated, and both have a specific failure mode fine-tuning is known to fix — rare output formats, niche jargon, refusal patterns, or latency and per-token cost only a smaller fine-tuned model can deliver.
- You have a labeled dataset of at least several hundred input-output pairs (several thousand is better), plus resources to keep generating examples.
- Usage and revenue justify the operational burden.
- You can keep up with the frontier-model release cadence — every new base forces a re-fine-tune decision that needs an engineering owner.
Most AI MVPs do not meet those four conditions and should not fine-tune before product-market fit. The exceptions are narrow: high-volume classification where a small fine-tuned model is dramatically cheaper, structured output in an unusual format prompting fails on, or refusal shaping in a domain (legal, medical) where the base is overly cautious.
The 2026 reality
The decision math has shifted. Articles from 2023 and 2024 over-recommend RAG and badly over-recommend fine-tuning, because the prompt-only frontier has expanded so much that techniques that were right two years ago are wrong now. Three changes drove it:
- Context windows grew by 50–100×. A frontier model with a multi-million-token context can ingest the entire policy library, product spec, and a sample of past interactions in one prompt. Tasks that required RAG in 2023 now ship prompt-only.
- Prompt caching changed the economics. The static prefix of a large prompt is cached provider-side and reused at a fraction of per-token cost. “Paste the whole manual into the prompt” is now viable at production volume.
- Base-model capability rose faster than fine-tuning gains. Prompt-only frontier models (Claude Opus 4.8, Sonnet 4.6, GPT-5, Gemini 2.5 Pro) outperform 2023-era fine-tuned models on the Artificial Analysis leaderboard. The gap between base + good prompt and fine-tuned + good prompt is narrow enough that the operational burden is hard to justify pre-PMF.
The 2026 default is prompt-only with a frontier model, scoped against a real eval suite. RAG and fine-tuning are upgrades the eval suite tells you to take — not the proposal.
Three worked founder scenarios
Scenario A: a help-desk email classifier
A SaaS founder wants to triage support email into five categories. Volume is several thousand emails a month. Definitions live in a 4-page internal doc. About 400 hand-labeled examples are available.
Technique: prompting. The definitions fit in the prompt with room to spare. Twenty few-shot examples cover the long tail. A frontier base scores well above threshold on a 200-input eval set on first pass. RAG adds infrastructure for no measurable lift because definitions are static. Fine-tuning is overkill — volume is too low to justify hosting, and per-token cost of a frontier base is already inside budget.
Scenario B: a contract-clause extractor
A legal-tech founder wants to extract a fixed set of clauses (term, renewal, jurisdiction, indemnification cap, change-of-control) from inbound vendor contracts. Contracts are 4–60-page PDFs. Output is structured JSON. 80 hand-extracted contracts with golden-label JSON are available.
Technique: prompting, with light retrieval for very long contracts only. Each contract is parsed to text and pasted into the prompt with a system prompt that includes the JSON schema and three few-shot examples. Under 40 pages, the whole document goes in; for longer contracts a section-router (regex on headings) extracts candidate sections first. Fine-tuning is tempting because the format is unusual, but a frontier model with a strict JSON schema and three few-shot examples hits the eval threshold without it.
Scenario C: a finance-research assistant
A fintech founder wants an assistant that answers analyst questions over 30,000 research notes, 50,000 SEC filings, and an inbound news stream. Citations to specific notes and filings are a hard requirement. The corpus grows by hundreds of items a day.
Technique: RAG. Textbook case. The corpus is too large for even the longest current context window, it is dynamic, and the citation requirement makes prompt-generated citations a known hallucination risk. The team builds an indexing pipeline, a hybrid retrieval layer (vector plus BM25), a generation prompt that includes the top-k chunks, and a two-stage eval suite (retrieval and generation). Fine-tuning is revisited at month 6, once the eval suite identifies failure modes RAG cannot fix.
Tying the decision back to evals
The single most important framing is this: the right technique is the cheapest, fastest technique that passes your eval suite. Not the most impressive. Not the one the vendor enjoys building. The one that clears the bar.
That framing only works if the eval suite exists before the technique choice. The Eval-First Build Playbook teaches the founder how to build that suite. For the broader build-vs-buy framing, see Decoding the AI Platform vs Tool Dichotomy for Build vs Buy.
The right sequence: write the PRD that an AI dev team can actually estimate, run eval-first scoping, pick the technique using the rules above, build the prompt-only baseline first (even if you suspect RAG), and re-evaluate after each upgrade against the same suite. Founders who skip the baseline pay agencies for systems that do not measurably improve outcomes — because no one ever measured the outcome before the upgrade. The companion guides RAG explained for founders and What are AI evals in plain English unpack the next two questions.
Frequently Asked Questions
Is RAG always better than prompting if the data is large?
No. RAG is better when the data is both too large for the prompt and dynamic or requires citation. If it is large but static and citations are not a hard requirement, large-context prompting with caching is often cheaper and faster. The decision rule is the eval result on a held-out test set, not corpus size.
Does fine-tuning make a model “know” my data better than RAG?
No — not the way founders sometimes assume. Fine-tuning adjusts weights toward your data distribution but does not implant facts. A fine-tuned model that has seen a document during training will not reliably quote it at inference. Fine-tuning teaches patterns; RAG grounds facts. Use fine-tuning for behavior shaping; use RAG for fact grounding.
Can I use prompt caching at production scale?
Yes — one of the major shifts of 2026. Frontier providers cache the static prefix of a prompt and reuse it across queries at a fraction of the per-token cost. Prompt-only solutions now cover ground that required RAG two years ago.
How much does fine-tuning a frontier model cost in 2026?
The training run is a few hundred to a few thousand dollars for a scoping-time dataset. The real cost is engineering time — labeled-dataset assembly, training pipeline, checkpoint integration, and re-tuning on every base update. Realistic all-in: a multi-week senior-engineer engagement on top of compute. Pre-PMF, that money usually buys more by improving the prompt-only baseline.
When does prompt engineering become “real engineering”?
Production prompt engineering includes templating, version control, eval against a held-out test set, refusal-pattern design, output-schema enforcement, and caching strategy. None of that is RAG or fine-tuning, but all of it is engineering. A prompt-only AI feature has a small infrastructure footprint and a non-trivial engineering footprint.
Can I mix techniques?
Yes, and most mature AI products do. A typical 2026 feature uses prompt engineering as the foundation, RAG for fact grounding, and a small fine-tuned model for a specific subroutine. The mix is built one layer at a time, each evaluated against the suite before the next is added. The mistake is building all three from day one because the proposal said so.
How does this affect my agency proposal review?
Ask three questions. (1) Did they build a prompt-only baseline? (2) What does the eval suite say about whether each upgrade beats the baseline? (3) What is the operational ownership plan after launch? If the proposal jumps to RAG or fine-tuning with no baseline, yellow flag. If it cannot answer (3), red flag.
What if the vendor insists fine-tuning is necessary?
Ask for the specific failure mode in the prompt-only baseline that fine-tuning fixes, with eval evidence. A vendor who names the failure, points to the eval results, and explains why neither a better prompt nor RAG closes the gap is making a defensible recommendation. A vendor who cannot is selling a longer project.
Does the choice of technique lock me into a vendor?
Prompting is most portable — the same prompt with minor adjustments works across providers. RAG infrastructure is portable in pieces (vector stores are interchangeable; orchestration code less so). Fine-tuned models are least portable — a checkpoint cannot be moved across providers, and re-fine-tuning on a new base is a fresh project. The lock-in argument favors prompting wherever the eval suite tolerates it.
Closing
The 2026 founder who scopes well does three things in order: writes an eval-first PRD, picks the technique using the rules above, and ships the prompt-only baseline before any upgrade. Applying the rules in order — and resisting the proposal that opens with RAG or fine-tuning — keeps the MVP on time and on budget.
Dirk Jan van Veen, PhD
