Three evals, frozen at the contract, beat thirty drafted in a spreadsheet that no one runs. Founders who try to ship MVP-1 with a comprehensive suite — happy path, worst case, fairness, latency, cost, refusal, prompt-injection, persona drift, jailbreak, regression, citation, format, tone — ship none. The suite never reaches a green run, the build slips, and the team substitutes a hand-curated demo set for the contract. Scope discipline on the eval set is the same as scope discipline on the feature set: pick the minimum that proves the product works, freeze it, ship it, let production telemetry tell you what to add next.
This grows out of the eval-first build playbook, part of the idea-to-product manifesto. It pairs with the eval rubric template every non-technical founder should ask for — that piece is the how of one eval; this piece is the how many. For the commercial frame, see stop budgeting AI projects in story points, budget them in eval runs.
Table of Contents
The 30-Eval Trap
Open Promptfoo, Inspect, or Braintrust and the documentation names fifteen eval categories: correctness, hallucination, faithfulness, refusal, toxicity, prompt-injection resistance, latency, cost, format adherence, persona, fairness, citation accuracy, regression, jailbreak resistance, drift detection. A diligent founder concludes the responsible move is to write one of each before launch. A diligent vendor, asked to ship “production-ready” evals, reaches the same conclusion. The team draws up a thirty-row spreadsheet and starts building.
Six weeks in, none are green. Each eval needs its own gold set, its own rubric, its own scoring function, and its own threshold negotiation. Half need a judge-as-LLM with its own meta-rubric. The team is doing eval engineering instead of building the product. Launch slips. Someone hand-curates eight inputs that all pass and calls it the suite. The contract has been quietly downgraded to the same un-evaluated demo set every AI MVP has always shipped on.
The 30-eval trap is a scope problem, not a tooling problem. The eval set is product scope, and the rule is the same as for any scope: pick the minimum that proves the thing works, freeze it, ship it, and learn from production. The maintenance cost of an eval is not zero — each consumes engineer attention every time the prompt changes, every time the model is swapped, and every time the team debates whether a near-miss is a pass.
McKinsey’s State of AI surveys have for several years reported that roughly four in five AI pilots fail to scale to production — not because the model fails, but because the operational discipline around the model is missing (McKinsey, The state of AI). Most teams diagnose this as a too-few-evals problem and respond by drafting more. The diagnosis is wrong. Teams that ship at quality ship with a small frozen eval set on day one and a clear trigger for what gets added later.
The 3 Evals That Matter
The case for three is not arbitrary. Each eval below covers a structurally distinct axis of failure. None is redundant. Removing any one leaves an axis uncovered. Adding a fourth on day one delays the only mechanism that tells you which fourth you actually need — production telemetry.
Eval 1. The Happy-Path Eval
What it covers: the modal input — the most common, most expected user request the feature is sold to handle.
What it does not cover: edge cases, adversarial inputs, low-frequency inputs, persona variation. One question only: when the feature is used the way the pitch deck says it will be used, does it return the right answer above threshold?
A happy-path eval is what separates an MVP from a demo. A demo is one input the founder picked because it works. A happy-path eval is twenty to fifty inputs sampled from the distribution the founder expects, scored against an explicit rubric, with a threshold written into the contract. The shift from one to twenty is the load-bearing move. One input is anecdote. Twenty from the expected distribution is evidence.
How a non-engineer founder reviews it without reading code: ask to see the gold set. Read every input. For each, ask “is this what a real user would type?” If three out of twenty are stylized or unrealistic, the eval is testing the demo, not the product. Threshold: 100% is wrong (too strict, the team will game the rubric); 60% is wrong (below the bar a buyer expects); 85–95% on a well-designed rubric is the typical range.
Eval 2. The Worst-Case Eval
What it covers: the failure mode with the largest blast radius. For most B2B AI features that is hallucination or unauthorized action — the system fabricates a fact a user will act on, or executes a step it should have refused. For a consumer feature, the worst case may be toxic output or unsafe advice. For a finance feature, leakage of confidential data into a model log. Pick the single worst case for this product, name it, and write the eval.
What it does not cover: every failure mode. It is not a comprehensive guardrail suite. It is the one guardrail whose failure ends the product.
The worst-case eval is asymmetric by design. The happy-path eval optimizes for the average case; the worst-case eval optimizes for the tail. Threshold logic is also asymmetric: where the happy-path eval allows a small failure rate, the worst-case eval typically targets zero failures on a curated set of adversarial inputs.
How a non-engineer founder reviews it: ask the team to articulate, in one sentence, the single worst thing the feature can do. If they cannot answer, the eval is not yet specified. Common answers: “fabricate a citation that does not exist,” “approve a contract clause containing an indemnity it should have flagged,” “advise a refund the user is not entitled to.” Pick the one with highest blast radius and build the eval against it.
Eval 3. The Regression Eval
What it covers: stability under change. When the team swaps a model — Anthropic ships Claude Opus 4.8, OpenAI ships a GPT-5 mini revision, Google ships Gemini 2.51 Flash — does the system still pass the inputs it passed yesterday? Regression evals are the only mechanism that detects silent quality drift on model migrations, and frontier release cadence in 2026 makes this drift structural (Stanford HAI AI Index 2025; Artificial Analysis Leaderboard).
What it does not cover: whether yesterday’s behavior was correct. A regression eval locks in the current pass set. If the pass set is wrong, the regression eval will help the team ship the wrong thing repeatably. The happy-path eval tests whether the pass set is right; the regression eval tests whether tomorrow’s deployment still passes it.
Construction is the cheapest of the three: a snapshot of pass-cases from the happy-path eval and the worst-case set, frozen. The threshold is “no regressions” — the set that passed yesterday must pass today. A tripwire, not a quality bar.
How a non-engineer founder reviews it: ask the team to run it after every model swap, every prompt change, and every retrieval-system change. Green = “this change did not break what worked.” Red = “fix it before merge or accept the trade-off explicitly.” Anthropic’s own guidance on building effective agents places regression discipline at the center of the production posture; asking for regression-eval visibility is asking for what the frontier labs already practice.
What You Do Not Eval in MVP-1
Three evals is the discipline of deliberate exclusion. Every category below is real, important, and ultimately worth covering — but not on day one.
- Latency. An ops concern, not a correctness concern. The easy fix (smaller model, more caching) is one engineer-day away. Measure in production; do not eval on a gold set in MVP-1.
- Cost-per-call. Same reasoning. Track it. Do not eval it.
- Persona fairness. Real, but not until the feature has enough users to surface the signal. Premature persona evals on synthetic personas are theater.
- Long-tail edge cases. By definition you have not seen them yet. Wait until production surfaces them.
- Prompt-injection red team. A specialist discipline. If the feature touches sensitive data or executes tools, hire someone to do it as a separate engagement — do not bury it in the MVP-1 suite.
- Citation accuracy on retrieval. Real on a RAG product, but a sub-rubric of the happy-path eval there, not an independent suite.
- Tone / brand voice. Subjective. The judge-as-LLM rubric required is itself a research project. Defer.
- Format adherence. Often deterministic enough to handle with output validators in code.
The shared logic: any eval that (a) requires production data you do not yet have, (b) requires specialist research effort, or (c) can be substituted with a non-eval mitigation is a poor candidate for MVP-1. The MVP-1 eval set establishes the product works. Later evals maintain it under conditions you have not yet seen.
When You Add Evals 4–10
Evals 4 through 10 get added on signal, not on schedule. The signal is always the same: a recurring production failure mode the triad did not catch. The trigger is a specific log pattern, a support ticket cluster, or a telemetry spike. The eval is built against that signal.
The discipline matters because the alternative — adding evals on schedule or vendor recommendation — produces bloat without coverage gains. A team adding a fairness eval before the product has enough users to detect fairness signal is engineering against a hypothesis, not a signal. A team adding a latency eval because a vendor doc recommended it is paying maintenance cost for a tripwire that may never fire.
In practice: a ticket cluster surfaces five customers complaining the AI got their account size wrong. The team traces the failure to a prompt that mishandles non-standard account names. Eval #4 is written against that failure — ten inputs with non-standard account names and the correct extraction in the rubric. It joins the suite. The regression eval is updated. The product is one eval stronger and the maintenance cost is justified because the eval was paid for by a real failure, not a hypothetical one.
Hamel Husain’s writing on practical eval discipline calls this the “look at your data” loop. Read the logs. Find the recurring failure. Write the eval. Repeat. That loop is the only mechanism that scales the suite without bloating it. By the time the suite reaches ten evals, every one is paid for by a failure the system saw in the wild.
A Worked Example: A Contract-Review MVP
Consider a hypothetical contract-review MVP — a feature that ingests a vendor contract and surfaces three things: missing indemnity clauses, non-standard payment terms, and auto-renewal language. The team is told to ship in eight weeks. What is in MVP-1?
Eval 1 — happy path. Thirty real vendor contracts sampled from the founder’s industry (SaaS sales contracts, MSAs, SOWs from the past two years), hand-annotated with the three target clauses. Rubric: did the system correctly flag presence/absence of the indemnity clause, surface payment terms, and identify auto-renewal language? Threshold: 90% pass rate averaged across the three sub-checks.
Eval 2 — worst case. The largest blast-radius failure is approving a contract as “clean” when it contains an indemnity hostile to the buyer. Twenty contracts each containing a known-hostile indemnity, drafted by an attorney on the founder’s network. Rubric: did the system flag the hostile clause? Threshold: zero misses. A miss blocks the release.
Eval 3 — regression. The combined pass set from Eval 1 (27 contracts) plus the 20 hostile-indemnity inputs from Eval 2, frozen. Re-run on every model swap, prompt change, and retrieval update. Threshold: no regressions.
Deliberately deferred: latency (tracked in production); cost per contract (tracked in production); fairness across drafting jurisdictions (deferred until production volume surfaces signal); prompt-injection on contracts containing malicious instructions (deferred to a separate security engagement); tone of the auto-generated summary (deferred until customer feedback surfaces a specific tone failure).
Total inputs: roughly 80. Total engineer-time to build the suite: one engineer-week. The team ships in eight weeks. The eval suite is the contract; the deferred list is also part of the contract — telling the vendor what they will not be graded on, and telling the founder what to watch for as evals 4–10 candidates.
Contrast: the same team given a thirty-eval mandate would do three engineer-months of eval engineering against an eight-week timeline. Shortcuts substitute — synthetic gold sets, unvalidated LLM-judge rubrics, and a week-six agreement to “ship with whatever passes.” The product launches with thirty evals in the repo, six green, and no one knowing which six matter.
What This Means for the Vendor SOW
The three-eval discipline maps directly to the engagement-model conversation a non-engineer founder is having with a vendor. A defensible AI MVP SOW includes:
| SOW section | What it specifies |
|---|---|
| Eval set, MVP-1 | The three named evals, gold-set source, rubric, thresholds |
| Freeze date | When the rubric and threshold are frozen for the build |
| Deferred eval list | The named evals that will not be built in MVP-1, with rationale |
| Trigger conditions for evals 4+ | Production-telemetry signals that trigger a new eval, with the engagement model for that build |
| Regression cadence | When the regression eval runs (every model swap, every prompt change, every release) |
| Quality gate | The pass condition on the three evals that gates production release |
A vendor that resists naming the three evals up front is selling demos. A vendor that lists thirty in the SOW is selling reassurance. A vendor that names three, ties each to a structural failure axis, and writes the trigger conditions for evals 4+ into the contract is selling a production posture. That is the vendor you want.
The reader of this article is the founder writing or reviewing that SOW. Three evals in the contract is the line you draw. Download the AI MVP Scoping Worksheet for the one-page version — three evals, deferred list, triggers, gate. Hand it to your vendor on day one. Grade them against it in production.
Frequently Asked Questions
Is three evals really enough to ship an AI MVP?
Three is enough to ship MVP-1 — the version that goes to your first ten production users. It is not enough to run the product at scale forever. The point of MVP-1 is to discover, from production telemetry, which evals 4 through 10 you actually need. Three is the deliberate floor that forces the team to ship and learn rather than draft and stall.
Why exactly three and not two or four?
Three covers three structurally distinct axes of failure: modal correctness (happy path), worst-case blast radius (worst case), and stability under change (regression). Two evals leave one axis uncovered. Four on day one buys no new structural coverage and delays the trigger-based loop that produces evals 4 through 10. Three is not a magic number; it is the floor for structural coverage.
What if my vendor insists on writing 30 evals?
Ask which structural failure axis each one covers. If the answer is “comprehensive coverage,” redirect them to the three-axis frame. If they cannot name the axis, the eval is not yet justified. Most “30-eval suites” collapse to three to five distinct axes when audited this way.
How big should each eval’s gold set be?
Happy-path: 20–50 inputs sampled from the production-expected distribution. Worst-case: 10–20 adversarial inputs targeted at the named failure mode. Regression: the union of pass cases from the other two, frozen at release. Roughly 50–100 total inputs is the right scale for MVP-1.
Who builds the gold sets?
A senior engineer with domain access — ideally the engineer building the feature, with sign-off from a domain expert (often the founder). Outsourcing gold-set construction to a junior labeller produces gold sets that test the labeller’s reading of the feature, not the feature itself. Founder review of every gold-set input is non-negotiable.
Where does prompt-injection or jailbreak testing fit?
Outside the three-eval MVP-1 suite. If the product handles sensitive data or executes tools, prompt-injection testing is a specialist red-team engagement run separately, often by a security firm. Burying it inside the MVP-1 suite produces a shallow tripwire that gives the team false confidence; running it as a separate engagement produces a real audit.
How often should the regression eval run?
On every model swap (Claude Opus 4.8 to Opus 4.7, GPT-5 to GPT-5, Gemini 2.51 to 3.2), every prompt change above a one-line tweak, every retrieval-system change, and every release candidate. The regression eval is cheap to run — a few hundred inferences — and the alternative is silent drift between releases.
What is the cost of running the three-eval suite?
A few hundred to a few thousand inferences per release. On current frontier-model pricing (Anthropic, OpenAI, Google equivalents), that is single-digit dollars per release. Engineering cost is roughly one engineer-week to build, then maintenance of an hour or two per release.
When do evals 4 through 10 get added?
On signal: a production-telemetry pattern (a recurring failure mode, a support ticket cluster, a metric spike) that the existing three did not catch. The eval is written against the signal. Schedule-driven, listicle-driven, or vendor-recommended additions are the failure mode this discipline prevents.
Closing
Three evals at the contract, frozen and shipped, beat thirty drafted in a spreadsheet no one runs. The constraint is the forcing function: it makes the team pick the load-bearing axes of failure, defer everything that does not earn its maintenance cost, and ship on a timeline that gets the product in front of real users. Real users surface real failure modes. Real failure modes earn their evals. The suite grows from there.
A founder writing or reviewing an AI MVP SOW has one sentence that compresses this article: “Show me the three evals, the deferred list, the triggers for evals 4–10, and the regression cadence.” A vendor who can answer in one page is ready to build. A vendor who cannot is selling a demo.
Pair this with the eval-first build playbook, the eval rubric template, what are AI evals in plain English, and stop budgeting AI projects in story points, budget them in eval runs. Then download the AI MVP Scoping Worksheet — three evals, deferred list, triggers, gate. One page. Hand it to your vendor.
Dirk Jan van Veen, PhD
