An eval is three things stapled together: a fixed set of representative inputs, a written rubric for what right looks like, and a pass-rate threshold the system has to clear. That is the whole concept. When an engineer says “we ran the evals” or “the pass rate dropped after the model update”, they mean that artifact. By the end of this piece you will read an eval doc, ask the right questions in a vendor meeting, and see why the eval set is the spec for any AI feature you are paying to build.
It is the concept that sits behind the eval-first build playbook and the broader idea-to-product manifesto. If you have read stop scoping AI projects in features, this is the explainer behind that argument.
Table of Contents
- The one-sentence definition
- Evals vs QA tests: the structural shift
- The anatomy of a single eval
- The four eval types and when each fits
- A worked example: a 5-input customer-support classifier
- Why the eval set is the spec
- Frequently Asked Questions
- Closing
The one-sentence definition
An eval is a fixed set of representative inputs, a written rubric for what right looks like on each input, and a pass-rate threshold the system has to clear.
Three nouns: inputs, rubric, threshold. Missing any of the three and it is not an eval — it is an opinion, a borrowed benchmark, or a screenshot of a happy demo.
Inputs are sampled from production traffic, not made up. The rubric is a written rule another person could apply and reach the same grade — not “did it feel okay”. The threshold is a numeric bar (“at least 85% of the 50 cases pass”) the team commits to in advance.
That paragraph is the whole concept. The rest is detail.
Evals vs QA tests: the structural shift
The closest mental model most founders have is QA testing — one test case per feature, an expected output, a binary pass or fail. That model is sound. An eval is the same shape with two upgrades that make it work on probabilistic outputs.
| Concept | Traditional QA test | AI eval |
|---|---|---|
| Input | One specific input per test | A set of representative inputs, often 20 to 500 |
| Expected output | One exact value or pattern | A rubric describing the properties a correct output must have |
| Grading | Exact match → pass / fail | Rubric per case, aggregated to a pass rate |
| Result | Green or red | A score against a threshold |
| Re-run trigger | Code change | Code change, prompt change, or model update |
First upgrade: inputs become input sets. A QA test for a save button uses one input because the function is deterministic. An AI feature is not — the same input can return different outputs, and an overnight vendor update can shift behavior. You grade with a sample, not one case.
Second upgrade: expected outputs become rubrics. A QA assertion is binary and exact. For an AI reply-drafter, “the reply is correct” has no single right answer. The rubric replaces exact match with a written rule (“addresses the stated question, neutral tone, no refund promise without authorization”).
If you already know QA, you already know 70% of what an eval is. The remaining 30% is the rubric and the threshold.
The anatomy of a single eval
Every eval row has the same four parts. Name them and you can read any eval doc that lands in your inbox.
-
Input. A concrete example of what the system sees in production — one realistic email for a classifier, one transcript for a summarizer. From production traffic, not synthetic data the vendor cooked up in a slide.
-
Expected behavior. What a correct output looks like for this input. For a classifier, the right label (“billing question”). For a generative task, a property list (“references the order number, offers one of three approved next steps, no refund commitment”).
-
Grader. The mechanism that compares actual to expected. Code (string-match), a human (domain expert), or another LLM (LLM-as-judge). The choice determines how repeatable, expensive, and trustworthy the eval is.
-
Threshold. The pass-rate bar the whole set must clear, not the row. A 50-row set at 85% threshold needs 43 passes. Set in advance based on the cost of errors — medical extraction might need 98%, tone rewrite 75%.
That is the entire surface area. An eval suite is many of these rows, grouped by capability, run in batch, reported as one number per capability. A vendor “eval plan” with all four parts is real; any missing or fuzzy and you have a vendor problem before you have a model problem.
The four eval types and when each fits
The four-part anatomy is constant; the grader slot has four shapes. Picking the wrong one is the most common reason eval suites quietly miss regressions.
Binary correctness. Grader is code. Right or wrong, no middle. Fits classification, extraction, and tool-calling decisions. Fast, free, perfectly repeatable. Reports an accuracy number (“47 of 50 correct”). Do not use for free-text generation — exact-match reports 0% even when the reply is excellent.
Scored rubric. Grader applies a multi-criterion rubric and produces a per-criterion score. Applied by a human, an LLM-as-judge, or both. Fits partial-credit tasks — right entity but missed a date, accurate but too long, right tone but wrong question. Watch for fuzz: a rubric two graders cannot apply consistently is worse than no rubric.
Human-judged. Grader is a domain expert reading by hand. Expensive and not scalable, but the most trustworthy grader for tasks requiring domain knowledge a model does not yet have. Fits medical, legal, regulated, and expert-judgment tasks early in a build. Also the calibration layer for an LLM-as-judge once the rubric has stabilized.
LLM-as-judge. Grader is another LLM, prompted with the rubric and the input and output. Cheap and scalable. Frontier models — Claude Opus 4.8 and Sonnet 4.6, GPT-5, Gemini 2.5 — apply a well-written rubric reasonably consistently (Artificial Analysis LLM Leaderboard). Gotchas: judges prefer longer answers, the first option in a pairwise, and over-weight surface fluency. They need a human-graded calibration sample. Do not use as the only grader for a new rubric.
Decision tree: one machine-checkable right answer? Binary correctness. Multi-criterion partial credit? Scored rubric. High-stakes or expert-only? Human-judged. Stable rubric, high volume? LLM-as-judge calibrated against humans.
A worked example: a 5-input customer-support classifier
Imagine an inbound email classifier that routes each message to one of five buckets: billing, shipping, return, product-question, complaint. The vendor’s first prototype is ready. You do not need 500 rows on day one — you need five inputs that cover the realistic shapes of the inbox.
Input set — five realistic emails:
| Row | Email (truncated) | True label |
|---|---|---|
| 1 | ”Hi, I was charged twice for order 4421, can you refund the extra?“ | billing |
| 2 | ”Where is my package? It’s been 9 days since the shipping email.” | shipping |
| 3 | ”The sweater I bought is too small. How do I send it back?“ | return |
| 4 | ”Does the blue jacket come in size XL?“ | product-question |
| 5 | ”Honestly this is the third time this has happened, I want to talk to a manager.” | complaint |
Rubric — three lines:
- The model returns exactly one of the five allowed labels.
- The label matches the true label above.
- If ambiguous, the model picks the label the primary request maps to.
Grader — binary correctness; each row has one right label. A human grades in two minutes; a string-match script in two seconds.
Threshold — 4 of 5 (80%) for a pilot smoke test. For production on a 50-row set, plan on 90%+ before routing customer email unsupervised.
The first run — say the prototype returns billing ✓, shipping ✓, return ✓, product-question ✓, return ✗ on row 5 (it read the complaint as a return because the message mentioned an item).
Score: 4 of 5 = 80%. Threshold cleared, barely. You now know two things: the smoke test passed, and complaint is the failure mode. The next iteration samples more complaint-shaped emails to see if row 5 is a fluke or a pattern. A vendor who runs that drill and reports “80%, failure on the complaint row, here are 20 more for iteration two” is doing the work. A vendor who says “looks great, ready to ship” without the drill is the exact failure mode the eval catches.
Same shape at every scale. Five rows on a Tuesday or five hundred across CI.
Why the eval set is the spec
The load-bearing claim. The eval set is the actual spec for an AI feature, because no other artifact survives a model migration.
A spec for a deterministic feature is a written description. “When the user clicks Save, the row persists” is precise enough to implement and test against. AI features do not work that way. The model changes on cycles of weeks. Anthropic shipped Claude Opus 4.8 and Sonnet 4.6 across the past year. OpenAI shipped GPT-5 and successor updates. Google shipped the Gemini 2.5 family. Open weights advanced through Llama 3.3 and the line. Each transition silently changes the behavior of any product pinned to a model alias.
A spec that says “the agent triages email correctly” is useless because correctly is a moving target. The eval set pins it down across a swap. Move from Opus 4.8 to Sonnet 4.6, re-run the set, get a new pass rate. Either it cleared the threshold (migration is safe) or it did not (you know which rows broke).
That property is also why the eval set is the right artifact for a commercial relationship. A founder hiring an agency without an eval contract has no way to grade the work, detect scope drift, or prove the product is broken when the vendor says it is fine. A vendor who refuses — or insists on writing one only after the build is “done” — is telegraphing how the engagement will go. The eval-first build playbook is the operational sequence; stop scoping AI projects in features is the commercial argument.
Same logic if you build it yourself with an AI coding assistant. The eval set is the only artifact that tells you whether today’s prompt is better than yesterday’s.
Adjacent concepts: RAG explained for founders (retrieval systems need a specific rubric shape — grounding) and what is an LLM agent, actually (agentic systems need trajectory grading).
Frequently Asked Questions
Do I need to know how to code to write evals?
No. The eval set, rubric, and threshold are domain-knowledge artifacts. A non-engineer founder is often the only person who can write them, because only the founder knows what correct means for the customer. The team writes the harness; the founder writes the cases and rubric.
How many cases does an eval set need?
Ten is the floor per capability. Below ten, the variance in a 70-to-90% pass rate is too wide to separate a regression from a flake. Most pilots ship with 20 to 50 per capability and grow to 100 to 500 as the product matures. Coverage beats count — 30 cases spanning the real distribution outperform 300 that all look like the demo.
What is a rubric exactly, and who writes it?
A written set of rules describing the properties a correct output must have. Written by whoever owns the quality bar — typically the founder, sometimes a domain expert. A good rubric is specific enough that two graders applying it to the same output agree at least 90% of the time. If they disagree more often, the rubric is too vague.
How is an eval different from a benchmark like MMLU or HumanEval?
A public benchmark grades the model on a generic task; an eval grades your product on your specific task. MMLU asks whether a model knows world facts. Your eval asks whether your classifier returns the right label on emails like your customers send. Benchmarks help you pick a model; your eval tells you whether that model, with your prompt and guardrails, does the job.
Can the AI grade its own work?
Sometimes, with calibration. LLM-as-judge works when the rubric is stable, a human-graded sample shows the judge agrees with humans, and you watch for known biases. Treat the LLM judge as a junior grader who scales infinitely but needs a senior to spot-check.
Why not just ask users in production whether the answer was good?
Production feedback is real signal but does not replace evals. It is biased toward edge cases and complainers, arrives slowly, and cannot tell you whether yesterday’s model update broke last week’s behavior. Evals are the pre-production contract; feedback is the post-production signal that updates the set over time.
When should I write the evals — before, during, or after?
Before. The eval set is the spec. A team that writes evals after the prototype works has encoded the prototype’s compromises into the rubric, and the threshold becomes whatever the prototype passes.
What does it cost to run an eval suite?
Less than founders expect. Binary-correctness evals on a 50-row set cost cents per run. LLM-as-judge runs cost a few dollars. Human-graded evals are the expensive line — domain experts at a few hundred dollars an hour. Pattern: binary correctness and LLM-as-judge for daily regressions, human grading for periodic calibration.
Is there a tool I should use?
Tooling is secondary. Open-source frameworks like Promptfoo, DeepEval, and the UK AISI Inspect harness, plus OpenAI’s Evals library, cover most early needs. Hosted platforms — Braintrust, Langfuse, Patronus AI — add UI and team collaboration. None write your cases or rubric.
What single sentence should I leave a vendor meeting with?
“Show me your eval set and pass-rate threshold for each capability.” If they have one, they are doing the work. If they promise to “write the evals once we have a prototype”, walk that back into a written eval contract before the SOW is signed.
Closing
That is the whole concept. Inputs, rubric, threshold. Four-part row anatomy. Four grader shapes. The eval set is the spec because it is the only artifact that survives a model migration.
If you are paying a team to build an AI feature, use this as a vendor-screening instrument. Ask to see the eval set. You will learn more in five minutes than in two hours of capability slides.
For the operational sequence, read the eval-first build playbook. For the commercial argument, stop scoping AI projects in features. For the wider thesis, start at the idea-to-product manifesto.
Subscribe to the SFAI Labs newsletter for a weekly piece in this shape: concept-first, no code, for the person paying for the build.
Dirk Jan van Veen, PhD
