You cannot write the code, but you can verify the rubric. A non-technical founder hiring a team to build an AI feature does not need to read Python or learn a YAML eval DSL. The founder needs a single artifact — a one-page rubric — that defines what correct means, what the pass bar is, and what blocks a future regression. If the team cannot fill that one page in plain English, the spec is wrong and the build will fail. The template below has six sections, is copy-pasteable, and is the verification surface that keeps a vendor honest from week one to production.
This template builds on the eval-first build playbook, part of the idea-to-product manifesto. Read it alongside the eval-first PRD and what are AI evals in plain English. The commercial companion is stop paying AI agencies for documentation, pay them for evals — that argues the why; this gives the how.
Table of Contents
- Why founders verify rubrics, not write them
- The Six Sections
- Worked Rubric: Customer-Support Classifier
- How to Use the Template With a Vendor
- The Founder’s Verification Checklist
- Frequently Asked Questions
- Closing
Why founders verify rubrics, not write them
The 2018 SaaS pattern was deterministic: engineer wrote the test, founder reviewed the demo, disputes resolved against the wireframe. AI features in 2026 are non-deterministic, drift on every model update, and fail on inputs the founder has never seen. The founder needs a written, signed artifact that says what good means before the engineer writes code. That artifact is the rubric.
Three reasons make verification load-bearing.
The rubric is a commercial artifact. Without it, quality disputes become scope-based and unwinnable. With a signed rubric, the dispute is technical: the suite passes the threshold or it does not.
The rubric is the migration test. Frontier models update on cycles of weeks — Anthropic shipped Claude Opus 4 through 4.8 and Sonnet 4.6 this year, OpenAI shipped GPT-5 with successors, Google shipped Gemini 2.5 and 3.1 (Artificial Analysis LLM Leaderboard). Each transition silently changes any feature pinned to a model alias. The rubric is the contract the vendor reruns to earn the right to upgrade.
The rubric is the only artifact the founder can sign without an engineer. The founder owns the domain — what correct means and the commercial cost of each failure mode. The engineer cannot decide that a wrong-priority-bucket error on a billing email costs five times what the same error costs on marketing. That call is the founder’s.
The template must be readable by a non-engineer, fillable in plain English, short enough to fit one page. Most vendor docs publish templates in YAML or JSON inside a harness DSL — those serve engineers. The template below was written for the founder.
The Six Sections
Every defensible AI eval rubric has the same six sections. Order matters — earlier sections constrain later ones. Skipping a section is the single most reliable predictor of a vendor dispute three months later.
Section 1. Task Definition
One sentence naming the single capability the rubric grades. The sentence cannot contain the word and. If the vendor needs an and, that is two capabilities — two rubrics.
Good: “Classify each inbound customer-support email into one of seven priority buckets using only the email body and the sender’s last 90 days of ticket history.”
Bad: “Triage support email and draft a reply and route to the right team.” Three capabilities.
End the section with the task type: classification, extraction, generation, or tool-use. Task type sets the shape of the scoring axes.
Section 2. Input Examples
Five representative inputs from the actual usage distribution (real or anonymized — synthetic hides failure modes). The full suite holds 80 to 150; the rubric names five. These are the rows the founder personally inspects when the model drifts. Picking them requires the founder — an engineer alone cannot know which patterns are commercially load-bearing.
Section 3. Output Spec
The exact shape of a correct output. For classification, the closed label set. For extraction, the field schema. For generation, the constraint set (length, structure, prohibitions). For tool-use, the allowed-tool set and call-schemas.
The discipline is that the spec is mechanically checkable: a non-engineer applies each constraint as a yes/no rule on any candidate output. Any output failing the spec scores zero on every axis before grading begins. That separates the rubric from a vibes-based review.
Section 4. Scoring Axes
The heart of the rubric. One to four anchored criteria graded binary (pass / fail) or 3-point (fail / partial / pass), with a one-sentence anchor per level.
The G-Eval paper (Liu et al., 2023) found LLM judges well-calibrated on one to four anchored axes and losing calibration past five. The MT-Bench paper (Zheng et al., 2023) found 1–10 Likert grading introduces verbosity, position, and self-preference bias; binary or 3-point reduces those biases substantially. Practitioner consensus — Eugene Yan, Anthropic’s empirical evaluation docs — is anchored grading over Likert.
Each anchor is one sentence. Not a paragraph. The founder should read each in five seconds.
Section 5. Threshold
The pass bar — per-axis and aggregate — that ships the feature. Vendors commonly omit this because naming a threshold creates accountability. Insist on it.
Per-axis floors: a minimum on each axis. Asymmetric — the floor on a commercially-load-bearing axis (under-calling severity, missing a refusal) is higher than on a debuggability axis. The asymmetry is the founder’s call.
Aggregate gate: every floor must clear simultaneously. No averaging — that lets the vendor over-tune one axis at the expense of another.
Set the threshold against the baseline run, typically 5 to 15 points higher per axis. The gap is the size of the build.
Section 6. Regression Rule
Blocks a deploy when a future change drops a score. Turns the rubric from snapshot into continuous contract.
Per-axis blocking delta: how much a score can drop before deploy is blocked. A 2-point absolute drop is a defensible default — tight enough to catch silent degradations, loose enough not to false-alarm on noise.
Model-update clause: any change to the model alias triggers a full suite run; the report is diff’d against the prior model; any axis dropping more than the delta blocks the migration. Closes the most common 2026 silent-regression vector.
Override signatory: who signs when the team deploys despite a regression. Default: founder. Every regression-allow is a signed decision.
Worked Rubric: Customer-Support Classifier
The page below is the entire rubric for one capability, on one page, plain English. Hand it to a vendor in week two of scoping; expect it back, signed, in week three.
RUBRIC — Customer-Support Email Classifier
Version 1.0 · Signed by founder + vendor · Reviewed quarterly
1. Task Definition
Classify each inbound customer-support email into one of seven priority buckets, using only the email body and the sender’s last 90 days of ticket history. Task type: classification.
2. Input Examples
| # | Label | Input (truncated) |
|---|---|---|
| 1 | p0 outage, enterprise | ”Our entire team is locked out as of 9:42am PT — we have a customer demo at 11:00. Pls help.” |
| 2 | p3 account, free tier | ”I forgot my password. Reset email isn’t arriving.” |
| 3 | p4 product question | ”Hey — quick one. Does the export support CSV with semicolon delimiters?“ |
| 4 | p6 marketing, false-urgent | ”URGENT: please remove me from your newsletter immediately.” |
| 5 | p1 billing urgent, non-English | ”Mi tarjeta fue cobrada dos veces este mes — la segunda no debería estar ahí.” |
Full input set (124 rows, stratified by label): /evals/customer_support_classifier/inputs.jsonl. The five above are the founder-inspected anchor rows.
3. Output Spec
Output is exactly one of seven labels in the closed set:
p0_outage · p1_billing_urgent · p2_billing_routine · p3_account · p4_product_question · p5_feature_request · p6_marketing
Plus an optional rationale (≤ 200 chars) and a refusal sentinel __out_of_scope__ for inputs the model should not classify (bounces, calendar invites, chain emails). Any output not matching this spec scores zero on every axis.
4. Scoring Axes
| Axis | Description | Grading |
|---|---|---|
| 1. Label correctness | Predicted label exactly matches the gold label. | Binary |
| 2. Severity floor | Model does not under-call severity. Predicting a lower-urgency label than gold is a fail; equal or higher urgency is a pass. | Binary |
| 3. Reasoning trace presence | Rationale exists, is ≤ 200 chars, references at least one specific phrase from the email body. | Binary |
| 4. Refusal correctness | On out-of-scope inputs, the model returns the refusal sentinel. On in-scope inputs, it does not. | Binary |
5. Threshold
Ship gate (all four floors must clear simultaneously):
- Axis 1 (label correctness): ≥ 92%
- Axis 2 (severity floor): ≥ 99%
- Axis 3 (reasoning trace presence): ≥ 95%
- Axis 4 (refusal correctness): ≥ 98%
Aggregate gate: the suite fails if any axis is below its floor. No averaging across axes.
Baseline (Sonnet 4.6, untuned prompt, no retrieval): 81% / 92% / 78% / 84%. Gap to ship gate is the size of the build.
6. Regression Rule
- Per-axis blocking delta: a drop of more than 2 absolute points on any axis blocks the deploy.
- Model-update clause: any change to the model alias triggers a full suite run; the report is diff’d against the prior model; any axis dropping more than the delta blocks the migration.
- Override signatory: founder. All regression-allow events captured in the deploy log with rationale.
The vendor fills the engineer-facing artifacts — YAML harness config, judge prompt, CI gate, deploy log — from this page. The founder signs the page.
How to Use the Template With a Vendor
Four meetings over two weeks.
- Capability decomposition. Founder and senior vendor engineer produce the capability inventory. Each row gets its own rubric.
- Task and output spec. Founder writes sections 1 and 3 per capability. Engineer pressure-tests for ambiguity; founder for hidden ands.
- Inputs and axes. Founder picks the five inputs (section 2). Engineer drafts axes (section 4). Founder revises anchors.
- Threshold and regression rule. Vendor runs the baseline. Both set per-axis floors (5) and regression rule (6). Both sign.
By meeting four, the founder has a signed rubric per capability. The eval-first PRD wraps them into a single deliverable. Vendors who resist this sequence are vendors to walk away from — the thesis in stop paying AI agencies for documentation, pay them for evals is that the eval suite is the asset and the rubric its spec.
The Founder’s Verification Checklist
For each rubric the vendor produces, verify before signing.
- Section 1, one sentence, no “and”. A conjunction joining verbs means two capabilities. Split.
- Section 2, five real inputs. Each row is real or anonymized real, recognizable as representative. No synthetic placeholders.
- Section 3, mechanically checkable. A non-engineer can apply each constraint as a yes/no rule.
- Section 4, 1 to 4 anchored axes. One-sentence anchor per level. Binary or 3-point only. No 1–10 Likert.
- Section 5, asymmetric floors. Floors reflect the commercial asymmetry of failure modes.
- Section 6, all three clauses. Blocking delta, model-update clause, override signatory.
A rubric passing all six checks is one the founder can sign. Cycle time: under twenty minutes per capability.
This is the artifact behind the closer: if your vendor can’t fill this template in one page, the spec is wrong. If the rubric needs more than six sections on one page, the capability is overloaded — split it.
For the full scoping packet — capability inventory, PRD shell, this template per capability, threshold-setting worksheet — download the AI MVP scoping worksheet.
Frequently Asked Questions
Does the founder write the rubric, or only verify it?
Verify. The vendor’s senior engineer drafts. Engineer knows the harness, metric library, model behavior; founder knows the user and the commercial cost of each failure mode. Neither writes a defensible rubric alone. The founder signs because the rubric is a commercial artifact.
Can the LLM write the rubric?
Not for scoping-time work. An LLM helps draft candidate anchor language and propose axes once the task is named, but cannot decide what correct means, cannot pick representative inputs from the actual usage distribution, and cannot calibrate the threshold against commercial constraints. The LLM is a drafting assistant; the founder is the source of ground truth.
Why binary or 3-point grading instead of 1–10 Likert?
LLM judges on a 1–10 Likert scale show verbosity, position, and self-preference bias at levels documented in the MT-Bench paper. Binary or 3-point grading reduces those biases substantially — fewer adjacent levels for the judge to drift between. Likert is the wrong default in 2026.
What if the rubric feels subjective on a generation task?
Decompose harder. Even creative tasks have measurable sub-criteria — groundedness, locale-correctness, refusal on out-of-scope, length, tone-match against a small reference set. The judge grades each sub-criterion against anchored language, with a human spot-check on a 20-output sample per cycle. Subjective usually means insufficiently decomposed.
How big is the full eval suite?
80 to 150 inputs per capability, stratified across the gold-label distribution, plus a 10 to 30-input out-of-scope sub-suite for refusal grading. The rubric names five anchor inputs; the rest live in a JSONL referenced from section 2. The suite grows over the build — every customer-reported failure becomes a new row.
How often does the rubric get rerun?
On every change that could affect the feature: prompt edits, retrieval swaps, model-alias upgrades, system-prompt revisions, tool-schema changes. In CI, the run gates the deploy. Out of CI, the suite runs on every model-alias upgrade and the diff is filed as part of the migration decision.
What harness should the vendor use?
The 2026 open-source standards are Promptfoo, Inspect AI, and DeepEval. Langfuse is the most-used hosted option. The choice is the vendor’s; this template is harness-agnostic.
Who signs the rubric and where does the signed copy live?
The founder and the vendor’s accountable engineering lead. The signed copy lives in the repo alongside the eval suite, version-controlled, with the rubric version matching the suite tag in CI. A rubric not version-controlled and not signed has not been agreed.
How does this template differ from a generic rubric template online?
Most generic templates ship a scoring grid and stop. This one insists on three sections that turn the rubric into a commercial contract: the output spec (3) makes outputs mechanically checkable; the threshold (5) names the pass bar with per-axis floors; the regression rule (6) makes the rubric continuous over the life of the feature. Without those three, the rubric is a one-time review.
What’s the biggest failure mode this template prevents?
A vendor returning a single Likert average — say, 4.2-out-of-5 — that hides catastrophic failure on the commercially-load-bearing axis. Binary or 3-point grading plus per-axis floors plus asymmetric weighting forces the failure to surface. A 4.2-out-of-5 with no per-axis vector is what every founder regrets accepting in week three.
Closing
The rubric converts a probabilistic system into a commercially gradable one. The six sections — task, inputs, output spec, scoring axes, threshold, regression rule — are the minimum that turns a rubric into a contract instead of a review. A founder who cannot write Python can verify every one of those sections. The verification surface is one page.
If your vendor cannot fill this template in one page, the spec is wrong. Push back, decompose the capability, try again.
Download the AI MVP scoping worksheet to get this template inside the full founder-facing scoping packet — capability inventory, PRD shell, threshold worksheet, regression-rule template ready to drop into your next vendor engagement.
Dirk Jan van Veen, PhD
