If you are a non-technical founder paying an AI partner to build your product, the single most diagnostic 45 minutes of your week is the eval review — where your partner walks you through how the model is performing against the test set that represents your users. Not the demo, not the standup, not the roadmap sync. This is where the relationship either compounds or quietly decays.
Most founders never sit in this meeting. They get a green-dashboard screenshot in Slack and assume the system is working. That is a mistake. Evals are the only artifact that tells you whether your partner is building a launchable product or an impressive demo. Below: five things to look at, five questions to ask, three anti-patterns to refuse, and a 15-minute agenda you can run from next Friday.
Decision Scope
This article is an editorial decision framework, not engineering, statistical, or vendor-management advice. The right specific metrics for your product will depend on your domain, risk tolerance, and the cost profile of your model calls. Treat percentage and dollar figures cited as illustrative ranges and validate them against your own eval logs and invoices.
Why the eval review is your meeting, not your partner’s
The standard founder failure mode is to treat evals as a technical artifact the partner owns. The dashboard belongs to the engineers; the buyer just gets a summary. By that logic, the buyer never reads the only document that predicts whether the product will hold up in front of real users.
The inversion: evals are the closest thing your partner produces to a regulatory filing — a structured claim that the system performed at a given level on a representative test set in week N. If the claim is wrong because the set was thin, the judge miscalibrated, or the example unrepresentative, your launch date and your runway are downstream of that error.
The 2024 Stack Overflow Developer Survey found fewer than half of professional developers report any structured eval process for the AI features they ship. Most partners do not run evals by default. A founder who does not insist on the meeting will not get one.
Treating the eval review as your meeting changes three things: the agenda becomes yours to set, the questions stop being engineer-to-engineer and start being buyer-to-vendor, and the partner has to defend the set itself — not just the score.
What evals are, and what they are not
An eval is a test set of inputs (user queries, document chunks, audio clips) paired with a graded expectation of the output. A grade can be deterministic (“output matched the gold answer”) or rubric-based, where a judge — often another model, sometimes a human — scores each output against a written rubric.
Three things evals are not:
- Unit tests. Unit tests check a function returns a specific value. Evals tolerate variance — the model is stochastic — and score against a population of acceptable answers.
- Demos. A demo shows what the system does on one input. An eval shows what it does on hundreds, spanning the distribution of real usage.
- Benchmarks. A benchmark (MMLU, HumanEval, GPQA) measures generic capability. Your eval set measures whether your product, in your domain, on your users’ inputs, performs to your threshold. Leaderboards like Artificial Analysis are useful for model selection, not shipping decisions.
If your partner is showing you benchmark scores in lieu of an eval set built on your domain, they are pointing at a thermometer from the next town over.
The 5 things to look at, every week
A weekly eval review should produce five artifacts on a single slide. If the partner needs more than five, the meeting is being run for the engineers, not the buyer.
1. Pass-rate trend (this week vs last 4)
The headline number is the percentage of cases that passed this week, plotted against the last four weeks. The trend matters more than the absolute number — eval sets are hard to calibrate in absolute terms, but a six-point drop week-over-week is unambiguous. Two to three points of variance is normal; five or more is a ten-minute conversation. Pass rate alone is a number, not a verdict — the verdict comes from the next four artifacts.
2. New failure modes (categorised)
Every run produces failures. The question is not how many — that is the pass rate — but how. The partner should classify failures into named buckets: hallucination, retrieval-miss, format-error, refusal-when-shouldnt, latency-timeout. A new bucket that did not exist before is a release-blocker. A bucket that grew from 3% to 18% is a regression. The set has done its job when failures are legible by class, not just by count.
3. Regressions (passed last week, failed this week)
The most actionable artifact in the review. A list of three or four cases tells you exactly what changed. If your partner cannot produce one, they are not running version-controlled evals — and cannot tell you with confidence what their code changes did to behaviour. That gap closes before the next release.
A non-technical founder reads a regression list by scanning inputs. Are they representative of your users? If yes, the regression is serious. If they are edge-case stress tests, it matters but is less urgent.
4. Cost-per-task (and the trend)
Every model call has a cost: input tokens, output tokens, judge-model tokens, retrieval-pipeline costs. The review should show cost-per-task — typically a few cents — and the four-week trend. A rising cost signals prompts getting longer, retries happening more often, or a model being switched to a more expensive tier without a quality justification. A 30% increase with a flat pass rate is a serious conversation. The unit economics of the product depend on this number staying under control.
5. New eval candidates from real customer logs
The most diagnostic artifact, and the one most likely to be missing. A disciplined partner extracts five to ten new candidate cases from real logs each week and proposes adding them. If the set is not being fed from real usage, it is frozen — testing against last quarter’s distribution, not this quarter’s.
The 5 questions to ask your partner
A founder does not need to read eval code. They need five buyer-side questions that surface drift, eval-theater, and judge miscalibration. These force the partner to defend the set, not just the score.
- “Show me three cases that passed this week. Walk me through why they passed.” Catches grading-rubric drift. If the partner cannot explain why a borderline case passed, the rubric is loose and your pass rate is overstated.
- “What changed in the eval set since last week?” If the answer is “nothing” and your users’ behaviour is changing, the set is going stale. If the answer is “we added five cases”, ask which customer logs they came from.
- “Has the judge model been audited against humans recently?” When a judge LLM grades outputs, it drifts. The expected practice is to manually grade a sample (say 30 cases) and check that human and judge agree on at least 85% to 90% of them. The Anthropic eval cookbook documents this calibration practice as standard.
- “Show me last week’s regression list and what got fixed.” A regression that was not fixed will recur. If last week’s list has carried into this week unresolved, the partner is shipping new features on top of quality debt.
- “What is the worst failure that did not block release?” Every release ships with known issues. Naming the worst one out loud surfaces buyer-side risk acceptance. If the partner cannot name a worst-case failure, they are not running an honest review.
These five questions are not technical. A non-engineer can ask all of them. The work is in listening to the answers — and refusing the three anti-patterns below.
Three anti-patterns to refuse
Anti-pattern 1: Eval theater (a green dashboard with no failing cases)
If every eval passes, the set is too easy. A well-designed set has 5% to 25% of cases failing on the current model — that is what makes it diagnostic. A 100%-green dashboard means the set has been pruned of anything that exposes weakness. Ask: “What is the hardest case in the set, and is the model passing it?” If no answer, the set is theater.
Anti-pattern 2: Single-input cherry-picking
A partner shows you one beautiful example and says, “see how it handled this query?” That is a demo, not an eval. The point of an eval set is statistical: across hundreds of inputs, what is the pass rate? One input proves nothing — in a stochastic model, the same input produces different outputs across calls. The unit of evidence is the population, not the example.
Anti-pattern 3: No judge-LLM calibration
If your partner uses a judge LLM to grade outputs and has never spot-checked it against human grades, the pass-rate signal is unreliable. Judges drift, have favourite phrasings, and over-credit verbose answers. The discipline is to manually grade a sample every few weeks and report the agreement rate. If the partner cannot say when they last calibrated, the review is built on sand.
What a good 15-minute eval review looks like
Run the meeting yourself, with the engineering lead presenting. The agenda is non-negotiable:
| Minute | Item | Output |
|---|---|---|
| 0–2 | Pass-rate trend | Buyer notes trend direction |
| 2–6 | New failure modes | Any new bucket? |
| 6–10 | Regressions + status | Buyer reads three in full |
| 10–12 | Cost-per-task trend | Unit economics hold? |
| 12–14 | New eval candidates from logs | Approve or de-prioritise |
| 14–15 | Decision: green-light? | Recorded in the meeting note |
The deliverable is a one-line decision: green-light, hold, or rollback. Anything that does not produce that decision is a status update, not a review. A founder who runs this meeting for six weeks will know more about their product’s real quality than most founders who pay an AI partner for the same length of engagement.
For the engagement this meeting fits into, see operating with an AI partner week-by-week and Inside the SFAI Labs operating cadence. The founder eval rubric template is the artifact you hand your partner before week one, and The AI agency quality system: evals, observability, and weekly review is the agency-seat companion read.
Frequently asked questions
What is a weekly eval review?
A 15- to 45-minute recurring meeting where the AI partner walks the founder through how the system performed against the eval set this week — pass-rate trend, new failure modes, regressions, cost-per-task, and proposed new cases drawn from real customer logs. It is the founder’s primary buyer-side accountability ritual.
My partner says evals are not worth the effort for an MVP. Should I believe them?
No. Eval discipline separates a partner who ships a launchable product from one who ships an impressive demo. A starting set (30 to 100 cases) is a few days of engineering. Without one, you have no signal on whether a model upgrade or prompt change improved or broke your product. If the partner pushes back, ask what they will do instead to catch regressions.
How big should the eval set be?
Start with 30 to 100 cases spanning common intents and the highest-risk edges. Grow to 200 to 500 over three months by harvesting real logs. A 1,000-case set of synthetic variations is worse than a 50-case set drawn from real users.
How long does it take to set up the first eval harness?
For a scoped MVP, two to five engineering days for the harness plus two to three to write the first 50 cases. The Anthropic cookbook and OpenAI evals framework are working starting points; Promptfoo, Braintrust, and LangSmith reduce setup further. More than two weeks is over-engineering.
What if my partner uses a judge LLM to grade outputs?
Common and reasonable for rubric evals. Insist that the partner manually grades a sample (typically 30 cases) every two to four weeks and reports the judge-human agreement rate. Below 80% to 85%, the verdict cannot be trusted and either the rubric or the judge needs revising.
How do I tell whether the eval set covers my real users?
Ask, for each case, where the input came from. A healthy mix: PRD intents, real customer logs from the last four weeks, and edge cases written to stress-test. If every case was hand-written in week one and never updated, the set is frozen and your coverage is decaying.
Should I ask for raw eval results or trust the summary?
Ask for the raw results. A good partner shares the failing-case list as a CSV or hosted dashboard. Spot-check three to five failing cases per week — read input, output, grade. Six weeks of this teaches you more about real behaviour than any other single activity.
What is the difference between online and offline evals?
Offline evals run against a static test set on a schedule. Online evals measure the live system on real user traffic — pass rate, latency, refusal rate, cost — and surface anomalies in production. Your partner should run both. Offline catches regressions before deploy; online catches drift after.
Where to go next
- The full operating manual: The founder–AI partner operating manual, week by week
- The bigger picture: The idea-to-product manifesto: how non-engineers ship AI products in 2026
- What is an AI development partnership, in plain English
- Discovery call vs paid pilot vs full engagement, explained
- From the agency side: The AI agency quality system: evals, observability, and weekly review
If your eval review does not yet produce the five artifacts above, subscribe to the SFAI Labs newsletter for founder-facing templates each week. The companion guides cover the four other rituals that make up the founder’s operating week.
Dirk Jan van Veen, PhD
