The weekly eval review is the founder’s primary accountability instrument on an AI MVP engagement, and most founders walk into it without an agenda. They listen, nod at a green dashboard, and leave. Six weeks later the partner ships a product that misses the threshold the SOW committed to, and nobody can point to the meeting where the drift started. The fix is not more eval expertise on the buyer side. The fix is four questions, asked in order, every week, with the founder doing the asking and the partner doing the defending.
This is the founder agenda. Pass rate against threshold. New failure modes. Regression catches. New inputs from real logs. Each question is structured so a non-technical buyer can ask it cold, recognise a good answer, and pattern-match the dodge. The order matters because the dodges chain together — a partner who fudges question one leans harder on question two, and a partner who avoids question three has usually quietly failed at question four. The agenda is designed to make evasion expensive.
This piece builds on The founder-AI partner operating manual, week by week, part of the idea-to-product manifesto. It is the question-card companion to the role of evals in your weekly partner relationship and the demo-side instrument described in the weekly AI MVP demo.
Decision scope
This is a buyer-side editorial framework, not engineering, statistical, or vendor-management advice. Specific pass-rate thresholds, eval set sizes, and judge-human agreement rates depend on your domain, risk tolerance, and cost profile. Figures cited below are illustrative ranges drawn from public engineering practice — validate them against your own SOW commitments and eval logs.
Why four questions, in this order
Four is the minimum that produces accountability. Three would let the partner steer past one of the failure modes; five becomes an engineering review a non-technical founder cannot lead. The four questions cover the four ways an eval set goes wrong over an MVP engagement:
- The set drifts away from the committed threshold. Question 1 — pass rate against threshold — surfaces this.
- A new failure category appears and is not named. Question 2 — new failure modes — surfaces this.
- The eval set is not version-controlled. Question 3 — regression catches — surfaces this.
- The set is frozen against last quarter’s distribution. Question 4 — new inputs from logs — surfaces this.
The order matters because each question constrains the next. If question one is fudged, question two becomes defensive. If question three is dodged, question four cannot be verified. Asked at a consistent weekly cadence, the four questions also produce a forecastable conversation — the partner knows what is coming and prepares an artifact for each, and the meeting compresses from a 45-minute status update to a 20-minute decision review with a one-line outcome: green-light, hold, or rollback.
Question 1: what is the pass rate against threshold this week?
This is the first question because it grounds the meeting in the contract. Pass-rate-against-threshold is sharper than “how are evals going” or even “what is the pass rate” — it forces the answer to reference a number the partner committed to in the SOW. A quality gate, not a trend line.
Why it matters
The threshold is the floor the product must clear to be considered launchable for the use case agreed in scoping. A pass rate trend without a threshold is a thermometer without a fever line — the founder knows whether the number went up, not whether the product ships. Anchoring to threshold turns the SOW into a live document. If scoping set the gate at 85% on the core intent set and the engagement is in week five at 71%, the conversation shifts from “evals are improving” to “we are 14 points below the gate we committed to, with N weeks to close it.” That is a different meeting.
A 2024 Stack Overflow Developer Survey finding is worth holding in mind: fewer than half of professional developers report any structured eval process for the AI features they ship. Most partners do not hold themselves to a numerical threshold by default. A founder who asks question one consistently is asking for accountability above the industry baseline.
What a good answer looks like
A good answer is a single number, the committed threshold, and a one-sentence narrative: “We are at 78% pass rate this week against a committed threshold of 85% on the core intent set. Up two points from last week. Six weeks to launch, seven points to close, concentrated in two buckets — retrieval-miss on long documents, and format-error on structured output.”
Three properties make that answer good. It cites the threshold from the scoping doc. It sites the number in context — both trend and gap to gate. It points to where the remaining gap lives, setting up question two. On the weeks it applies, a good answer also names a calibration check: “We re-graded 30 cases manually this fortnight; judge-LLM agreement was 88%, above our 85% floor.” That tells the founder the number is trustworthy, not just present.
What a dodge looks like
Three dodges recur. The trend-only answer — “pass rate is up four points week-over-week” — omits the threshold and is calibrated to reassure. The founder cannot tell whether the engagement is on track or eight points from a launch miss. The rubric-softening answer — “we adjusted the rubric this week to better match user intent, pass rate is now 87%” — is sometimes legitimate, often the set being made easier. Follow up: “What changed in the rubric, and what would pass rate be if we re-ran on last week’s rubric?” A partner who cannot produce both numbers is not running version-controlled evals. The threshold-not-set evasion — “we have not formalised a threshold yet” — is fine in week one and a structural failure by week four. A threshold a partner refuses to commit to is one that cannot be missed, which is exactly the dynamic that produces a week-fourteen miss.
If any of the three appear, pause the rest of the agenda and resolve question one. The other three cannot be honestly answered if the pass-rate number is not trustworthy.
Question 2: what new failure mode showed up?
The second question asks the partner to name a new bucket. If the answer is “nothing”, the answer is wrong — production AI systems produce novel failure modes every week, and a partner who claims none did is not looking. The question is structured to extract a categorical, named failure type — not a count or a percentage.
Why it matters
New failure modes are the leading indicator of regression risk and scope risk. When the model fails in a new way, three things are usually true: the failure was not in the rubric, the set under-represented the affected input distribution, and the behaviour change is recent — typically a prompt edit, a model upgrade, or a retrieval change in the last seven days. Named early, a new failure mode is absorbed into the work. Not named, it accumulates — three uncategorised week-two failures become a 14% drop on a customer demo intent by week six, and the set never tracked it because nobody named the bucket.
Question two is also the cheapest accountability question a non-technical founder can ask. A founder cannot evaluate rubric design or judge calibration. A founder can absolutely ask “is there a new kind of failure this week” and tell whether the partner sounds prepared or surprised.
What a good answer looks like
A good answer names the bucket, gives an example input, attributes the cause, and proposes the next action: “Yes — new bucket called ‘refusal-on-ambiguous-PII’. Showed up after we tightened the safety classifier on Tuesday. About 4% of customer-support cases hit it. Example: a user asking ‘can you summarise the last email from John’ triggers a refusal because ‘John’ is ambiguous. We propose either loosening the classifier on this intent or adding a clarification step. Need a decision from you by Friday.”
Four properties: named bucket, representative input, attributed cause, decision request. A partner running a healthy review produces all four without being asked. The honest “no new failure mode” answer is acceptable but only when paired with the bucket distribution alongside last week’s — without the comparison, “no new failure mode” is indistinguishable from “I did not check”.
What a dodge looks like
The variance frame — “a couple of cases failed in an unusual way, looks like noise, we will watch it” — pushes the failure into the no-action bucket without ever naming it. The founder cannot follow up next week because there is no named thing to follow up on. The rebucketing answer — “we adjusted how we categorise failures, buckets look cleaner” — sometimes hides failure modes being relabelled rather than tracked. Follow up: “Show me old and new buckets side by side.” The deferral — “we need more data before we can name what is happening” — is fine in week one and usually a volume problem by week four.
Question 3: did regression catch anything?
The third question is the cheapest verification ritual the founder has. It asks whether anything that passed last week failed this week. The answer is either a list, an empty list with evidence, or a dodge.
Why it matters
Regression catching is the proof that the set is run as a version-controlled artifact. The set has to be the same set, week to week, for “regression” to be meaningful. If the partner cannot produce a regression list, either the set was silently rewritten (in which case question-one numbers are also untrustworthy) or the harness is not being run consistently (in which case the review is theatre).
A founder asking question three is doing work no one else in the engagement will do. Engineers run regressions because it is good practice; they rarely surface them upward because the buyer rarely asks. A partner who knows the buyer asks question three every week starts producing a clean regression artifact by week three — the discipline emerges from the asking. The Anthropic eval cookbook and the OpenAI evals repo document version-controlled regression suites as standard practice; the gap between “documented as standard” and “actually run weekly” is wider than most founders assume.
What a good answer looks like
A good answer is a short list — typically two to five cases — with the input, the previous output, the current output, and a one-sentence diagnosis per case. “Yes, regression caught three. Two in format-error — the structured-output prompt now misses the closing brace 6% of the time after we shortened the system prompt Monday. The third is a retrieval-miss on a long document; intermittent, watching it. We rolled back the prompt change this morning; verifying on tomorrow’s run.”
Four parts: list of cases, named buckets, attribution to a recent change, action taken or proposed. A clean regression answer takes ninety seconds. “Zero regressions” is fine but should be supported by the artifact: the set was run, every case from last week passed again, harness logs are linked.
What a dodge looks like
The set-shift dodge — “we added new cases this week, comparison is not clean” — follow up: “What was the regression count on the cases in both weeks’ sets?” The harness-down dodge — “harness had an issue this week” — is sometimes true and by week four is a process problem, not a tooling one. The category-rollup dodge — “regression caught a few things, all in known buckets we are working on” — elides the cases. Follow up: “Read me the inputs.” A partner who cannot read three inputs out loud is not running regression review at the case level.
Question 4: what new inputs should we add to the set?
The fourth question closes the loop. It asks the partner what inputs from the last week of real customer logs should join the set. The answer should be a small list — three to ten cases — proposed by the partner, with one-sentence rationale per case.
Why it matters
The set is only as useful as its match to real user behaviour. In week one, it is hand-written and approximates the PRD’s intent distribution. By week six, real users have shown the partner inputs the PRD never anticipated — phrasings, edge cases, multi-turn patterns, domain quirks. A set that does not absorb these grows less diagnostic over time, even as the pass rate stays high.
Question four is also the buyer-side verification that real logs are being read. A partner who is not reading logs cannot propose new candidates. A partner who is reading logs but never proposes new candidates is reading without listening. There is a softer reason too: question four invites the founder into the eval-curation conversation. Over six weeks, the founder ends up with a mental map of the user distribution that no other ritual produces. It is one of the highest-yield 60 seconds of the week.
What a good answer looks like
A short list of cases with source attribution: “Five new cases this week. Three came from customer-support logs Tuesday — a new phrasing pattern where users ask for summary plus next action in the same message; the model currently splits them. Two from the analytics dashboard intent — a customer asked for a metric in a unit we do not support, model hallucinated a conversion. We would weight the first three heavier. Add all five, or hold the analytics ones until we have a unit-conversion plan?”
Four properties: short list, source attribution per case, prioritisation rationale, buyer decision request. The honest “no new candidates this week” should be paired with traffic data: “240 sessions this week. Distribution looked similar to last week. No new patterns worth adding.” Rare but legitimate; without the traffic data, it usually means logs were not reviewed.
What a dodge looks like
The volume dodge — “not enough customer traffic to surface new patterns” — is plausible in week one or two and unconvincing by week four. Even ten sessions per day produces inputs worth examining. The synthetic-substitute dodge — “we generated additional cases by varying phrasings with a model” — has a role but is a poor substitute for real-log harvesting. Follow up: “How many of this week’s new cases came from real logs versus synthetic?” A healthy mid-engagement mix is at least 60% real-log-sourced. The defer-to-engineering dodge — “engineers are batching candidate review for end of sprint” — is fine only if the candidates reach the founder at the meeting. The whole point of question four is that the buyer is in the loop on what goes into the set, because what goes into the set defines what the threshold in question one is measured against.
The 4-question founder card
The four questions print on a single index card. The founder carries it into the meeting. The partner walks through each in order. The meeting closes in 20 minutes.
| # | Question | What good sounds like | What a dodge sounds like |
|---|---|---|---|
| 1 | Pass rate vs threshold this week? | A number, the committed threshold, and a one-sentence gap-to-gate narrative. | Trend-only, rubric-softened, or “threshold not set”. |
| 2 | What new failure mode showed up? | A named bucket, example input, attributed cause, decision request. | Variance frame, rebucketing, or “need more data”. |
| 3 | Did regression catch anything? | Two-to-five-case list with previous output, current output, and action taken. | Set-shift, harness-down, or category-rollup. |
| 4 | What new inputs should we add? | Short list with source attribution and prioritisation rationale. | Volume excuse, synthetic substitute, or defer-to-engineering. |
The card is deliberately stripped. Four lines per question, twelve dodge labels, one threshold to anchor on. Print it. Bring it to every Friday review for the first eight weeks. By week eight, the partner produces the answers in order without the card; the card has done its job.
What good looks like by week six
By the sixth weekly review, three things should be true.
First, the partner walks in with the four answers structured before the founder asks. The questions do not need to be read off the card any more — the partner has internalised the agenda. The meeting compresses from forty-five minutes of narration to twenty minutes of decision review.
Second, the pass-rate-vs-threshold answer becomes load-bearing in the engagement narrative. Sprint planning, scope discussions, and roadmap reviews all reference the threshold. The number stops being a metric the engineers report and starts being the gate the engagement is organised around.
Third, the eval set is visibly evolving each week. By week six, the set has roughly 50% to 70% more cases than week one, with 60% or more of the additions sourced from real customer logs through question four. The set is closer to the user distribution than the week-one hand-written version. The threshold means more because the set means more.
If any of the three are not true, the issue is upstream of the review — the SOW threshold was never specified, the engagement does not yet have real users, or the partner does not have eval engineering as a routine capability. Each is a separate conversation. The closely related role of evals in your weekly partner relationship covers the artifact-level review; the weekly AI MVP demo covers the parallel demo ritual; and Stop paying AI agencies for documentation — pay them for evals covers the commercial framing of eval discipline as the thing you are actually buying.
If you are six weeks into an engagement where these four answers are not yet structured, bring the card to next Friday’s review and run the meeting yourself.
Frequently asked questions
What is the founder’s eval review?
A weekly 20- to 45-minute meeting where a non-technical founder asks their AI partner four structured questions about how the eval set performed this week — pass rate against threshold, new failure modes, regression catches, and new inputs from real customer logs. The founder runs the meeting; the partner defends the answers.
Why exactly four questions?
Four is the minimum that produces accountability across the four ways an eval set goes wrong: threshold drift, missed failure modes, lost version control, and frozen coverage. Three questions would let the partner steer past one. Five becomes an engineering review a non-technical founder cannot lead.
Do I need to understand evals to run this meeting?
No. The questions are designed to be asked cold by a non-technical buyer. You need to recognise the shape of a good answer (specific, attributed, decision-ready) and pattern-match the dodges. Rubric design, judge calibration, and tooling stay on the partner side.
What is a pass-rate threshold and where does it come from?
A numeric pass-rate floor for the core eval set, written into the scoping document or SOW. It represents the quality gate the product must clear to be considered launchable for the use case. If your engagement does not have one, that is the first conversation, not the first eval review.
How often should the eval set grow?
In a healthy MVP engagement, the set grows roughly 5% to 15% per week from week three onward, with 60% or more of additions sourced from real customer logs. By week eight the set is usually 50% to 100% larger than week one and substantially closer to the real user distribution.
Should I attend if I am not technical?
Yes. The agenda is designed for a non-technical buyer to run. The partner brings engineering depth; the founder brings the four questions and threshold accountability. If the meeting feels too technical, that is usually a sign the partner is not translating well, not that the founder should skip it.
What if my partner does not have evals set up at all?
Then the four-question agenda is on hold and the right conversation is whether eval engineering is being treated as a deliverable in the engagement. Stop scoping AI features in user stories — scope them in evals covers the scoping fix. The four questions resume once the harness exists.
How long should the meeting take?
Expect 45 minutes in week one — the partner is learning the agenda. By week three, 25 to 30 minutes. By week six, 20 minutes. If the meeting still takes 45 minutes in week six, the partner is narrating rather than reporting, and the founder should ask for a tighter format.
How do I know if my partner is willing to be held accountable this way?
Run the four questions in week three and watch the answers. A partner with eval discipline answers cleanly and improves the structure over the next two weeks. A partner without it produces dodges; if the dodges persist past week five, the engagement is structurally drifting and the conversation moves from the four questions to a scope-and-deliverable conversation with the partner principal.
Where to go next
- The full operating rhythm — The founder-AI partner operating manual, week by week
- The broader model — The idea-to-product manifesto: how non-engineers ship AI products in 2026
- A companion read — The role of evals in your weekly partner relationship
- Also useful — The weekly AI MVP demo: what to question, what to celebrate
- On the commercial side — Stop paying AI agencies for documentation — pay them for evals
If your weekly eval review does not yet produce these four answers in order, bring the founder card to next Friday’s meeting and run it. Six weeks of disciplined asking compounds into a partner who answers without being asked — which is the operating state every AI MVP engagement should converge toward.
Dirk Jan van Veen, PhD
