Scope AI features by eval pass-rate, not by user stories. A scope written as a feature checklist assumes the implementation is deterministic. AI features are probabilistic — a feature is not done when the code compiles; it is done when the system clears a quality threshold the founder will defend. If your current scope is a list of user stories with no eval threshold attached, you have written a 2018 scope on a 2026 product, and the build will drift into a quality bar nobody signed off on.
This article builds on the eval-first build playbook within the idea-to-product manifesto. The playbook covers the full engineering arc; this article gives the founder-side method for cutting scope before it begins.
Table of Contents
- Why feature-list scoping breaks for AI
- The 6-step method
- Worked example — scoping the same idea both ways
- The founder’s role
- The most common scoping mistake
- Frequently Asked Questions
- Closing
Why feature-list scoping breaks for AI
A 2018 SaaS scope was a checklist with deterministic acceptance per line: “User can upload CSV. System parses rows.” A 2026 AI scope cannot run on that machinery. Three structural facts force the shift.
AI outputs are distributions, not values. A line that says “the assistant drafts a reply” is silent on whether 70% acceptable drafts versus 90% is “done” — and that 20-point gap separates a product the founder uses daily from one she abandons by week three.
The model layer moves underneath the team. The state-of-the-art mix tracked by the Artificial Analysis leaderboard is GPT-5, Claude Opus 4.8, and Gemini 2.5 Pro — none of which were leading a year ago. A scope without an eval pass-rate cannot detect when a model update silently shifts behavior overnight.
The pilot-to-production gap is the dominant failure mode. McKinsey’s State of AI tracks 78% organizational adoption but a small fraction capturing material value. BCG’s AI at Work traces the same pattern: pilots that demo well stall when the quality bar is implicit.
The only artifact that survives a model swap, a prompt rewrite, or a vendor change is the eval threshold attached to each task in scope.
The 6-step method
The method is sequential. Each step depends on the one before it. A founder running it solo on a focused product can finish in an afternoon; a multi-task workflow takes a week.
Step 1 — Capability inventory
List every distinct AI capability the product requires, atomically. One per line. A capability is one thing the system has to do — not a feature, not a screen. For a customer-support reply assistant:
- Classify an inbound message into a category (refund, bug, sales, feedback)
- Detect the sender’s emotional tone
- Retrieve relevant context from a knowledge base
- Draft a reply that matches a brand voice
- Detect out-of-scope messages and escalate
Five capabilities. A scope that says “build a support assistant” hides all five behind a phrase. If a line contains “and” linking two outputs, split it. Atomic capabilities are the unit the rest of the method operates on.
Step 2 — Task taxonomy
Group capabilities into tasks. One task = one eval set. The rule for bundling: a task is one unit if the founder cares about the joint output, not the intermediate steps (retrieve-then-draft becomes one task because the founder grades the final draft).
| Task | Capabilities bundled |
|---|---|
| Classification | classify message into category |
| Tone detection | detect emotional tone |
| Reply drafting | retrieve context + draft reply |
| Escalation | detect out-of-scope + escalate |
Tasks are the unit you can scope cut against — capabilities are too granular, products are too coarse.
Step 3 — Threshold setting
For each task, name the pass-rate the founder will accept as shipped. A single number, set before any code is written.
| Task | Threshold | Rationale |
|---|---|---|
| Classification | 90% | Mis-routing breaks the downstream queue |
| Tone detection | 75% | Used for ranking, not routing; lower cost of error |
| Reply drafting | 80% pass + 0% factual contradictions | Drafts go to a human; factual errors are uncapped cost |
| Escalation | 95% recall on out-of-scope | A missed escalation is the worst outcome |
Three properties make a threshold defensible. It names the metric (pass-rate, recall, contradiction count). It names the asymmetry between tolerable and intolerable errors. It is a number the founder can defend without hedging. “Production-grade quality” is not a threshold. “80% pass on the eval, zero factual contradictions, measured on Claude Opus 4.8 at temperature 0” is.
Step 4 — Eval set authoring
For each task, write 10–30 hard inputs. Real inputs, not synthetic ones — the hardest cases the system will face in the first 90 days of production. For reply drafting:
- A customer wrong about a refund policy but politely framed
- A customer right about a bug but using insulting language
- A bilingual message where the actionable request is in the second language
- A multi-paragraph message where only the last sentence is the ask
- A message referencing a fact the team has never written down
Ten cases is the floor — below that, the variance between a 70% and 90% pass rate is too high to distinguish a real regression from a flake. The eval grows during the build, but the seed set must exist at scoping time. The implementation team cannot author these cases — they do not know which inputs will embarrass the team on day three. A team writing its own eval is grading its own homework.
Step 5 — Scope cut against threshold
Run a feasibility probe. For each task, take a candidate model from the current leading mix, write a baseline prompt, and run the eval. Record the pass rate.
| Task | Threshold | Day-1 baseline | Decision |
|---|---|---|---|
| Classification | 90% | 87% | In scope — closable with prompt + few-shot |
| Tone detection | 75% | 78% | In scope (already over) |
| Reply drafting | 80% / 0% contradictions | 60% / 4% contradictions | Descoped to suggestion mode only |
| Escalation | 95% recall | 92% | In scope — closable with rubric tuning |
Tasks within 5–10 points of the threshold stay in scope. Tasks with a 20+ point gap or a breached safety budget get cut or descoped.
This is the load-bearing step. A scope cut against eval feasibility is defensible: we descoped reply drafting because the day-1 baseline showed 4% factual contradictions against a 0% threshold. A scope cut against opinion is not. Anthropic’s Building Effective Agents makes the same point from the engineering side: complexity is justified by the eval, not intuition.
Step 6 — Acceptance criteria
Write the contract. For each surviving task, name the threshold, the eval, the model, and the deliverable.
Task: classify inbound support messages into one of four buckets — refund, bug, sales, feedback. Eval set: 24 messages from the founder’s last 60 days. Threshold: 90% pass, measured on Claude Opus 4.8 at temperature 0. Failure-mode budget: refund↔bug miscalls ≤ 5%; refund↔feedback miscalls ≤ 0%. Deliverable: classifier, eval harness, baseline report, one-pager describing the rubric.
The vendor runs the eval on day one and reports a baseline. Milestone billing ties to the threshold passed, not to documents shipped. The scope is now falsifiable in both directions. For the spec-side artifact this slots into, see the eval-first PRD.
Worked example — scoping the same idea both ways
Idea: an AI customer-support assistant for a 12-person SaaS company handling ~80 inbound emails per day. The founder wants the assistant to take over triage and drafting so the two-person support team can focus on edge cases.
Scope A — 2018-shape (user stories)
- The system shall classify inbound emails into the support taxonomy.
- The system shall draft a reply to each classified email.
- The system shall escalate out-of-scope messages.
- The system shall match the company’s tone in drafted replies.
- Acceptance: the team reports 80%+ usable drafts in a 2-week pilot.
By week two, the team is prompt-tuning against the founder’s intuition. By week six, the founder pilots it on a real inbox and discovers the assistant fabricates refund-policy details in 8% of drafts. The “acceptance” criterion is unrunnable, and rollback cost at week six is high.
Scope B — 2026-shape (eval threshold)
Task 1 — Classification. Eval: 24 messages, last 60 days. Threshold: 90% pass on Claude Opus 4.8, temp 0. Budget: refund↔feedback miscalls ≤ 0%.
Task 2 — Reply drafting (suggestion mode only). Eval: 18 message-reply pairs hand-graded by the founder. Threshold: 75% pass + 0% factual contradictions against a knowledge-base ground truth. Mode: suggestion only — drafts route to a human-review queue, no auto-send.
Task 3 — Escalation. Eval: 20 messages, half in-scope, half out-of-scope. Threshold: 95% recall on out-of-scope; 90% precision.
Out of scope for MVP: tone matching (no falsifiable measure at this surface size); auto-send mode (factual contradiction risk exceeds budget).
The team runs the eval on day one and reports baselines. By week six, the founder either accepts the threshold or knows precisely which task is short and by how much. Rollback cost is low because the contract is falsifiable. Scope A is four promises with no measurement; Scope B is three contracts with three thresholds. For the polemic that motivates this shift, see stop scoping AI projects in features, scope them in evaluations.
The founder’s role
The method has a clean division of labor. The handoff is at the threshold table.
Founder-owned: capability inventory (Step 1), threshold setting (Step 3), eval set authoring (Step 4). Only the founder knows the workflow, the quality bar that is defensible to stakeholders, and the hard cases the system will see on day three.
Implementation-team-owned: task taxonomy (Step 2), scope cut (Step 5), acceptance phrasing (Step 6). Engineering judgment on bundling, closability, and contract language.
The most common founder mistake is delegating Step 3. A founder who says “the team will tell us what good looks like” has handed the implementation team a blank check — they will set a threshold they can clear, not one the founder will defend. A founder who says “we’ll figure out the eval cases when the system exists” has reversed the order; the eval cases are the spec, and the spec must precede the build. For the explainer behind the eval concept itself, see what are AI evals, in plain English.
The most common scoping mistake
Scoping at the model layer instead of the eval layer. “We will use Claude Opus 4.8 for drafting and GPT-5 for classification” is not scope — it is a vendor decision dressed up as scope. The eval threshold is upstream of the model; the model is one of several knobs the team turns to clear the threshold. A scope that names the model but not the threshold has solved a downstream problem while leaving the load-bearing question — what does shipped mean for this task? — unanswered.
A second common mistake: writing the eval after the team has been prompt-tuning for a week. The eval written then encodes the team’s compromises. Only an eval authored before code begins is one the founder can defend.
Frequently Asked Questions
Who is qualified to run the 6-step method on a product idea? The founder, working with one engineer or technical advisor for Steps 2 and 5. The founder cannot delegate Step 3 (threshold) — that is the founder’s signature.
How long does the method take for a typical MVP scope? An afternoon for a single-task product; a week for a 3–5 task product; two weeks for a multi-agent workflow with 10+ tasks. The bottleneck is Step 4 (eval authoring) — surfacing real domain cases cannot be rushed.
What if the day-1 baseline is far above the threshold? Raise the threshold first. A task where the candidate model clears 95% on a 75% threshold is a task where the threshold was set too low. Raise it to a defensible production number, then re-check.
How does this differ from a regular PRD? The PRD is the document; the scoping method is the decision process that fills the PRD’s eval section. A PRD without the 6-step method behind it can still contain an eval section, but the cases, thresholds, and scope cuts will be guesses. See the eval-first PRD for the document the method feeds into.
What if no model can clear the threshold during the feasibility probe? Three options: narrow the task (split into a sub-task the model can clear plus a sub-task a human handles); lower the threshold to a defensible production minimum; or cut the task from the MVP. Do not lower the threshold to fit the model — that is the failure mode the method is designed to prevent.
Does this work for non-LLM AI features — classification, retrieval, recommendation? Yes. Classification has the cleanest eval shape (input → label). Retrieval evals on top-K recall or rank metrics. Recommendation on engagement-rate uplift over baseline. The structural point — scope cut against a measurable pass-rate — holds across model architecture.
How does the scoping method interact with a fixed-price contract? The acceptance criteria from Step 6 become the contract milestones. Payment ties to eval threshold passed per task, not to documents delivered. Milestone billing against eval thresholds is the only structure that aligns founder and vendor incentives on a probabilistic product.
Can the implementation team run the method on behalf of the founder? Partially — they can run Steps 2 and 5. They cannot run Steps 1, 3, or 4 without the founder; a team that says “we’ll figure out the eval cases later” will write evals matching their build, not your bar.
What is the single biggest sign a scope is broken? A hand-wave word — “production-grade”, “high-quality”, “enterprise-ready” — without a threshold next to it. Each such term lets the implementation team grade their own homework. Replace with a number and a rubric or cut the line.
Closing
A 2026 AI scope is a table of tasks, each with an eval set, a threshold, and a failure-mode budget. The 6-step method makes the table defensible: capability inventory, task taxonomy, threshold setting, eval set authoring, scope cut, acceptance criteria — in that order, no step skipped.
If you are about to commission an AI build and the scope on the table is a feature list, send it back with one sentence: show me the eval threshold per task, or I cannot grade what you ship me.
Get the AI MVP scoping worksheet — a one-page template with the capability inventory, threshold table, and acceptance contract pre-formatted.
Dirk Jan van Veen, PhD
