Every AI feature in a 6 to 12 week build hits a scope-test moment in week 3 or 4. The founder looks at a half-finished feature and decides three ways: cut to MVP-2, ship as-is, or extend the build. Cut a feature one week from shipping and the product loses its anchor. Ship a feature two weeks from a viable pass-rate and customers churn on first touch. Extend a build that should have been cut and the engagement drifts from 8 to 16 weeks, blowing the hardening budget. The right answer is not a gut call. It is the output of five measurable signals read against named thresholds.
This article names the signals, the threshold pattern for each, the founder’s role in reading it, and what each color triggers. It builds on the idea-validation playbook and the idea-to-product manifesto.
Decision Scope
An editorial framework, not legal, security, or procurement advice. Thresholds are heuristic — derived from patterns we see in 6–12 week AI builds, not from a published benchmark. Calibrate the bands against your customer tolerance and risk budget before acting.
Why week 3 to 4 is the scope-test moment
A 6 to 12 week AI build has a predictable shape. Weeks 1 to 2: validation and PRD. Week 3: first integration cut — model behind the actual product surface, end-to-end, on real data. Week 4: first honest read. By week 5 the team should be hardening; by week 6 a customer should be looking at it. The scope-test lives between the first honest read and the first hardening sprint.
Before week 3 the founder lacks real data — synthetic-prompt pass rates do not predict customer-input pass rates. After week 5 the calendar runs out: a 10-week build that cuts a feature in week 6 has burned half the budget on something the customer never sees. The model layer is also moving underneath — Anthropic’s Claude Opus 4 to 4.6, OpenAI’s GPT-5 to 5.4, and Google’s Gemini 2.5 Pro each silently changed behaviour for systems pinned to a model alias (Artificial Analysis LLM Leaderboard). Run the scope-test at week 3 to 4 and you read against the model the product will ship on.
Five signals. Run them in order. The signal you are tempted to skip is the one carrying bad news.
Signal 1: Eval pass-rate
Whether the feature passes the eval set you wrote on day 1, at the threshold you committed to. If you did not write an eval set on day 1, this signal cannot fire — read stop scoping AI projects in features, scope them in evaluations first.
| Color | Pass-rate vs day-1 threshold |
|---|---|
| Green | At or above the committed threshold on the frozen eval set |
| Yellow | 5 to 15 points below threshold, above honest baseline |
| Red | More than 15 points below, or below baseline |
The founder asks the engineer three questions. What was the day-1 threshold, and the pass rate today? Has the eval set been touched since day 1? A drifting eval set is the most common silent failure — the team softens the test until the pass rate looks green. What is the variance across customer-shaped inputs? A 90 percent rate on uniform inputs and 40 percent on outlier customers is a yellow pretending to be green.
Green moves on. Yellow extends with a cap — one to two weeks of focused work on the failure cluster, then re-run; if still yellow, ship with explicit fallback (Signal 4) or move to MVP-2. Red is a hard cut. Founders who push past a red blow 60 percent of the budget on a feature the customer rejects in week 8.
Signal 2: Cost-per-task
Whether unit economics work at month-one load. Cost-per-task is fully-loaded inference cost — input plus output tokens plus retries plus retrieval — divided by one successful task, not one call. A 70 percent pass-rate feature pays for its failures.
| Surface | Pattern |
|---|---|
| Consumer (sub-cent target) | Green at or under budget; yellow up to 2x; red above 3x |
| B2B (cent-to-dollar target) | Green at or under budget; yellow up to 2x; red above 3x |
| Internal tool | Green if it does not dominate variable cost; red if it does |
A red at week 4 almost always means the wrong model tier. The fix is rarely “optimise the prompt” — it is to route frontier calls to mid-tier where the eval permits, or to cache a step. Ask: what is cost-per-task today, what is the budget, what tier is the call on? Top tier for every call: red is usually recoverable by routing. Already on the lowest tier and still over budget: structurally too expensive, MVP-2. Deeper reasoning lives in the AI MVP economics playbook.
Signal 3: Latency at the user surface
Whether the latency budget fits the user surface. A 3-second response is fine in chat, bad in autocomplete, irrelevant in a nightly batch.
| Surface | Green | Yellow | Red |
|---|---|---|---|
| Autocomplete / inline UI | Under 500 ms p95 | 500 ms to 1.5 s | Over 1.5 s |
| Chat / Q&A | Under 3 s p95 | 3 to 8 s | Over 8 s |
| Form-fill / on-demand | Under 8 s p95 | 8 to 20 s | Over 20 s |
| Bulk / batch | Under 60 s per task | 60 to 180 s | Over 180 s |
Ask for p95, not the median — average latency hides the long tail customers complain about. A red is sometimes recoverable in-build by streaming tokens (a 6-second answer that starts streaming at 0.4 seconds reads as fast), paralleling tool calls, or moving non-critical steps to background. Otherwise: wrong model tier, wrong context strategy, or wrong surface — MVP-2 or change the surface.
Signal 4: Failure-mode recoverability
Whether failures are recoverable or trust-damaging. Founders skip this signal most often — they read pass-rate as the quality bar and forget that a 90 percent pass-rate with hallucinated-clause failures is a worse product than an 80 percent pass-rate with “I’m not sure, can you confirm” handoffs.
| Color | Failure shape |
|---|---|
| Green | Failures are detected, produce graceful handoff, do not damage trust |
| Yellow | Failures partly detected; some leak through as visible weirdness, not silent error |
| Red | Failures are undetected, silent, trust-damaging — wrong answers presented as confident answers |
A red here is the most expensive failure mode in AI products and the one most likely to convert paying customers into churning customers. The fix is a fallback — a deterministic step that catches the model’s uncertainty and routes to a human, a different model, or a safe default. The deeper argument lives in why most AI MVPs ship the wrong thing first. A feature that cannot be wrapped in a recoverable failure path should be cut to MVP-2 regardless of pass-rate.
Look at five recent failures and ask: what did the user see? Did the system know it had failed? What was the recovery path? Three confident-wrong answers in five is red. Three “I’m not sure” handoffs in five is a green failure shape even at a low pass-rate.
Signal 5: Build-cost-to-finish
Engineering’s honest estimate of engineer-weeks remaining to reach day-1 thresholds on Signals 1 through 4. Not features. Not story points. Engineer-weeks to threshold.
| Build-cost-to-finish | Runway remaining | Reading |
|---|---|---|
| 1 week or less | 4 plus weeks | Green — finish |
| 1 to 2 weeks | 3 plus weeks | Yellow — extend with a cap |
| 3 plus weeks | Variable | Red — cut, unless the feature is the product |
The most easily misread signal. Engineering defaults to optimism; founders to sunk-cost. Both push past a red. A 3-week estimate at week 4 in a 10-week build ships in week 7 and gets one week of hardening — too little. Cut, freeze the half-built feature on a branch, ship what is shippable. Ask for the estimate in days, and ask what could push this from 1 week to 3 weeks? That answer is usually the failure mode the team has not yet faced.
Reading the five signals together
| Pattern | Decision |
|---|---|
| All green | Ship as scoped |
| 4 green, 1 yellow | Extend with cap on the yellow; ship after re-test |
| 3 green, 2 yellow | Ship with explicit fallback path on the two yellows |
| 1 red on quality, latency, or failure-mode | Cut to MVP-2 |
| 1 red on cost or build-cost-to-finish | Recoverable in-build only with a named, dated fix |
| 2 reds | Cut to MVP-2, no exceptions |
The “ship with explicit fallback” pattern is the option founders most often miss. Not “ship broken” — “ship with the product surface designed around the yellow signal”. Yellow on latency ships with a streaming token UI and a “this may take a moment” affordance. Yellow on pass-rate ships with a confidence indicator and an easy correction path. Yellow on failure-mode ships with a human-in-the-loop confirmation step on the failure cluster. The fallback makes the yellow honest to the customer and protects against the trust-damage an undisclosed yellow creates.
If the rubric exposes a missing eval set, missing cost budget, or no named user surface, the scope-test is telling you the day-1 work was incomplete. Go back to it.
Worked example: contract-review at week 4
A two-person team, four weeks into an eight-week build of a contract-review feature for a vertical SaaS. MVP-1 surface: highlight clauses that deviate from the customer’s standard template, one-line explanation per highlight. Integration just landed; eval data is in; the founder is at the sprint review.
Signal 1, pass-rate. Day-1 threshold 85 percent correct-clause-flagged on a frozen 100-contract eval. Today: 78. Yellow — within band, above a regex baseline of 31.
Signal 2, cost-per-task. Budget 35 cents per contract; today 41 cents (Claude Opus 4.8 on long sections, Sonnet 4.6 on shorter, retries on flagged clauses). Yellow — within 2x of budget.
Signal 3, latency. On-demand form-fill surface; p95 14 seconds. Yellow — recoverable by streaming per-clause highlights.
Signal 4, failure-mode. Five recent failures: three under-flags (missed deviations, no false flag invented), two over-flags (clauses standard for the customer’s template). Under-flagging is visible; over-flagging is a one-click dismiss. Green failure shape.
Signal 5, build-cost-to-finish. Push pass-rate to 85: one focused week on clause-classification prompts. Push latency to 8 seconds: parallel the classification calls, half a week. Total 1.5 weeks. With 4 weeks of runway, green-yellow border.
The decision. Three yellows, one green, one green-yellow border, zero reds. Ship with explicit fallback on the two most operationally consequential yellows — latency (stream per-clause results) and pass-rate (confidence indicator, easy-dismiss). The team extends 1.5 weeks for focused quality work, ships at week 5.5 with two weeks of hardening. Cost yellow accepted — within bracket, model-routing on the MVP-2 backlog.
Reading three yellows as red and cutting loses a product 1.5 weeks from quality. Reading them as green and shipping at week 6 with no fallback gives the customer a slow, occasionally-wrong feature with no affordance for the failure mode — permanent trust damage. The rubric protects against both errors.
Frequently asked questions
No eval set on day 1? You cannot run Signal 1. Stop the scope-test and run eval-design first — one to two engineer-days. Without an eval set every other signal lacks a denominator. The argument lives in stop scoping AI projects in features, scope them in evaluations.
Can I skip a signal that “obviously does not apply”? Only Signal 3 latency, for true offline batch jobs where the user is not waiting. Every other signal applies to every AI feature, including internal ones. The signal you are tempted to skip is the one whose threshold you have not committed to.
My engineering team says these thresholds are too strict. They are heuristic. Calibrate against customer tolerance — legal, medical, or financial surfaces need tighter green bands. The wrong move is softening thresholds to turn the signal green. Write down the calibrated threshold on day 1 and read against that.
Founder and engineer disagree on a signal color? Common case: engineering reading a yellow as green (“close enough”). Ask the threshold question in writing — day-1 commitment, current value, where it falls. If the engineer cannot answer those three on the spot, the signal is not yet readable.
Does this rubric work for agentic features? With one modification. Agentic features compound error — 95 percent per-step compounds to roughly 60 percent across ten steps. Signal 1 reads end-to-end task success on the frozen eval, not per-step. Signal 4 must include “the agent recognises when it has gone wrong mid-plan and stops”.
What if Signal 5 says 1 week and I am 1 week from end of contract? Green on build-cost-to-finish but red on engagement structure. Ship, then negotiate a hardening extension framed around what the scope-test surfaced. A feature with two weeks of hardening is a product; with none, a liability.
How does this interact with the day-1 PRD? The PRD names the thresholds the scope-test reads against. A PRD without a pass-rate threshold, cost budget, latency target, failure-mode policy, and eval set makes the scope-test impossible to run. Fix the PRD — the idea-validation playbook covers what a defensible PRD looks like.
What about a yellow the team estimates is unrecoverable? A yellow the team cannot move to green in remaining runway is functionally a red. The politeness of “yellow” hides a structural problem. Cut to MVP-2 and ship the workflows that went green.
Is there a version of this rubric for model migration? Yes — add a “compatibility against incumbent” signal reading whether the migrated model preserves behaviour the eval set was tuned around. The five signals still apply.
Closing
The scope-test is not when the founder is supposed to know everything. It is when five measurable signals are stable enough to read honestly. Read each against its day-1 threshold and let the pattern produce the decision — do not backfill the decision the team already feels like making.
If you are mid-build, the AI MVP scoping worksheet packages the five signals into a one-page artifact for your next sprint review. Upstream: the smallest-possible-AI-feature framing. Downstream: the AI MVP economics playbook.
Read the signals. Ship the green-yellows. Cut the reds. The scope-test will be wrong sometimes. Founders who run it consistently still ship at quality more often than those who do not.
Arthur Wandzel
