Most “is this feasible?” questions get answered after fifty thousand dollars of engineering work. The team scopes, builds the harness, wires the model, runs the eval, and discovers four weeks in that the frontier cannot do the task reliably enough. The check that should have killed the project on day one happened on day twenty-eight. A senior engineer can answer the same question in four hours with three chat consoles and ten well-chosen test inputs.
This is the technical companion to the idea validation playbook, a chapter inside the broader idea-to-product manifesto. It zooms into the single step that decides everything: the capability test.
Decision Scope
This is a feasibility protocol, not a production eval suite. It is designed for one senior engineer in an afternoon with no infrastructure. Production AI features require a larger-sample eval pipeline — see the eval-first build playbook. Treat the thresholds here as planning heuristics, not contractual guarantees.
Why a 4-hour feasibility check beats a $50K build sprint
A build sprint that finds a feasibility problem in week four has burned the same capital as a small early-stage round. Two engineers at a loaded four hundred dollars an hour, plus product and design overhead, plus opportunity cost — four weeks compounds north of fifty thousand dollars. A feasibility check that answers the same question in four hours costs the engineer’s afternoon. The asymmetry is not subtle.
Teams skip it because “feasibility check” sounds like infrastructure. It is not. The protocol uses three browser tabs and a spreadsheet. The discipline is in test-set design and rubric writing, not tooling.
Step 1: Name the AI capability explicitly
Every AI feature reduces to one or more of seven capabilities. Name yours first.
| Capability | Example task |
|---|---|
| Extraction | Pull party names, dates, dollar amounts from a contract PDF. |
| Classification | Triage support tickets into routing categories. |
| Generation | Draft an outbound sales email from a prospect dossier. |
| Planning | Decompose “book me a flight” into executable steps. |
| Multi-turn dialog | Coach a salesperson live during a discovery call. |
| Tool-use | Call APIs, retrieve documents, write to a database in a loop. |
| Multi-modal | Read a chart in a PDF, parse a screenshot, transcribe a video. |
The capability matters because the frontier is uneven. As of June 2026: excellent at extraction with small schemas, strong at classification on short clean inputs, fluent but not always factual at generation, good at planning up to four or five steps, mixed at long multi-turn dialog, increasingly competent at tool-use, rapidly improving on multi-modal.
If your idea uses two capabilities, name both. A chained system is upper-bounded by its worst link. Founders who skip this step test the wrong primitive and ship into the right one.
Step 2: Build the 10-input test set
Ten inputs. Small enough to score in one session, large enough to register a curve.
Selection rules:
- Real inputs, not synthetic. Use anonymized examples from the actual data your product will see. Aspirational examples give a false-positive read.
- Cover the distribution. Three easy (clean, short, common). Four medium (some messiness, typical jargon). Three hard (long, ambiguous, the real user inputs you fear).
- Include the failure modes you already worry about. Acronym collisions, OCR artifacts, mixed languages — the hard cases are the test.
- Label expected output in advance. Without this, you will rationalize during scoring.
The common mistake is to fill the test set with the demo. The demo is the easiest case the model will ever see. The medium-hard mix predicts production behavior.
Step 3: Write the rubric — useful, recoverable, dangerous
Numeric pass/fail rubrics hide the failure mode that matters most: how bad is wrong. A model that gets eight of ten right and confidently invents a fact on the other two is dangerous in production. A model that gets six of ten right and obviously refuses on the other four is shippable with a human-in-the-loop fallback. A three-tier rubric distinguishes them; a numeric score does not.
Three lines, one per outcome class:
- Useful. A real user accepts it with light edits or none. Extraction fields correct and verbatim. Classification label matches your own judgment. Generation draft on-spec and on-tone.
- Wrong but recoverable. Wrongness visible to a competent user, detectable by an automatic check, or easy to undo. The model refused, said “I’m not sure,” or failed a simple validator. Recoverable failures cost trust slowly and absorb with UX.
- Wrong and dangerous. Confidently incorrect in a way the user will not catch. Invented contract clause. High-risk ticket labeled low-risk with no signal of doubt. Convincing prose with a factual error. Dangerous failures cost trust instantly.
Write the rubric before running any inputs. Three sentences, tied to the capability and test set.
Step 4: Run the test against three frontier chat consoles
Three browser tabs:
- claude.ai with Claude Opus 4.8.
- chatgpt.com with GPT-5.
- gemini.google.com with Gemini 2.5 Pro.
These are the three frontier chat-console offerings as of June 2026. Run all three. The leaderboard shifts quarter by quarter and different capabilities have different leaders. The model you assume will win frequently does not.
Send the same prompt for each of the ten inputs to all three consoles. Keep it clean — one paragraph of instruction, the input, one sentence on output format. Do not multi-shot, do not paste untested system prompts. The check should reflect what the model can do with a reasonable first prompt, not what a prompt engineer can extract after a week of tuning. If the floor is shippable, the ceiling is much higher with engineering investment.
Thirty cells in a spreadsheet. At ninety seconds per cell, ninety minutes including re-runs.
Step 5: Score and tally
Label each cell useful, recoverable, or dangerous. No half-credit. Tally per model:
| Model | Useful | Recoverable | Dangerous |
|---|---|---|---|
| Claude Opus 4.8 | 7 | 2 | 1 |
| GPT-5 | 6 | 3 | 1 |
| Gemini 2.5 Pro | 5 | 3 | 2 |
The headline number is useful out of ten. The disqualifying number is dangerous out of ten. A model with a higher useful count but more dangerous failures may be the worse choice — dangerous failures break products and erode trust fast.
Two discipline rules: score independently before you discuss (group discussion drifts toward consensus optimism), and score as a user, not as the model’s friend (founders rationalize; users do not give credit for being close).
Step 6: Read the curve
Three tally patterns matter. Use them as decision rules.
7/10 useful — shippable with eval discipline
Seven or more useful and one or fewer dangerous means the idea is shippable. The gap closes with prompt engineering, retrieval, light fine-tuning, and a production eval suite. Every quality investment compounds against a working base. Ideas at 7/10 typically reach 9/10 by month three of eval-driven development.
5/10 useful — research project
Five or six useful with the rest split is the hardest zone. Feasible enough to be tempting, unreliable enough to be expensive. Closing the gap requires real engineering: retrieval, a fine-tuned model on top of the frontier, an eval harness with thousands of labels, or an architectural redesign that breaks the task into sub-tasks. Twelve weeks turns into twenty-four. If your runway can absorb that, proceed. If not, narrow scope.
Below 5/10 — wait or pivot
Four or fewer useful, or more than two dangerous, is below the bar. Two paths:
- Narrow the scope. Find a sub-task inside the idea that scores 7/10 and rebuild around it. Most successful AI products are scoped at the narrowest version that hits the bar, not the maximally ambitious vision.
- Wait. The frontier improves fast. Re-run in three months.
The path you do not want is building anyway because the founder has announced the product or fallen in love with the demo. The feasibility check protects against this — but only if you act on it.
Step 7: Write the one-page feasibility memo
The memo is the deliverable. One page, five sections.
- Idea. One sentence: what the AI does, for whom, when.
- Capability. The primitive(s) from step one. If two, order and dependency.
- Test set. A line per input — easy/medium/hard split, what the failure cases looked like.
- Results. The score table from step five plus one paragraph on where dangerous failures cluster.
- Decision. Shippable, research project, or wait/pivot — plus one sentence of rationale.
The team in week one of build references this document. The eval suite that contracts the engagement is seeded from the test set. The PRD that follows (the AI PRD explained) quotes the decision as its scope constraint. A serious AI build partner will not engage without something like this on the table — the editorial-600 essay stop scoping AI projects in features, scope them in evaluations makes the case in detail.
Worked example: a support-ticket classifier
A founder with a customer-support background wants an AI that classifies inbound tickets into twelve routing categories (billing, refund, account access, integration bug, feature request, churn risk, abuse, etc.) for a fifteen-person support team.
Step 1 — capability. Classification, fixed twelve-label space.
Step 2 — test set. Ten anonymized tickets from a previous-company export. Three easy (single clear topic). Four medium (two topics in one ticket, customer venting, ambiguous wording). Three hard (churn risk disguised as a feature request, refund buried in an integration complaint, a non-English ticket).
Step 3 — rubric.
- Useful. Correct label, or closest correct label when two apply.
- Recoverable. Wrong label, low confidence or “other” / “unclear” sentinel returned.
- Dangerous. Wrong label, high confidence, misroute delays response to a high-stakes ticket.
Step 4 — run. Same one-paragraph prompt to all three consoles, label space included verbatim. Thirty-cell grid in ninety minutes.
Step 5 — score.
| Model | Useful | Recoverable | Dangerous |
|---|---|---|---|
| Claude Opus 4.8 | 8 | 1 | 1 |
| GPT-5 | 7 | 2 | 1 |
| Gemini 2.5 Pro | 7 | 1 | 2 |
Claude’s dangerous failure was a churn-risk ticket misrouted to feature-request with no expressed uncertainty. Gemini’s two were similar. GPT-5 mislabeled the non-English ticket.
Step 6 — read. Claude Opus 4.8 scores 8/10 useful, 1 dangerous. Shippable. The dangerous failure mode is mitigable with a parallel urgency-signal classifier. Recoverable failures absorb with a confidence-threshold “send to human” fallback.
Step 7 — memo.
Idea. An AI that classifies inbound support tickets into twelve routing categories for a fifteen-person support team. Capability. Classification, twelve-label space. Test set. Ten anonymized tickets, three easy / four medium / three hard. Hard cases: churn-disguised-as-feature-request, refund-buried-in-integration, non-English ticket. Results. Claude Opus 4.8: 8/1/1. GPT-5: 7/2/1. Gemini 2.5 Pro: 7/1/2. Dangerous failures clustered on churn-risk tickets dressed as feature requests. Decision. Shippable. Build against Claude Opus 4.8 as default. Add a parallel urgency-signal classifier and a confidence-threshold fallback. Twelve-week MVP plausible; the test set seeds the eval suite.
Four hours of senior-engineer time. The PRD writes itself in a day. A build partner reads the memo, asks three follow-up questions, and scopes a fixed-fee engagement against the eval thresholds.
FAQ
How is this different from running a public benchmark?
Public benchmarks (MMLU, GPQA, SWE-Bench Verified, AIME 2025) report average performance on a standardized task. They do not tell you whether a model can do your task on your inputs. A model can be SOTA on a public benchmark and fail on yours because your inputs are longer, messier, or in a domain underrepresented in the benchmark.
Why ten inputs and not fifty?
Ten is the smallest that registers a curve and the largest one engineer can score carefully in one session. The point is rapid go / no-go signal, not statistical significance.
Why three models and not just one?
Different capabilities have different leaders, and the leaderboard moves quarter by quarter. Running Claude Opus 4.8, GPT-5, and Gemini 2.5 Pro in parallel takes thirty extra minutes and hedges against picking the worst fit. Divergent scores give model-selection guidance for the build.
Should I include open-weight models like Llama 3.3?
Not for the feasibility check — the question here is whether the closed frontier can do the task at all. Llama 3.3 is typically a notch below. Once feasibility is established, it re-enters as a cost lever during the build phase.
What if my idea requires tool-use or agentic behavior?
For agentic tasks, use the providers’ agent-mode interfaces and score on end-to-end task completion, not single-prompt quality. A useful score requires the full plan to execute with no human intervention. Agentic feasibility scores are usually two or three points lower on the same scale.
My test scored 6/10. What now?
Three options. Narrow scope until the version scores 7+. Budget a six-to-nine-month build with retrieval and fine-tuning. Or re-test in three months — 6/10 is the zone most likely to flip with the next frontier release.
How do I tell a recoverable failure from a dangerous one?
Would a competent user catch the failure without external information? If yes, recoverable. If they would believe the wrong output and act on it, dangerous. A confident, plausible wrong answer is dangerous. A confused, partial, or “I’m not sure” wrong answer is recoverable.
Can I delegate this to a junior engineer?
The protocol can be executed by a junior engineer. The judgment cannot. Test-set selection, rubric writing, and result interpretation need a senior eye. Mechanical scoring is fine to delegate.
Does this replace the full validation playbook?
No. Feasibility is the technical question. The full idea validation playbook also covers demand validation and the AI-lift-vs-wrapper question. Feasibility is one chapter, and the most expensive one to skip. See the sibling pre-prompt validation guide for the end-to-end procedure.
Next step
The feasibility memo is the bridge between idea and engineering. With a go decision, the next deliverables are the PRD (what an AI PRD looks like in 2026) and the eval-first build scoping (the eval-first build playbook). The idea-to-product manifesto sets the pipeline in context. If you have a draft memo and want a second opinion before committing, that conversation is what our idea-review sessions are for.
Dirk Jan van Veen, PhD
