Sizing AI features by story points is the cheapest way to mis-budget an AI project in 2026. Story-point sizing prices the code. AI features cost what the eval costs, and the eval is most of the engineering. A “large” AI feature is a six-week eval construction — a scored rubric, an LLM-as-judge, a regression harness — sitting on top of a model call you wrote in an afternoon. Sizing by code time hides the actual project.
This article gives non-technical founders a small / medium / large sizing rubric denominated in eval engineering cost. Three tiers, three eval shapes, three illustrative build-cost ranges, and decision rules for when to size up or down. It sits inside the broader idea-to-product manifesto, pairs with the eval-first build playbook (scope by evals) and the eval rubric template (rubric format). The sizing framework tells you how big each rubric should be.
Decision Scope
This article is an editorial sizing framework, not legal, financial, or accounting advice. Dollar ranges, prompt counts, and tier thresholds are illustrative planning heuristics drawn from published vendor pricing and the public eval literature. Validate the numbers against your own contracts and budget model before signing a build.
Why story-point sizing breaks for AI features
Story-point sizing is a vocabulary from deterministic engineering. Atlassian’s agile estimation guide defines a story-point as the effort to write the code that satisfies the acceptance test. That definition presupposes the acceptance test is cheap to write. For OAuth, it is. For an AI feature, it is not.
An AI feature has two engineering budgets that story-point framing collapses:
- The cost of producing the candidate output — one model call on a frontier API, often a few lines of code. This is the part that looks like “the feature.”
- The cost of producing the eval suite that proves the output is good enough to ship — prompt sets, anchored rubrics, judge calibration, regression harnesses. This is most of the project.
Story-point sizing reads the first budget and hides the second, which is how AI MVPs get quoted at $30K and overrun to $120K. The editorial argument for budgeting AI projects in eval runs develops the broader case; this article gives the per-feature rubric.
The fix is to denominate sizing in eval cost. A feature is small if a small eval suite proves it works, large if a large suite is required. The model call is incidental. The eval is the unit of work.
The three eval shapes that define the tiers
Three eval shapes recur across the literature — Eugene Yan’s practitioner writing, Hamel Husain’s eval posts, Anthropic and OpenAI eval docs, the G-Eval paper, MT-Bench, and the Shankar et al. work on rubric validation. The sizing framework maps onto those three shapes directly.
Shape 1 — 10 prompts + binary rubric (Small). Ten representative inputs, each with a single pass/fail criterion. “Correct tag, yes or no.” “Correct JSON shape, yes or no.” No anchored scale, no judge model, no regression harness — a Python script and a CSV is enough. Right shape for narrow, observable, well-defined tasks where correctness is binary and the failure mode is bounded.
Shape 2 — 30 prompts + scored anchored rubric (Medium). Thirty curated inputs graded on a scored rubric with anchored examples per score (binary or 3-point per axis, per G-Eval and MT-Bench). Two or three quality axes — accuracy, format adherence, refusal behaviour — each scored independently. A human grades the set, rubric anchors are tight enough that two graders agree above 0.85, and the suite runs in CI on every model or prompt change. No LLM-as-judge yet.
Shape 3 — 100+ prompts + LLM-as-judge + regression harness (Large). 100 or more prompts across sampled difficulty bands, graded by a calibrated LLM-as-judge whose outputs have been spot-checked against the human rubric until agreement passes 0.85. A regression harness runs in CI on every PR, captures per-axis scores, and gates merges against a delta threshold. A model-update clause specifies a re-calibration window when the provider releases a new checkpoint.
These shapes are not arbitrary. They are what survives contact with real engagements. A founder reading a proposal for a “large” AI feature should be able to find shape 3 inside the eval section. If it is not there, the feature has been mis-sized — usually downward.
The three tiers, side by side
The same three shapes, presented as a sizing rubric a founder can read in under a minute:
| Tier | Eval shape | Example feature | Build-cost range (illustrative) | When to use this tier | Ship-gate threshold |
|---|---|---|---|---|---|
| Small | 10 prompts + binary rubric | Customer-support tag classifier, single-table SQL generator, intent detector | $5K–$15K | Narrow, observable, well-defined task. Single output type. Correctness is binary. Failure mode is bounded. | 95% pass rate on the 10-prompt set, re-run on each prompt change. |
| Medium | 30 prompts + scored anchored rubric (binary or 3-point per axis) | Structured extraction from semi-structured PDFs, content generation with format constraints, multi-tag classification, RAG QA over a single corpus | $15K–$40K | Multi-axis quality, format constraints, partial credit matters. Inter-rater agreement above 0.85 on the rubric. | 0.85+ average across axes, no axis below 0.80, on the 30-prompt set in CI. |
| Large | 100+ prompts + LLM-as-judge + regression harness | Multi-step research agent, conversational tutor, regulated-domain extraction (legal, medical, financial), tool-calling agent with > 3 tools | $40K–$120K | Regulated domain, multi-step trajectory, drift-sensitive, or production traffic above ~1K requests/day. | 0.90+ on each axis with judge-human agreement above 0.85, regression delta < 0.02 per axis on every PR, model-update re-calibration clause active. |
The ranges cover eval engineering, model and judge inference across the build period, integration, and regression infrastructure. They exclude product, design, and full-stack costs priced elsewhere in the build. The AI MVP economics playbook gives the project-level roll-up.
Size up: three triggers
A founder who sizes a feature one tier too small ships an under-evaluated system. Three triggers justify sizing up:
Regulated or safety-critical domain. Legal, medical, financial, insurance, or any domain where a wrong output triggers a regulatory consequence. Sample sizes of 10 or 30 prompts cannot detect a 2% hallucination rate on a safety-critical axis. Move up one tier. A “small” intent detector for a legal-extraction product is a Large eval suite once the safety axis is in scope.
Model-drift sensitivity. Any feature whose quality swings on a provider’s silent model update — RAG retrieval, structured extraction at format boundaries, agentic tool-calling, long-context reasoning. If the feature runs unattended for more than six months, the regression harness from tier 3 is the price of preventing a silent quality drop. Move up one tier.
Multi-step agent or trajectory. Any feature with more than two tool calls, a planning step, or stateful conversation. Trajectory grading is its own eval shape — partial credit, ordering, recoverability after a tool error. A medium rubric cannot grade a multi-step trajectory honestly.
Size down: three triggers
The opposite mistake — sizing one tier too large — inflates the SOW for a feature that does not need the harness. Three triggers justify sizing down:
Research-grade demo, not production. A prototype shown to investors or an internal team with no live user-facing deployment. The eval can be a smoke test. A medium-feature demo can ship at the small-tier shape, with the explicit understanding that production hardening means moving back up.
Single-user prototype. Founder-built workflow tool for the founder’s own desk, or an internal ops tool with a single named user. The eval is the user, running the tool. The risk is bounded — a single-user tool that breaks is a Slack message, not a customer incident.
Deterministic post-LLM step. Features where the LLM produces a candidate and a deterministic step verifies it before the output reaches the user. SQL generation with a query-plan validator. Structured extraction with a JSON-schema validator. Code generation with a unit-test harness. The validator absorbs most of the eval burden — a tier-3 LLM-as-judge collapses to a tier-1 binary check on the validator output.
The trigger pattern is symmetric. Trigger one in each direction reflects domain risk, trigger two reflects deployment surface, trigger three reflects architectural shape. Three sizing-up signals at once means the feature is Large no matter what the implementation looks like.
Worked example: sizing three founder feature requests
The framework is most useful applied to concrete feature requests. Three illustrative cases.
Feature A — Customer-support email tag classifier. A support-ops founder wants to route incoming emails into one of twelve canonical tags. One tag per output, mis-routing is contained within an internal team, the domain is not regulated, and tags are reviewed by a human on every triage pass.
This is Small. Ten prompts (one or two per tag, plus edge cases), a binary rubric, a CSV scoring script, ship-gate at 95% pass rate. Build cost in the $5K–$15K range, dominated by the email-queue integration rather than the eval.
Feature B — Invoice line-item extraction from PDF. An AP-automation SaaS founder wants to extract vendor name, date, line items with descriptions and amounts, and total from arbitrary vendor PDFs. Output is structured JSON, a wrong number costs more than the feature, format adherence matters, and PDFs vary across hundreds of templates.
This is Medium. Thirty representative PDFs sampled across vendor template clusters, a three-axis scored rubric (line-item accuracy, format adherence, total reconciliation), ship-gate at 0.85 average with no axis below 0.80. A deterministic post-step (line-items sum to extracted total within rounding) absorbs part of the eval burden, but format-adherence and line-item-accuracy axes still require human grading. Build cost in the $15K–$40K range.
Feature C — Clinical research agent. A digital-health founder wants an agent that takes a clinical question, searches a curated literature corpus and trial registry, synthesises a cited answer, and refuses when evidence is insufficient. Multi-step trajectory, regulated domain, drift-sensitive across provider updates, projected traffic above 1K queries/day in year one.
This is Large — three size-up triggers fire. The eval needs 100+ prompts across clinical question types and difficulty bands, an LLM-as-judge calibrated against an in-house clinical reviewer until agreement passes 0.85, axes for citation grounding / refusal behaviour / synthesis faithfulness / retrieval recall, and a CI regression harness gating merges and tracking per-axis deltas. Build cost in the $40K–$120K range, with the eval suite the dominant line item.
A PRD that wrote “AI extraction” three times produces three different feature shapes once the framework is applied. The framework makes those differences legible at the scoping table, before a vendor quote anchors on the wrong tier.
What this framework is not
The framework is a per-feature sizing rubric. Not a project plan, not a roadmap, not a pricing sheet. Three explicit boundaries:
It does not price the project. Per-feature build-cost ranges roll up to a project budget once you add discovery, product and design, integration, and infrastructure. Use the AI MVP economics playbook for the project-level number.
It does not replace the eval rubric. The framework tells you how big to make the eval; the eval rubric template tells you how the rubric is structured inside each tier. Both are needed.
It is not a one-shot exercise. Features get re-sized as the engagement learns. A feature that looked Small at kickoff and turned out to be Medium after the first eval sprint is normal — the sizing exercise has done its job by surfacing the mis-classification early.
How to use this in a vendor conversation
Three practical uses:
Ask the vendor to size each feature against this rubric in writing, before the SOW is drafted. A vendor who returns “Large feature, 25 prompts, binary rubric” is under-quoting the eval — that conversation belongs before the contract is signed.
Require the eval shape to be named in the SOW per feature. “Tier S — 10 prompts + binary rubric, ship-gate 95%” or “Tier L — 100+ prompts + LLM-as-judge + regression harness, ship-gate 0.90 / axis.” If the SOW only contains feature names, the eval cost is hidden. The companion editorial on scoping AI projects in evaluations rather than features develops the contract-language version.
Re-run the sizing at the discovery-week artifact gate. A discovery week ends with a written sizing per feature, signed by the founder and the technical lead. If the sizing changed from kickoff, record what moved it and why.
The payoff sentence: you size the eval, not the feature. Founders who internalise that stop arguing about feature scope and start arguing about eval scope, which is where the money actually sits.
FAQs
How do I know which tier my feature is in? Start by writing the eval. If a person on your team can hand-grade ten representative inputs in a morning with a binary rubric and feel confident, the feature is Small. If a scored rubric across two or three axes on thirty inputs is needed before two graders agree, the feature is Medium. If a production-grade harness with an LLM-as-judge and regression infrastructure is needed before you would put the feature in front of paying users, the feature is Large.
Who sizes the feature — the founder or the vendor? Both, separately, then reconcile. The founder sizes from the domain side (“how bad is a wrong answer?”) and the vendor sizes from the engineering side (“how big is the eval suite that proves this works?”). Mismatches are the most useful signal you will get during scoping — if the founder sizes a regulated extraction as Small and the vendor sizes it as Large, that conversation belongs before any code is written.
What if the vendor disagrees with my sizing? Ask the vendor to write down the eval suite they propose at their tier — prompt count, rubric grain, judge setup. Then ask whether that suite defends against the failure mode you care about (regulatory miss, drift-sensitive output, multi-step trajectory failure). The disagreement usually collapses to a specific eval line item, which is a cheaper argument than feature scope.
What changes when the model upgrades? At Tier S, re-run the ten-prompt suite by hand. At Tier M, re-run the thirty-prompt suite in CI and check no axis dropped by more than the change-control threshold. At Tier L, the regression harness runs automatically, the judge is re-calibrated on a sampled subset, and a one-week re-calibration window precedes re-locking thresholds. Provider model updates are the dominant ongoing cost for Tier L features.
Can a single eval suite cover several features? Sometimes, when the features share a task shape — three classifiers across three label sets can share one binary-rubric harness. But prompt sets and rubric anchors are still per-feature. The economy of scale is in the infrastructure (CI, judge, regression hooks), not the rubric. Sharing across tiers is usually a mistake.
How does this map to overall project cost? Add the per-feature ranges, then layer discovery, product and design, full-stack integration, deployment infrastructure, and a contingency for at least one feature being re-sized upward. A typical $80K six-week MVP build sits at roughly two Small features and one Medium feature, with discovery and integration absorbing the rest.
Is “extra-large” or “extra-small” ever a tier? The three-tier rubric is deliberate. Features that look “extra-large” are usually two Large features stacked (research agent plus citation-grounding component); features that look “extra-small” are usually deterministic features with a wrapper LLM call. Decompose them and the sizing reads cleanly.
Does this framework apply to fine-tuning or RAG? Yes, with one adjustment. Fine-tuning and RAG features are sized on the eval, not on the training-data preparation or the retrieval-pipeline build — those are deterministic engineering with their own estimation methods. The eval (does the fine-tuned model actually answer better, does retrieval recall meet target?) is what sorts the feature into S, M, or L.
What if my feature does not fit any tier? Either it is two features in a trench coat — decompose it — or it is research, not engineering. Research-grade features (open-ended generation, novel agent architectures, multimodal reasoning where the eval shape is unknown) need a discovery phase before sizing. The cleanest move is a paid one-to-two-week discovery sprint whose deliverable is a written sizing, which is what the eval-first build playbook recommends as the default first artifact.
Next step
The AI MVP Scoping Worksheet packages this framework as a one-page rubric you can hand to a vendor — three feature slots, three eval shapes, three cost ranges, and a sizing-up / sizing-down checklist. Download the worksheet and bring it to your next vendor call.
Dirk Jan van Veen, PhD
