Production-ready AI is not a feeling — it is a seven-criterion artifact list a buyer can demand before signing the SOW: an eval pass-rate threshold met on a frozen test set, a fallback path for every named failure mode, an observability dashboard with a named on-call engineer, a pinned model version, a written failure budget, a runbook for the top-10 known issues, and a 30-day post-launch on-call window inside the contract. A vendor who cannot produce those seven artifacts is shipping a demo, not a product. By the end of this piece you will read any “production-ready” claim as a list of seven verifiable items and know the cost of skipping each one.
It builds on the eval-first build playbook and the idea-to-product manifesto. The vendor-side companion is decoding production-ready in AI agency proposals; this is the buyer-side definition.
Table of Contents
- Why “the demo worked” is not a launch criterion
- The seven criteria at a glance
- 1. Eval pass-rate on a frozen test set
- 2. Fallback path for every named failure mode
- 3. Observability dashboard + named on-call engineer
- 4. Model version pinned
- 5. Failure budget agreed in writing
- 6. Runbook for the top-10 known issues
- 7. 30-day post-launch on-call window
- Retrofit cost when these are missing at launch
- Frequently Asked Questions
- Closing: ask for the artifact, not the assertion
Why “the demo worked” is not a launch criterion
The standard form of the production-readiness mistake is one sentence in a vendor email: “the demo worked end-to-end, we’re ready to ship.” A demo runs the happy path on inputs the team has seen. Production is the long tail of inputs the team has not seen, sent at hours when nobody is watching, against a model whose behaviour the vendor does not own.
Production-ready is a set of artifacts. Each closes a specific failure mode. If the artifact does not exist, the failure mode is open — a customer hits it, the system answers wrong with confidence, the founder finds out from a complaint thread on Tuesday afternoon.
This is the definitional explainer. How to build evals: what are AI evals, in plain English. How to scope the build: the eval-first build playbook. This piece names the seven artifacts that prove a build is past the dev-to-production line, so you can grade any proposal in a forty-minute meeting.
The seven criteria at a glance
| # | Criterion | The artifact that proves it |
|---|---|---|
| 1 | Eval pass-rate on a frozen test set | Eval suite + pass-rate report + a frozen set the vendor cannot edit |
| 2 | Fallback path for every named failure mode | A failure-mode list + a per-mode fallback action |
| 3 | Observability dashboard + named on-call engineer | A dashboard URL + a name + a paging schedule |
| 4 | Model version pinned | A version string in config + a written swap protocol |
| 5 | Failure budget agreed in writing | A numeric budget per quarter + a rollback rule |
| 6 | Runbook for the top-10 known issues | A document, one page per issue, with detection + remediation |
| 7 | 30-day post-launch on-call window | A line in the SOW with start date, end date, and escalation path |
Seven items. All are objects on disk, not statements of intent. The rest of this piece walks each one — evidence, the failure mode of missing it, and retrofit cost.
1. Eval pass-rate on a frozen test set
The headline criterion. Without it, the others are decoration.
Evidence: a written eval suite (inputs, rubric, threshold) and a recent run report at or above threshold. The set must be frozen — the vendor commits the cases before tuning starts, and the production gate is the pass rate against that set. Tuning against the gating set is the LLM equivalent of training on the test set. Cases edited mid-build means the gate is a moving line.
Missing: the feature works fine for two weeks and then quietly degrades. Internal threads use “feels worse” instead of percentages. Disputes get settled by whoever is loudest.
Retrofit: $15K–$30K across two to four weeks to build a 50-row gating set, rubric, and pass-rate harness. The hidden cost is worse — every prompt change between launch and the retrofit was ungraded, so the team cannot tell which week the regression entered.
2. Fallback path for every named failure mode
A LLM-based feature has a non-zero probability of getting any single call wrong. Production-readiness is not “zero failures” — that is impossible. It is “every named failure has a named action.”
Evidence: a failure-mode list (six to twelve modes per feature) and a per-mode fallback. For a customer-support classifier: unknown label, no label, high-confidence label on an unsupported category, timeout, rate-limit, content-policy refusal. Each has a designed action — route to human, default to lowest-risk bucket, retry on a smaller model, graceful error. Failure-mode taxonomy in AI hallucination explained.
Missing: the application crashes, hangs, or returns a confident wrong answer. Engineering wraps a try/except, swallows the exception, logs nothing.
Retrofit: $8K–$16K across one to two weeks per feature. The trap is retrofits cover the obvious modes (timeout, rate-limit) and skip the LLM-specific ones (low-confidence, refusal, off-topic). Those are the ones that bite.
3. Observability dashboard + named on-call engineer
The “we will know about the problem before the customer does” criterion.
Evidence: a live dashboard URL the founder can open, minimum showing request volume, latency p50/p95, model-call cost per day, fallback-trigger rate per mode, and eval pass rate on a re-run cadence. Plus a named engineer with a paging schedule. “The team will monitor it” is not an artifact.
Missing: outages get discovered when a customer emails support. Cost spikes get found at month-end. A vendor model swap changes behaviour and nobody notices for ten days because pass-rate is not re-run.
Retrofit: $3K–$6K across three to five days if the team already uses LangSmith, Langfuse, Helicone, or Braintrust. With no tracing in place, add a week. Negotiating the on-call rotation is the harder part.
4. Model version pinned
The LLM-specific criterion. Most teams do not realise they have to do this until a model swap breaks their feature in production.
Evidence: a specific version string in config — claude-opus-4-8-20251101 or gpt-5-2025-09-15, not claude-3-opus-latest or gpt-4-turbo. Plus a written swap protocol: the vendor commits to not changing the alias without notice, re-running the eval set before any swap, reporting the new pass rate, and getting founder approval.
Missing: behaviour changes overnight. The team digs and finds the vendor swapped to a newer alias — usually higher on public benchmarks, different on the specific prompt the product is built around. See AI model selection 101.
Retrofit: $1K–$2K and a day if already on a versioned alias. The expensive case is when the prompt has co-evolved with months of silent *-latest swaps; then pinning is meaningless until criterion 1 is satisfied.
5. Failure budget agreed in writing
The most-skipped criterion, and the one that prevents the most arguments.
Evidence: a numeric statement. “The system is allowed to fail on no more than X% of production calls per quarter, measured by the eval pass rate on a rolling 200-row sample of real traffic. Above the budget, the most recent change is rolled back; sustained burn triggers a redesign.” A failure budget is the LLM analogue of the SRE error budget — a contract that some failure rate is acceptable, and what happens when it is breached.
Missing: every individual failure becomes a debate. The founder thinks “this should never happen.” The vendor thinks “this is within tolerance.” With a budget, the conversation collapses to a number. (For how undefined budgets compound into project blow-ups, see anatomy of a runaway AI project.)
Retrofit: $2K–$5K of facilitation across one to three weeks. Cheap on paper, expensive in calendar.
6. Runbook for the top-10 known issues
The “the next person on-call needs to be able to fix this” criterion.
Evidence: one page per known issue — a one-sentence symptom, the detection signal (log line, dashboard alert, customer report shape), the diagnostic steps, the remediation steps, and the escalation path. Ten pages, living document.
Missing: knowledge sits in one engineer’s head. When that engineer is on holiday, the next failure takes four times longer to resolve. The same incident gets investigated three times in the first six months because nobody wrote down the diagnosis.
Retrofit: $3K–$5K across three to five engineering days for the first pass. The pain is not the writing — it is reconstructing diagnoses for incidents that were already resolved but not documented.
7. 30-day post-launch on-call window
The criterion most often left out of the SOW and most often regretted in week three.
Evidence: a contract line with a start date, an end date 30 days later, an escalation path, and a response-time SLA — inside the contract value, not as a billable extension. “We will be available for support” is not a deliverable. Post-launch on-call · D+0 to D+30 · 1-business-day response · escalation to <named engineer> is.
Missing: the team disbands at launch. A bug surfaces in week two. The agency replies “we can scope a support engagement, $X for Y weeks.” The bug sits open while the new SOW is negotiated. Customer trust erodes. This is the single largest source of “we shipped but it failed” stories in the founder community.
Retrofit: $10K–$25K, priced at 1.5×–2× what it would have cost inside the original SOW because the pricing power has flipped to the vendor. Insist on this line item during initial negotiation.
Retrofit cost when these are missing at launch
The summary table founders should screenshot before their next vendor meeting.
| Criterion | Retrofit cost | Calendar | What makes it expensive |
|---|---|---|---|
| 1. Eval pass-rate on frozen set | $15K–$30K | 2–4 weeks | Rubric written in retrospect; ungraded weeks unknowable |
| 2. Fallback paths | $8K–$16K | 1–2 weeks | LLM-specific modes (refusal, low-confidence) get missed |
| 3. Observability + on-call | $3K–$6K | 3–5 days | Negotiating the rotation, not wiring the dashboard |
| 4. Model version pinned | $1K–$2K | 1 day | Cheap unless co-evolved with silent swaps |
| 5. Failure budget | $2K–$5K | 1–3 weeks | Calendar alignment, not engineering |
| 6. Runbook | $3K–$5K | 1 week | Reconstructing diagnoses already in heads |
| 7. 30-day on-call window | $10K–$25K | re-negotiation | Pricing power has flipped |
Total: $40K–$90K and four to eight weeks of partial calendar time to retrofit a build that shipped without the seven. The same artifacts cost a fraction during the initial build — this is the underlying argument for stop scoping AI projects in features, scope them in evaluations: scope the artifacts, not the screens.
Retrieval-augmented systems carry one additional criterion: a numeric grounding rate on a test set, proving each claim traces to a source document. RAG concepts in RAG explained for founders.
Frequently Asked Questions
What single thing most distinguishes “production-ready” from a working demo?
A frozen eval set the vendor cannot edit, with a pass-rate threshold the build has cleared. Without it, every other criterion is unmeasurable. The demo works on inputs the team chose. The eval set works on inputs the team did not get to choose.
How is this different from the standard MLOps maturity model?
MLOps maturity was written for bespoke ML models the team owned end-to-end. LLM products do not work that way — the model is a third-party API whose behaviour changes on a vendor cadence. The seven criteria assume model swaps are routine, the prompt is the new spec, and the eval set is the new test suite.
Are these the same for an internal tool versus a customer-facing product?
The shape is the same; the thresholds differ. An internal tool can run at 70–80% pass rate because a human is in the loop. A customer-facing product typically needs 90%+ on the gating set and a 99.5% availability SLA. All seven artifacts still have to exist.
How much does it cost to ship an AI MVP that meets all seven?
Inside a 6–12 week MVP build, the seven artifacts add roughly 10–20% to the line items. Most of the work would have happened anyway, just less explicitly. The seven are not a premium tier — they are the floor.
Who writes the failure-mode list — the vendor or the founder?
Both, in two passes. The founder writes the first pass from a customer-experience view (“the system should never refund without authorisation”). The vendor extends it with LLM-specific modes the founder will not think of (content-policy refusal, partial JSON parse failure, low-confidence label).
What if the vendor refuses to commit to a failure budget?
That refusal is the answer. A vendor unwilling to commit to a numeric quality bar is reserving the right to define quality in retrospect. Walk.
Is “production-ready” the same threshold at pilot, MVP, and GA?
No, but the seven artifacts have to exist at each stage. A pilot can ship with a smaller frozen set and a more permissive failure budget; GA ratchets both. Each gate is “the next version of the seven artifacts.”
How do I run this checklist in a 40-minute vendor meeting?
Ask seven questions, one per criterion, and ask for the artifact, not the assertion. Show me your frozen eval set and the latest pass-rate run. Show me the failure-mode list and per-mode fallback design. Show me the dashboard URL and tell me who is on-call. What model version is pinned and what is the swap protocol? Where is the failure budget written? Show me the runbook. Is the 30-day on-call window in the SOW? Three missing is a no-go. Five is a re-scope. Seven of seven is a build worth paying for.
Closing: ask for the artifact, not the assertion
Production-ready AI is not a vibe. It is seven artifacts a buyer can ask to see: frozen eval pass-rate, fallback paths, dashboard plus on-call, pinned model version, failure budget, runbook, 30-day on-call window.
Use this as the launch gate. A vendor who can produce all seven on a Tuesday afternoon has crossed the dev-to-production line. A vendor whose answer to any of the seven is “we will get to that after launch” is selling a demo in launch language. The retrofit cost table above is what that answer actually costs.
For the operational sequence that produces these artifacts, read the eval-first build playbook. For the broader thesis, the idea-to-product manifesto. For the vendor-side mirror, decoding production-ready in AI agency proposals.
Subscribe to the SFAI Labs newsletter for one piece a week: concept-first, no code, for the person paying for the build.
Dirk Jan van Veen, PhD
