The vendor says “we’re production-ready” in week 11. You, the non-engineer founder, have no way to verify that. This piece gives you seven criteria you can verify yourself in under an afternoon, plus a verification artifact for each one that you — not the vendor — must own at handoff. If a criterion is missing, the system is not production-ready. It is MVP-shipped, a different milestone with a different cost of failure.
Most failed AI engagements do not fail in the build. They fail in the four weeks after handoff, when a model version shifts, a fallback path was never tested, an eval suite was never wired into CI, or a runbook lists the wrong on-call number. Production-ready handoff is the contractual line that prevents that. This piece sits inside the wider Founder-AI-Partner Operating Manual as the closeout checklist for the final week.
Decision Scope
This article is an editorial decision framework for non-engineer founders, not legal, contractual, security, or accounting advice. Treat the criteria as the operational bar to set with your AI partner before handoff. Validate specific contractual terms — IP transfer, post-launch support, escalation SLAs — against your engagement letter.
The two-line definition
A production-ready AI handoff is one where (1) the founder can operate the system without paging the vendor for the first 30 days after launch, and (2) the founder owns every artifact required to operate the system at the account, repo, and URL level.
Most marketing definitions skip line 2. They describe the build, not the transfer. The result is a “handoff” that is actually a lease: dashboards live in the vendor’s Datadog, evals run in the vendor’s CI, runbooks list the vendor’s on-call number. The founder cannot operate the system without renting back the vendor’s infrastructure. That is a managed-services contract dressed up as a delivery, not production-ready.
The 7 criteria of a production-ready AI handoff
- Eval pass-rate stable across a 4-week window — not “we have evals,” but evidence the pass-rate held under load.
- Fallback paths tested with chaos cases — every fallback has been triggered deliberately and produced the documented degraded behavior.
- Observability dashboards live and owned by the founder — named dashboards in the founder’s account, with alerts routed to the founder’s pager.
- On-call schedule signed, with named humans and escalation triggers — a rota the founder controls.
- Runbook for the top-10 production issues — extracted from real production traffic, not invented at kickoff.
- Model version pinned, with a re-evaluation gate before any upgrade.
- 30-day post-launch on-call — a written, contracted commitment with weekly delta reports and a final exit memo.
If your vendor cannot point to the verification artifact for each criterion in 30 seconds, the criterion is not met. Production-ready is artifact-backed; everything else is theatre.
Why this matters now
Provider model versions shift faster than annual release cycles — a system that scored 92% on its eval suite in May can score 78% in July if the underlying model was not pinned. The cost of a working demo collapsed, widening the gap between “demo runs” and “system operates.” McKinsey’s State of AI 2025 puts the pilot-to-production failure rate at 64% across the surveyed enterprise sample. The Stack Overflow Developer Survey 2025 finds 41% of professional developers do not trust LLM accuracy without a human review step. The operating discipline that turns an MVP into a production system is what most engagements skip — and the founder pays the cost when that skip surfaces in week 14.
Criterion 1: eval pass-rate stable across a 4-week window
Production-ready means the eval suite has run on every merge for at least 28 days, and the headline pass-rate held within a defined tolerance (typically ±2 percentage points) across that window. The suite covers the major failure modes — accuracy, hallucination, refusal, latency budget, cost budget — with documented thresholds and traffic sampled from real production usage, not just kickoff cases.
Verification artifact: a pinned eval report showing pass-rate by week for the last four weeks, with cases version-controlled alongside the code. You should be able to click into a single failed case and read input, output, expected output, and the reason for failure.
What you ask the vendor: “Send me the eval report for the last four weeks, with pass-rate per week and the top three regressed cases per week.” For more on the eval depth that should have been running through the weekly cadence before handoff, see The role of evals in your weekly partner relationship.
Red flags: the suite exists only as a notebook on the vendor’s machine; the threshold is a “we feel good about it” judgment; pass-rate jumps more than ±5 points week-over-week with no documented cause.
Criterion 2: fallback paths tested with chaos cases
Production-ready means every fallback path — provider outage, rate-limit hit, malformed output, timeout, cost cap exceeded — has been deliberately triggered in a controlled chaos test and produced the documented degraded behavior. Chaos cases are part of the eval suite. A one-page fallback matrix lists every fallback, trigger condition, degraded behavior, and latency / cost impact.
Verification artifact: the fallback matrix plus chaos-test entries in the eval report from Criterion 1. You can also ask for a screen recording of one chaos test end-to-end.
What you ask the vendor: “Show me the fallback matrix and walk me through one chaos test you ran in the last sprint.”
Red flags: fallbacks described in slides but never code-tested; chaos cases last run two months ago; the matrix lists “graceful failure” without saying what the system returns to the user.
Criterion 3: observability dashboards live, in your account
Production-ready means every LLM call is logged with full prompt, full response, model version, latency, token counts, cost, trace ID, user ID, and request ID — and logs are queryable within five minutes. At least three dashboards are live: latency (p50 / p95 / p99), cost per request, and eval-score-on-production-traffic. Alerts route to the founder’s pager, not the vendor’s.
Verification artifact: dashboard URLs you can open today, in your own Datadog / Grafana / Langfuse account, provisioned to your billing. You should be able to log in, change an alert threshold, and have the change persist after the vendor’s seat is revoked.
What you ask the vendor: “Send me the dashboard URLs and confirm they live under my account, on my billing.”
Red flags: dashboards on the vendor’s tenant only; logs sampled at one percent so most calls are invisible; alerts route to the vendor’s pager; dashboards become invisible after handoff because they were behind the vendor’s SSO.
Criterion 4: on-call schedule signed, with named humans
Production-ready means a written rota for the 30 days after launch, with named humans, phone numbers, time zones, escalation triggers, and response-time commitments. The rota is signed by both sides. The vendor is first responder for the first 14 days; the founder side shadows every incident from day one and takes over as first responder by day 15.
Verification artifact: the countersigned rota plus contact info loaded into the founder’s incident system. If there is no pager, the rota names “phone call to [number]” with response-time SLAs.
What you ask the vendor: “Send me the signed on-call rota for days 0–30, with names, numbers, and escalation triggers.”
Red flags: the commitment is “we will respond on Slack during business hours”; the rota names roles (“engineer on duty”) rather than humans; escalation triggers are not defined.
Criterion 5: runbook for the top-10 production issues
Production-ready means a single runbook in the repo (typically RUNBOOK.md) listing the top-10 production issues in priority order, with symptom, diagnostic command or dashboard, remediation steps, and rollback procedure. The runbook is written from real traffic — the issues actually fired in the four weeks before handoff, not the ones imagined at kickoff. Each entry has a date stamp.
Verification artifact: the runbook in your repo, with date stamps showing entries updated within the last 30 days. You should be able to read it as a non-engineer and understand the diagnostic step for at least eight of ten entries.
What you ask the vendor: “Open the runbook and walk me through the most recent issue, end-to-end.”
Red flags: hypothetical issues that have never fired; language only an engineer can interpret; the runbook lives in the vendor’s Notion rather than your repo.
Criterion 6: model version pinned, with re-evaluation gate
Production-ready means every model call specifies an explicit version string (not “latest” or “default”). The repo has a documented model-upgrade procedure: before any version changes, the eval suite runs against the new model on the existing test set, the delta is reported, and a human signs off. If pass-rate drops below the production threshold, the upgrade is blocked.
Verification artifact: the model-pinning audit (a grep showing every model call has a version string) plus the upgrade procedure. As a non-engineer you should be able to read it and understand who signs off and what evidence they require.
What you ask the vendor: “Show me the grep result for model= across the codebase and the upgrade procedure.”
Red flags: unversioned strings like “claude-opus” or “gpt-default”; no documented upgrade procedure; the eval suite is not wired to run against a candidate model before upgrade.
Criterion 7: 30-day post-launch on-call
Production-ready means a contracted 30-day window in which the vendor remains the escalation point for incidents the founder cannot resolve, with weekly delta reports (eval pass-rate, cost per request, incident count, runbook entries added) and a written exit memo on day 30. The exit memo records operational state, open known issues, suggested next-quarter priorities, and a final go / no-go on whether production-ready status held across the window.
Verification artifact: the 30-day clause in the contract, the weekly delta report template, and the exit memo template. All three should exist before handoff so there is no negotiation in week 13.
What you ask the vendor: “Send me the contract clause for the 30-day on-call, the weekly report template, and the exit memo template.”
Red flags: support is “best-effort” with no SLA; weekly reports are not committed in writing; the exit memo is not part of the deliverable; the vendor offers a “month of free Slack support” instead.
What is out of scope
Production-ready handoff is a checkpoint, not a guarantee of business outcome. It does not promise market fit, a defensible moat, eval pass-rate holding a year from now, or that providers will not deprecate the pinned model. Pinning buys a re-evaluation gate, not eternal stability. For the proposal-stage version of this conversation — what “production-ready” should mean before you sign — see Decoding “production-ready” in AI agency proposals. For the full week-by-week structure that ends in this handoff, see how an idea-to-product engagement actually works, week by week.
The founder’s 30-minute pre-handoff audit
Schedule a 30-minute call with your vendor in week 11. Bring the seven criteria. For each one, ask the verification question. Mark a check or an X.
- Seven checks: proceed to handoff, then run the 30-day post-launch on-call.
- Five or six checks: name the missing criteria, set a deadline, delay handoff.
- Four or fewer: the system is not production-ready. Delay launch.
A handoff that ships with three missing criteria sets up the founder to be paged at 2 a.m. on day 8 without the dashboards, runbook, or on-call rota required to respond. The cost of that incident is higher than the cost of slipping launch by a week. A vendor that pushes back on delay or written descope is signaling they treat production-ready as a marketing claim.
FAQ
What is “production-ready” in plain English for a non-engineer founder?
A production-ready handoff is the point at which your AI system can run for 30 days without you paging the vendor for help, and you own every artifact (dashboards, runbook, on-call rota, evals) required to operate it. Anything less is “the build is done” — a different milestone.
How long does the pre-handoff sprint usually take?
On a 12-week engagement, weeks 9–11. If your vendor proposes a single-week handoff at week 12 with no pre-handoff sprint, that is a strong signal they have not run a serious handoff before.
What if my vendor says “we use OpenAI’s dashboards” for observability?
Provider dashboards cover billing and rate-limit visibility but rarely cover prompt / response logging, trace correlation, or eval-score-on-production-traffic. You want named dashboards in your account, with at least the three in Criterion 3.
Do I really need a runbook if my vendor is on call for 30 days?
Yes. The runbook is what the founder side reads during the on-call shadow period. Without it, the founder cannot become first responder on day 15. It is also the document that survives the engagement.
What if my vendor says these criteria are “enterprise overkill”?
Eval pass-rate, runbook, observability, and on-call are not enterprise-only — they are the minimum for any LLM system facing real users. A vendor that calls them “overkill” has not operated an LLM in production through a real incident.
What is the difference between this checklist and an SLA?
An SLA promises uptime numbers. This checklist promises operating capability. You can have a 99.9% uptime SLA and still be unable to operate the system if the runbook does not exist and dashboards live on the vendor’s tenant.
What if I do not have a pager or incident system?
Then your on-call rota is “phone call to [number]” with documented response-time SLAs. You need a written rota with named humans, time zones, and one phone number you trust to ring loud at 3 a.m.
Can I delegate this audit to a friendly engineer?
Yes — many founders do. But you should understand each criterion at a non-engineer level, because the audit is also a calibration of your vendor relationship for the post-handoff months.
How does the 30-day post-launch on-call differ from a “support contract”?
A support contract is open-ended and reactive. The 30-day post-launch on-call is a closed window with named humans, weekly delta reports, and a final exit memo. It transfers operating capability rather than locking in managed-service revenue.
What if everything passes but my system still fails in week 6?
That is what the exit memo’s “open known issues” section is for. Production-ready handoff is not a guarantee of zero failures — it is a guarantee that when failure happens, you have the dashboards, runbook, evals, and on-call rota required to respond without re-engaging the vendor.
Next steps
If you are mid-engagement, copy the seven criteria into a shared doc with your vendor this week and ask which week each will ship. If you are pre-engagement, write the criteria into your engagement letter as the definition of “production-ready” before signing. The criteria are most valuable when agreed before the build, not negotiated after it.
For the wider frame on how non-engineers ship AI products in 2026, the idea-to-product manifesto sets the strategy. For occasional founder-grade essays — subscribe to the SFAI Labs newsletter.
Arthur Wandzel
