What to ask in a discovery call with an AI MVP partner (15 questions)

A 60-minute discovery call with the right 15 questions is the single best diligence instrument a non-technical founder has before signing a $100K-to-$250K AI MVP contract. The wrong questions invite a sales pitch. The right ones invite evidence — or expose its absence. This is the checklist we hand to founders in the final-two stage of partner selection: three groups of five questions covering team and scope, eval and production discipline, and contract and IP. Each question is paired with what a good answer sounds like and what a hand-wave sounds like, so a non-technical founder can score on signal, not on confidence.

It builds on the founder-AI partner operating manual, part of the broader idea-to-product manifesto. Companion reading: the discovery call vs paid pilot vs full engagement explained, discovery call vs paid pilot: how to start an idea-to-product engagement, and idea-to-product red flags: 7 signals you are about to overpay.

How to use this checklist

Send the partner a one-page summary of your idea, the problem, the user, and budget range before the call. A good partner reads it. A hand-wave partner asks you to re-summarise it at the top of the call.

Ask the questions in order. Take notes verbatim — phrasing matters more than general shape. Score each answer 0-2: 0 hand-wave, 1 adequate, 2 operator-grade. A total of 22 or higher earns the partner a paid pilot. Under 18 justifies walking away. The point is a structured rubric so a non-technical founder does not mistake confidence for capability.

A discovery call is not a paid pilot. It produces evidence of partner identity — who they are, how they think — not partner ability. The call-then-pilot sequence is the subject of the companion discovery call vs paid pilot piece.

Group 1: Team and scope (questions 1-5)

Question 1: Who specifically on your team would lead this engagement, and what have they shipped in the last 12 months?

Testing: whether the principal you are speaking to is the person who would do the work, or whether you would be handed off to a junior after signing.

Good answer: a name, a role, two or three specific projects from the last 12 months with the technical role each person played, and an offer to introduce you to that person if it is not them on the current call.

Hand-wave: “We will assign the best-fit team after we scope the engagement.” Or a vague “senior team” without names. Or a bait-and-switch where the discovery-call principal is a salesperson and the build team is unnamed.

Senior-to-junior ratio matters in AI MVPs more than in traditional software. Eval design, prompt engineering, and architecture are judgement work. A 30-percent-senior team is a real AI build team. A 10-percent-senior team with the rest bootcamp-trained generalists is a 2019-era web shop with an AI wrapper.

Question 2: How many AI MVPs has your team shipped to production in the last 18 months, and how many are still running today?

Testing: shipped count and survival rate.

Good answer: a specific count, roughly three to twelve for a healthy small-to-mid partner. The partner mentions one or two that did not survive and explains why (usage did not materialise, founder pivoted, model API pricing shifted unit economics). They distinguish “shipped” from “still running today” comfortably.

Hand-wave: “Dozens” or “over fifty” — usually inflated by counting demos as MVPs. A 100-percent survival rate is statistically implausible. Ask for two still-running products by URL. A partner who has shipped real MVPs has artifacts to point at.

Question 3: What is the first technical decision you would make if you took this engagement, and what would inform it?

Testing: whether the partner reasons from your problem or from a default stack.

Good answer: a specific decision tied to your problem — “the first decision is whether the core capability needs an agent loop or a single-shot extraction; we would test both on 30 samples of your data in week one before committing.” A decision, an input, a falsification method.

Hand-wave: “We would start with our standard architecture.” Or generic “discovery and requirements gathering” without a specific decision. Or a stack-first answer naming a vector database and an orchestration framework before they understand your problem.

The first-technical-decision question separates partners who think from partners who deploy templates.

Question 4: What is the riskiest technical assumption in our idea, and how would you falsify it cheaply?

Testing: whether the partner can name a specific failure mode for your idea within 60 minutes.

Good answer: a specific assumption — “the riskiest assumption is that your domain documents are structured enough for retrieval to work at the quality bar you described; we would falsify that with a 50-document RAG eval in three days for $4K.” Specific risk, cheap falsification, stated cost.

Hand-wave: generic AI risks (hallucination, bias, drift) with no tie to your idea. Or “we would scope a full discovery phase to identify risks” — a non-answer that defers the question by a month and four figures.

A partner who cannot name a specific falsifiable risk after one hour will not name one after twelve weeks. They will build what you described and discover the risk during your own user testing — when the budget is gone.

Question 5: If we had only $30K to spend, what would you build, and what would you refuse to build?

Testing: scoping muscle. Whether the partner refuses work or always scales to the budget heard.

Good answer: a specific cut. “For $30K we would build the single highest-risk capability — the document-extraction pipeline — with an eval, a CLI for you to demo to investors, and zero UI. We would refuse to build a polished web app, a multi-tenant backend, or auth.” The partner is willing to refuse work.

Hand-wave: “We would do as much as fits the budget.” Or a yes-to-everything reshape. Or visible discomfort with the constraint because the partner sells fixed packages and does not actually scope.

The refusal half is the signal. A partner with scope discipline names what they would not build. A partner without it lists only what they would, leaving the founder to discover the gaps after signing.

Group 2: Eval and production discipline (questions 6-10)

These five questions separate an AI agency from a software agency that called itself an AI agency in late 2024. The listening cues are non-technical — you are testing for presence of practice, not jargon density.

Question 6: Walk me through the last eval set your team built. How many samples, who labelled them, what threshold did you ship at?

Testing: whether evals are routine practice or a marketing word.

Good answer: a specific story. “Last month for a legal-research startup we built a 200-sample eval set; their head of research labelled the ground truth, we labelled the predictions, we shipped at 87-percent exact-match on the headline metric with a documented 13-percent failure analysis.” Size, labeller, metric, threshold, failure analysis.

Hand-wave: “We use industry-standard benchmarks.” Or “we test thoroughly.” Or confusion between evals (your task-specific test set) and benchmarks (public datasets like MMLU). A partner who cannot distinguish the two is not shipping AI MVPs at quality.

Eval-set construction is the most important skill in an AI MVP partner. Public benchmarks measure model intelligence; your eval measures whether your specific product works on your data. A partner without an eval practice has no way to know whether what they ship meets your quality bar.

Question 7: How do you decide when a feature is good enough to ship, and when to keep iterating?

Testing: explicit quality bar and stop rule, or shipping decisions on vibes.

Good answer: a specific rule. “We ship when the eval passes at the threshold we agreed in the SOW and failure modes are either acceptable or have a human-in-the-loop fallback. We keep iterating when the eval is below threshold. The threshold is negotiated up-front; we do not change it mid-engagement.” A number is attached.

Hand-wave: “When the client and we both feel good.” Or “when QA passes.” Or any answer that does not name an evaluable threshold.

A vibes-based quality bar is how AI MVPs ship at 60-percent accuracy and the founder discovers it from the first 20 users.

Question 8: When a deployed AI feature regresses in production — output quality drops — what is your detection-and-rollback procedure?

Testing: production discipline. Separates partners who have run AI in production from partners who have only shipped it.

Good answer: a specific procedure. “We deploy behind a feature flag with a 5-percent canary. We run the production eval daily and alert if pass rate drops 3 points. Rollback is a feature-flag flip, MTTR under 10 minutes. We keep previous prompt versions in version control and can A/B test if regression is prompt versus model-API.” A runbook.

Hand-wave: “We monitor and roll back if needed.” Or “we have not had a major regression yet.” Or a deer-in-headlights pause followed by generic answers about logging.

Model providers ship silent updates and retrieval indices drift as content grows. A partner who treats production AI as set-and-forget hands you a degrading product six months in.

Question 9: What does observability look like for the AI parts of a product you shipped? Show me a screenshot or describe a dashboard.

Testing: whether the partner instruments AI calls or treats them as opaque.

Good answer: a specific tool or homegrown stack. “We instrument every model call with input, output, latency, cost per call, token counts, and trace sampling via Langfuse / Helicone / OpenTelemetry. We can pull up our most recent client’s dashboard and show you.” Specific dimensions, real evidence.

Hand-wave: “We have logging.” Or “we use the provider dashboard.” Or “we add observability after launch if needed.”

Observability lets the partner answer Q8 in the first place. Cost observability matters as much as quality — a 2-cent-per-query AI feature with 10,000 daily queries is $200 a day, which a founder must know before it becomes a surprise bill.

Question 10: How would you hand this product off to a future in-house engineer 6 months from now? What does that handoff include?

Testing: whether the partner builds with handoff in mind or builds dependency.

Good answer: a specific list. “Handoff includes: documented architecture diagram, runnable local dev environment with seed data, the eval set with labels and methodology, all prompts in version control with a changelog, the observability dashboard transferred to your accounts, and a one-week pairing with our team.” They have done handoffs.

Hand-wave: “We provide documentation.” Or “we are always available for ongoing support.” Or visible discomfort because the partner’s business model depends on you not hiring an in-house engineer.

A partner who builds for handoff has incentives aligned with yours. A partner who builds for dependency has a retainer that grows without commensurate value.

Group 3: Contract and IP (questions 11-15)

The legal and commercial structure. These matter more in AI MVPs than in traditional software builds because value sits partly in code and partly in artifacts — prompts, eval sets, fine-tuned model weights — whose ownership is not always obvious.

Question 11: Who owns the prompts, the eval set, and the model fine-tunes at the end of the engagement?

Testing: AI-specific artifact ownership.

Good answer: “You own everything. Prompts are checked into your repository, the eval set is in your repository, any fine-tuned model weights are your property and we transfer the training-data manifest and the training code. Our contract template names these explicitly.” Comfortable with full transfer.

Hand-wave: “We retain rights to reusable components.” Or “industry-standard IP terms apply.” Or argument that prompts and evals are “their methodology” and you license rather than own them.

If their contract retains prompts and the eval set, they can resell what they built for you to your competitor. A clean ownership clause is non-negotiable.

Question 12: What is your average idea-to-product engagement cost, what does it include, and what does it not include?

Testing: pricing transparency. Partners who hide pricing in a discovery call hide pricing for a reason.

Good answer: a specific range tied to scope. “Our typical engagement is $80K to $180K depending on data complexity and integration count, billed in milestones over 8 to 12 weeks. It includes PRD, architecture, build, eval, deployment, and a 30-day post-launch warranty. It excludes hosting and inference costs (your direct accounts), third-party APIs, design beyond functional UI, and post-warranty support.” Quoted without seeing your contract.

Hand-wave: “It depends on scope.” Or “we will know after the discovery phase.” Or “let us send you a proposal.” A partner who has shipped 10 AI MVPs knows their typical band within 15 percent before the call ends.

Question 13: How do you bill — fixed-fee milestones, time-and-materials, or hybrid — and which would you recommend, and why?

Testing: defensible point of view on billing structure, or default to whichever earns most.

Good answer: a recommendation with reasoning. “For an idea-to-product MVP we recommend fixed-fee milestones with three to four milestones, because scope is well-defined enough to estimate and the founder needs cost certainty. We do time-and-materials only for open-ended exploratory work. For your engagement specifically we recommend fixed-fee because [reason tied to your idea].” A model, a default.

Hand-wave: “Whatever works best for you.” Or “we are flexible.” Or a strong push toward time-and-materials without a reason tied to your situation.

Founders new to agency work often assume time-and-materials is “safer”. It is not. It transfers risk to the founder and removes the partner’s incentive to scope efficiently.

Question 14: What does your termination clause look like? If we are not happy after milestone one, what happens?

Testing: clean exit pathway versus a contract that traps you.

Good answer: a specific clause. “Either side terminates after any milestone with two weeks’ written notice. Work accepted at the most recent milestone is paid in full; in-progress work is billed pro-rata. All artifacts — code, prompts, evals, docs — transfer to you on termination. We honour a 30-day post-warranty period on shipped milestones regardless of termination.” Comfortable with you leaving.

Hand-wave: “We have a standard termination clause.” Or “early-termination penalty” beyond pro-rata billing. Or a 90-day notice. Or artifact ownership retained if termination is “for cause” — a clause lawyers argue about for months.

A partner whose contract makes leaving expensive is preparing for you to want to leave.

Question 15: What conflict-of-interest clauses apply? Are you currently working with anyone in our space?

Testing: simultaneous competitor work and whether the contract addresses it.

Good answer: a direct answer. “We are not currently working with anyone in your space. Our standard contract includes a 12-month non-compete on direct-competitor work in the specific vertical and use case, defined narrowly to not block general industry work.” The partner answers without flinching and the clause exists.

Hand-wave: “We work across many industries.” Or “non-compete clauses are not standard.” Or visible discomfort.

Non-competes are negotiable but not absent. A 12-month, specific-vertical, specific-use-case clause is reasonable. A 24-month blanket is unreasonable.

Scoring the call

Score each question 0-2: 0 hand-wave, 1 adequate, 2 operator-grade. Total possible: 30.

Total score	Interpretation	Next step
26-30	Operator-grade partner	Book a paid pilot
22-25	Strong partner	Book a paid pilot, follow up on weakest answers
18-21	Mixed signals	One follow-up call to clarify
14-17	Weak partner	Walk away
0-13	Sales pitch, not a partner	Walk away

Scoring is asymmetric. A partner scoring 9-10 in Group 2 but 6-7 elsewhere is still worth a paid pilot — AI-specific skills are the moat. A partner scoring 9-10 in Group 1 but 6-7 in Group 2 interviews well but may not ship at quality. Weight Group 2 most heavily.

When to walk away

Three signals justify walking away immediately, regardless of overall score:

1. Refusal to name the project lead. If a partner cannot commit to a named principal, the discovery-call principal is a salesperson and the build team is unknown.
2. No eval practice. If a partner cannot describe an eval set they built for a previous client (Q6), they do not ship AI MVPs at quality. They ship software with AI features and hope.
3. Hostile contract terms surfaced casually. IP retention on prompts, 90-day notice periods, non-pro-rata termination fees — the contract is the relationship’s floor, not its ceiling.

Other weak answers can be probed in follow-up. These three cannot.

What to do after a good discovery call

A 22-plus call earns the partner a paid pilot, not a main contract. The call confirms partner identity; the pilot confirms partner ability. You need both before a $100K-plus engagement.

The sequence:

1. Send a memo within 48 hours restating what you heard. The partner should produce one too. Compare the two memos for alignment.
2. Scope the pilot to one falsifiable risk — the risk the partner named in Q4.
3. Run two pilots in parallel only if you have the cash — typically $10K-$25K each. If not, pick the top-scoring partner; if they fail the pilot, run a second with second-place.
4. Do not skip the pilot to save time. Main contract risk is two-to-five times pilot cost.

If you would like a structured review of your specific situation, we run a 30-minute idea review where we apply this rubric to your candidates and recommend which to pilot. No pitch. Just the rubric, applied, in 30 minutes.

FAQ

How long should an AI MVP discovery call be?

60 to 90 minutes. Shorter and you cannot get through 15 substantive questions with listening cues. Longer and the conversation degrades into sales narrative. A partner insisting on 30 minutes is running an intake call, not a diligence call.

Should the partner ask me questions during the discovery call too?

Yes. A good partner spends roughly 40 percent of the call asking — about your users, domain, data, runway, timeline. A partner who only talks is selling; a partner who only listens is interviewing. Your 15 questions need to leave room for theirs.

What if the partner refuses to answer specific questions?

That is itself a high-signal answer. A partner who refuses to discuss pricing, termination clauses, or named team members is selling on opacity. Ask the question a second time with different phrasing, or score it as a hand-wave. Three or more refusals justifies walking away.

Should I record the discovery call?

Ask first. Most partners agree if framed as “for our internal note-taking, since I am a non-technical founder evaluating multiple partners.” A partner who refuses recording on a call where they are saying nothing they would not put in writing is a partner with something to hide.

Can I send the 15 questions to the partner before the call?

We recommend against it. Sending in advance lets the partner prepare polished answers; the listening cues you most want are the unscripted ones — how they react to a question they have not pre-baked. The exception is Q12 (pricing), which a partner may legitimately want to prepare before quoting.

Is 15 questions too many for a 60-minute call?

Two minutes per question on the partner’s side plus 30 seconds of your follow-up is roughly 37 minutes of question-time, leaving 23 minutes for the partner’s questions and rapport. Tight but workable. If running long, prioritise Group 2 — those five are the moat.

What if I am not technical enough to evaluate the Group 2 answers?

The listening cues in Group 2 are deliberately non-technical. You are testing for presence of practice, not technical correctness. A partner who describes a specific eval set with a specific size, labeller, and threshold has the practice. You do not need to know what the threshold means to detect whether one exists.

Should I do a discovery call with more than three partners?

Diminishing returns set in after three. The first call calibrates your sense of a good answer. The second and third give comparison. The fourth adds noise. Pick three from a longer shortlist using public artifacts — case studies, GitHub, conference talks — and run the 15-question call with those three.

Do these 15 questions apply if I am hiring a freelancer instead of an agency?

Mostly yes. Q1 collapses (the freelancer is the team). Q15 (conflict of interest) becomes more important — freelancers often work on multiple concurrent projects. Group 2 becomes more important for freelancers, not less, because there is no team to backstop a weak individual.

When in the process should I run the 15-question discovery call?

After you have a shortlist of three partners and before you have signed anything. The discovery call is the gate between partner-shopping and partner-pilot. The 15 questions exist to make that gate explicit so a non-technical founder, without engineering instincts, has a structured way through.