Mid-market enterprises; roughly $50M to $1B revenue, with engineering orgs of 5 to 50 engineers; face a sourcing question the legacy build-vs-buy framing answers wrong. Building a production-grade agent platform is a 10-to-30-engineer multi-quarter project, well beyond mid-market staffing depth; doing it anyway burns 12 to 24 months of capacity for an outcome the buy alternative would deliver for a fraction of the cost. Buying generic evaluators fails for a different reason: the evaluator encodes the org’s domain expertise and quality bar, and no vendor’s generic harness can replace that judgment. The right pattern is split: buy the agent platform from a vendor with vertical specialization, multi-model backend, and clean exit terms; build the evaluator with a 1-to-3-engineer team and a 4-to-8-week first version. This piece is the operating model: which agent vendor to buy from, what the in-house evaluator team looks like, what the integration glue looks like, and the failure modes if either side of the split goes wrong.
It applies the AI build-vs-buy-vs-hire decision matrix for 2026. The matrix’s eighth principle is that the default verb is compose; buy the rails, build the moat, hire the judgment. The mid-market split is the most explicit operational instance of that principle: agents are the rails, evaluators are the moat.
The mid-market staffing constraint
Mid-market AI strategy lives or dies on staffing depth. A large enterprise can stand up a 30-engineer AI platform team without the rest of the org noticing. A mid-market enterprise with 30 engineers total cannot. Most engineer assigned to AI platform work is one fewer assigned to the product. The math forces a sourcing pattern where leverage matters more than control.
The right read is not “we’re smaller, so build less.” It is “build the things where staffing depth produces moat; buy the things where staffing depth produces a worse version of what a vendor already sells.”
Two layers are load-bearing: agents (the orchestration layer deciding what the AI does step by step) and evaluators (the judgment layer deciding whether what the AI did was good). Agents are large investments with broad applicability. Evaluators are small investments with narrow applicability. The staffing math points the same way most time: buy the large investment, build the small one.
Why buy agents
Three reasons.
Engineering depth required. A production-grade agent platform handles tool-use orchestration, error recovery, retries, multi-step workflow composition, observability, prompt evaluation, latency budgeting, cost control. 10 to 30 engineers steady-state for a production vendor; mid-market orgs cannot field that depth.
Long tail of failure modes. Agents fail in domain-specific and model-specific ways. The vendor sees failures across many customers and patches once; the in-house build sees only its own failures and patches n times across n edge cases. Vendor amortization is real.
Velocity of model changes. The platform must adapt to most model release; extended context, native tool use, agent-specific capabilities. The vendor’s full-time team adapts faster than a mid-market team where agents are one of many responsibilities.
The buy is not “outsource the AI strategy”; it is “rent the platform the AI runs on, keep the strategy in-house.” The strategy lives in which vendor you pick, which workloads you put on it, and most importantly the evaluator that grades whether the workload is working.
Why build evaluators
Three reasons.
Domain expertise is the input. The evaluator encodes what “good output” means for this specific business. No vendor has that judgment; no vendor can acquire it without re-acquiring the org’s full domain context.
Build is small. A 1-to-3-engineer team produces a first version in 4 to 8 weeks: versioned eval set, CI-style harness, regression-triage workflow, quarterly model-comparison report. Fits mid-market staffing depth in a way the agent build doesn’t.
Portability across vendors. The evaluator stays when the agent vendor swaps. The investment compounds across vendor decisions; the agent platform investment doesn’t.
Detail in the case for buying your AI evaluation stack and building your AI evaluator. The mid-market enterprise sits inside that pattern with the additional constraint that “buy” is the right verb for the agent platform too.
What kind of agent vendor to buy from
Five criteria.
- Vertical or workflow specialization. Sales-agent, support-agent, ops-agent, or similar narrow focus tends to outperform horizontal “build any agent” platforms on the workloads in their specialty.
- Eval and prompt-registry interoperability. The vendor exposes outputs and intermediate steps in a form the buyer’s evaluator can grade. Prompts are accessible to the buyer’s registry; at minimum readable, ideally writable. Vendors that black-box the eval surface should be declined.
- Multi-model backend. Multiple frontier providers, with the buyer able to route across them. Single-provider lock-in at the agent layer compounds the lock-in at the foundation-model layer.
- Clean exit path. On termination, the buyer receives prompts, workflow definitions, and any vendor-trained components in a runnable or reconstructable form. “Training data is vendor IP” or “workflow definitions are vendor confidential” are red flags.
- Pricing aligned to value, not seats. Per-call or per-workflow tracks workload value better than per-seat. Per-seat for agents tends to misprice at both ends.
A vendor meeting many five is a credible buy. A vendor missing two or more is a near-future regret.
What the in-house evaluator team looks like
- One eval-fluent senior engineer. Owns eval set, harness, scoring rubric, CI integration, regression-triage workflow. Load-bearing role.
- One product or domain expert. Provides judgment for the eval set; what items to include, what counts as correct, what edge cases to surface. Can be a fractional commitment from a senior PM or domain SME, but consistent.
- Optional one ML-ops support. Infrastructure: model serving for eval, dashboard delivery, CI pipeline. Often shared from the broader engineering org.
Deliverable is a quarterly eval report grading whichever agent platform the org has bought against the workload the org runs; cost-per-useful-task, quality-percentile distributions, regression history, recommendation on the contract. The team owns the leverage: with the report, the buyer can credibly threaten to switch and negotiate on evidence.
The integration glue
The glue between the bought agent platform and the built evaluator is itself a small build; 0.5 to 1 engineer.
- Output capture. A wrapper around the vendor’s API capturing most response (and intermediate steps where available) into a buyer-controlled store.
- Eval invocation. A scheduler running the evaluator at the right cadence; synchronously per call, nightly, or on a sample. Workload-dependent.
- Regression-triage hook. Routes flagged failures to the right team; agent vendor (platform issue), in-house (prompt issue), data team (upstream-data issue). Triage detail in the AI insourcing wave.
The glue stays in-house even as the agent vendor swaps. It is the bridge that lets the org migrate vendors without rebuilding the evaluation stack.
Failure modes
Failure mode 1: trying to build agents. A mid-market org with 20 engineers assigns 5 to building an agent platform. Twelve to twenty-four months later the platform exists partially with a fraction of the buy alternative’s capability. The product team waiting on the platform team didn’t ship. Opportunity cost is 5 engineers plus product capacity. Wreckage: a regretted hire round, a deprecated framework, an exec who avoids the AI conversation.
Failure mode 2: trying to buy evaluators. A mid-market org buys “eval-as-a-service” on the theory that eval is too specialized to build. Six months later the dashboard is green and the product is regressing; the eval scored a generic problem; the actual problem was different. The org rebuilds the eval in-house anyway, having spent six months and a SaaS budget on the masked version.
The frame avoids both by naming the split explicitly: agents bought, evaluators built.
Sequencing the build
From zero; no agent platform, no evaluator, just a need:
- Months 0 to 1: scope the workload (what, on what data, for what user). Produces eval set and platform requirements simultaneously.
- Months 1 to 2: select the agent vendor. Pilot against two or three using a draft eval. Pick on the five criteria plus pilot performance.
- Months 2 to 4: build the evaluator and the integration glue.
- Months 4 to 6: deploy to production with continuous evaluation. Tune prompts, routing, eval thresholds. First quarterly report ships.
- Months 6+: steady-state with quarterly evaluator updates and annual vendor review.
Six months from zero to a working production system with the right sourcing pattern. Compressed sequences skip the evaluator and pay for the omission later.
Frequently asked questions
What counts as a mid-market enterprise for this frame?
Roughly $50M to $1B in revenue, 200 to 5,000 employees, and an engineering org with 5 to 50 engineers; large enough to buy enterprise software, too small to staff a 30-engineer AI platform team. The frame’s logic also applies to upper-mid-market (up to $5B) and to engineering-heavy startups; the staffing constraint is the load-bearing variable, not the revenue band.
Why buy agents and not build them?
Building a production-grade agent platform is a 10-to-30-engineer multi-quarter project. Mid-market enterprises lack the engineering depth, the agent-research talent pool, and the operational tolerance for the long tail of agent failure modes. Vendors who specialize in agents amortize that cost across many customers; the mid-market buyer rents the platform and ships features at a fraction of the build cost.
Why build evaluators and not buy them?
Evaluators encode the org’s domain expertise; what “good output” means for this specific business. No vendor has that judgment; no vendor’s generic eval harness can replace it. The build is small (1 to 3 engineers, 4 to 8 weeks for a first version), the moat is large (the org’s quality bar is encoded), and the artifact is portable across whichever agent platform the org buys this year or next.
Doesn’t this contradict the principle that eval is build-or-hire, rarely buy?
It applies the principle to the mid-market. The build path is open to mid-market orgs because the eval build is small. The hire path is also valid; bring in an eval-fluent contractor for 8 to 12 weeks, transfer the artifact, then run it in-house. The buy path stays closed for the same reason it’s closed for enterprise: generic eval vendors cannot test the workload that matters.
What kind of agent vendor should mid-market buy from?
A vendor with a vertical or workflow specialization (sales agents, support agents, ops agents) over a generic horizontal platform; with eval and prompt-registry interoperability so the buyer’s evaluator can grade the vendor’s output; with multi-model backend support so the agent isn’t locked to one frontier provider; and with a clean exit path on contract termination.
What does the in-house evaluator team look like?
One eval-fluent senior engineer plus one product or domain expert plus optional one ML-ops support, owning the eval set, the harness, the regression-triage workflow, and the model-comparison reports. The team’s deliverable is a quarterly eval report that grades whichever agent platform the org has bought against the workload the org runs.
How does the buyer keep leverage over the agent vendor?
The evaluator is the leverage. The buyer can credibly threaten to switch agent vendors because the buyer owns the eval set that grades any agent platform. Without an in-house evaluator, the buyer is dependent on the vendor’s self-reporting; with one, the buyer has a portable benchmark and a real exit option.
What about the orchestration layer between agents and evaluators?
Buy the agent platform, build the evaluator, and treat the integration glue between them as build (light, 0.5-to-1 engineer); typically a thin wrapper that calls the vendor’s agent API, runs the in-house evaluator on the response, and triggers the regression-triage workflow on failures. The glue stays in-house even as the agent vendor swaps.
What’s the failure mode if mid-market tries to build agents?
Twelve to twenty-four months of engineering capacity sunk into a platform that lags the buy alternative by a generation, with no business outcome to show for it because the team was building the substrate when the competitor was shipping the feature. The wreckage typically includes a regretted hire round, a deprecated framework, and an exec who avoids the AI conversation.
What’s the failure mode if mid-market tries to buy evaluators?
A vendor eval that scores a generic problem rather than the workload, masked by green dashboards until the first production regression that the eval missed. The org then rebuilds the eval in-house anyway, having spent 12 months and a SaaS budget on the masked version. Insourcing first is the cheaper path.
Key takeaways
Mid-market AI sourcing is staffing-constrained. The right pattern is split: buy agents (no leverage to build), build evaluators (your domain, your data, your judgment). Blanket-buy and blanket-build both fail.
The agent vendor is selected on five criteria: vertical specialization, eval and prompt-registry interoperability, multi-model backend, clean exit path, value-aligned pricing. Vendors missing two or more are near-future regrets.
The in-house evaluator team is small: one eval-fluent senior engineer, one product or domain expert, optional ML-ops support. Deliverable is a quarterly eval report that grades the agent platform against the org’s actual workload. The team owns the leverage that keeps the vendor honest.
Failure mode 1 is trying to build agents; burns 12 to 24 months for a platform that lags the buy alternative. Failure mode 2 is trying to buy evaluators; green dashboards mask production regressions. The split avoids both. Sequence the build at six months from zero to working production with the right sourcing pattern in place.
Arthur Wandzel
