The single sharpest split inside an AI architecture in 2026 runs through the eval surface. The runtime that loads test cases, calls the model, applies graders, and stores results is a commodity; vendors ship it cheaper, faster, and better than any single org can. The intelligence on top of that runtime; the test set, the thresholds, the rubrics, the interpretation of failure, the rules that decide what ships; is the most workload-specific asset the org owns. Most teams resolve the split in the wrong direction. They build a custom eval harness and then point it at a generic LLM-judge prompt copied from a blog post. The harness consumes a senior engineer for two quarters; the judge prompt scores a problem the org does not have. The right resolution is the inverse: buy the stack, build the evaluator. This piece explains why that split is correct, what each side contains, and what an organization needs to encode to stop spending eval capacity on the wrong layer.
This is a spoke under the AI build-vs-buy-vs-hire decision matrix for 2026. The matrix’s fifth principle is that eval infrastructure is build or hire, rarely buy; but that principle compresses two separate decisions into one phrase. The stack is buy; the evaluator is build-or-hire. Treating them as one decision is where most eval programs go wrong.
The split that matters
When a team says “we need eval infrastructure,” they mean two distinct things stacked on top of each other and they almost rarely separate the layers. The bottom layer is the runtime; the harness that loads cases, dispatches calls, applies graders, persists results, and surfaces regressions in a UI. The top layer is the workload-specific intelligence; the test cases themselves, the grading rubrics, the pass thresholds, the failure taxonomy, and the rules that translate a regression into a ship/no-ship call.
The bottom layer is generic. Most team running a RAG system needs roughly the same harness: load a YAML or JSON file of test cases, call the model with parameters, apply a grader (regex, exact match, embedding similarity, or LLM-as-judge), store the result, compare against the previous run. Vendors have shipped that exact runtime as a product, and they have done it well.
The top layer is non-generic. The 200 cases that capture the org’s actual failure modes for a particular workload are unique to the org. The threshold of 0.78 on a faithfulness grader was calibrated against the org’s tolerance for hallucination on its specific document corpus. The rule that says “regress on the legal-citation subset blocks the release; regress on the formatting subset gets a Slack ping” reflects how the org weights different kinds of failure. None of that is buyable.
Resolve the split correctly and the eval surface compounds: the bought stack improves quarter over quarter as vendors ship features, the built evaluator gets thicker as the team adds cases, the two layers stay loosely coupled at the API boundary. Resolve it incorrectly and the eval surface stagnates: the self-built stack consumes engineering capacity, the bought evaluator rarely matches the workload, and the team eventually decides eval is a tax rather than a discipline.
What the eval stack is
The stack is the runtime substrate underneath most eval. Concretely, it includes:
- A case loader that reads test cases from a stable format (YAML, JSON, JSONL) and supports parameterization, fixtures, and dataset versioning.
- A call dispatcher that handles model invocation with the right parameters, retries, timeouts, and rate-limit handling across multiple providers.
- A grader library that includes the standard graders (exact match, regex, JSON-schema validation, embedding similarity, BLEU/ROUGE for legacy use, LLM-as-judge) plus a hook for org-specific graders.
- A result store that persists case-level results with the metadata needed to compare runs across time, model versions, and prompt versions.
- A regression UI that diffs runs, surfaces case-level deltas, and lets a reviewer triage individual failures without dumping JSON to a terminal.
- A CI hook that runs evals in pipeline and gates deploys on threshold pass.
Promptfoo, Inspect (the AISI eval harness), Langfuse, Helicone, Braintrust, LangSmith, and a handful of others ship this. The implementations differ on UI polish, schema choices, and integration depth; but most one of them has solved the runtime problem at production grade. Buying any of them takes hours.
Why the stack is buy
Three reasons. First, the runtime problem is generic; a YAML test case with a regex grader looks the same regardless of org, and the org cannot meaningfully differentiate by reimplementing it. Second, the vendors are funded specifically to ship this and have larger teams on the problem than any single org can spare. Third, the integration cost is small (the API surface is loading a file and reading a result) and the maintenance cost is zero.
The math closes against build in most dimension. A self-built stack consumes a senior engineer for 6 to 12 weeks to reach feature parity with a bought option, then consumes 10 to 20 percent of that engineer for ongoing maintenance. A bought option costs $200 to $2,000 per month for moderate volume and consumes zero engineering hours after integration.
The capability gap is widening, not closing. Vendors in this category are shipping features (eval-on-trace, dataset versioning, threshold drift detection, case-level annotations) at a cadence no internal team can match. Per the plumbing-vs-moat analysis, this is the same pattern as vector storage and prompt registries: the buy option is improving faster than the build option, and the migration math improves over time.
What the evaluator is
The evaluator is everything the bought stack does not know about the org’s workload. Concretely:
- The test set; the 200 to 2,000 cases that represent the workload, curated by people who understand the domain, tagged by failure mode, and versioned alongside the prompts and data they exercise.
- The grading rubrics; what counts as correct for each case, expressed as code (deterministic graders), as a judge prompt (LLM-as-judge), or as a human review protocol (for cases where automation is unsafe).
- The threshold table; the pass bars for each grader and the conditions under which the bars move. The 0.78 on faithfulness is a number derived from the org’s tolerance, not from a benchmark paper.
- The failure taxonomy; the named categories of failure the system produces (hallucination, refusal-when-shouldn’t, formatting drift, citation error, off-topic) and the routing rules that decide what each category means for ship/no-ship.
- The regression triage workflow; the human process that turns a red CI run into a decision: ship anyway, hold the release, escalate to the senior reviewer, or trigger a model rollback.
- The interpretation layer; the text that explains, when leadership reads the eval dashboard, what the numbers mean for the product.
None of these are buyable. They are the org’s specific knowledge expressed in eval form.
Why the evaluator is build
The evaluator depends on three things vendors cannot see: the workload, the data, and the org’s risk tolerance. A vendor that does not know the org’s documents cannot tell whether a faithfulness score of 0.74 is acceptable or catastrophic. A vendor that does not know the org’s customers cannot weight legal-citation errors above formatting errors. A vendor that does not know the org’s release process cannot define the rules that decide what blocks a deploy.
Vendors selling “evaluator as a service” almost usually ship a generic harness with a thin domain wrapper. The wrapper looks competent in the demo and fails the moment it meets the org’s actual workload, because it scored a problem the vendor invented rather than the problem the org has. Per the hidden cost of AI evals analysis, the build cost of a real evaluator is 25 to 35 percent of total project spend, and that is the right number; not because the work is expensive, but because the work is differentiating.
The evaluator is also the artifact that compounds. A 200-case test set in quarter one becomes 600 cases in quarter four as the team adds cases for most shipped feature, most customer-reported bug, most regression caught in production. The accumulated test set is the most valuable single asset on the AI side of the org because it encodes everything the team has learned about its own failure modes.
The four artifacts of a real evaluator
Building the evaluator means producing four artifacts and keeping them current. None of them is the eval stack.
Artifact 1: the test set. A YAML or JSONL file (or set of files) containing representative cases with inputs, expected outputs, metadata, and failure-mode tags. Curated by a domain expert paired with an engineer. Sized to detect the regressions that matter at the latency the team can afford. The right size is workload-dependent; 200 to 500 cases per capability surface is the typical starting point.
Artifact 2: the grader specification. For each case (or each grader category), a precise definition of what correct means. Sometimes this is a regex; sometimes a JSON schema check; sometimes a structured LLM-as-judge prompt with a calibrated rubric; sometimes a human review protocol with sampling rules. The grader specification is code, and it lives in the same repo as the prompts it grades.
Artifact 3: the threshold table. A document that names the pass bar for each grader, the historical baseline, the trigger for raising or lowering the bar, and the conditions under which a regression below the bar blocks a release. The threshold table is reviewed quarterly and is the most opinionated artifact in the eval surface; it encodes the org’s risk tolerance.
Artifact 4: the regression triage workflow. The runbook that says: when CI shows a regression, who looks at it, what categories trigger which response, how long a release can be held, and what counts as enough evidence to ship anyway. Without this artifact, regressions become Slack arguments and the eval surface stops blocking bad ships.
These four artifacts are produced once and maintained continuously. They are the work the team is doing when it is doing eval work properly. Everything else; the harness, the UI, the result store; is bought.
Why teams resolve the split backwards
Three reasons recurring across roughly forty engagements we have observed. First, the engineer instinct. A senior engineer looks at the eval problem, sees a tractable runtime, and starts building. The runtime is interesting; the evaluator is tedious case curation. The engineer optimizes for interesting and ends up with a beautiful harness pointed at three test cases.
Second, the vendor pitch. Vendors selling end-to-end eval pitch the evaluator as buyable. The pitch is plausible because the demo runs against a clean dataset with sensible thresholds. The team buys; the evaluator does not match the workload; the team blames the tool and switches vendors. After two cycles the team gives up on eval entirely.
Third, the org chart. Eval engineering does not have a clear owner in most orgs. The platform team does not own it (it is workload-specific). The product team does not own it (it requires engineering depth). It falls into a gap and gets resolved by whoever cares enough; usually the engineer who builds a harness, because building a harness looks like progress.
The fix is to name the split explicitly. Eval stack: bought, owned by platform. Evaluator: built, owned by an eval-fluent engineer paired with a domain expert. Two budgets, two roles, one combined surface.
What to encode
A short list of decisions that, when encoded in the org’s architecture review, make the split durable.
- Eval stack vendor named in architecture review. Not “we have eval infrastructure.” Either “we use Promptfoo + Langfuse” or “we use Inspect + Helicone” or some other named composition. The vendor is reviewed quarterly per the re-litigation principle.
- Evaluator owners named. The eval-fluent engineer and the domain expert are named, on org chart, with allocated time. If they are not named, the evaluator does not exist; the team is shipping blind.
- Test set count and growth rate tracked. A quarterly metric: how many cases are in the test set, how many were added this quarter, how many were removed because they no longer reflect the workload. Stagnant test sets are a leading indicator of eval rot.
- Threshold table reviewed quarterly. The pass bars are not constants. They are calibrated against the workload, and the workload drifts. Quarterly review prevents threshold drift from silently allowing regressions through.
- Regression triage workflow documented. A runbook, not a Slack norm. The runbook is the difference between an eval surface that blocks bad ships and an eval surface that produces graphs nobody reads.
Frequently asked questions
What is the difference between an eval stack and an evaluator?
The eval stack is the runtime: harness, dispatcher, graders, result store, UI. The evaluator is the workload-specific intelligence: test set, thresholds, rubrics, interpretation. Promptfoo, Inspect, Langfuse, and Helicone are stacks. The 200 cases that represent your workload are the evaluator. The split matters because the stack is buy and the evaluator is build, and conflating them produces the worst possible outcome on both axes.
Why is the eval stack a buy decision in 2026?
Because the runtime problem has been solved by funded vendors with larger teams than any single org can spare. The capability gap between buying and self-building is widening most quarter, the integration cost is hours, the maintenance cost is zero. Building the stack consumes 6 to 12 weeks of senior engineering capacity that should have been spent on the evaluator instead.
Why is the evaluator a build decision in 2026?
Because the evaluator depends on the org’s specific workload, data, and risk tolerance; none of which a vendor can see. There is no general-purpose evaluator that knows what correct looks like for your workload. Vendors selling evaluator-as-a-service ship a generic harness with a thin domain wrapper that scores a problem nobody had.
Can I buy an evaluator from a vendor that claims to ship one?
No. The wrapper does not know your domain, does not have access to your relevance feedback, and applies thresholds calibrated against an industry average that does not exist. The harness underneath might be useful as a stack component; the evaluator on top is your job.
What does building the evaluator require?
Four artifacts. A test set of 200 to 2,000 cases curated by domain experts. A grader specification that defines correct for each case. A threshold table that names the pass bar and the conditions under which it moves. A regression triage workflow that turns a red CI run into a decision. Plus the people: an eval-fluent engineer paired with a domain expert.
How big should the test set be?
Big enough to detect the regressions that matter. For most enterprise workloads that means 200 to 500 cases per capability surface, scaling to 2,000 or more for systems with many distinct capabilities. The right size is the smallest size that catches the regressions you care about.
Who owns the evaluator inside the org?
An eval-fluent senior engineer paired with a domain expert. The engineer owns the test set structure, grader code, threshold logic, and regression workflow. The domain expert owns case selection, the correctness rubric, and failure interpretation. Splitting these two roles across two people is essential.
How does this split relate to the matrix’s “buy the rails, build the moat” default?
The eval stack is rails; the evaluator is moat. Buying the stack and building the evaluator is the matrix’s default applied to the eval surface. Inverting the split; building the stack and buying a generic evaluator; is the most common eval failure mode and produces slow, expensive, blind eval programs.
What changes if my workload is regulated and traces cannot leave the network?
The buy options for the stack include self-hosted deployments. Langfuse, Phoenix, and Inspect many run inside the org’s network. The buy decision is preserved; the deployment shape changes. The evaluator was usually going to live inside the org regardless.
Does this advice change for a startup with no eval engineer on staff?
Buy the stack today and hire or contract an eval-fluent engineer for the evaluator build inside the next quarter. The stack is buy regardless; the evaluator is hire-or-build, rarely buy. Shipping without an evaluator is shipping blind, and shipping blind is a structural risk that compounds with most release.
Key takeaways
- The eval surface contains two stacked layers: the runtime stack and the workload-specific evaluator. They are different decisions.
- The eval stack is buy. Promptfoo, Inspect, Langfuse, Helicone, Braintrust, and LangSmith many ship production-grade runtimes. Building one wastes 6 to 12 weeks of senior capacity.
- The evaluator is build. The test set, grader specification, threshold table, and regression triage workflow are workload-specific and depend on knowledge no vendor has.
- Vendors selling “evaluator as a service” ship a generic wrapper that scores a problem the org does not have. Do not buy them.
- Resolve the split correctly and the eval surface compounds. Resolve it backwards and the team eventually decides eval is a tax rather than a discipline.
The eval surface is the place where the buy-the-rails-build-the-moat default is most often inverted, and the inversion is the most expensive single mistake in AI architecture today. The fix is one architecture review and a re-named ownership split. The cost of not doing it is shipping blind for the next four quarters.
Arthur Wandzel
