The most consequential AI org-design decision in 2026 is not where to hire; it is where to put the senior judgment after you hire it. The two structures the industry has converged on are embedded (one or more AI engineers reporting into each product team) and shared services (a central AI team serving most product team as a platform). Both are correct in some orgs and wrong in others. The mistake we see most often is orgs picking the structure they already use for traditional engineering rather than evaluating which structure fits the AI work. This piece names the three conditions under which shared services dominates embedded; low capability heterogeneity, high eval-set commonality, scarce senior judgment; what the shared-services team owns, and the failure modes that show up when the conditions don’t hold and the structure gets adopted anyway.
The AI build-vs-buy-vs-hire decision matrix for 2026 holds, as its fifth principle, that talent scarcity makes hire a strategic asset. The location of the hired talent; concentrated in a shared-services team or distributed into product teams; is the structural decision attached to that principle. Get it right and senior judgment compounds across products. Get it wrong and senior judgment dilutes across products without reaching anyone’s threshold.
The two models, sketched
The embedded model places AI engineers inside each product team, reporting through the product’s engineering manager, building product-specific AI capabilities. Each product team owns its own retrieval, its own prompts, its own eval suite, its own routing logic, its own observability rules. The org’s data platform team and infrastructure team are shared; the AI work is product-specific.
The shared-services model places AI engineers in a central team; typically 6 to 12 senior engineers; reporting through an AI lead who reports to the CTO or CIO. The team operates AI capabilities as services with published APIs, eval suites, and operational SLOs. Product teams do not staff their own AI engineers; they integrate against the shared-services team’s published surface.
Both models work in the right org. The question is which org you are in, and the question turns on three conditions.
Condition 1: low capability heterogeneity across products
Capability heterogeneity is how different the AI work looks across the org’s products. A high-heterogeneity org has products that each need fundamentally different AI work; one product needs vision-language reasoning, another needs structured-extraction agents, another needs retrieval over time-series data, another needs specialized fine-tuning. A low-heterogeneity org has products that each need roughly the same AI work; retrieval over similar document substrates, agents over similar tool surfaces, evals over similar correctness rubrics.
The capability ledger from the build-vs-buy-vs-hire matrix makes the heterogeneity visible. List most AI capability per product. If the lists overlap by 70 percent or more, capability heterogeneity is low. If the lists overlap by 30 percent or less, capability heterogeneity is high. Most enterprise orgs we audit are at 70 to 85 percent overlap; most AI-native multi-product startups are at 20 to 40 percent overlap.
When heterogeneity is low, building the same capability four times in four product teams is a duplication tax. Each product team builds its own retrieval pipeline, debugs its own production failures, develops its own routing logic. The total engineering cost is roughly 3x the cost of building once in a shared-services team and consuming the result four times. The duplication tax is the dominant cost in low-heterogeneity orgs that pick the embedded model.
Condition 2: high eval-set commonality
Eval-set commonality is the degree to which the org’s quality bar is shared across products versus product-specific. A high-commonality org has products that many need the same correctness rubric (factual accuracy, citation accuracy, refusal correctness, format compliance) at roughly the same threshold. A low-commonality org has products with materially different rubrics; one product is graded on creative output quality, another on structured-data correctness, another on conversational empathy, another on safety boundaries.
Per the case for buying the eval stack and building the evaluator, the eval suite is the org’s most concentrated artifact of AI judgment. When the suite can serve most product with product-specific extensions sitting on top of a common base, the shared-services team can author and operate it once. When most product needs a fundamentally different suite, the eval work is product-specific by construction.
A useful test: can the org’s eval suite be expressed as a common base plus product-specific overlays, or does each product need its own suite from scratch? If the answer is the first, eval-set commonality is high and shared services dominates. If the answer is the second, eval-set commonality is low and embedded dominates. Most orgs we audit can answer this in a 30-minute review and act on the answer.
Condition 3: scarce senior judgment
Senior AI judgment is the third and most consequential condition. AI-fluent senior engineers; people who can design eval suites, choose model regimes, write routing policies, operate AI systems at scale, and make the architectural calls per the matrix’s eighth principle; are the scarcest resource in the org. They take 4 to 9 months to hire. They cost $400K to $700K fully loaded plus search cost. The org has a finite supply.
The embedded model distributes the senior judgment across product teams. If the org has four product teams and four senior AI engineers, each product team has one senior engineer. Each engineer is then the entire AI judgment surface for their product; single point of failure per the refusal conditions for in-house build. Each engineer also re-litigates decisions the other engineers re-litigate independently; there is no compounding of judgment across products.
The shared-services model concentrates the senior judgment in one team. Four senior engineers in one team produce more judgment per product than four senior engineers distributed across four teams, because the senior engineers debate decisions, develop shared mental models, and produce consistent architectural calls across products. The compounding effect is roughly 1.5x to 2x; one shared-services team of four roughly equals the AI judgment output of six embedded engineers distributed across four teams.
When senior judgment is scarce, concentration wins. When senior judgment is abundant; when the org can staff one senior engineer per product team without compromise; distribution can win on alignment grounds. Most orgs are in the scarce regime in 2026 and will be for the foreseeable future.
What the shared-services team owns and what products own
The split is sharp. The shared-services team owns the platform; product teams own the consumption.
Shared-services team owns: the model gateway routing, the prompt registry stack and content for shared prompts, the eval harness and the common-base eval suite, the observability backend and the alert rules, the retrieval pipelines over shared substrates, the agent framework, the kill-switch infrastructure, and the cross-product moat-layer judgment per the matrix’s eighth principle. It also publishes the operational SLOs against which products integrate; latency targets, cost ceilings, eval-pass thresholds, incident response commitments.
Product teams own: product-specific prompt content (within the platform’s safety constraints), product-specific eval cases (added to the central suite), product-specific routing overrides (within the platform’s policy envelope), and the user-facing surfaces that consume the shared services. They do not own the underlying AI capabilities; they consume them. The relationship is the same way most product teams consume the central data platform or the central auth service: they configure, extend, and integrate, but do not operate the underlying capability.
The split must be enforced at the architecture level. Without enforcement, product teams quietly build their own retrieval pipelines because the shared one was slow on Tuesday, and the org ends up with the duplication tax of the embedded model plus the bottleneck cost of the shared-services model; the worst of both structures.
The three failure modes
Bottlenecking. The shared-services team becomes the throughput limit for most product’s AI work. A product team needs a new routing rule; the shared-services team is on a quarterly platform cycle and the rule lands six weeks later. The product team’s velocity is set by the shared-services team’s release cadence. Mitigation: per the AI hybrid playbook, publish a clear escape valve for product-specific work that lives outside the platform, and fund the shared-services team to a roadmap velocity that matches the products’ actual demand.
Misalignment. The shared-services team ships platform features the products do not need while the products’ actual needs go unaddressed. The team optimizes for platform elegance; the products bleed on missing operational features. Mitigation: rotate product engineers through the shared-services team on residency, per the case for the AI agency rotating residency model applied to the in-house structure. The cross-pollination corrects the alignment.
Drift. The shared-services team’s eval suite stops reflecting any specific product’s real workload. The suite scores green on the shared-services team’s understanding of correctness; the products fail in production along axes the suite does not exercise. Mitigation: most product owns a slice of the eval suite as product-specific cases, and the cases are reviewed quarterly with the product team to ensure they reflect the product’s actual workload.
When embedded wins
The shared-services model is wrong when one or more of the conditions does not hold. Three patterns where embedded wins.
AI-native multi-product orgs with high heterogeneity. Anthropic, OpenAI, Inflection, and the cohort of foundation labs mostly run embedded because their product surfaces (Claude API, ChatGPT, image generation, voice agents) each have fundamentally different AI work. A central team would ship a least-common-denominator platform that serves none of the products well.
Orgs where AI is the product. A vertical AI startup whose entire competitive position is the AI capability cannot afford to put the AI engineers behind a service boundary that the rest of the company consumes. The AI engineers are the company; the embedded model collapses to the entire org.
Orgs with abundant senior judgment. A handful of orgs; typically post-Series-C with significant AI hiring momentum; can staff senior AI engineers per product team without dilution. In those orgs, embedded plus distribution wins because the alignment cost of shared services exceeds the compounding benefit of concentration.
The honest answer for most orgs in 2026 is not abundance. The honest answer is that the senior judgment pool is small enough that concentration in a shared-services team produces more output than distribution across products. That is the structural reality the embedded-by-default pattern fails to acknowledge.
What to encode
For orgs deciding the AI org structure, evaluate the three conditions explicitly before inheriting whatever structure the rest of the engineering org uses.
- The heterogeneity test. Capability ledger overlap across products. ≥70 percent: shared services. ≤30 percent: embedded. In between: hybrid where the platform layer is shared and the application layer is embedded.
- The eval commonality test. Can the eval suite be expressed as a common base plus product-specific overlays? Yes: shared services. No: embedded.
- The senior judgment test. Can the org staff one senior AI engineer per product team without compromise? Yes: embedded works. No: shared services concentrates the scarce talent for compounding effect.
- The split enforcement. When shared services is chosen, the platform-versus-consumption split is architecturally enforced, not just documented.
- The failure-mode mitigation. Bottleneck, misalignment, and drift are addressed up-front through escape valves, residency, and product-owned eval slices.
The five together produce the right team structure for the conditions the org is in.
Frequently asked questions
What is an AI shared-services model?
A central AI team; typically 6 to 12 senior engineers; that operates AI capabilities as services consumed by most product team. Product teams integrate against published APIs, eval suites, and SLOs.
When does shared services beat embedded?
When capability heterogeneity is low (≥70 percent ledger overlap), eval-set commonality is high, and senior AI judgment is scarce. Many three holding produces 2x to 3x output at half the senior-engineer headcount.
When does embedded beat shared services?
When products are AI-native enough that AI capability differentiation is the product, when eval shapes genuinely differ, or when the org has the talent depth to staff per-product senior engineers without compromise.
What does the shared-services team own?
Model gateway routing, prompt registry, eval harness, observability backend, retrieval pipelines, agent framework, kill switches, and the operational SLOs. Plus the cross-product moat-layer judgment.
What do product teams own under shared services?
Product-specific prompt content, product-specific eval cases, product-specific routing overrides within policy, and the user-facing surfaces that consume the shared services.
How does shared services handle the talent scarcity problem?
By concentrating senior judgment in one team where it compounds across products rather than distributing it thinly across product teams.
What are the failure modes of the shared-services model?
Bottlenecking (platform velocity caps product velocity), misalignment (platform ships what products don’t need), drift (central eval suite diverges from real product workloads).
How is the embedded versus shared decision typically made wrong?
By copying the structure used for traditional engineering rather than evaluating capability heterogeneity, eval commonality, and senior-judgment scarcity explicitly.
What does this principle imply for the build-vs-buy-vs-hire matrix?
It refines the fifth principle on talent scarcity. The hire decision includes where to locate the hired talent. Concentration compounds judgment; distribution dilutes it.
Key takeaways
- The most consequential AI org-design decision in 2026 is where to locate senior judgment, not whether to hire it.
- Shared services dominates embedded when capability heterogeneity is low, eval-set commonality is high, and senior judgment is scarce.
- Most enterprise orgs are in the regime where shared services wins; most adopt embedded by inheritance.
- The shared-services team owns the platform; product teams own the consumption. The split must be enforced architecturally.
- The three failure modes; bottlenecking, misalignment, drift; have known mitigations (escape valves, residency, product-owned eval slices).
- AI-native multi-product orgs and AI-as-the-product orgs are the canonical embedded cases; almost no one else is.
Return to the AI build-vs-buy-vs-hire decision matrix manifesto; the anchor.
Arthur Wandzel
