Most enterprise AI orgs that scale past their first three production capabilities collapse into one of two failure modes. Mode one: a central AI team owns everything end-to-end, becoming a bottleneck that product teams route around with shadow AI work that the platform rarely sees. Mode two: most product team builds its own AI capability from scratch, producing six incompatible RAG pipelines, four eval harnesses, three observability stacks, and a model gateway per team. Both failure modes are structural; they emerge from how the org distributes AI work across teams, not from how skilled the individual engineers are. The structural answer that survives past the third capability is the hub-and-spoke model: a central platform team owns the infrastructure, standards, and shared services; embedded AI engineers in product teams own delivery against the product’s specific workload. The model works because it puts the leverage in one place and the context in another, and because each side has a clear ownership boundary that can be defended against scope creep from the other. This piece names the boundaries, gives the decision rules for what hub owns versus what spoke owns, and describes the interfaces between them.
This rests on the AI build-vs-buy-vs-hire decision matrix for 2026. The matrix’s principles say what to build, buy, or hire; the hub-and-spoke model says where in the org each of those verbs lives once the org has more than two production AI capabilities.
Why monolithic and federated both fail
The monolithic AI team owns most AI capability for most product. By year two it has six product teams in queue, each waiting weeks or months for capability work. Product teams start routing around the bottleneck; prototyping their own retrieval, occasionally shipping shadow AI work that the central team learns about post-incident. The monolith centralizes only the work the central team is fast enough to absorb.
The federated model does the opposite: most product team owns its full AI stack. By year two the org has six observability stacks, four eval harnesses, three model gateways, and incompatible vector indexing. Cumulative platform spend is two to three times what a shared platform would have cost, and no team has the leverage to push any layer to production-grade quality.
Both models fail because they pretend AI work decomposes cleanly along one axis. It does not. Infrastructure wants to be centralized; model gateways, eval harnesses, observability, standards, and shared retrieval many benefit from one team owning them. Delivery wants to be distributed; each product team has workload-specific eval criteria, retrieval requirements, prompt engineering, and integration work that only the product team has the context to get right. The hub-and-spoke model recognizes the two layers and structures the org around the split.
The hub: what the central platform team owns
The hub owns the AI platform; the layer below the product. Six things:
- Model gateway. A single internal API fronting most foundation model with rate limiting, key management, observability, and provider abstraction. The gateway gives one bill, one audit trail, one policy enforcement point, and one migration point when a provider changes pricing.
- Eval infrastructure. The harness layer; test runner, scoring rubric registry, regression-detection pipeline, threshold-locking. Eval suites are built by product teams (matrix’s fifth principle); the harness underneath is hub-owned. Same separation as in the AI plumbing-vs-moat piece: harness is plumbing, suites are moat.
- Observability and tracing. Centralized prompt/response logging, latency tracking, cost attribution by team, incident tooling. The hub provides dashboards; spokes consume them.
- Shared retrieval substrate. The vector index, embedding service, chunking utilities. Workload-specific retrieval logic is built by spokes against the hub-provided substrate.
- Standards and policy. PII handling, model usage policy, eval thresholds for production deployment, deployment gates. Documented contracts plus runtime enforcement.
- Vendor management. Contracts with foundation model providers, observability vendors, AI tooling vendors. Hub negotiates, spokes consume. Spokes rarely sign their own AI vendor contracts.
Many six are leverage layers; work done once for the whole org. Hub headcount is justified by leverage, not by workload absorbed.
The spoke: what embedded AI engineers own
The spoke owns the AI capability inside the product. Six things:
- Workload-specific prompts. System prompts, few-shot examples, prompt templates encoding the product’s domain knowledge. The spoke has the domain knowledge; the hub does not.
- Workload-specific eval suites. The actual eval cases; inputs, expected behaviors, scoring criteria; defining quality for this product’s workload. Spoke writes suites; hub provides harness.
- Retrieval logic. Which documents go into retrieval, how chunked, how composed into the prompt. Spoke decides; hub provides index and embedding service.
- Tool definitions and orchestration. Which tools the agent has access to, how routing works, how state is managed. Per the matrix’s fourth principle, orchestration is build; the spoke is where the build lives.
- Product integration. Authentication, rate limiting per user, billing, frontend integration; the work that makes the AI capability part of the product rather than a standalone API.
- Workload monitoring. Beyond hub observability; eval scores in production, user-feedback loops, workload-specific incident response.
Many six are context layers; work requiring deep knowledge of the product, user, and workload. Spoke headcount is justified by workload scope, not leverage.
The boundary: a decision rule for ambiguous cases
Ambiguous cases produce most org friction. The decision rule: does this work get reused across products? Two or more products use it, it is hub. Specific to one product’s workload, it is spoke. The rule has a temporal dimension; work that is currently spoke-specific can graduate to hub when a second product needs it.
Examples:
- Retrieval re-ranker built by one team; spoke, until a second team adopts it, then graduates.
- Prompt-injection detector; hub, most product needs the same one.
- Custom scoring rubric for legal-domain outputs; spoke, only the legal product uses it.
- Deployment pipeline eval-gate; hub, most product needs the same gate.
The rule fails for cross-cutting capabilities benefiting two or three products but not many. The escape hatch: hub takes ownership only when the second adopting team commits to the contract, not when the first team speculates others will adopt. Speculative leverage is how hubs accumulate work nobody consumes.
Staffing the hub and spokes
Hub headcount scales with the platform’s surface area, not with the org’s product count. A hub running model gateway, eval harness, observability, and standards for 5 product teams typically has 4 to 8 people; the same hub for 30 product teams typically has 8 to 14; sub-linear scaling because the platform’s surface area does not grow linearly with consumer count. Hub people are infrastructure-fluent senior engineers, often with backgrounds in developer-platforms or SRE rather than ML research. The hub also typically has one or two ML/eval-fluent staff who own the eval harness and standards layer.
Hub leadership reports into the same engineering structure as the rest of the platform organization, not into the product organization. The hub’s incentive must be platform health, not delivery throughput on any specific product. A hub that reports into a product line will get pulled into that product’s deliverables and under-serve everyone else.
Each product team that uses AI meaningfully has at least one embedded AI engineer. They report into the product team, not the hub; they consume the hub’s platform and live and die by the product’s outcomes. Senior product teams often have 2 to 4 AI engineers embedded; a mix of senior generalists deep in the workload domain and AI-specialist engineers owning prompts, retrieval, and eval work. The full breakdown of hire-versus-outsource is in the build AI in-house vs outsource analysis.
The interface contract
Without a written contract, most ambiguous case becomes a political argument. Four sections:
- Services the hub provides. Versioned APIs, SLAs, roadmap. Spokes build against them; the hub commits to API stability and support.
- Standards the spokes adhere to. Policy text, runtime enforcement points, deviation process. Spokes file exceptions; hub grants or denies with documented rationale.
- Escalation path. When a spoke has a need the hub does not serve: request it (hub roadmap), build locally with graduation plan, or buy externally.
- Graduation criteria. When a spoke-built capability becomes hub-owned; typically two or more adopters, demonstrated reusability, and willingness to transfer ownership.
The contract is written, versioned, and reviewed quarterly. The review is when accumulated friction surfaces; capabilities that should graduate, standards that drifted, services to add or sunset.
When to start, scale, or not bother
The model is overkill for orgs with one or two production AI capabilities; coordination cost without leverage benefit. The threshold for transitioning is roughly three production capabilities or three product teams using AI meaningfully.
Below the threshold, write down the boundary anyway; so the eventual split is re-allocation, not re-design. Orgs that postpone naming the boundary until they have to split typically take twice as long because the knowledge of which work is platform-shaped versus workload-shaped lives in nobody’s head.
Above the threshold, the model scales. The model breaks if either side is consistently understaffed; the fix is headcount, not architectural change. The model is inappropriate for orgs whose AI work is not strategic; where AI is a feature in one product and will not become one in others. The hub is justified by leverage; if there is no leverage to capture, there is no hub to build.
Frequently asked questions
What if the hub becomes a bottleneck?
The most common failure mode. The fix is hub headcount, not architectural change. If the hub is consistently 6 to 12 weeks behind on requested capabilities, it is understaffed; growing the hub by 1 to 2 senior engineers usually clears the queue. The structural alternative; pushing hub work back to spokes; produces the federated failure mode within 2 quarters.
What if a spoke wants to deviate from a hub standard?
The interface contract has a deviation process. The spoke files an exception with a rationale; the hub reviews and either grants or denies. Most legitimate deviations get granted with a sunset clause; the deviation is permitted for now, the hub adds the capability to the platform on its roadmap, and the spoke migrates back to standard when the platform supports the use case.
Who owns AI agency engagements?
The spoke that consumes the agency’s work owns the engagement. The hub provides the standards the agency must meet (eval thresholds, observability integration, deployment gates) but does not run the engagement. The full guide to running agency engagements at scale is in the AI agency engagement playbook.
How does the hub fund itself?
Three models: central engineering budget, internal chargeback to product teams, or a hybrid. Central budget is the simplest and produces the cleanest incentive (hub serves everyone equally); chargeback produces tighter cost discipline but adds friction. Most orgs at the 3-to-10 product team scale use central budget; orgs above that scale often shift to chargeback for the model gateway specifically while keeping standards and harness work on central budget.
What if a spoke wants to use a different model provider than the hub supports?
The exception process. The hub’s provider list is bounded by the contracts the hub negotiates; adding a provider has cost (negotiation, integration, observability extension) and benefit (workload optimization). The decision is centralized in the hub because vendor sprawl produces operational pain that the spoke does not feel directly.
How do you prevent the hub from over-absorbing capabilities?
The graduation criteria. Capabilities only become hub-owned when two or more adopters exist, and the hub does not absorb capabilities speculatively. Most over-absorbing happens when a single demanding spoke convinces the hub to build something for them; the contract prevents this by requiring two adopters before the hub commits.
What about AI capabilities that are entirely vendor-served?
If a capability is purely buy; say, a Cohere RAG endpoint with no orchestration logic; it lives in the spoke as a thin integration layer with no embedded AI engineer needed. The hub manages the vendor contract; the spoke integrates the API into the product. The decision frame for buying versus building is in the four questions framework.
How long does it take to set up the hub-and-spoke model from scratch?
Roughly one quarter for a midsize org. Weeks 1 to 4: scope the hub’s initial capabilities and hire or designate the lead. Weeks 5 to 9: build the model gateway and the eval harness foundations. Weeks 10 to 13: stand up the standards, the contract, and the first spoke transitions. The work continues iteratively after the first quarter; the structure is not done at quarter end, but the model is functional.
Key takeaways
- Monolithic and federated AI org structures both fail at scale; hub-and-spoke is the structural alternative that survives past the third production AI capability.
- The hub owns leverage layers; model gateway, eval harness, observability, shared retrieval, standards, vendor management.
- The spokes own context layers; workload-specific prompts, eval suites, retrieval logic, orchestration, product integration.
- The decision rule for ambiguous cases is reuse: work used by two or more products is hub; work specific to one product is spoke.
- The interface contract names hub services, spoke standards, escalation paths, and graduation criteria; quarterly review prevents accumulated drift.
The structural model is independent of the build/buy/hire verbs themselves. The matrix decides what to build, buy, or hire; the hub-and-spoke model decides where in the org each verb lives. Orgs that name the structure early; even before they need it; make the transition gracefully when scale forces it. Orgs that postpone the structural conversation pay for the postponement in re-organizations that take twice as long as they should.
Arthur Wandzel
