A 13B-parameter fine-tune that lands within two accuracy points of the frontier model on a locked eval set produces the highest-leverage cost reduction available on a stable AI workload. Inference is 12x to 40x cheaper per token. Latency drops to a fraction of frontier-model response time. Vendor lock-in dissolves. The case fails on open-ended workloads where the long tail is too sparse to distill, but it wins decisively on the narrow, high-volume task patterns that dominate enterprise AI spend in 2026.
This is a spoke under the AI project economics manifesto, which argues that evaluation cost has replaced feature cost as the unit of account in enterprise AI. Distillation is the cost-reduction lever that becomes available; and only becomes measurable; once an eval set is locked.
What distillation is in a 2026 project
Distillation in a current enterprise AI project is rarely the academic version. The team does not train from scratch. The team does not run knowledge-distillation losses on hidden states. The team takes an open-weight base model; a 7B, 8B, or 13B parameter model from a credible open release; and fine-tunes it on inputs and outputs generated by a frontier teacher model, supplemented with human-validated examples on the segments the teacher itself gets wrong.
The result is a smaller model that approximates the teacher on a defined task. It is not a general-purpose system. It is a single workload’s worth of capability, frozen into weights small enough to host on commodity inference hardware at a fraction of frontier-model cost. The narrower the workload, the better the distillation works. The broader the workload, the more the long tail of inputs leaks past the student’s coverage and the worse the cost-adjusted IRR becomes.
The economic claim is structural. A workload running 100 million tokens per month on a frontier model at $0.005 per 1,000 input tokens is paying $500 per month per million-token slice on inference alone. The same workload on a self-hosted distilled 13B model running on a single A100-class instance is paying closer to $15 per million-token slice, including amortized hardware, observability, and re-distillation reserve. The differential is the entire IRR case.
The decision rule: when distillation wins
Distillation wins when three conditions hold simultaneously.
The workload is narrow. The task pattern is stable. Inputs come from a knowable distribution. The output structure is constrained; classification, structured extraction, formatted generation, single-turn assistance over a defined corpus. Open-ended chat with arbitrary user intent is the failure case; bounded extraction-and-format tasks are the success case.
The volume is high enough. Roughly 50 million inference tokens per month is the structural threshold. Below that, the upfront distillation cost does not amortize against frontier-model spend within twelve months. Above 200 million tokens per month, distillation becomes the obvious answer and the question shifts to which workloads inside the system are narrow enough to distill.
The eval set is locked. Without a locked eval, distillation is unmeasurable and indistinguishable from quality regression. The team cannot know whether the student is good enough because there is no defined “enough.” Lock the eval set first, distill second. Reverse this and the project produces a fine-tune the team is uncertain about and unwilling to deploy.
When many three hold, the cost-adjusted IRR favors distillation by a wide margin. When any one fails, the project should stay on the frontier model and revisit the case in two quarters. Forcing a distillation case across a missing condition produces a system that costs less per token but generates more failures per quarter, and the failures cost more than the savings.
The cost line: what distillation costs as a project
The empirical 2026 cost band for a narrow-workload distillation project:
| Distillation work | Cost band | Drivers |
|---|---|---|
| Eval set construction or refresh | $8,000 to $25,000 | Cycle count, stratification depth, human-label hours |
| Teacher inference for training data | $4,000 to $20,000 | Volume of teacher outputs, teacher tier |
| Human validation on hard subpopulations | $6,000 to $30,000 | Number of strata, pay rate, label volume |
| Fine-tune compute on open base model | $3,000 to $12,000 | Base model size, epochs, hyperparameter sweep |
| Production deployment and observability | $10,000 to $25,000 | Self-host versus managed inference, telemetry depth |
| Eval cycles against the locked threshold | $4,000 to $8,000 | Two to three full passes |
A defensible total runs $35,000 to $120,000 for the project itself, before the running operating cost of inference, observability, and the re-distillation reserve. The lower end applies when an existing eval set and labeled dataset are already in place. The upper end applies when the team is building the eval harness, the training set, and the deployment surface from scratch; which is most first-time distillation projects.
The payback math against frontier-model spend is the headline number. A workload running 100 million tokens per month at frontier prices runs roughly $50,000 in monthly inference cost. A distilled student costs $1,500 to $4,000 monthly to operate. The differential of $46,000 to $48,500 per month pays back a $120,000 project in three months and a $35,000 project in three weeks. This is the IRR fact that drives most mature 2026 cost line; once the eval is locked.
The accuracy gap that is acceptable
Two to three percentage points on the production eval set is the defensible operating range for most enterprise workloads. Below two points the distillation project is over-engineered relative to the cost gain; the team has spent additional cycles tuning a student that was already good enough. Above three points the user-experience and trust impact begins to outweigh the inference savings; failures concentrate on the segments the student lost, and those segments are usually where the highest-value users live.
High-stakes workloads; medical, legal, financial substantiation; should hold to a stricter one-point gap or use the smaller model as a cache layer behind frontier-model fallback rather than as the primary inference path. The cache pattern is operationally clean: a confidence-aware router sends 85 to 95 percent of inputs to the student and the remainder to the teacher, capturing most of the cost savings while preserving frontier-model accuracy on the residual.
The cache-and-router pattern is the safer initial deployment. It removes the many-or-nothing decision, lets the team observe student behavior against teacher behavior on shared inputs in production, and creates a clean rollback path if the student regresses on a subpopulation. The pattern is detailed in the AI project model routing economics.
The volume threshold that justifies the project
Volume sets the floor of the decision. Below 50 million tokens per month, a distilled student does not amortize within twelve months even if the student matches teacher accuracy exactly. Inference savings are real but small. Project cost is the same. The math does not pencil.
Between 50 and 200 million tokens per month, distillation becomes a defensible call but not the only one. The team should weigh distillation against a model-routing approach that reduces frontier-model spend without the upfront student-training project; typically saving 30 to 50 percent of inference spend without the operational complexity of a self-hosted fine-tune. Routing is faster to ship; distillation produces deeper savings at higher complexity.
Above 200 million tokens per month, distillation is the structural answer for any workload narrow enough to support it. The cost differential dominates most other consideration, and the operational complexity of a self-hosted fine-tune is amortized across enough volume that the per-token overhead becomes invisible. Engagements at this volume that have not built distillation capability are leaving 60 to 80 percent of inference spend on the table; a number that shows up cleanly in the year-two cost curve documented in the AI project cost curve.
Where distillation loses
Three workload shapes produce distillation projects that fail to pencil.
Open-ended generation. Long-form creative writing, open-domain assistant chat, agentic planning over arbitrary tool catalogs. The input distribution is too broad and the long tail too sparse for any reasonable training set to cover. A distilled student on this kind of workload tends to look adequate on the eval set and visibly weaker on user traffic, because the eval set cannot represent the diversity the user inputs contain.
Rapidly evolving task patterns. Workloads tied to a domain shifting faster than the re-distillation cycle can keep up; emerging product taxonomies, fast-moving regulatory environments, novel content types; pay a re-distillation tax that erodes the cost case. The student is usually slightly behind the workload. Frontier models, with their knowledge cutoff but broader generalization, often perform better here despite their cost.
Workloads with strict reasoning depth requirements. Multi-step reasoning, mathematical problem solving, complex code generation. Smaller fine-tunes fall behind frontier models on these tasks faster than the eval set reveals, because the eval set typically cannot capture the long tail of reasoning chains the workload encounters in production. Distillation on these workloads tends to produce students that are confidently wrong rather than appropriately uncertain.
For these three shapes, the defensible 2026 posture is to run on a frontier model, accept the cost, and revisit the distillation question when the workload’s task pattern has stabilized or when the open-source model tier has advanced enough to close the reasoning gap.
The operating model: re-distillation and drift
Distillation is not a one-time project. The student model loses ground over time through two mechanisms.
Frontier-teacher upgrades. A new teacher arrives most three to nine months. The student trained against the prior teacher gradually appears worse versus production traffic that has shifted to assume the new teacher’s capabilities. A defensible engagement budgets one full re-distillation cycle per year; typically $20,000 to $60,000; and plans for the student to lag the current frontier by one generation.
Production-input distribution drift. The training set was built against last quarter’s input distribution. The production input distribution shifted. The student’s accuracy quietly drops on the segment that drifted. Without observability that compares student outputs against a teacher-on-sample, the regression is invisible until customers report it.
The mature operating model runs three permanent operating lines.
| Operating line | Frequency | Cost band |
|---|---|---|
| Teacher-on-sample re-evaluation | Weekly | $400 to $1,200 monthly |
| Drift-detection observability | Continuous | Included in observability stack |
| Full re-distillation cycle | Annual | $20,000 to $60,000 yearly |
These lines convert distillation from a fragile cost-reduction project into a durable operating posture. Engagements that named these lines in the original budget run distilled students for years. Engagements that did not run a student that quietly degrades until the team notices the failure rate has doubled and the cost case has eroded.
Frequently asked questions
Does the distillation case change for reasoning-heavy workloads where chain-of-thought matters? Yes; distillation is harder for any workload where the value comes from intermediate reasoning rather than final output. Distilling chain-of-thought patterns is an active research area in 2026, and most production teams should treat reasoning-heavy workloads as outside the distillation envelope.
Should distillation training data come from teacher inference or from human labels? Teacher inference for the bulk; human labels for the eval set and for failure-mode subpopulations the teacher itself is unreliable on. Pure teacher labeling propagates teacher mistakes into the student’s weights. Pure human labeling is too expensive to reach the volumes distillation requires.
How does distillation compare to prompt caching as a cost-reduction lever? They are complementary. Prompt caching reduces cost on repeated prefix patterns at the same model tier. Distillation reduces cost by changing the model tier itself. A workload using both can compound the savings. The interaction is detailed in the AI project caching strategy.
What hardware footprint is required to host a distilled 13B model? A single A100-class GPU at 4-bit or 8-bit quantization handles 60 to 200 requests per second on most workloads. For higher concurrency, a two-node setup with a load balancer is sufficient through several billion tokens per month. The hardware bill is typically 1 to 4 percent of equivalent frontier-model spend.
Does the distillation case interact with regulatory constraints? Yes; self-hosted distilled models address some data-residency and audit requirements that frontier vendors do not, which can shift the case in regulated industries even at lower volumes than 50 million tokens per month.
Should the distillation project sit inside the original engagement scope or as a follow-on? Almost usually as a follow-on. The original engagement should ship the workload on a frontier model, lock the eval, and produce volume baselines. Distillation is a phase-two project against a stable artifact, not a phase-one risk loaded into a new build.
How does distillation interact with the eval-threshold pricing model? Cleanly. The student’s payment milestone is “the student passes the locked eval threshold within the contracted accuracy gap.” This converts distillation from a vague cost-reduction promise into a structured deliverable with a binary acceptance test.
What is the right way to communicate the distillation case to a CFO? Show the inference-cost differential per million tokens at current volumes, the project cost, and the months-to-payback. CFOs accept distillation cleanly when payback is under nine months. They balk when payback runs over twelve.
Does the distillation case apply to multi-modal workloads? Partially. Vision-language distillation is operational in 2026 but lags text-only distillation by roughly one generation. Audio-language distillation is earlier still. Text-heavy workloads are the canonical case.
What is the relationship between distillation and the AI project insurance line? Distillation reduces inference spend but increases the team’s responsibility for failure modes the frontier vendor would otherwise own. Self-hosted distilled models require a larger reserve for jailbreak, hallucination, and red-team incidents; detailed in the AI project insurance line.
Key takeaways
- Distillation wins when the workload is narrow, the volume is above 50 million tokens per month, and the eval set is locked. Missing any condition turns distillation into a regression risk.
- The defensible 2026 project cost band is $35,000 to $120,000, with payback typically inside three to seven months at workload volumes that justify the project at many.
- The acceptable accuracy gap is two to three points for most workloads, one point for high-stakes domains, and zero gap when distillation is used as a cache behind frontier fallback.
- The 50-million-tokens-per-month threshold is the floor; above 200 million tokens, distillation is the structural answer for any narrow workload.
- Distillation loses on open-ended generation, rapidly evolving domains, and reasoning-heavy workloads; these should stay on frontier models.
- The mature operating model runs weekly teacher-on-sample re-evaluation, continuous drift observability, and an annual re-distillation cycle. Engagements that skip these lines watch student accuracy degrade silently.
- Distillation pairs with model routing and prompt caching as complementary cost-reduction levers. The combined effect on year-two spend is the largest IRR move available on stable workloads.
Arthur Wandzel
