Back to Guides
Enterprise Software 17 min read

The Attribution Problem: separating training effects from tool effects

The Attribution Problem: separating training effects from tool effects

Your team got faster after the workshop. The question your CFO will ask is whether the workshop is why. This is the AI training attribution problem, and it has a specific shape: the licenses and the training arrived close together, so the gain you measured has four parents and your report credits one of them.

Only one published experiment has separated a training arm from a tool arm. It found the training’s marginal contribution was real, statistically significant, and roughly a tenth of the total quality gain. It also found the better-trained group made more errors on the one task the model could not handle. That is an uncomfortable result and a usable one, because it tells you which part of a session is worth paying for and which part is already in the license.

The four parents of any post-training gain

Treat the observed change as a sum of four quantities. Most reports collapse them into one and label it with whatever line item is under review.

Component What it is Who usually gets the credit
Tool effect What the model adds — drafting, retrieval, summarizing — to anyone with a login The training budget
Capability effect Prompting skill, verification habits, judgment about what to route to the model The license renewal
Interaction The part existing only when both are present. Complements, not addends, so subtraction cannot split it off Nobody
Drift Novelty, learning by doing, a quiet model upgrade, a process change shipped the same quarter The training budget

The fourth row is where mid-market programs lose the argument. A figure measured six weeks after a session sits inside a window that also contains a vendor release, a seasonal workload shift, and three weeks of people trying something because it is new.

So: the first two rows are separable with planning, the third is not separable at all, and the fourth is controllable only by comparison. That is the whole method.

The one experiment that split the arms

The jagged-frontier study by Dell’Acqua and colleagues is quoted constantly for its headline quality number. Almost nobody reports that it ran a training arm.

The pre-registered experiment put 758 Boston Consulting Group consultants — about 7% of the firm’s individual-contributor consultants — into three conditions, after first establishing a baseline on a comparable task. The conditions were no AI, the model alone, and the model plus a prompt-engineering overview of instructional videos and documents on effective usage.

That third arm is a short, structured enablement intervention. It is the closest thing in the literature to a clean read on what training adds once the license is already in hand.

Inside the model’s capability frontier, across 18 realistic consulting tasks:

Arm Human-graded quality vs. control mean of 4.1 Task completion
No AI (control) — 82%
Model only +1.56 (+38%) ≈91%
Model + overview +1.75 (+42.5%) ≈93%

The paper reports that the overview arm “consistently exhibits a more pronounced positive effect,” with the equality test against the model-only arm “below or around the conventional 5% threshold in all specifications.” Training’s marginal contribution is statistically real. It is also about four and a half points of a roughly forty-point gain.

Then the finding that should change how you design a session. On a task deliberately chosen to sit outside what the model could do, the control group was correct 84.5% of the time. The AI arms scored 60% and 70%, and the regression attributes −24 percentage points to the overview arm against −13 for the model-only arm. The better-trained group was more wrong. The authors supply the mechanism: the overview increased “retainment” — copying and pasting the model’s output — and retainment predicted better performance on work the model handled well.

Read together, those two results give you the most useful sentence in this literature. A short overview that teaches prompting moves usage behavior; usage behavior raises throughput on suitable work and confident error on unsuitable work. Teaching people to prompt harder is not neutral. Teaching them where the frontier sits is what protects the downside, and it is the part a vendor demo almost never covers.

Two anchors, from opposite directions

Before crediting a workshop, set a floor for what the tool does on its own.

Brynjolfsson, Li and Raymond studied the staggered introduction of a generative AI assistant across 5,172 customer support agents, published in the Quarterly Journal of Economics in 2025. Access alone raised issues resolved per hour by 15%. Less experienced and lower-skilled workers improved on both speed and quality; the most experienced saw small speed gains and small quality declines. (Earlier working-paper versions report 14% across 5,179 agents — quote one version, not a blend.)

The mirror image is newer. In IZA Discussion Paper 18224 (October 2025), Fouarge and co-authors examined a financial services company where an AI tool generated performance feedback on call center agents to guide their training. Identification came from the staggered rollout of that system. The AI-augmented training cut call handling time by 10%, with larger effects for short-tenured workers and better communication style among long-tenured ones. The treatment there is coaching, not access.

Two consequences. If your post-rollout gain is in the low double digits, the published tool effect alone can account for it, and a report crediting the workshop asserts something the data does not contain. And any claim of a 40% improvement needs to explain which of these two channels it beat.

Five designs, ranked by what you can afford

The textbook answer is a 2×2 factorial: randomize tool access and training independently, then read the interaction. It is usually unavailable. Powering an interaction takes roughly four times the sample of a main effect of the same size, and withholding a paid license from one group for a quarter is a conversation most operations leaders decline to have. Below it, in descending order of affordability at 50–500 people:

Design What it identifies Cost What breaks it
Staggered cohorts — train team A in March, comparable team B in May, same deliverable, same dates Training effect, tool and calendar held constant Low; a scheduling decision Teams that are not comparable; talk across the boundary
Order reversal — A gets the tool first, B the training first Both incremental effects, each group its own control Low to moderate Needs control over provisioning dates
Dose-response — all licensed; randomize a 60-minute briefing vs. a half-day vs. ongoing coaching Marginal return to training intensity, the actual decision Moderate Needs enough people per dose
Tool-removal check — at eight to twelve weeks, score a work sample with the assistant off Whether capability transferred or dependence formed Low, and uncomfortable Reads as a test; frame it as a baseline exercise
Event study on two dates — plot the metric against activation and against the training date Separation of two effects, from data you already hold Almost none Requires that the two dates differ

Staggering is the default and the cheapest credibility available. Nothing is withheld permanently, everyone is trained inside a quarter, and it absorbs the confound nobody schedules: a vendor shipping an upgrade mid-window. Our note on the 30/60/90 checkpoints treats that comparison as a gate rather than a footnote, and it only works if the before-state was recorded first — the argument in our look at what to measure the week before training starts.

The tool-removal check deserves more use than it gets. It is the only design that distinguishes a capability you now own from a throughput gain you are renting, and it is the honest test behind questions about what still holds six months after a session.

The two dates you already own

Most companies have a natural experiment sitting in two spreadsheets and have never plotted it.

License provisioning dates and training dates rarely coincide. Seats switch on when procurement closes; sessions get scheduled when calendars allow. The gap is often six to sixteen weeks. Plot your outcome metric against each date separately, as an event study with the week of the event at zero. Three shapes, three readings:

  • A step at activation and a second step at the training date — two separable effects, and the size of each step is your decomposition.
  • A step at activation, a flat line at the training date — the session moved sentiment, not output. Check whether behavior moved before concluding it was worthless.
  • A smooth ramp with no step anywhere — learning by doing, not instruction. A model with both a step term and a cumulative-usage term separates these; a before-and-after average will not.

This costs an analyst an afternoon, and it is the highest-return hour in measuring AI adoption.

Three confounds that will invert your result

Composition. The tool’s documented gains concentrate among less experienced workers. Volunteer cohorts and “pilot with the keen team” selection both skew young, so a junior-heavy trained group hands training the credit for the tool’s novice skew. Stratify assignment by tenure and baseline performance, and fix the splits before you look.

Spillover. Trained people teach untrained colleagues, which contaminates the comparison group and biases the training effect toward zero. Randomize at team or site level rather than individual, and ask the comparison group whether anyone has shown them anything.

Mid-window releases. These assistants ship capability changes continuously, so a gain appearing in week three may belong to a model update. Keep a dated log of vendor release notes beside your metric and overlay the two before you present.

Two further hazards are worth a line each. Novelty and observation effects take months to decay, so a window shorter than a quarter captures some of both. And speed gains often carry quality costs, so any decomposition has to run on a measure that is quality-adjusted and net of rework — the arithmetic worked through in our decode of time-savings claims.

When you cannot attribute dollars, attribute behavior

Below roughly a hundred people in a comparable role, no design will give you a defensible dollar attribution. Stop trying, and attribute the thing training is supposed to change.

Training should shift intermediate behavior, and that behavior should predict output. Both halves are measurable from telemetry and work samples:

  1. Task selection — the share of work routed to the model that it is suited to, and the share correctly kept away from it. Given the outside-frontier result above, this carries the most downside protection.
  2. Verification rate — how often output is checked against a source before it leaves the building.
  3. Refinement depth — single-shot prompts against multi-turn iteration, and prompt specificity.
  4. Reach — the share of eligible work touched, not logins.

Those four are the AI enablement metrics that survive a question about causation, because each one is a behavior training is meant to change rather than an outcome with many parents. Identical distributions across trained and untrained groups mean the session delivered awareness and nothing else, whatever the satisfaction scores say. Divergence that output follows gives you a mechanism rather than a correlation. That is a weaker claim than an ROI percentage and a much harder one to dismiss, which is the trade the metrics in our rules for setting AI enablement KPIs are built around.

Be wary of the opposite genre. Trade case studies reporting 616% returns after 60 days, or four-figure annualized percentages, usually rest on participant self-estimates of hours saved with no adjustment for self-report bias, and on a comparison group the author describes as too small to reach significance. For context, the Association for Talent Development’s 2019 evaluation research found 38% of organizations measuring at the business-outcomes level and 16% at the financial-results level. The discipline is thin across the board, not only for AI.

A worked example

A 180-person services firm, 120 licenses provisioned in January, two comparable 30-person operations teams. Figures are illustrative; the sequence is the point.

Step Decision or finding What it buys
Record the before-state on one deliverable, both teams, two weeks Cycle time and a graded quality sample A baseline predating any incentive
Train team A in March, team B in May Staggered cohorts A comparison group that costs nothing and expires
Plot both teams against the January activation date Step of +9% in both, no divergence The tool effect, separated and dated
Plot both teams against the March training date A further +5% in team A only, holding to May The capability effect, team B as counterfactual
Overlay the vendor’s release log One upgrade in February, inside the shared window Removes the obvious challenge pre-emptively
Score a work sample with the assistant off in week ten Team A above its baseline, team B at baseline Evidence of transfer, not dependence
Report the interaction as unidentified Stated as a limit A defensible range instead of a false point estimate

The report says: the tool accounts for roughly nine points, the training adds roughly five on top, the two cannot be cleanly separated beyond that, and the capability survived the tool being removed. A workshop in the $2,000–15,000 market range that produces that paragraph has been measured properly. One that produces a 616% figure has not.

The sequencing is the expensive part, and free if you decide it early — the argument running through our guide to proving capability moved and the broader case that buying licenses is not the same as building capability. Decided afterwards, attribution is a rhetorical exercise.

Frequently asked questions

How do you separate AI training effects from AI tool effects?

Create variation in one while holding the other fixed. The cheapest version is staggered cohorts: train one team six to eight weeks before a comparable team, both already holding licenses, then compare the same deliverable on the same dates. The difference during the gap is the training effect, net of the tool and the calendar.

Do we need a control group to prove AI training ROI?

You need a comparison, which is not the same as withholding something permanently. A later-trained cohort is a control group that dissolves on its own, which clears the fairness objection a true holdout does not.

Can usage dashboards attribute a gain to training?

No. Logins, seats and assisted-hour tiles measure reach and habit. They contain no counterfactual — no read on what would have happened anyway — so they cannot isolate the session’s contribution. Use them to check that the population you trained is the population using the tools, then measure outcomes separately.

How much of the productivity gain is the tool and how much is the training?

In the only clean three-arm experiment, the tool produced a 38% quality gain over control and adding a prompt-engineering overview took that to 42.5%. On work inside the model’s capability, training’s marginal share was about a tenth of the total. Treat that as a prior, not a benchmark; the split shifts with how close the work sits to the model’s limits.

Does better training always mean better results?

Not on work the model cannot do. In the same experiment the better-trained arm was 24 percentage points less likely to reach a correct answer on an out-of-scope task, against 13 points for the arm with the tool alone. Instruction that raises usage without teaching where the limits sit raises confident error along with throughput.

How long should the measurement window be?

At least three months before you treat a result as durable, because novelty and observation effects need that long to decay, and the same window shows whether the gain holds or fades. Shorter windows are fine for behavior checks and misleading for financial claims.

What if the trained group is more junior than the comparison group?

Then your result is partly a composition effect, because documented tool gains concentrate among less experienced workers. Stratify assignment by tenure and baseline performance up front, and pre-commit to the subgroup splits you will report.

Is a 2x2 factorial design realistic for a mid-market company?

Rarely. Powering the interaction takes roughly four times the sample needed for a main effect, and the training-only cell requires people trained but unlicensed. Below a few hundred people in comparable roles, use staggered cohorts or a dose-response design and state the limit plainly.

What should the report say if we cannot attribute the gain at all?

Report the bundle as a bundle, state the identification limit in one sentence, and claim only the behavior change you can evidence: task selection, verification rate, refinement depth, reach. A mechanism with a stated limit survives scrutiny that a single unexplained percentage does not.

Key takeaways

  • Any post-training gain sums the tool effect, the capability effect, their interaction, and drift. The interaction cannot be separated by subtraction, so the honest output is a range with a stated limit.
  • The one three-arm experiment puts the tool at +38% quality over control and the tool plus a prompt-engineering overview at +42.5%. Training’s marginal share was real, significant, and about a tenth of the total.
  • On work outside the model’s capability, the better-trained arm was 24 percentage points less likely to be correct, against 13 for the tool alone. Teaching the frontier matters more than teaching prompt syntax.
  • Tool access alone raised issues resolved per hour by 15% across 5,172 support agents. A gain in the low double digits may be entirely the license.
  • Coaching without changing access cut call handling time by 10% in a staggered financial services rollout. Both channels work; your report has to say which one it beat.
  • Staggered cohorts are the cheapest defensible design and the only one that automatically absorbs a mid-window vendor release.
  • Plot your metric against the activation date and the training date separately. Two steps mean two effects; a smooth ramp means learning by doing.
  • Composition, spillover and model upgrades are the confounds most likely to invert a mid-market result. Handle them in the design, not the discussion.
  • When dollars cannot be attributed, attribute behavior. Task selection and verification rate are what training should move, and both are measurable.

If you want the sequencing set up before anyone sits down — which deliverable to time, which team to hold back, what the number will and will not prove — book a free AI readiness call. The decisions that make attribution possible are all made in the two weeks before a session, not the quarter after it.

Last Updated: Nov 14, 2026

AW

Arthur Wandzel

SFAI Labs helps companies build AI-powered products that work. We focus on practical solutions, not hype.

Make your team fluent in AI — then automate what proves out

  • Hands-on training applied to your own documents and workflows
  • Often reimbursable through state small-business training programs
  • Built for 5–50 person firms with no IT department

Related articles