Most AI training metrics die in the same thirty seconds. Someone presents a slide, a finance reviewer asks where the comparison number came from, and the answer is that there isn’t one. The metric was never wrong, only uncheckable, which in a budget meeting is worse.
This is a field guide to the metrics that get put in front of a CFO, sorted into five families, each with a verdict. Some survive questioning. Some survive only with a stated discount, and three of them look rigorous and are not.
The test that separates them is short enough to apply in a hallway.
Two different things get called AI training metrics
Search the phrase and half the results are about training models, not people: floating-point operations, loss curves, the cost of a frontier run. That reading has its own literature and none of it applies to you.
The other reading is the one you need: whether teaching your staff to use the tools you already pay for changed anything you can point to. Keep the two apart in your own documents, because the frightening numbers from the first reading get quoted in meetings about the second, where they settle nothing.
The three questions every metric gets asked
A finance reviewer is not hostile to AI enablement metrics. They are hostile to numbers whose construction they cannot inspect. Every metric gets asked three things, usually in this order.
Where did the before-value come from? Not an estimate of it. A record, taken at a specific time, by a named person, from a source you can reopen. This is where most presentations end, because nobody took the reading before the first session and a baseline is the one input you cannot reconstruct afterward. Our look at the cost of an unmeasured program works through why that gap is unrecoverable.
What else could have caused the change? A new tool version, a quieter quarter, two strong hires, a process fix that shipped the same month. You do not need to rule out every rival cause, only to name them before someone else does, which is the substance of separating what the training did from what the tool did.
Which decision does this change? If the number moves and nothing happens differently, it is a reporting habit, not a metric. Renew or do not renew. Run the session for the next department or stop. Give the eleven people who went quiet a refresher or accept the loss.
A metric that answers all three survives. One that answers two survives with a caveat. One that answers the third alone is a story.
Family 1: activity metrics
These come free from the admin console, which is why they are on every slide. Microsoft’s own reporting is a fair sample of what the category gives you. Its Copilot Chat usage report returns active users, average daily active users, total prompts submitted and average prompts per user, plus a per-person table of prompts submitted, active days and last activity date, over 7, 28, 90 or 180-day windows, with each day’s data landing about 48 hours later.
That is a real instrument and it answers one question well: has anybody stopped. A person whose last activity date is six weeks old is a fact, and it names a decision, which is to call them.
What it cannot do is carry weight as a result. Prompt volume rewards itself the moment you set it as a target, and nothing in the console knows whether a prompt produced usable work. EY makes the same point to boards, that usage, license counts and basic engagement are useful signals rather than answers to the strategic question of whether the business runs differently.
Verdict: survives as a hygiene check, fails as evidence. Report it in one line, by role, never as a headline. Collection cost: minutes, already paid for.
Family 2: learning metrics
Completion rates, satisfaction scores, quiz results. These are the metrics a vendor volunteers, and the reason is structural rather than sinister: a vendor is paid on delivery, so it measures delivery.
Completion measures attendance. A room can finish a session at 100% and leave with nothing, and for AI work that gap is wider than usual, because fluency is built by repeated use in the weeks after the room empties. Satisfaction is weaker still. It tracks how engaging the session felt, which is not what you bought.
Quiz scores sit a little higher. A recall test proves someone retained a definition, not that they can get a usable draft out of a tool on their own documents.
Verdict: fails. Completion stays in your operations tracker, where attendance belongs. Keep satisfaction if you want to know whether to rebook the same facilitator, and label it as that. Collection cost: zero, which is part of the temptation.
Family 3: capability metrics
This is the family almost nobody collects, and the one that carries a finance review best relative to its cost.
The method is unglamorous. Pick one artifact the role produces: a client update, a scope summary, a reconciliation note, a first-pass job description. Have each person produce one before any training, under a timer, and score it on a short rubric written in advance.
Repeat it after. You now own a before-value and an after-value on the same task, same people, same scoring sheet.
Two things make this hold up. The rubric is fixed before anyone sees results, so the grader cannot drift, and the artifact is the real work product, so nobody has to accept a proxy. The approach also keeps the claim at an altitude where checkable numbers sit. Measured gains in the published literature are task-level rather than company-level: Stanford’s AI Index reports around 14 to 15% in customer support and 26% in software development for AI use on specific tasks.
The cost is the honest objection: two timed rounds for 30 people runs about a day of someone’s attention plus 40 minutes per participant. For most mid-market programs that is still the cheapest defensible evidence available, and our measurement playbook sets out where it sits against everything else.
Verdict: survives. Collection cost: one day of design, 40 minutes per person.
Family 4: work metrics
Capability shows someone can. Work metrics show whether they do, on live output, after the session.
Three are easy to produce. Cycle time on one named deliverable, from the same two system dates you already capture. Rework rate, the share of drafts sent back, often the first number to move and the one that persuades skeptics, because nobody believes a speed gain that arrives with more errors. Hours returned on one defined task, narrow enough to time and to argue about honestly.
The fourth is newer and the most CFO-legible instrument published this year. In July 2026, OpenAI’s finance chief Sarah Friar proposed cost per successful task as the unit of account, with the denominator counting only work that clears a stated quality bar and the numerator carrying the full cost, including employee time, human review, retries and rework. Cheaper tokens, the argument runs, do not mean a cheaper outcome.
Read that denominator again from an enablement angle. Retries, review time and rework are the terms that move when someone learns to brief a tool properly and to check what comes back. Training vendors have not picked the framing up. Your CFO may well have, since the business press covered it the same week.
One caution on self-report, which belongs to this family and keeps escaping it. Asked estimates are uncalibrated in both directions, and our guide to measuring hours actually returned covers the randomized evidence in full. Collect the estimate, but never let it stand alone.
Verdict: survives, cycle time and rework most easily. Collection cost: low for the system dates, moderate for anything requiring a quality bar.
Family 5: money metrics
At some point the question becomes AI training ROI in the narrow sense: what did the money buy. Three constructions get used for it and they are not equally sturdy.
Hours saved multiplied by a loaded rate is the standard move and it is only as good as its footnotes. It needs the task named, the hours measured rather than asked, and an explicit statement of what happened to the freed time. Freed hours that went into more work are a capacity gain, not a cost saving, and calling them a saving is the single fastest way to lose the room.
Payback period survives well, because it is a date rather than a percentage, and a date can be checked.
Attributed profit impact is where programs overreach. McKinsey’s 2026 survey, fielded between May 4 and June 8 across more than 1,700 respondents, found 80% reporting better personal productivity while 37% attributed any EBIT impact to AI, with about 6% attributing at least 5% of EBIT. That spread is not their measurement failure. It is the real difficulty of the claim, and a 40-person department claiming cleanly what most large enterprises cannot is claiming too much.
Verdict: survives with a stated discount. Present the range, name the assumption you are least sure of, and let the reviewer argue with the assumption rather than with you.
The field guide on one page
| Metric | Family | Survives a finance review? | Cost to collect |
|---|---|---|---|
| Weekly active share of licensed users | Activity | Hygiene check only | Minutes |
| Prompts per user | Activity | No, rewards itself | Minutes |
| Days since last activity, by person | Activity | Yes, for one decision | Minutes |
| Completion rate | Learning | No | Zero |
| Satisfaction score | Learning | No | Zero |
| Scored artifact, before and after | Capability | Yes | One day plus 40 min per head |
| Timed task on real work | Capability | Yes | Same two rounds |
| Cycle time on a named deliverable | Work | Yes | Low, from system dates |
| Rework rate | Work | Yes, often moves first | Low |
| Cost per successful task | Work | Yes, with a stated quality bar | Moderate |
| Self-reported hours saved | Work | Only beside a measured figure | One survey question |
| Hours saved times loaded rate | Money | Yes, with footnotes | Inherits its inputs |
| Payback period | Money | Yes | Low |
| Attributed profit impact | Money | Rarely at this size | High |
Three metrics that look rigorous and are not
The composite index. Four dimensions, weighted 30/30/20/20, rolled into one score out of 100. The weightings are almost always invented, and the format’s real function is to hide which part is failing. If adoption is strong and output quality is flat, a blended 74 tells a reviewer nothing and tells you less.
The company-wide average. Averaging a finance team that uses AI daily with a field team that has not logged in since the session produces a number true of nobody. Report by role or do not report. The gap between roles is usually the most useful finding in the whole exercise.
Any metric with no owner’s name on it. A number nobody is accountable for drifts in definition between quarters until two reports disagree and both are defensible. One owner, one source, one window, written down before the first session.
What to report
Three metrics, one page. A capability score with its before-value and its rubric. One work metric, by role, with the two dates it came from. One money figure with its assumption stated and a range around it.
Underneath that, one line of activity data as a hygiene check, and the names of the rival explanations you considered.
That page is harder to produce than a dashboard and far harder to dismiss. It makes the argument our case against treating license count as capability makes at the level of strategy, in a form a reviewer can check line by line.
If you are earlier than that, in the position of approving rather than reporting, the questions worth asking before you sign are a better starting point than any metric list.
If you want a second read on which three numbers your situation can support, a free AI-readiness call is the fastest way to get one. We run LLM fluency sessions on a team’s own documents, and we treat the measurement design as part of the scope rather than something bolted on afterward.
Frequently asked questions
What are the most useful AI training metrics?
A capability score taken before and after on a real work artifact, one work metric such as cycle time or rework rate, and one money figure with its assumptions stated. Programs that report fifteen metrics are usually compensating for having no baseline on any of them.
Is usage data from our admin console enough?
No, though it is worth one line. Consoles report active users, prompt counts and last activity dates. None of that knows whether a prompt produced usable work, and prompt volume stops measuring anything the moment you set it as a goal. Use it to find people who have gone quiet.
Are completion rates worth reporting at all?
Keep them in your operations tracker, not in the business case. Completion measures attendance. For AI work the gap between attending and being able is unusually wide, because fluency is built by repeated use in the weeks after the session rather than inside it.
What is cost per successful task?
The full cost of producing work that clears a stated quality bar, divided by the number of times it cleared. OpenAI’s CFO proposed it in July 2026, and the cost side includes employee time, human review, retries and rework. A mid-market company can compute it on one workflow, which is the right scope to start with.
We never took a baseline. Can we still measure anything?
Yes, with a smaller claim. Run the capability assessment now and treat it as the baseline for the next phase, compare roles against each other rather than against a missing past, and pull historical cycle times from whatever system already holds the dates. What you cannot recover is a timed before-reading on a deliverable, which is why the next program should open with one.
Can we use self-reported hours saved?
Only beside a measured figure. Asked estimates are uncalibrated in both directions, and the usual pattern is that people report large gains that objective timing does not confirm. Collect the estimate, record the measured number separately, and treat the difference between them as its own finding.
How soon should any of these move?
Activity data moves within days and tells you little. Rework rate is often the first real signal, within four to six weeks. Capability scores need a second assessment, usually at 60 to 90 days. A money figure anyone will defend takes two quarters, because it needs enough cycles to rule out a quiet month.
Who should own the number?
One named person who is neither the vendor nor the program’s sponsor. The vendor’s survival depends on the score, and the sponsor’s credibility does. Finance can hold the definition while the sponsor holds the work, but the definition belongs in writing before the first session.
How many metrics should an AI enablement program have?
Three to five, each tied to a decision somebody will actually make. If you cannot name the decision a metric changes, drop it. The point of measuring AI adoption is not a fuller dashboard. It is being able to answer renew, repeat, or stop.
Key takeaways
- Every metric gets asked the same three questions: where the before-value came from, what else could explain the change, and which decision it changes.
- Activity data is a hygiene check. It finds people who have gone quiet and proves nothing about capability.
- Completion and satisfaction scores measure delivery, which is what the seller is paid for, not what you bought.
- A scored artifact before and after is the cheapest evidence that holds up, at roughly 40 minutes per person.
- Cost per successful task counts retries, review and rework in the denominator, and those are the terms fluency training moves.
- Report three metrics by role on one page, with assumptions stated, rather than a blended score that hides which part is failing.
Arthur Wandzel