Most AI training KPIs fail for construction reasons, not because somebody picked the wrong metric. The number has no owner, or no before-value, or no threshold that would have counted as failure, so when the quarter ends there is a dashboard and no decision. Nine rules fix that, and all nine are about how a KPI is built rather than which one you choose.
Search this topic and you get ten versions of the same metric list: adoption rate, time saved, completion, sentiment, quality. The lists are fine. What none of them supplies is the test a candidate KPI has to pass before it earns a row on the page that goes to your CFO. Here are the nine tests and the five-row specification they produce.
Rule 1: Every KPI names the decision it will change
Write the decision next to the metric, in the same row, before you start collecting. If no decision moves when the number moves, you have a report line, not a KPI.
A company of 50 to 500 people has three real decisions after a first round of training: whether to fund a second phase, who gets trained next, and whether the license allocation matches where the work is. That is the whole set. A measurement that cannot change one of those three costs someone an afternoon a month and is not paying for it.
The test is blunt. Take each candidate metric and finish this sentence: “If this lands at X, we will do Y instead of Z.” A metric that cannot complete the sentence goes in an appendix, where it is useful context without pretending to be evidence. The sheet these numbers end up on has its own structure, covered in our walkthrough of the one-page business case.
Rule 2: Write the failure number before the first session
Name the result that would mean the program did not work, in a number, with a date, signed by whoever authorized the spend. “If median turnaround on the client update has not moved by 20 January, we do not buy phase two.”
A threshold written after the result arrives is not a threshold. It is a description of the result with a threshold’s grammar, and anyone in finance recognizes it on sight. The only version that carries weight is the one dated before the first session, which is also the version that can make the sponsor look wrong later. That discomfort is the entire value.
It is the cheapest item in the exercise and the most often skipped. It belongs in the pre-training record with everything else you collect that week, laid out item by item in our guide to what to measure the week before training starts.
Rule 3: No KPI without a before-value you can produce this week
If you cannot produce the before-value in the next five working days, from a system you already have access to, the KPI is a wish. Drop it and pick one you can baseline.
This is where the classic evaluation ladder breaks. The Kirkpatrick model runs reaction, learning, behavior, results, and most programs report the first two levels because reaction and quiz scores need no history. Behavior and results need a before-value nobody captured, so they get narrated instead of measured.
Two before-values are irrecoverable once training is on calendars: how long a named deliverable took, and an untouched work sample. Console figures and your cost basis can both be pulled late. Sort the candidate list by that distinction and collect the fragile items first.
Rule 4: Fix the denominator before you argue about the numerator
Adoption rate is active users divided by eligible users. Every source gives that formula and almost none defines “eligible,” which is where the number goes wrong.
Licensed, eligible, and headcount are three different populations. Licensed is a procurement artifact; headcount includes people whose work the tool does not touch. Eligible is a named list of people for whom using the tool on a specific task is the expectation, frozen in writing at the start, with a stated rule for joiners and leavers.
The failure mode is documented by the vendors themselves. Microsoft’s Copilot Dashboard documentation notes that the active-user count can exceed the licensed-user count, because it counts everyone who took an action in the 28-day window including people whose license was removed afterward. An adoption percentage above 100% is not a data error there; it is two populations being divided by each other. If your denominator is a frozen list of named people, that cannot happen.
Rule 5: One owner, one source, one window, written down
Each KPI gets one owner who can produce it unaided, from a system they already have access to, plus a named source system and a named window. Four fields, one line of text, decided before collection starts.
The owner test matters more than it sounds. A metric that needs three people to coordinate gets produced twice and then quietly stops, so put each one on whoever already holds the system: finance for the cost basis, the tool administrator for the console pull, the function lead for deliverable timing.
The window has to be the source’s window, not one you prefer. Microsoft counts an active user as someone who performed at least one intentional action in a supported app within the previous 28 days, and its dashboard pages carry a reporting delay. Google’s administrator reporting has fixed retention periods that administrators cannot change or delete. You inherit those definitions; what you control is writing down which one you used, because switching sources between the before and the after ends the comparison without announcing it.
Rule 6: Never let a self-reported hour stand alone
Ask people how much time a tool saved them and you get a number whose direction can be wrong, not just its size.
The cleanest evidence is a randomized trial published by METR in July 2025. Sixteen experienced open-source developers worked 246 real issues from repositories they knew well, with AI assistance randomly allowed or disallowed per issue. They forecast a 24% speed-up beforehand. Measured, they took 19% longer with assistance. Afterward, they still believed it had made them 20% faster.
Sixteen developers on early-2025 tooling is a small study and not a verdict on current assistants. It is enough to retire self-report as a standalone KPI. Keep the survey — it tells you where people think the friction is, which is useful for curriculum — and pair every claimed hour with one observable: a timestamp on an approved artifact, a turnaround on a named deliverable, a count of review rounds. Turning that pairing into a defensible claim is the method in our playbook on proving capability moved.
Rule 7: Usage is a hygiene check until it names a task
The share of people active in a window is a hygiene check. It tells you licenses are reachable and nothing about whether anyone’s work changed. Report it, do not promote it.
Prompt counts are worse. The same task takes three prompts from a person who writes a good one and eighty from someone iterating blindly, so a rising count is as consistent with confusion as with fluency.
A usage metric becomes a KPI the moment it names a task and an approval: the share of eligible client updates drafted with assistance and approved without extra review rounds. That version carries a decision. A low figure says the training did not reach the work; a high figure with rising rework says it reached the work and degraded it.
Rule 8: Report by role, never as one company average
One company-wide average is the fastest way to describe nobody. Effects in this field are concentrated, not evenly spread.
The largest field study available remains Brynjolfsson, Li and Raymond’s Generative AI at Work, covering 5,179 customer support agents. Average productivity rose 14% in issues resolved per hour. Novice and lower-skilled agents gained 34%. Experienced, higher-skilled agents saw minimal effect. It is a working paper covering one job with one assistant, but the shape of the finding is the point: average that distribution and you get a figure matching neither cohort.
For a 40-person company that means five rows rather than one, plus a note on tenure where a function has both new and long-serving people. Rows comparable across roles need one rubric used twice, which we set out in our framework for scoring a team before and after.
Rule 9: Every KPI gets a review date and a retirement rule
Set both when you create the metric: the date somebody looks at it, and the condition under which it leaves the page.
Retirement conditions are the part nobody writes. Reasonable ones: a hygiene check retires once it holds above its line for two consecutive reporting periods; a capability score retires when its cohort finishes the curriculum; a time-on-deliverable measure retires when the decision it fed has been made. Without a retirement rule, dashboards only grow, and a page of eleven numbers gets read as a page of none.
Monthly and quarterly review dates hide different things: a monthly read catches an access problem while it is still fixable, and a quarterly read is the only one finance treats as a trend. That trade-off is worked through in our comparison of monthly and quarterly reporting.
The five-row specification these rules produce
Five KPIs is the whole set for a company in this size band. Each row below carries the four fields the rules demand.
| KPI | Decision it changes | Owner | Source and window | Failure line |
|---|---|---|---|---|
| Median time on one named deliverable, by function | Fund phase two or stop | Function lead | Timestamped drafts and the approval log, three instances per person per quarter | No movement by the dated threshold |
| Share of eligible deliverables produced with assistance and approved | Re-scope the program or re-train | Function lead | Approval log, monthly | Below 40% at 60 days |
| Review rounds per approved deliverable | Whether any time claim survives | The reviewer | Same approval log, monthly | Above the pre-training figure |
| Capability score on dated samples, by role | Who gets trained next | Whoever scored the baseline | Scored work samples, one rubric, before and after | Fewer than half the cohort moves a band |
| Active in the window, against the frozen eligible list | Access and licensing, not training | Tool administrator | One named vendor console, its own window | Below 60% at 30 days |
Four of the five come out of systems that already exist: a drive, an approval trail, a console. The fifth needs somebody to score samples twice against one rubric, a few hours per round for a group of 40.
Keep the measurement proportionate to the spend. Team workshops in this market generally run between $2,000 and $15,000, and a program in that band does not justify an apparatus that costs more than the training. The argument underneath all nine rules — that capability is something you build and verify rather than something a license purchase delivers — is made in full in our piece on why buying licenses is not the same as building capability.
Frequently asked questions
How many AI enablement KPIs should a program have?
Five, with a hard ceiling of seven. Vendor guides commonly suggest three to five for a first program and five to seven at maturity, and the binding constraint is attention: a page of eleven numbers gets skimmed and none of them drives a decision. Pick one capability measure, one work-output measure, one quality measure, one coverage measure, and one hygiene check.
Who owns these KPIs when nobody owns AI?
Spread them across the people who already touch each system, and have one person coordinate rather than collect. With no dedicated AI lead, that means finance owns the cost basis, the tool administrator owns the console figure, each function lead owns deliverable timing for their own team, and the sponsor owns the failure line. Nobody takes on a second job and no number depends on three calendars aligning.
What is a realistic adoption target in the first 30 days?
Set it against your frozen eligible list rather than a published benchmark, because vendor figures count different populations over different windows. A reasonable first-month line is that most of the eligible list has used the tool on a real task at least once. The useful signal is not the percentage but which names are missing: absences cluster by team, and the cause is usually access or workload rather than willingness.
Can vendor dashboards supply the whole KPI set?
No. They report activity and seats, which covers one of the five rows. They cannot tell you whether judgment improved, whether output quality held, or whether the saved time merely moved to a reviewer. Use the console for the hygiene check, record which console and window you used, and build the other four rows from your own approval trail and work samples.
Are completion rates ever worth reporting?
Only as a delivery check during rollout. Completion runs near-universal in mandated programs and predicts almost nothing about behavior, so it belongs in an operational update rather than on the page that justifies the next phase. If your learning system produces it free, keep it off the executive sheet.
How do we handle self-reported hours saved?
Collect them, never publish them alone. Pair each claimed hour with one observable from a system that does not depend on recall — a timestamp, an approval date, a count of review rounds — and report the pair. The METR trial is the reason: participants there believed assistance made them 20% faster while measurement showed them 19% slower, which means self-report can be wrong about the direction and not merely the magnitude.
Do we need a control group?
Something that isolates the program’s effect, yes, though a formal control group is rarely available at this size. The Phillips ROI approach, which adds a fifth return-on-investment level above the four classic ones, treats isolating that effect from other influences as a required step rather than a refinement. The practical versions are a staggered rollout where the team trained second serves as the comparison, or a comparable untrained function measured over the same window.
When should a KPI be retired?
On the condition you wrote when you created it: a hygiene check once it holds above its line for two reporting periods, a capability score once its cohort finishes the curriculum, a deliverable-timing measure once the decision it fed has been taken. Retiring a metric is a result, not an admission, and it keeps five rows from becoming eleven.
How soon can a KPI show movement?
Coverage and hygiene figures move within two to four weeks, which is also the window where a new habit is easiest to see and easiest to lose. Deliverable timing needs a full production cycle, so a monthly artifact means roughly one quarter before a trend is readable. Quality measures move last, and sometimes the wrong way first, as reviewers catch what the drafter stopped checking.
Key takeaways
- A KPI is valid when it carries four fields: the decision it changes, one owner who can produce it unaided, a named source and window, and a failure number dated before the first session.
- Define “eligible” as a frozen list of named people. Licensed and headcount are different populations, and vendor documentation confirms an active-user count can exceed a licensed count.
- Self-reported time savings cannot stand alone. In a randomized trial, participants believed they were 20% faster while measurement showed 19% slower, so pair every claimed hour with one observable.
- Report by role. A 14% average that was 34% for novices and near zero for experienced staff is a number that describes neither group.
- Five rows, five owners, one afternoon a month, each row with a review date and a retirement condition. Dashboards that only grow stop being read.
Working out which five numbers your program can actually produce, and whether the training you have been quoted would move any of them, is most of what a first conversation covers. Book a free AI-readiness call and we will go through your candidate set against the deliverable you want to claim.
Arthur Wandzel