Back to Guides
Enterprise Software 15 min read

The end of seat-license reporting as an adoption metric

The end of seat-license reporting as an adoption metric

The slide says 94% adoption. What the dashboard counted is the share of licensed people who took one deliberate action inside a 180-day window. Those are not the same claim, and the gap between them is where most AI training vanity metrics live. Seat-license reporting is a reach metric read as a capability metric, and once you see the substitution you cannot report the number the same way again.

Reach is the one thing an AI enablement program can always produce. Licenses are easy to assign and count; capability is neither. The easy number gets promoted into the slot the hard number should occupy, and a year later nobody can say whether the spend changed how anyone works.

What the dashboard actually counts

Read the vendor’s own words before your own slide. Microsoft documents the Copilot usage report in the Microsoft 365 admin center with three headline figures:

Metric Microsoft’s definition What it measures
Enabled Users “the total number of unique users in your organization with Microsoft Copilot licenses over the selected timeframe” Purchasing
Active Users “the total number of enabled users in your organization who tried a user-initiated Microsoft Copilot feature, in one or more apps” Reach
Active users rate active users divided by enabled users Reach, as a percentage

The verb is tried. The report’s FAQ is precise about the threshold: a user counts as active “if they perform an intentional action for an AI-powered capability.” Opening the Copilot pane from the Word ribbon does not qualify. One prompt does.

Now add the timeframe control. The report runs over 7, 28, 90, or 180 days. A single prompt in April makes a person an active user in a report pulled in September. At the 180-day setting, the active users rate is a lifetime-reach counter wearing the clothes of an adoption rate.

Two more details matter before you quote the figure. The user-level table lists everyone who held a license at any point in the past 180 days, “even if the user later removed the license or never had any Copilot active usage.” And since December 11, 2025, viewing a Loop document generated by the Teams Facilitator feature counts toward active usage. Reading AI-written meeting notes is now an adoption event.

None of this is a defect. The report is a reach counter and documents itself as one. The failure is downstream, in the slide that turns it into a capability claim.

Nor is the pattern specific to one vendor. GitHub exposes a last_activity_at field per assigned Copilot seat, and its rollout guidance treats a seat as idle after 30 days, but its changelog sets any value older than 90 days to null. A blank means dormant or never started, and nothing in the field separates the two.

Four questions a seat number cannot answer

Four questions decide whether an enablement program earned its budget. Reach answers none.

Question What reach says What would answer it
Are people using this on real work? Nothing. One prompt about the weather counts. Share of a team’s deliverables that passed through the tool
Did the habit survive? Nothing. The window hides the decay. Active days per user, plotted over three months
Did output get better, or just faster? Nothing. A quality check on a fixed sample, scored the same way before and after
Which people are stuck? Nothing. An aggregate hides every person. The distribution, not the mean

The fourth row is the expensive one. A percentage has no shape, so a team where six people are fluent and twenty-four logged in once reports identically to a team where all thirty sit mid-curve. Those two cases call for opposite fixes. Building a view that shows the shape is the subject of our guide to an adoption dashboard a manager will actually open.

The test that sorts a useful metric from a flattering one

Eric Ries drew the line in The Lean Startup: some numbers help you learn, and some help you look good. Cumulative registered users only climbs, so it always flatters; retention by cohort can fall, so it can teach you something. Two screens come out of that, and they catch nearly everything.

If it can only go up, be suspicious. Seats assigned, prompts ever sent, people who ever tried the tool: monotonic by construction. A number that cannot carry bad news cannot inform a decision.

If you cannot name the decision it would change, it is decoration. Ask what you would do differently at 94% versus 71%. If the honest answer is “put a different figure on the slide,” you have a vanity metric. If it is “pause the next license tranche and run a session for the bottom two teams,” you have a useful one.

Applied to the familiar set:

Reported number Can it fall? Decision it changes
Seats assigned No None
People who have tried the tool No None
Workshop attendance No None
Post-session satisfaction score Yes Who facilitates next time, little else
Active days per user, this month vs. last Yes Where to put reinforcement effort
Share of a defined task now done with the tool Yes Extend, pause, or redesign

The bottom two rows are harder to produce. That is the whole reason the top three keep appearing.

The vendor writes the definition

The structural problem under all of it: the party selling seats defines “active,” reports it, and benefits when it reads high.

That is not an accusation of bad faith. Microsoft publishes its definitions openly, which is more than most software companies do. But the incentive runs one direction, and it shows in choices that are defensible alone and favorable together: a threshold of one intentional action, a window out to 180 days, a user table that keeps people who gave the license back, a passive document view promoted into an activity event.

The same report warns you off the obvious shortcut. Microsoft says audit-log data is built for security and compliance and is “not, however, intended to be used as the basis for Copilot usage reporting,” and that metrics built on it may not match the official figures. The one dataset with real per-interaction detail is unsupported for this purpose, and the supported number is the generous one.

Treat the active-user rate the way you would treat a supplier’s own quality score: worth collecting, not the figure you present as proof. Gallup’s Q2 2026 wave, fielded May 6 to 20 among 22,573 employed US adults, found 52% using AI in their role, 30% a few times a week or more, and 15% daily. An internal reach figure far above that frequent-use band is measuring something other than habit.

Depth is where the return lives

The strongest argument against reach reporting is not that it overstates things. It is that it measures the wrong axis. The same Gallup wave asked users how many distinct purposes they put AI to, then whether it had improved their productivity. The gradient is steep:

Purposes used for Share reporting a productivity gain
1–2 45%
3–4 66%
5–6 78%
7 or more 90%

Read that as a scorecard. A person who found one use is roughly a coin flip; a person who found seven or more is close to a certainty. Reach reporting cannot see the difference, because both land in the same column.

The most useful number in the Microsoft report is not on the summary tiles. The exported user table carries Active Days, Prompts submitted (any app), and a per-app Last activity date: enough for a depth distribution and a decay curve, at the cost of a spreadsheet rather than a new analytics purchase. The honest metric is already in the tool you pay for, one tab further in.

Four layers worth reporting

Replace the single rate with four layers of AI enablement metrics. Each answers a different question, and each can get worse.

Layer What it answers Where it comes from
Reach Did the license and the session land on the right people? Enabled users, against the roster you meant to train
Depth How many distinct jobs is each person using it for? Active days and prompts per user, as a distribution
Retention Is the habit holding at 30, 60, and 90 days? Last activity date, plotted against the training date
Outcome Did the work get faster or better, on a named task? A fixed sample of real deliverables, before and after

Report reach once, as a denominator, never as the result. Report depth and retention monthly, with a median and a bottom quartile. Report outcome quarterly against a baseline taken before anyone was trained.

Retention exposes a training problem earliest, because skills fade without reinforcement faster than most programs expect, the subject of our piece on what actually sticks six months after a session. Outcome is the layer that carries any AI training ROI claim, and the one most likely to rest on a time-savings estimate nobody checked; decoding time-savings claims covers how to measure hours actually returned rather than hours asserted.

One caution there. Licenses and training usually arrive within weeks of each other, so a gain measured afterward has more than one plausible parent. That is a design problem, not a reporting problem, and the attribution question deserves its own treatment before you put a dollar figure in a board pack.

The layer no dashboard can supply

Three of the four layers come from telemetry. The fourth does not, and no product will ever ship it, because capability is not an event a client can log.

Whether someone can tell a good model output from a plausible wrong one does not appear in a usage report. Neither does whether they know which work to route to a model and which to keep. Those judgments are the whole difference between a team that is faster and one that is confidently wrong faster, and the only way to read them is to look at the work.

The mechanics are not exotic. Before a session, take a small fixed sample of real team output and score it against a short rubric: was the claim checked, was the source named, would you send this to a client. Keep the sample, the rubric, and the scorer.

Re-score a fresh sample from the same workflow six to eight weeks later. Thirty items is enough to see a shift, and the pair is worth more in a finance conversation than any percentage a dashboard can generate. The method is set out in full in our measurement guide for proving capability moved.

This gets skipped because it needs someone to read work and form a judgment, which a dashboard refresh does not. That cost is the moat. Teams that keep a baseline sample can still defend the spend a year later. Teams that took none are left arguing from a reach number against a CFO who has stopped believing it.

What to put in front of your CFO

A one-page report that holds up looks like this:

  1. The population you meant to change, as a count, with the share holding a license. One line, and a denominator.
  2. A depth distribution, not an average: how many people used the tool on 10 or more days last month, on 1 to 9 days, and on none. Three numbers against the prior month.
  3. A retention curve: the cohort’s median active days at 30, 60, and 90 days after their session.
  4. One outcome, on one named task, against a baseline you took first, method in a sentence.
  5. A capability read: before-and-after rubric scores on a sample of real work, with the sample size.
  6. The decision you are asking for: extend, hold, redesign, or reallocate licenses.

That page can fail, which is why a finance team will treat it as evidence. The broader case for buying capability rather than access is in our argument that buying licenses is not the same as building capability.

Two numbers to leave out. Skip the much-quoted claim that 95% of generative-AI pilots deliver no measurable impact; it is directional at best. And skip any vendor’s benchmark for a “normal” seat-to-active gap, since those carry no disclosed sample and exist to size a services proposal.

If your reporting is a single adoption percentage and you want a second opinion on what it hides, a free AI-readiness call is the fastest way to find out. We will look at the fields you already have and name the layers you can build this quarter.

Frequently asked questions

What are AI training vanity metrics?

Numbers that look like progress but cannot inform a decision: seats assigned, people who have tried the tool, prompts sent, workshop attendance, post-session satisfaction scores. Each rises monotonically or measures a feeling rather than a behavior, so none can carry bad news.

Is the active users rate in my admin console a vanity metric?

As a headline, yes. Microsoft defines an active user as an enabled user who tried a user-initiated feature in the chosen window, and that window runs out to 180 days. It is a sound denominator and a poor result.

What should we measure instead of seat utilization?

Four layers: reach as a denominator, depth as a distribution of active days and prompts per user, retention as a decay curve at 30, 60, and 90 days, and outcome on one named task against a pre-training baseline. Add a capability read on a sample of real work, which no dashboard produces.

How do we measure AI adoption without buying an analytics tool?

Export the user-level table from the usage report you already have. It carries active days, prompts submitted, and a last activity date per person, which is enough for a depth distribution and a retention curve in a spreadsheet.

How many people should be using AI for the program to be working?

Compare against external frequency rather than against 100%. Gallup’s May 2026 wave put frequent use, a few times a week or more, at 30% of US employees and daily use at 15%. A reach figure far above that band is measuring trial, not habit.

Why does depth of use matter more than how many people tried it?

The return scales with breadth of application. In the same Gallup wave, 45% of people using AI for one or two purposes reported a productivity gain, against 90% of those using it for seven or more. Reach reporting cannot tell the two groups apart.

Can we use audit logs to build better usage metrics?

Microsoft advises against it, stating that audit-log data is built for security and compliance, is not intended as the basis for usage reporting, and that metrics derived from it may not match the official figures. Use it for governance questions, not adoption ones.

How do we know whether the training worked or the tool did?

You design for the question before you spend. The cheapest approach is staggered cohorts: train one team several weeks ahead of a comparable team while both already hold licenses, then compare the same deliverable on the same dates. Without that structure, any split you report afterward is an estimate.

Should we reclaim licenses from inactive users?

Not before you have run enablement, and never on a null field. GitHub sets a last-activity value to null once it is older than 90 days, so a blank cannot separate a dormant user from one who never started. Check the field’s definition, then fix the capability problem before cutting the seat.

Key takeaways

  • Seat and active-user reporting measures reach. The threshold for “active” is one intentional action inside a window that runs out to 180 days.
  • The damage happens in the slide that turns a reach counter into a capability claim, not in the dashboard.
  • Two screens catch most AI training vanity metrics: a number that can only rise cannot carry bad news, and a number tied to no decision is decoration. The company that defines “active” is also the company selling the seats.
  • Return scales with depth, not breadth of trial: 45% of Gallup’s one-or-two-purpose users reported a productivity gain, against 90% at seven or more.
  • Report four layers: reach as a denominator, depth as a distribution, retention as a decay curve, outcome against a baseline you took first.
  • The capability layer is in no dashboard. A scored sample of real work before and after is the only way to read it, and the only number that survives a finance review.

Last Updated: Nov 19, 2026

AW

Arthur Wandzel

SFAI Labs helps companies build AI-powered products that work. We focus on practical solutions, not hype.

Make your team fluent in AI — then automate what proves out

  • Hands-on training applied to your own documents and workflows
  • Often reimbursable through state small-business training programs
  • Built for 5–50 person firms with no IT department

Related articles