Back to Guides
Enterprise Software 15 min read

Measuring output quality, not just output speed

Measuring output quality, not just output speed

Every AI enablement program reports time saved, because time saved is the half of the claim that is free to collect. A timestamp costs nothing; a quality judgment costs a manager’s afternoon. So the report says the team drafts proposals 40% faster and says nothing about whether the proposals got better, got worse, or all started to sound the same — which is the half a CFO was buying.

That asymmetry is not an oversight. It is what happens when a program measures whatever the admin console hands over free, and it is where a training budget’s return quietly leaks out.

The phrase has been claimed by the wrong discipline

Search “AI quality measurement” and you land in machine-learning engineering: reference-based and reference-free metrics, retrieval groundedness, model-as-judge scoring, drift monitoring, token cost per call. The NIST AI Risk Management Framework organizes the lifecycle into Govern, Map, Measure and Manage; ISO/IEC 25059 extends the standard software quality model to AI systems.

All real work, and almost none of it is your work: those instruments score a system’s outputs. After a training program you need to score a person’s deliverable — the proposal, the case summary, the board memo — produced with assistance, against what that person produced before. There is no benchmark for that.

Three ideas do import cleanly from that side, and they are the only three worth carrying:

  • Goodhart’s law. When a measure becomes a target it stops being a good measure. Attach a bonus to a quality score and you get the score, not the quality.
  • Criteria are not metrics. “Usable without rework” is a criterion; “revision rounds before approval” is one metric that partly operationalizes it. Collapse them and you can no longer ask whether the metric was right.
  • Blinding. A grader who knows which draft used assistance is grading the hypothesis, not the document.

What happens to quality when speed goes up

The best controlled evidence says quality and speed move together on some tasks and in opposite directions on others, and that the people doing the work cannot reliably tell which case they are in.

The largest trial of this kind remains BCG’s, run with academic collaborators on 758 of its own junior consultants. Participants completed a baseline task unassisted, then were randomized into three arms: assistance plus about thirty minutes of training, assistance with none, and a control group. Human graders scored every output blind to which arm produced it — the detail that makes the result quotable.

On a creative product-innovation task, the assisted group converged on performance about 40% higher than the unassisted group. On a business problem-solving task sitting outside the tool’s competence, assisted participants performed 23% worse than people who used nothing at all.

Same tool, same firm, same week, opposite sign. The authors call that boundary a jagged frontier, and the consequence is that one quality number averaged across a department tells you nothing: it averages the 40% and the 23% into a shrug.

The same trial found something subtler that no vendor page on this subject mentions. Idea diversity fell 41% among assisted participants. Every answer got better and every answer got closer to every other answer.

A reviewer reading twenty submissions in a row sees that immediately and describes it as “these all read the same” — not a phrase that survives into a quarterly report. No speed metric and no satisfaction survey will catch it.

Then there is the reporting problem. In a randomized trial of 16 experienced open-source developers across 246 real issues in repositories they maintained, METR measured work taking 19% longer with assistance while those developers had forecast a 24% speedup and still believed afterward they had gained about 20%. METR marks that figure as superseded by later data on newer tools, and the durable finding is not that assistance slows people down — it is that self-reported and measured speed came apart by roughly 40 points in the same subjects. Any program whose evidence is a post-session survey stands on that gap.

Quality can also change shape rather than level. In a pre-registered field experiment with 776 professionals at Procter & Gamble, unassisted R&D people proposed technical solutions and commercial people proposed commercial ones; assisted participants produced balanced solutions regardless of background. Google Cloud’s DORA research puts the general case plainly: AI’s primary role is an amplifier that magnifies an organization’s existing strengths and weaknesses.

Four things a speed metric cannot see

Quality is not one dimension. Four matter after an enablement program, and each fails differently.

Dimension The question What failure looks like
Correctness Does every claim trace to something checkable? A confident number with no source; a cited case that does not exist
Fitness Is it right for this reader and this decision? Accurate, generically addressed, needs rewriting to send
Revision burden How much work does the next person do? Arrives early, gets approved late
Variance Does the team still produce distinguishable work? Twenty documents in one voice; the 41% diversity collapse

Correctness is easiest to instrument, because someone other than the author can verify it. Fitness costs the most to fix and never reaches a dashboard. Revision burden silently cancels the speed gain, and variance takes a quarter to notice and a year to undo.

Pick the dimensions before training starts: the before values cannot be reconstructed later. What to capture in that week is in our note on building the baseline.

The instrument: blind paired review on a sample

The instrument is unglamorous and it works: ten artifacts per role, five criteria, scored 1–5 by the person who already approves that artifact, blind to condition. The specifics are the method.

  • One artifact type per role. Not “written work.” The weekly client update, the first-pass case summary, the renewal proposal — one named deliverable the role produces often enough to sample.
  • Ten before, ten after. Ten is not statistical power and should never be reported as if it were. It shows a direction, and it is few enough that a manager will do it twice. Pull the before set from work already filed the week prior to training; take the after set at the same point in a comparable cycle, not the week after the session when everyone is performing.
  • Strip the attribution. Remove names, dates and any tell about which set a document came from, shuffle all twenty, score in one sitting. Teams skip this, and skipping it makes the result unusable. BCG blinded its graders; a program that will not blind one manager is confirming, not measuring.
  • The existing approver scores. Not the training provider, not a committee, not an external evaluator reading work they hold no standard for.
  • Two scorers on at least one role. A second reviewer scores the same twenty independently. Wild disagreement means the rubric is ambiguous and both sets of numbers are noise.

This measures work products, not what a person can do. That is a separate instrument on a separate clock, covered in our framework for scoring a team before and after.

A rubric a manager will actually fill in

Five criteria, 1–5, no weights. Weights invite argument about the weights.

Criterion 1 3 5
Accuracy Contains a claim that is wrong Right, but two claims unsourced Every claim traces to a source
Fitness for reader Would embarrass us as sent Needs the opening and ask rewritten Could go out unchanged
Completeness Missing what the reader will ask for Covers the obvious, misses one objection Anticipates the questions back
Judgment No recommendation, or a wrong one A recommendation, thin reasoning A defensible call, reasoning visible
Distinctiveness Could be about any client Specific in places, generic in framing Unmistakably about this situation

Defend the fifth row when somebody tries to cut the rubric to four: distinctiveness is the only criterion here that a faster, more confident draft can make systematically worse. Score the mixed set in one sitting, then unblind and compare medians — with ten documents, one disaster drags a mean somewhere misleading.

Review burden is the second instrument

The second instrument is nearly free, because the data already sits in a calendar: how long the approver spent, and how many rounds it took.

Recovered time does not vanish, it moves. In a survey published by the Upwork Research Institute on July 23, 2024, 77% of employees using AI said the tools had added to their workload, against 96% of C-suite leaders who expected AI to raise productivity. Separately, 47% of those AI users said they did not know how to achieve the gains their employer expected.

The survey is two years old and self-reported — the evidence class the METR result warns about. Treat it as a hypothesis to test on your own team, not a finding to quote at a board.

Testing it takes two numbers per artifact: revision rounds to approval, counted before and after, and approver minutes, noted on ten documents for four weeks rather than on everything forever. A draft that arrives two days early and takes three rounds instead of one has saved nobody anything.

If drafting time falls 30% while approver minutes rise 50%, that is not a productivity gain. It is a cost transfer to a more expensive calendar — and it is invisible to every metric an admin console produces.

What it costs in hours

Pricing the instrument honestly is what gets it approved.

Activity Time
Define artifact, rubric and dimensions 2 hours, once
Collect and strip the before set 1 hour per role
Score 20 documents, one role 2–3 hours
Second scorer on one role 2–3 hours, once per program
Approver time log, 4 weeks 10 minutes a week
Write up one role 1 hour

Call it six to eight hours per role plus a one-time overhead of about five. Three roles is roughly twenty-five hours of senior time — against a workshop engagement in the $2,000–15,000 market range, a sane fraction of the spend.

Resist any platform promising to automate this. Quality scoring by the approver is the measurement; a tool that never has to sign the document returns a score with nothing behind it.

What to do when the quality number drops

A drop is a diagnosis, not a verdict, and there are three useful responses.

If correctness fell, the gap is verification rather than capability: people are shipping drafts without checking the checkable parts. That is a changed definition of done, not a second workshop. If fitness fell while accuracy held, people are accepting the first usable draft instead of directing a second one — the most common pattern and the most trainable, because the skill is specifying reader, constraint and ask up front. If distinctiveness fell, check whether the team works from one shared starting prompt; shared prompts converge output by design, so vary the input rather than the tool.

If the drop sits on one task while another improved, you have located your own jagged frontier. Narrow the program to the good side and name the rest out of scope, in writing. Which checkpoint carries that decision is in our walk-through of the 30/60/90 gates.

What these numbers cannot tell you

Every number has a claim people want to make from it and a smaller claim it supports. Printing the gap yourself is cheaper than having a CFO find it.

Measure What gets claimed What it supports Cheap correction
Median rubric score rose “Training improved our work quality” Twenty documents scored higher, one reviewer, one period Blind the scorer; report the second scorer beside the first
Revision rounds fell “We cut rework” Rounds fell on one artifact type, no control group Track one untrained team on the same dates
Both moved “The program caused this” Something moved in a window that also held tool releases Record what else changed, in the same document
Distinctiveness held “No downside” It held on the sampled artifact Name the artifact; say others were not sampled

None of this supports a causal claim, and a program that pretends otherwise gets dismantled the first time someone competent reads it. The defensible version is narrower and more useful: on this artifact, with this reviewer, in this window, the work scored higher and took fewer rounds.

Rolling those rows up into something an executive can read without being misled is the subject of our measurement guide for proving capability moved. The wider argument, that a license count is not a capability, is in the case against mistaking purchase for capability.

Frequently asked questions

What is AI quality measurement for a training program?

Scoring the work your team produces with AI assistance against what they produced before, on named criteria, by the person who already approves that work. It is distinct from AI model evaluation, which scores a system’s outputs on benchmarks and is what most results for the phrase hand you.

Why is speed the metric everyone reports?

Because it is free. Timestamps, session counts and license activity come out of an admin console with no human effort; a quality judgment needs a qualified person to read two sets of documents and score them.

How many documents do I need to sample?

Ten before and ten after, per role. Ten gives no statistical significance and should never be presented as if it did. It shows a direction, and a busy manager will finish it twice — which they will not do with fifty.

Who should score the work?

The person whose approval the deliverable already needs; they hold the standard. An external evaluator reading unfamiliar work applies a generic one, and a training provider scoring its own output has an obvious conflict.

Does the reviewer really have to be blind to which set is which?

Yes, and it is the step most often skipped. The BCG trial that produced the clearest evidence here blinded its graders. An unblinded reviewer scores the hypothesis rather than the document, in whichever direction they expect.

Can faster output be lower quality without anyone noticing?

Routinely. The clearest demonstration is the BCG result: assisted consultants scored 23% worse than unassisted ones on a task outside the tool’s competence, alongside a 41% fall in idea diversity. Both are invisible to speed and volume metrics.

What if quality scores stay flat?

Flat is a real result and often the right one. If the work holds its standard and takes measurably fewer rounds to approve, that is the return, stated correctly. Flat quality with no time saved and no drop in rework means the program did not land — worth knowing at week eight, not at renewal.

Should the quality score be tied to anyone’s bonus?

No. A quality score with money attached becomes a target and stops measuring anything, which is Goodhart’s law doing what it always does. Keep it as a program diagnostic, separate from individual performance review.

Key takeaways

  • Speed gets reported because it is free; quality goes unmeasured because it costs a qualified person’s hours. That asymmetry, not a shortage of frameworks, is why most programs cannot say whether the work got better.
  • Controlled evidence shows quality moving both ways with the same tool: BCG’s blinded trial of 758 consultants found work about 40% better on one task and 23% worse on another, with idea diversity down 41%. Self-reported speed is not evidence either — METR measured work taking 19% longer while the same developers believed they were 20% faster.
  • Measure four separable dimensions — correctness, fitness, revision burden and variance — and choose them before training starts, because the before values cannot be reconstructed later.
  • The instrument: ten artifacts before, ten after, five criteria scored 1–5 by the existing approver, blind to condition, with a second scorer on one role. Six to eight hours per role.
  • Track approver minutes and revision rounds beside drafting time. A 30% drafting gain paid for with a 50% rise in senior review time is a cost transfer, not a return.
  • Print what each number cannot support. A narrow claim survives a CFO’s reading.

If you are about to buy AI enablement for a team and want the before set collected properly, book a free AI-readiness call. We will work through which artifact to sample, which criteria your approvers will actually score, and what the result will and will not prove.

Last Updated: Nov 14, 2026

AW

Arthur Wandzel

SFAI Labs helps companies build AI-powered products that work. We focus on practical solutions, not hype.

Make your team fluent in AI — then automate what proves out

  • Hands-on training applied to your own documents and workflows
  • Often reimbursable through state small-business training programs
  • Built for 5–50 person firms with no IT department

Related articles