Back to Guides
Enterprise Software 15 min read

The AI Fluency Assessment Framework: scoring a team before and after

The AI Fluency Assessment Framework: scoring a team before and after

An AI fluency assessment that can be scored before and after an AI enablement program needs four things: a small set of competencies, a level scale with observable evidence at each step, one piece of the person’s real work to score against, and a rater calibrated before they score anything. Drop any one and you get a number that moved, with no way to say why.

Search the term and you find something else. The top-ranking artifact is a self-rating ladder from a workflow-automation company, which says plainly that it is not a scorecard and not a pass or fail test. One learning platform ships its rubric as a flat image. An assessment vendor argues correctly that you should score demonstrated work rather than self-ratings, then publishes no rubric at all.

So here is the table, the scoring procedure, and the part every vendor leaves out: what the delta between your two scores is and is not evidence of.

What fluency has to mean before you can score it

Three definitions circulate, and only one is measurable.

Usage. How often somebody opens an assistant. Admin consoles produce it for free, which is why it gets used. A person can open ChatGPT daily and paste the same unchanged request for a year.

Self-reported confidence. A five-point survey question. Cheap and weakly related to performance: Zell and Krizan pooled 22 meta-analyses in Perspectives on Psychological Science and put the mean correlation between ability self-evaluations and actual performance at about .29, with individual effects from .09 to .63. Moderate, not worthless, far too loose to carry a budget decision — and the measure most likely to move for reasons unrelated to capability, because somebody leaving a workshop feels more capable on the way out.

Demonstrated judgment on real work. Whether this person, on a task they own, decided correctly when to involve an assistant, described it well enough to get something usable, caught what came back wrong, and handled the sensitive parts properly. Scoreable, and the only one a console cannot produce. It sits at level three of the Kirkpatrick model — behavior, not reaction or knowledge — which is why it costs something to measure and why most programs skip it.

Structure is what buys the validity. Sackett, Zhang, Berry and Lievens published revised estimates in the Journal of Applied Psychology in 2022 putting structured interviews at .42 against .19 for unstructured ones, with work samples at .33. Same raters, same people, roughly double the predictive value from structure alone. A rubric is not bureaucracy. It is what makes a judgment mean anything.

The four competencies worth scoring

Anthropic publishes an AI fluency framework built with Joseph Feller at University College Cork and Rick Dakan at Ringling College, organized around four competencies: delegation, description, discernment, and diligence. It is a free course about collaborating with AI rather than a measurement instrument. The decomposition is still the best available, and worth scoring against because the four fail independently. Strong describers delegate badly. Fast, accurate and careless is a common profile.

Two structural details settle arguments most buyers are still having. Prompting sits inside description — one component of one of four competencies — so a prompt-writing test measures a small fraction of the thing. And of the 24 behaviors the framework names, only 11 happen inside the chat window; the other 13 happen around it. That is why no admin console will produce a fluency score, however good the dashboard gets.

The four also do not move together, and the gap has been measured. Anthropic’s AI Fluency Index classified those 11 observable behaviors across 9,830 multi-turn conversations in a seven-day window in January 2026. Iterating and refining appeared in 85.7%. Questioning the model’s reasoning appeared in 15.8%. Checking facts and claims that mattered appeared in 8.7%.

That is one population and one product, so read it as a shape rather than your team’s number. Description behaviors are common, discernment behaviors are rare, and the ratio gets worse when the work looks finished: conversations producing a polished artifact iterated more (94.1% against 84.5%) and fact-checked less (5.5% against 9.2%). A team scoring well on description and badly on discernment is producing more and checking less. One blended number hides exactly that.

The rubric

Four levels per competency. Each cell is evidence a rater can look for in a work sample, not a disposition.

1 — Absent 2 — Developing 3 — Working 4 — Fluent
Delegation No assistant, or used on a task it cannot do One familiar task type, regardless of fit Chooses per task; can say why this one and not that one Splits a task, keeping the parts that need a person
Description One-line request, no context or constraint Longer requests, inconsistent context, no success criteria Gives goal, context, format, constraints; iterates deliberately Gives examples and anti-examples; builds a reusable pattern
Discernment Output used as returned Spot-checks surface errors: tone, format, length Checks claims, numbers and sources against something external Knows where this task type fails and checks there first
Diligence Sensitive material in an unapproved tool; no attribution Knows the rule, applies it inconsistently Approved tools for sensitive work; involvement disclosed Flags edge cases upward and improves the team’s rule

Score four numbers per person against one work sample. Do not average or weight them. A person is a 3 on description and a 2 on discernment, and that sentence is the finding.

Two deliberate choices: there is no zero, because “has not tried” is level one and belongs on the same scale; and level four is reachable, because a scale whose top means mastery nobody attains is a two-point scale wearing four labels.

The baseline you cannot reconstruct

Most of this instrument can be rebuilt late. The before-score cannot.

Score the week before the first session. Afterwards the sample is contaminated two ways. People have seen the rubric, so they hand you the work they think it wants. And the rater has watched the training, so a sample that read as a 2 in January reads as a 1 in March against what they now know is possible.

The mechanics are small. Pick one recurring artifact each person genuinely produces — a client update, a case summary, a weekly analysis. Collect the most recent one, dated, before anybody announces a program. Score it and store the numbers with the sample. Score the same artifact type again at least four weeks after the session, so you are reading a habit rather than a performance. Four weeks is the shortest window we would defend; the behavior-change timeline runs in months, which is why the broader method sits in our guide to proving capability moved.

Already trained people and have no baseline? Do not reconstruct one from memory. Score an untrained part of the organization now and train that group next.

Who scores, and how they get calibrated

The functional manager scores, because only they can tell whether a work sample is good for that job. An outside reviewer cannot judge whether a case summary served the case. The trade-offs are worked through in our comparison of self-assessment against an external capability audit.

Calibration is the step every published rubric omits, and it is about three meetings’ work.

Anchor together before anyone scores alone. Three real samples, agreed in advance as weak, middling and strong. Every rater scores all three in the same room and argues until the numbers agree. Keep them and re-read them before each round. Without anchors, two managers’ 3s are different scales.

Double-score a fifth of the people. Two raters, independently, on 20%. Disagreement of more than one level on any competency means the rubric wording needs fixing before you score the rest. It is the only check on whether the instrument works.

Keep the post-scorer blind to the pre-score. Not blind to whether training happened, which is impossible, but the earlier numbers stay out of sight until both rounds are done. Expectancy runs in the direction you are hoping for, and hiding one column is free.

Managers also score their own teams generously, which is fine for a delta and not for comparing departments. Report movement within a team, not a league table.

What the delta is not evidence of

This is the section no vendor publishes, because every enablement vendor’s interest runs toward a large delta. A pre and post comparison with no control group, raters who know which round is which, and a cohort that volunteered carries five confounds. Name them before a CFO does.

Confound Effect on your number Cheap partial fix
No control group Any company-wide change gets credited to the training Score one untrained team on the same schedule
Rater expectancy Raters who expect improvement find it Blind the post-scorer; re-read the anchors
Tool changes mid-window Gains from a product update, not the people Record the tools in use at each round
Self-selection The pilot volunteer was already moving Score a whole team, or state the selection
Regression to the mean Lowest scorers drift up on re-measurement anyway Report the distribution, not just the mean

With those fixes in place, the sentence you have earned is specific and small: forty-one people were scored on their own work in January and again in April; median discernment moved from 2 to 3, median description from 2 to 3, delegation and diligence did not move, and an untrained comparison team did not move at all.

That survives a skeptical finance reading. “AI fluency improved 47%” does not, and it is what most vendors hand over. The rest of the questions finance will ask are collected in our note on what a CFO should ask first.

What the instrument costs in hours

Published rubrics never state this, which is the main reason assessments get designed and never run twice. Per 50 people, per round: two hours collecting and dating samples, 8.5 hours scoring at ten minutes a person, 1.7 hours double-scoring a fifth of them, two hours writing it up. Around 14 hours, plus a one-time 1.5-hour anchor meeting per rater. Two rounds for a 50-person department is roughly 30 hours of manager time across a quarter — a defensible cost, and the reason we would not run this on 400 people at once. Score the function you intend to change.

Protect the ten minutes. Raters trying to be thorough drift to 30, the round takes three times as long, the second round never happens, and the before-score becomes a file nobody opens.

Reporting it without inventing a composite

The temptation is a single index. Resist it. A composite is how a team that got faster without getting more careful reports as improved.

Report four medians and four distributions, per competency, per round, plus the count of people who moved up at least one level on each. Eight numbers and a sentence, on the same page as the usage and output rows in our breakdown of an enablement scorecard.

Two rules worth holding. Report at team level, because one person’s four-point score is too coarse to be an individual verdict and will be read as a performance rating if you let it. And keep the instrument separate from the curriculum: if one vendor writes the rubric, delivers the training and scores the result, the result is a sales document. Score it yourself, or have somebody with no stake in the delta do it. Tooling that carries parts of the job is surveyed in our look at the options for assessing fluency across a workforce.

A worked before-and-after

A 180-person company trains one 40-person operations group. Figures are illustrative; the shape is the point.

Competency Median, January Median, April Moved up ≥1 level
Delegation 2 2 9 of 40
Description 2 3 26 of 40
Discernment 1 2 22 of 40
Diligence 2 2 7 of 40

Untrained comparison team of 18: no median movement on any competency.

Read it honestly. Description moved most, which is the skill the session drilled. Discernment moved from the worst start and is the one to reinforce. Delegation and diligence did not move, which says the session taught prompting rather than task selection and handling rules — a finding about the curriculum, not the people. Nobody reached level 4 in a quarter, and a program claiming otherwise is reporting enthusiasm.

No published benchmark will tell you whether your own team got better. Two columns of your own are the only evidence available, which is the argument running through why buying licenses is not the same as building capability.

Frequently asked questions

What is an AI fluency assessment?

A structured scoring of how well somebody uses an assistant on their own real work, across a small set of competencies, against a rubric with observable evidence at each level. A calibrated human rater reads a work sample; nothing is generated from usage data or a survey.

How is it different from an AI readiness assessment?

A readiness assessment looks forward and rates conditions (data, infrastructure, leadership, policy) to decide whether to invest. A fluency assessment looks backward at people and asks whether capability moved. If licenses are out and sessions are run, the forward-looking version answers a question nobody is still asking.

Can we use a self-assessment survey instead?

Only as a secondary measure. Pooled meta-analytic evidence puts the self-evaluation to performance correlation around .29, and self-ratings rise right after training for reasons unrelated to capability. Fine as a morale signal, not as evidence for finance.

How many competencies should we score?

Four, scored separately. Published frameworks run from three to eight dimensions with no canonical set, so pick ones that fail independently and that a manager can see in a work sample. Past about five, scoring time rises and rater agreement falls.

Who should do the scoring?

The person’s functional manager, because judging whether a work sample is good for that job requires knowing the job. Calibrate them on three shared anchor samples first, double-score a fifth of the people, and keep the post-training scorer blind to the earlier numbers.

How long after training should we re-score?

At least four weeks, ideally eight to twelve. Scoring in the week after a session measures the session, not the habit. Past a quarter you start picking up skill decay — useful to know, and a different question.

What does a four-point scale miss?

Movement inside a level. Somebody who got meaningfully better at describing tasks but stayed a 3 reads as flat. That is the price of a scale raters can apply consistently in ten minutes. Pair the scores with a count of people who moved rather than adding levels nobody can tell apart.

What should we do if the scores barely move?

Read which competency did not move, because that names the problem. Flat description after a prompting-heavy session usually means too little practice on the team’s own documents. Flat discernment means verification was demonstrated rather than practiced. Flat diligence is usually a policy problem. That diagnosis is the point of scoring separately.

Key takeaways

  • Four competencies, a level scale with observable evidence, one piece of real work, and a calibrated rater. Three out of four produces a number nobody can defend.
  • Usage counts and self-ratings are not fluency. Self-evaluations correlate with performance around .29 and rise on their own right after a workshop.
  • Structure buys validity: revised estimates put structured assessment at roughly double the predictive value of unstructured judgment.
  • Score the week before the first session. The before-score is the only part you cannot reconstruct later.
  • Calibrate on three shared anchors, double-score 20%, keep the post-scorer blind.
  • Budget about 14 hours of manager time per 50 people per round, and protect the ten-minutes-per-person rule.
  • Report four medians and four distributions, never a composite index, and never let whoever delivers the training own the rubric and the scoring.

If you want this built against your own work samples and the function you need to move, book a free AI readiness call. We will tell you which competency to baseline this week and which one your next session should be built around.

Last Updated: Nov 13, 2026

DJ

Dirk Jan van Veen, PhD

SFAI Labs helps companies build AI-powered products that work. We focus on practical solutions, not hype.

Make your team fluent in AI — then automate what proves out

  • Hands-on training applied to your own documents and workflows
  • Often reimbursable through state small-business training programs
  • Built for 5–50 person firms with no IT department

Related articles