Back to Guides
Enterprise Software 15 min read

Why the 30% productivity number in every AI deck is unfalsifiable

Why the 30% productivity number in every AI deck is unfalsifiable

Ask the person who put 30% on the slide what result would have made them take it off. Most AI productivity claims have no answer, and a claim with no possible disconfirming result has stopped being a claim at all. It is a sentence shaped like a measurement.

That matters because you are being asked to fund AI enablement against it. The figure is probably not a lie, and it may be close to right for some task somebody somewhere timed. It is still unusable as the basis for a budget decision, for a reason that has nothing to do with whether AI works.

Where the 30% actually comes from

It does not come from one study. It is a round number sitting in the middle of a band that real experiments have produced on bounded tasks, which is what makes it so hard to argue with.

Three of those experiments are worth knowing by name. Shakked Noy and Whitney Zhang ran 453 professionals through writing tasks and found about a 40% cut in time taken and an 18% quality improvement. Fabrizio Dell’Acqua, Karim Lakhani and colleagues put 758 consultants through realistic tasks and found over 12% more tasks completed, over 25% faster. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied 5,179 customer support agents and found roughly a 15% rise in issues resolved per hour.

So 30% is in the right neighborhood for a short, well-scoped task performed by one person under observation. The deck drops every one of those conditions and keeps the number, which is how a finding reaches your desk with nothing left attached that evidence could contradict.

Measuring a workforce rather than a workbench gives a different order of magnitude. Economists at the Federal Reserve Bank of St. Louis found time savings of 5.4% of work hours among people who used the tools, falling to 1.4% once non-users are counted in.

Those figures and the 30% figure answer different questions, and only one is the question your budget turns on.

Four questions that make an AI productivity claim testable

Karl Popper’s test for a scientific statement was whether some observation could show it false. Applied to a vendor slide, it collapses into four questions you can ask inside a single meeting.

Question A usable answer sounds like A dead end sounds like
What was measured? Issues resolved per hour; minutes to first draft of a named document “Productivity”
Measured against what starting point? A timed figure for the same task, same people, before the tool “Compared to before”
Compared with whom? A group doing the same work without the tool over the same weeks “Compared to the industry”
What result would have disproved it? A named threshold the study would have reported as a null Silence

The fourth question does the work, and nobody rehearses an answer to it. A supplier who can tell you what their study would have counted as a failure has run a study. A supplier who cannot has run a testimonial.

None of this asks you to doubt that AI raises output. It asks you to separate a claim that could have come out the other way from one that could not.

The gap between what people feel and what the clock says

The strongest evidence that self-reported gains cannot be trusted comes from a group of people with every reason to report honestly.

In 2025 the research organization METR ran a randomized trial with 16 experienced open-source developers across 246 real issues from their own repositories, each issue randomly assigned to allow or forbid AI assistance. Tasks took 19% longer when AI was allowed, with a confidence interval running from 2% to 39%. The developers had forecast a 24% speedup beforehand; after finishing, having lived through the slowdown, they still estimated that AI had sped them up by 20%.

That is roughly a 40-point gap between perception and the clock, in the same people, on the same tasks. Hold it next to the method behind most figures quoted at you, which is a survey asking staff how much time AI saves them.

Two caveats belong on the result and both cut toward caution. The sample was 16 people doing expert work on code they knew well, so it is no general law, and METR says so.

METR also now treats the finding as historical. Neither caveat shrinks the perception gap.

When the comparison group stops existing

The part of this story that shows the problem is structural, rather than a matter of insufficient rigor, is what happened when METR tried again with more resources.

The second study covered 57 developers, 143 repositories and over 800 tasks. It returned an 18% speedup for returning developers, on an interval running from a 38% speedup to a 9% slowdown, and 4% for new recruits, on an interval from a 15% speedup to a 9% slowdown. Both intervals cross zero.

More striking than the width is what METR wrote about its own data: the experiment “gives us an unreliable signal of the current productivity effect of AI tools.” Developers declined to take part at much higher rates than before. Among those who did, between 30% and 50% held back some tasks because they did not want to do that work without AI, so the study was missing the tasks with the highest expected uplift.

Read that as a finding about measurement, not about developers. Once a tool is embedded, the version of your team that works without it is no longer available to compare against, and the honest consequence is wider error bars rather than a cleaner number. The mechanics of that problem are the subject of our guide to separating training effects from tool effects.

Reported gains and implied gains are different numbers

The cleanest demonstration that the error runs in one direction comes from a Federal Reserve survey that asked the same firms the same thing two ways.

The CFO Survey, run jointly by the Richmond and Atlanta Feds with Duke University, collected about 748 responses from senior financial decision-makers between November 2025 and January 2026. Asked directly, firms reported that AI raised output per worker by 1.8% in 2025.

The authors then derived productivity a second way, from the revenue firms attributed to AI set against the employment change they attributed to AI. The implied gains were much smaller, across every major industry, in both years.

Note the magnitudes. Finance chiefs forced to put a number on it said 1.8%, not 30%, and even that looked generous against their own revenue arithmetic. The authors offer two innocent explanations, delayed output and quality gains revenue does not capture, and both may hold.

The direction is still the finding: when the same people estimate and then show the arithmetic, the estimate is the larger number.

The average is the least useful part of any study

Even a well-run study reports a mean that may apply to nobody on your team.

The customer support trial is the clearest case. The headline was roughly 15% more issues resolved per hour, but the gain landed overwhelmingly on the least experienced agents and had minimal impact on experienced, highly skilled workers. A company staffed mostly by seniors and a company staffed mostly by new hires would both be quoted 15% and both be misled.

The consulting study supplies the other half. Inside the range of tasks the tool handled well, results improved. On tasks just outside it, consultants using AI were less likely to reach the correct answer, because plausible wrong output is persuasive.

That boundary is jagged and shifts with each product release, which is why a figure from last year’s tooling says little about this year’s.

The useful question about a 30% claim is never whether 30% is right. It is which part of which population moved, on which tasks, and whether your team resembles them. Our measurement guide sets out how to answer that for your own staff.

How the question gets framed decides the answer

A pilot study often tells you more about its own survey design than about AI.

The Ada Lovelace Institute compared two government pilots of the same assistant software, run in the same summer, and found that the one framed around benefits reported strong gains while the one framed around risk and added workload found time savings but no productivity improvement. Same tool, opposite conclusions, different question wording.

The brief names four failure modes worth checking in any pilot you are shown:

  • Recall and social-desirability bias in self-reported time
  • Samples with no benchmark for how representative they are
  • Simulated tasks without the accuracy demands of real work
  • Horizons short enough to catch only the novelty period, where one cited study saw AI use fall from twice as likely to 1.6 times as likely over two months

Savings models then assume work is spread evenly across tasks, that adoption is immediate, and that salary is the only cost. Each assumption pushes the number the same way.

What a testable claim looks like

Three versions of one claim, to be concrete about the standard.

Unfalsifiable. “Teams see a 30% productivity improvement after AI enablement.” No metric, no baseline, no comparison, no stated failure condition.

Weak but testable. “Across 40 participants at six companies, median minutes to produce a first-draft client summary fell from 52 to 38 over 30 days. We would have reported no effect below a 10% move. We ran no control group, so a product update in that window is a live alternative explanation.”

Strong. The same, with a comparable team trained two months later as the comparison, and the day-90 drop-off reported alongside the day-30 figure.

The weak version is the more useful one, and not because it is modest. You can tell what it would have taken to be wrong, and it names its own alternative explanation. A supplier who volunteers the limits of their own evidence is showing you how they will report on your program.

Ask one more thing: who paid for the study. The largest reported effects in this field come disproportionately from companies selling the tools, and the much-quoted figure that 95% of enterprise AI pilots show no return rests on about 150 survey responses and a success test that counts only custom builds reaching measurable profit-and-loss impact. Numbers travel far past their method in both directions.

Building a number you own

The practical reply to an unfalsifiable industry average is a small local measurement your own finance team can check, not a better average.

Three inputs cover a first round. Pick one deliverable your team produces, time it for a handful of people before anyone trains, and time the same deliverable 30 days afterward. Write down in advance the result you would report as no effect, then name the alternative explanation you cannot rule out, usually a product update or a seasonal workload change.

The cost is modest. A hands-on AI enablement workshop sits in a market range of roughly $2,000 to $15,000 by scope and group size, and the measurement wrapped around it is a spreadsheet, a stopwatch and about four hours a month of somebody’s attention. Turning those timings into hours that survive scrutiny is covered in our walkthrough of time-savings claims.

One window closes for good: the before figure can only be taken before, and no supplier has a commercial reason to remind you, which is the subject of our piece on what an unmeasured program costs.

A result of 6% you can show your working for will survive a finance review. A 30% figure you cannot source will not, and the gap between the two is the argument in our case for treating capability as something you build rather than something you buy.

Frequently asked questions

Where did the 30% AI productivity number come from?

Not from a single study. It sits in the middle of a band controlled experiments have produced on bounded individual tasks: about 40% less time on writing tasks among 453 professionals, over 25% faster completion among 758 consultants, roughly 15% more issues resolved per hour among 5,179 support agents. Those results are real and their conditions are specific. The deck keeps the number and drops the conditions.

Does unfalsifiable mean the claim is false?

No, and that is the point to hold onto. An unfalsifiable claim might be accurate, but you cannot find out, so it cannot carry a decision. A wrong number can be fixed with better data; a claim with no possible disconfirming result cannot be fixed at all, which leaves only one move: replace it with a measurement you took yourself.

Should we discount vendor-funded studies entirely?

Discount them, do not dismiss them. Funding predicts effect size in this field, so read a supplier’s figure as an upper bound and ask for the method. Several of the better vendor studies are genuinely randomized, and they are also run on their own best-case tasks.

Why do our employees report bigger gains than our financials show?

Because people are inaccurate at timing themselves and the error runs one direction. In the METR trial, developers estimated a 20% speedup while the clock showed a 19% slowdown, a gap of about 40 points. The CFO Survey found the same pattern among finance chiefs: 1.8% reported, less than their own revenue arithmetic implied.

What is a realistic internal number to aim for?

Anything you can source. Survey work puts time savings near 5% of work hours among people who use the tools, a reasonable prior for a whole team rather than one task. A measured 5 to 10% on one named deliverable, method written down, beats any industry average at a budget meeting.

Can we measure this without buying an analytics platform?

Yes, for a first round. One deliverable, a handful of people, a timing before and a timing 30 days after, plus a pre-written definition of no effect. Usage telemetry from licenses you already pay for shows whether the behavior stuck. Software earns its place at several hundred people across many departments.

Does a flat or negative result mean the program failed?

It means that deliverable did not move in that window, which is a different and more useful statement. Gains concentrate on the least experienced staff and on tasks the tool handles well, so a senior team on hard problems can be fluent and still show a flat first number. Report the null, then pick a better-matched task.

How do I test a productivity number in one meeting?

Four questions: what was measured, against what starting point, compared with whom, and what result would have disproved it. Each should get a specific answer in under a minute. The fourth is the one nobody prepares for, and that answer tells you most of what you need.

Is measuring AI adoption worth the effort if the research is this messy?

The mess is the argument for local measurement, not against it. Studies disagree because they cover different tasks, people and tooling, none of which are yours. A small before-and-after on work your team does sidesteps the dispute, and it is the only figure a CFO can audit.

Key takeaways

  • The 30% figure is a real task-level result with its conditions stripped off, which is why no evidence could contradict it.
  • Four questions settle whether a claim is checkable: what was measured, against what baseline, compared with whom, and what result would have disproved it.
  • People misjudge their own speed by about 40 points. Developers in a randomized trial estimated a 20% speedup while taking 19% longer.
  • Once a tool is embedded, the comparison group disappears. METR published exactly that about its own 2026 study, where 30 to 50% of participants withheld the tasks they would not do without AI.
  • A measured 6% you can source beats a quoted 30% you cannot, and the before figure can only be taken before.

If a number on a slide is driving your AI enablement budget and nobody can say what would have disproved it, that is the normal starting position rather than a sign anything went wrong. We run a free AI-readiness call that works through which deliverable to time, what your realistic before figure looks like, and what you can honestly claim at day 30. Book a time here.

Last Updated: Nov 19, 2026

AW

Arthur Wandzel

SFAI Labs helps companies build AI-powered products that work. We focus on practical solutions, not hype.

Make your team fluent in AI — then automate what proves out

  • Hands-on training applied to your own documents and workflows
  • Often reimbursable through state small-business training programs
  • Built for 5–50 person firms with no IT department

Related articles