Every document AI vendor selling into commercial real estate leads with the same number: 95% accuracy, give or take a few points. It sounds like a specification. It is a marketing figure, and on its own it tells you almost nothing about whether the tool will read your lease stack correctly. A 95% accuracy claim omits four things a buyer needs to know: 95% of which fields, measured how, on what documents, and counted which way. Decode those four, and the same number can mean “reliable enough to trust with a light check” or “wrong on one in five documents.” This piece shows a small firm with no data scientist how to interrogate an accuracy claim and turn it into a verification plan you can actually run.
Why a single accuracy number is not a spec
Accuracy, in document AI, is not one measurement. It is a family of measurements that behave very differently, and a vendor is free to quote whichever one flatters the product. “95%” with no qualifier is the extraction equivalent of a used-car ad that says “runs great”: probably true in some sense, silent on the sense that matters to you.
The reason this is not pedantry is that CRE documents punish the ambiguity. A lease is not a uniform block of text where every value is equally hard to read. The tenant name is trivial; the CAM gross-up mechanic buried in a 1998 amendment is not. An accuracy figure that averages across those extremes, on a clean test set the vendor chose, can look excellent while being wrong on exactly the clauses that cost money. Before you weigh one tool against another, you have to force the number to say what it is measuring. The four sections that follow are the four things it usually hides.
95% of which fields?
The first question a document AI accuracy claim dodges is which fields the number covers. Almost every vendor benchmark is field-level: it scores each extracted field against a known-correct value and reports the average. That average is dominated by the easy fields.
A lease abstract mixes fields of wildly different difficulty:
- Header fields: tenant, landlord, premises, execution date. Printed cleanly on page one, extracted correctly almost every time. These drag the average up.
- Simple economics: base rent, square footage, commencement and expiration dates. Usually clean, occasionally buried in an amendment.
- Structured clauses: escalations, renewal options, notice windows. Harder, because the value depends on reading a mechanic, not copying a number.
- Bespoke clauses: CAM caps and gross-ups, co-tenancy, exclusives, recapture rights. Written differently in every lease, and where extraction genuinely struggles.
If a tool is 99% accurate on the first group and 80% on the last, and your abstract is mostly header and economic fields, the blended figure lands in the mid-90s, and it hides the fact that the tool is unreliable on precisely the clauses that trigger disputes and six-figure decisions. The accuracy you care about is not the average. It is the accuracy on your highest-cost fields, which for most firms are the options, the critical dates, and the recovery clauses. Our guide to the fields worth extracting and why sorts every lease field by what an error actually costs; read a vendor’s accuracy claim through that lens, not as a single blended percentage.
Measured how? Read accuracy versus extraction accuracy
The second hidden variable is what “correct” means, and it splits two ways.
The first split is reading versus understanding. Optical character recognition (OCR) accuracy measures whether the system turned the pixels into the right characters. A vendor can honestly report 99% character accuracy while the extraction (did it identify that this number is the renewal notice period and not the default cure period) is far less reliable. OCR accuracy on a clean digital PDF is nearly a solved problem. Extraction accuracy, the part that decides whether your abstract is right, is the harder number, and the one more likely to get quietly swapped for the easier one in a pitch.
The second split is how a match is scored. Three methods are common, and they produce different percentages from identical output:
- Exact match: the extracted value must equal the reference string character for character. Strict, and it penalizes harmless format differences (“$4,500” versus “4500”).
- Fuzzy or normalized match: values are compared after cleaning up formatting, so “NNN” and “triple net” count as equal. More forgiving, and closer to how a human would grade it.
- Human-judged: a reviewer decides whether the extraction is usable. Most realistic, least reproducible, and the most flattering if the grader is lenient.
None of these is wrong, but they are not comparable. A tool reporting 92% on strict exact match may be more reliable than one reporting 97% on lenient human grading. When a claim does not state the scoring method, the number is not decodable, which is itself the answer to whether you should trust it.
On what documents?
The third question: what was the test set? Accuracy on clean, digitally native leases is the easy case, and it is almost always the case a benchmark uses. Your portfolio is not that.
A real small-firm document pile includes the scanned 1998 lease with six stacked amendments, the fax-quality estoppel, the rider with a hand-marked escalation, and the PDF assembled from photos of a stapled original. Extraction accuracy on those degrades sharply, because the model is now fighting bad OCR, missing context spread across documents, and clauses that reference other clauses. A vendor number generated on pristine test data can overstate real-world performance by ten or more points on a stack like this.
This is why the operating conditions matter as much as the model, and why so many extraction projects stall despite a strong demo, a pattern we unpack in why most lease abstraction projects fail. The honest test is not the vendor’s benchmark. It is a pilot on your documents, including the ugly ones, before you commit. Any serious tool will let you run a sample; a reluctance to be measured on your stack is a data point about the number.
Precision, recall, and the dropped field you never see
A single accuracy figure also collapses two failure modes that a buyer should keep separate: getting a field wrong, and missing a field entirely. The metrics that keep them apart are precision and recall.
- Precision answers: of the values the tool extracted, how many were correct? Low precision means visible errors: a wrong number sitting in a cell where you can see and question it.
- Recall answers: of the values that were in the document, how many did the tool find? Low recall means silent omissions: the option window that was never extracted, so it never made it to your calendar.
The dangerous failure for a CRE firm is low recall, because a missing field is invisible. A wrong rent figure looks wrong and gets caught in review. A renewal option that the tool never surfaced looks like a lease that simply has no renewal option, and nobody checks a field that is not there. The F1 score, the harmonic mean of precision and recall, is the metric that refuses to let a vendor hide a recall problem behind high precision, which is exactly why serious document AI evaluations report F1 rather than a bare accuracy number. If a claim quotes “accuracy” and cannot tell you precision and recall separately, assume the number is not measuring the omissions that hurt most.
The compounding problem: per-field versus per-document
Here is the arithmetic that reframes every accuracy claim, and that no vendor page shows you. Field-level accuracy and document-level accuracy are not the same number, and the gap between them is large.
Say a tool is genuinely 95% accurate per field, and honestly so. Your abstract has 30 fields. The chance that a single document comes out with every field correct is roughly 0.95 raised to the 30th power, about 21%. A per-field number that sounds excellent means that, left unchecked, close to four out of five documents contain at least one error somewhere.
The sensitivity to that per-field figure is the part worth internalizing:
| Per-field accuracy | 30-field document fully correct |
|---|---|
| 95% | ~21% |
| 97% | ~40% |
| 98% | ~55% |
| 99% | ~74% |
The jump from 95% to 99% per field looks like four percentage points. At the document level it is the difference between a fifth of your abstracts being clean and three-quarters of them being clean. This is not an argument against document AI. Extraction at these rates is still far faster and often more consistent than manual keying. It is an argument against reading a per-field percentage as a per-document promise, and for building review into the workflow rather than assuming the tool removes the need for it.
Four questions that turn a marketing number into a real one
You do not need a data scientist to decode an accuracy claim. You need four questions, asked of any vendor, in a demo or an email:
- 95% of which fields? Ask for accuracy broken out by field, or at least for the hard clauses (options, notice windows, CAM recoveries) reported separately from the header fields. A vendor who can only give a blended number is telling you they do not measure the fields you care about.
- Measured how? Exact match, normalized match, or human-judged, and read accuracy or extraction accuracy? Pin down which. Insist the number describes extraction, not OCR.
- On what documents? Clean digital leases or a realistic mix including scanned amendments and riders? Then ask to run the tool on a sample of your worst documents.
- Precision and recall, separately? A single accuracy figure hides omissions. Ask specifically how often the tool misses a field that was present, because that is the error you will never catch by eye.
The answers matter less than the willingness to give them. A vendor who quantifies their own limits is more trustworthy than one who repeats “95%” and changes the subject.
Turning the honest number into a verification budget
Once you have decoded the claim, the point is not to demand perfection, because no tool clears the bar, and manual abstraction does not either. The point is to spend your limited review attention where the accuracy claim tells you the risk lives.
The workflow follows directly from the four variables. Let the model extract everything. Then have a person verify only the fields where accuracy is lowest and error cost is highest — for most firms, the option windows, notice deadlines, and recovery mechanics. The header and economic fields, where extraction is near-perfect, get a spot check. This is how a firm with no abstraction department gets institutional-grade output: the machine reads at scale, and a person spends fifteen minutes on the five fields that can cost real money. The compounding math is what makes this non-negotiable: with a fifth of documents carrying an error somewhere, a targeted human pass is the mechanism that keeps the abstract trustworthy without re-reading every lease.
That verification discipline is the connective tissue of every reliable extraction workflow, and it sits at the center of our broader playbook for turning a lease stack into structured data. It is also the capability that lets a lean team match a much larger one: the ability to run a current, trustworthy portfolio view without a back office, which is the argument at the heart of our manifesto on how small CRE firms out-operate the giants. A decoded accuracy number is not a reason to distrust the tool. It is the input that tells you exactly where to point the one scarce resource you have: human attention.
FAQ
What does “95% accuracy” mean in document AI?
By itself, very little. It is almost always a field-level average: each extracted field is scored against a known value, and the results are averaged. That average is dominated by easy fields like names and dates, so it can look high while the tool struggles on hard clauses. To make the number meaningful, you need four qualifiers: which fields it covers, how a match was scored (exact, fuzzy, or human-judged), what documents it was tested on, and whether it separates precision from recall. Without those, 95% is a marketing figure, not a specification.
Is field-level accuracy the same as document-level accuracy?
No, and the gap is large. Field-level accuracy is the per-field average; document-level accuracy is the chance an entire document comes out with every field correct. If a tool is 95% accurate per field and your abstract has 30 fields, the odds of a fully correct document are roughly 0.95 to the 30th power, about 21%. A strong per-field number can still mean most documents contain at least one error, which is why review stays part of the workflow even with an accurate tool.
What is the difference between precision and recall in extraction?
Precision is how many of the tool’s extracted values were correct; recall is how many of the values actually present in the document the tool managed to find. Low precision produces visible errors you can catch in review. Low recall produces silent omissions: a field that was never extracted, so nobody checks it. For CRE, missing an option window or notice deadline (a recall failure) is more dangerous than a visibly wrong number, because you cannot review a field that is not there.
Why do vendors report F1 score instead of accuracy?
Because F1 is harder to game. It is the harmonic mean of precision and recall, so a tool cannot post a high F1 by being cautious and accurate on the few fields it does extract while quietly missing many others. Plain “accuracy” can hide a recall problem; F1 cannot. When a vendor reports F1 broken out by field, it usually signals they measure their system honestly. A bare accuracy percentage with no precision or recall behind it should be treated as incomplete.
How accurate is AI at reading commercial leases in practice?
On clean, digitally native leases, current business-tier assistants and purpose-built platforms extract standard fields at accuracy in the mid-to-high 90s. On a realistic stack (scanned older leases, stacked amendments, riders, poor-quality faxes), accuracy on the hard clauses drops meaningfully. The reliable read is that extraction is strong on standard fields and standard documents, weaker on bespoke clauses and degraded scans, and always in need of a targeted human check on the highest-cost fields. Test any claim on your own worst documents before trusting it.
What questions should I ask a document AI vendor about accuracy?
Four. First, 95% of which fields: ask for the hard clauses (options, notice windows, CAM recoveries) reported separately from header fields. Second, measured how: exact match, normalized match, or human-judged, and extraction accuracy rather than OCR read accuracy. Third, on what documents, and can you run a sample of your own scanned amendments. Fourth, precision and recall separately, so you know how often the tool silently drops a field. A vendor’s willingness to answer is itself the most useful signal.
Does higher accuracy mean I can skip human review?
No. Even at 99% per field (better than most tools deliver), a 30-field abstract is fully correct only about three-quarters of the time, so roughly one in four documents still carries an error. The accuracy number tells you where errors are least likely, not that there are none. The workflow that works is to verify by error cost: a person checks the highest-cost, lowest-accuracy fields (option windows, recoveries), while near-perfect header fields get a spot check. That targets scarce review attention instead of removing the need for it.
Is OCR accuracy the same as extraction accuracy?
No, and conflating them is a common way accuracy claims mislead. OCR accuracy measures whether the system converted pixels to the correct characters, a near-solved problem on clean digital documents. Extraction accuracy measures whether the system correctly identified what each value is: that a given date is the renewal notice deadline and not the default cure period. Extraction is the harder task and the one that determines whether your abstract is right. If a vendor quotes a very high number, confirm it describes extraction, not OCR read accuracy.
Key takeaways
- A bare “95% accuracy” claim is a marketing figure. Force it to answer four questions: 95% of which fields, measured how, on what documents, and precision versus recall.
- Field-level accuracy is not document-level accuracy. At 95% per field across 30 fields, only about 21% of documents come out fully correct, so plan for review accordingly.
- The blended average is dominated by easy header fields. The accuracy that matters is on your highest-cost clauses: options, notice windows, and CAM recoveries.
- Low recall is the silent danger. A dropped field is invisible in a way a wrong value is not, which is why F1 and per-field breakdowns beat a single percentage.
- Decode the number, then verify by error cost: let the model extract everything, and spend human attention only on the fields that are both hardest and most expensive to get wrong.
Trying to judge one vendor’s accuracy claim against another, or figure out where your firm’s review time should actually go? That is exactly the kind of question a short working session sorts out against your real documents and workflow. Book your free AI-readiness assessment →
Dirk Jan van Veen, PhD
