You can evaluate any AI vendor for your firm in a single meeting with ten questions. Run them in order, get the answers in writing, and you will know before you sign whether a tool solves a real workflow, keeps your deal data safe, and costs less than the problem it fixes — or whether it is a demo dressed as a product. This checklist is built for a principal at a 4–20 person commercial real estate firm with confidential deal data and no IT department to hide behind. Every question is one you can ask a salesperson to their face, and every red flag is one you can act on without a technical background.
Most vendor-evaluation checklists are written for enterprises with a procurement team, a security officer, and an RFP process. A ten-person brokerage has none of those, so the standard advice — “review the vendor’s SOC 2, involve IT, run a formal RFP” — is either impossible or wildly out of scale. What follows is scaled for the firm that makes the decision in a partner meeting on a Thursday. Before you evaluate a single tool, though, there is a question one step upstream: whether buying software is even the right move, which the buy-versus-build playbook works through in full.
First, Decide Whether You Should Buy at All
The cheapest AI vendor is often the one you do not hire. Two firms with the same problem can make opposite correct decisions: one buys a $200-a-month tool, the other writes a prompt over ChatGPT it already pays for and solves the same task for nothing.
Before you evaluate vendors, name the workflow, count how often it happens, and estimate the hours it eats. A task that consumes analyst time every week is worth a serious tool or a build. A task that happens twice a quarter rarely justifies a subscription, and almost never justifies a custom project. Run the checklist below only once you have decided the workflow is worth paying for — otherwise you are shopping for a solution to a problem you have not sized.
The 10-Point AI Vendor Evaluation Checklist
Ask these ten questions of any AI vendor — a proptech subscription or a custom-build partner. Score each one pass or fail. A tool that fails on data terms, auditability, or accuracy is a walk-away regardless of how well it demos.
1. Does It Solve a Workflow You Actually Have
Start with your problem, not the vendor’s feature list. A tool that ranks inbound deals is useless if your bottleneck is lease abstraction, no matter how impressive the ranking is. Write down the one workflow where hours go to die — abstracting leases, triaging broker blasts, reconciling CAM, drafting market write-ups — and make the vendor show you that exact workflow on your data. If the demo drifts to features you did not ask about, that is a sign the tool was built for someone else’s problem.
2. Test It on Your Worst Documents, Not the Demo
The accuracy that matters is on your ugliest files, not the vendor’s clean sample deal. Modern AI tools read tidy, well-formatted PDFs well and stumble on the scanned, amended, typo-ridden documents that fill a real CRE inbox. Hand the vendor ten of your own documents during evaluation, including the two worst — a scanned T-12, a lease with three amendments — and check the extraction line by line. The failure to fear is not an obvious error, which anyone catches, but a clean-looking number pulled from the wrong column. If the vendor will not test on your documents, you have learned what you need to know.
3. Get the Three Data Questions Answered in Writing
Your deal packages are confidential and often under NDA, and a contract is the only security department a small firm has. Before you upload anything, get three answers in writing:
- Where is our data stored, and who can access it?
- Do you train shared models on our uploads? The answer you want is no.
- How do we delete and export our full history, and how fast?
“Your data is secure” is a marketing line. A data-processing addendum that commits the vendor not to train on your data and to delete it on request is an enforceable one. Treat vagueness here as a reason to walk, not a detail to sort out later.
4. Is It Real AI or a Thin Wrapper
Many proptech tools now advertise “AI” that turns out to be a light layer over a general model like ChatGPT or Claude, sold at a steep markup. That is not automatically bad — the packaging and workflow can be worth paying for — but you should know what you are buying. Ask directly: what does your tool do that we could not do ourselves with a general AI tool and a good prompt? A confident vendor answers with specifics — connections to your systems, a trained pipeline for CRE documents, a maintained workflow. A vendor who deflects is selling you a wrapper you could rent for a fraction of the price.
5. Demand Auditable Output and a Human in the Loop
Any tool that touches a number, a dollar figure, or a go/no-go decision must show its work and leave the call to a person. A screening score with no visible reasoning is a liability the first time it ranks a bad deal highly. Require that every extracted figure links back to its source page so a human can verify the ones a decision hinges on. The correct division of labor is AI for the reading and drafting, human judgment for the decision. A vendor who pitches the tool as replacing your analyst’s judgment is describing a risk, not a feature.
6. Price the Full Twelve Months, Including Your Hours
The monthly sticker is rarely the real cost. Setup, integration, training your team, and the hours to maintain the tool usually make up most of the first-year total. Price twelve months with your own time included: subscription fee, onboarding, and the staff hours the tool consumes before it saves any. For custom builds, the market range runs roughly $25,000 to $150,000 as a one-time project depending on scope, plus modest monthly run costs — the full breakdown of what moves that number is in our guide to what a custom AI automation project costs a small firm. Compare the true twelve-month figure against the labor hours the tool returns; if payback does not arrive inside a year for a weekly task, question the purchase.
7. Confirm It Connects to the Systems You Run
A tool that cannot reach your inbox, your spreadsheet, your CRM, or your accounting package makes you the integration. Ask exactly how it connects to the systems you already run — Outlook, Excel, a modern CRM, Yardi or AppFolio or Buildium — and whether that connection is built in or something you paste between by hand. A tool that requires manual copy-paste every day adds a chore while claiming to remove one. Verify the connections you need exist today, not on a roadmap.
8. Ask Who Fixes It When It Breaks
A firm with no IT department cannot maintain software itself, so support is not a nicety, it is the difference between a working tool and a dead login. Ask who you call when the tool stops reading a document format, what the response time is, and what happens when the underlying AI models are updated. For a subscription, this is the vendor’s support tier. For a custom build, it is a named maintenance arrangement with response times — its absence is the most common and most expensive omission in a proposal, and our guide to choosing the right AI development partner covers what separates a partner from a vendor who disappears at handoff.
9. Verify Your Data Portability and Exit
You should own your outputs and be able to leave. A tool that locks your structured lease data or deal pipeline inside its own format recreates the vendor trap you were trying to escape. Confirm before signing that you can export your full data in a usable format on demand, and that leaving does not mean starting over. The subscription you can walk away from is the one with real bargaining power; the one holding your data hostage will raise its price the moment it can.
10. Weigh the Vendor’s Staying Power
Proptech is a churny market, and a tool that vanishes takes your workflow with it. You are not underwriting the vendor’s balance sheet, but a few signals are fair game: how long the company has operated, whether it serves firms like yours, and whether it is a focused product or a side feature the company may drop. A newer vendor can be an excellent choice — just make the data-export answer in question nine airtight, so a shutdown is an inconvenience, not a catastrophe.
Red Flags That Should End the Conversation
Some answers should stop the evaluation on the spot, regardless of how good the tool looks:
- Vague data terms. No clear answer on storage, training, and deletion means confidential deal data is at risk. This is the fastest walk-away.
- Silent guessing. A tool that invents a missing number instead of flagging it is worse than no tool, because it looks finished. Ask what it does when a document is unreadable.
- A black box you cannot audit. A score or summary with no traceable source is a liability waiting for its first bad deal.
- “It replaces your analyst.” Overreach in the pitch predicts overreach in the product. The tool should remove busywork, not judgment.
- No maintenance answer. For a build, a code handoff with no support plan is half a product. For a subscription, thin support means you are on your own.
- Won’t test on your data. A vendor confident in the tool will run it on your worst documents. One who insists on the demo deal is hiding the accuracy that matters.
Any single one of these is enough to end the meeting. They are the failures that lose money or leak data, not the ones that merely disappoint.
Same Checklist, Different Answers: Subscription vs Custom Build
The ten questions apply to both an off-the-shelf tool and a custom automation, but the strong answers differ.
| Question | Strong answer from a subscription | Strong answer from a custom build |
|---|---|---|
| Data terms | Standard DPA, no training on your data, self-serve export | Runs in infrastructure you control; you own everything |
| Accuracy | Proven on your documents in a trial | Tuned to your specific documents during the build |
| Cost | Predictable monthly fee at your head count | One-time build plus a modest, named monthly run |
| Maintenance | Vendor support tier with response times | A named retainer with response times and a model-update plan |
| Fit | Serves your workflow out of the box | Built for a workflow no vendor connects |
Buy when a standard tool fits your asset classes and volume, because a subscription is faster and cheaper to start. Build only when your workflow is specific enough that no product fits, or when it crosses systems no vendor connects. The three-year cost comparison between the two paths is worked out in our analysis of proptech subscriptions versus a custom automation project. Whichever path you choose, the discipline of buying capability deliberately — rather than accumulating logins — is the through-line of the approach that lets small firms out-operate far larger ones.
Frequently Asked Questions
What questions should I ask an AI vendor before buying?
Ask ten, in order: does it solve a workflow you actually have; how accurate is it on your worst documents; where is our data stored and do you train on it and how do we delete it; is this real AI or a wrapper on a general tool; can we audit the output and keep a human on the decision; what is the full twelve-month cost including our hours; how does it connect to our systems; who maintains it when it breaks; can we export our data and leave; and how stable is the vendor. Get the answers to the data and accuracy questions in writing. Any tool that fails those two is a walk-away regardless of how well it demos.
How do I know if a proptech tool is real AI or just AI-washing?
Ask the vendor exactly what the tool does that you could not do yourself with a general AI tool like ChatGPT or Claude and a good prompt. A real product answers with specifics: connections to your systems, a pipeline trained on CRE documents, a maintained workflow, support. A thin wrapper deflects or talks only in benefits. This does not make a wrapper worthless — packaging and workflow can be worth paying for — but you should know you are paying a markup for convenience, not for technology you could not otherwise access, and price it accordingly.
How do I protect my confidential deal data when evaluating an AI vendor?
Get three answers in writing before you upload anything: where the data is stored and who can access it, whether the vendor trains shared models on your uploads (you want a firm no), and how you delete and export your full history. A contract is the only security department a small firm has, so require a data-processing addendum that commits the vendor on training and deletion. “Your data is secure” is marketing; an enforceable clause is protection. Vague answers are a reason to walk, because deal packages are confidential and often under NDA.
How much should a small CRE firm pay for an AI tool?
It depends on buy versus build. Off-the-shelf subscriptions are typically a monthly per-seat or per-firm fee, while a custom automation project generally lands in the range of $25,000 to $150,000 as a one-time build plus modest monthly run costs, depending on scope. The subscription sticker is rarely the real cost — setup, integration, training, and maintenance make up most of the first-year total. Price the full twelve months with your own hours included and weigh it against the labor the tool returns, not the monthly number alone.
Should I buy a proptech subscription or build a custom automation?
Buy when a standard tool fits your asset classes and volume, because a subscription starts faster and cheaper. Build only when your workflow is specific enough that no product ranks or reads it well, or when it crosses systems no vendor connects. Most 4–20 person firms should default to buy and earn the build case with real volume data. Many land in the middle: a general tool for standard tasks plus custom prompt workflows for the parts unique to how the firm makes money.
Can I just use ChatGPT or Claude instead of buying a proptech tool?
Often, yes, for the first pass and at low volume. A well-written prompt over ChatGPT, Claude, or Gemini plus a spreadsheet will extract and summarize many documents using tools your firm already pays for. The limits: general models can confidently misread a figure off a messy scan, and they do not connect to your inbox or keep a pipeline, so you manage by hand. Use them to get fluent and to prove whether you need a dedicated tool at all — and verify every number they pull before it moves money.
What contract terms matter most in an AI vendor agreement?
Three: a commitment not to train on your data, a deletion-and-export right you can exercise on demand, and a defined support arrangement with response times. For a custom build, add a named maintenance retainer and a plan for when the underlying models update. These terms are the small firm’s real protection, because you do not have an IT team to compensate for a weak contract. The absence of any of them should lower your offer or end the conversation.
How do I test an AI tool’s accuracy before I trust it?
Run it on ten of your own documents during the evaluation, including the two ugliest you can find. Check the output line by line against the source. The error to hunt for is not an obvious mistake — those get caught — but a clean-looking, wrong number pulled from the wrong place. Require that every figure the tool extracts links back to its source page so a person can verify the ones a decision depends on. A vendor who will only demo on a clean sample deal is hiding the accuracy that actually matters to you.
What are the biggest red flags in an AI vendor pitch?
Vague data terms, a tool that guesses instead of flagging missing information, output you cannot audit, a pitch that claims to replace your analyst’s judgment, no maintenance plan, and a refusal to test on your own documents. Any single one is enough to walk. These are the failures that lose money or leak confidential data, not the ones that merely underwhelm, so treat them as disqualifiers rather than negotiating points.
Where to Start
A checklist is only useful once you know which workflow to point it at, and most firms start shopping for vendors before they have decided which problem is worth solving first. That inventory — which of your workflows a tool should fix, in what order, and whether the honest answer is a subscription, a build, or a prompt you already own — is the real first step. A free AI-readiness assessment produces exactly that: a working session that maps your firm’s workflows, flags the ones where a tool pays for itself, and hands you a ranked plan with real cost ranges before you sit across from any salesperson. Book a free AI-readiness assessment if you want that map first. If no vendor is the right spend for your firm right now, the assessment will tell you so — and you will still leave with the plan.
Arthur Wandzel
