Function calling is the contract that lets a language model do things in the real world: the model emits a structured JSON object naming a function and its arguments, your code runs the function, and the result is fed back to the model. Without it, an LLM is a text completion engine. With it, an LLM is a runtime that can search a database, charge a credit card, send an email, or refund a customer. It is the single primitive that turned chatbots into systems that ship work.
This piece is for a non-engineer founder reading vendor proposals full of phrases like AI-powered, tool use, agent runtime, and structured output. You get the 2026 definition, the four ingredients to demand by name, the fit test, the eval shape, the three founder-relevant failure modes, and a worked customer-support example. The article builds on the eval-first build playbook, part of the broader idea-to-product manifesto.
Table of Contents
- The 2026 definition (plain English)
- Why function calling exists at all
- The 4 ingredients (the buyable line items)
- When function calling fits — and when it doesn’t
- The eval shape for function-calling features
- 3 founder-relevant failure modes
- A worked example: the refund-tool agent
- Procurement red flags in 2026 proposals
- Frequently Asked Questions
- Closing
The 2026 definition (plain English)
Function calling is a request/response contract between a language model and your code. You describe a set of functions — each with a name, a plain-English description, and a typed schema for its arguments. Given a user request, the model decides whether to answer directly or call one of the functions. If it calls one, the model returns a JSON object with the function name and arguments. Your code runs it, returns the result, and the model continues.
Three points to fix in mind. The model does not run the function — your application does; the model only requests the call. The request shape is JSON, not English. And the function does whatever your code does — search a database, call Stripe, send Slack. The model has no special privileges; it inherits what your application is allowed to do.
OpenAI introduced this pattern with GPT-4-0613 in June 2023; Anthropic and Google followed. The Model Context Protocol (MCP), open-sourced by Anthropic in November 2024, has emerged as the cross-provider standard. The mechanism is now a commodity.
Why function calling exists at all
Before function calling, the only way for an LLM to act on the world was for an engineer to parse the model’s free-text output with regular expressions. That approach broke constantly — the model phrased the same intent five different ways, missed arguments, or hallucinated parameters.
Function calling solved that with two changes. The model was post-trained to emit a structured request directly. And the provider’s API surfaces a separate field — tool_calls on OpenAI, tool_use blocks on Anthropic — so the request is never confused with the assistant’s reply. The contract is now machine-readable and schema-enforced. OpenAI’s Structured Outputs (Strict mode), released August 2024, lifted argument-schema compliance from roughly 83% to 99.9% on their benchmark. Function calling is no longer a research bet; it is a load-bearing primitive.
The 4 ingredients (the buyable line items)
Every credible function-calling implementation in 2026 contains exactly four parts. They map onto line items in a scope of work; a non-engineer founder should be able to name all four.
1. The tool schema. A list of functions the model can call — each with a name (e.g. lookup_customer), a plain-English description, and a typed JSON Schema for arguments. Without it the model has no contract to honor and your code has no contract to parse. Demand the JSON Schema for every tool in the proposal appendix.
2. The model decision. Given the user’s request, the conversation, and the tool list, the model decides whether to call a tool — and if so, which one with what arguments. This is the load-bearing intelligence and the eval target: when the model picks the wrong tool, your feature ships the wrong action. Demand a tool-call accuracy benchmark.
3. Your runtime. The code that receives the tool_call, validates arguments, executes the function (DB query, Stripe charge, Slack post), catches errors, and returns the result. Security lives here — authentication, authorization, rate limiting, audit logging. Demand a runtime architecture diagram.
4. Result handling. The output flows back as a tool message (OpenAI) or tool_result block (Anthropic), and the model continues — either answering the user or calling another tool. The shape of the result determines whether the model can act on what it sees. Demand a result-schema spec.
If the proposal does not contain all four artifacts, the function-calling line item is undefined. You are buying behavior the engineering team has not yet specified.
When function calling fits — and when it doesn’t
Function calling fits when the model needs to cause a side effect outside its context window. Three cases where it fits:
Real-world side effects. The model must do more than write text — search a database, charge a card, send an email, update a record, fetch a live price, file a ticket. The side effect is the value the user came for.
Live data the model cannot have memorized. The user asks “what is my current order status?” No frontier model can have memorized your customer’s order. The model must call a tool to read your database. Same for prices, schedules, inventory, account state.
Composition across systems. The request requires touching two or more independent systems — find the customer in our CRM, then check the payment in Stripe, then update the ticket in Zendesk. Function calling, usually inside an agent loop, bridges them without a brittle procedural script.
Three cases where function calling is the wrong shape:
Pure generation. “Draft a marketing email.” No external dependencies; wrapping it in a function-calling loop adds tokens and latency for zero quality gain.
Pure classification or extraction. “Classify this ticket. Extract line items from this invoice.” These are structured output problems — use a JSON Schema response_format directly. Cheaper and just as reliable.
Deterministic flows. “On every new lead, look up the company, generate a first email, save to CRM.” The sequence is fixed. Write it in code; call the LLM only at the step that needs language understanding.
The disciplined founder bids down — start at structured output, escalate to a single function call only when a real-world side effect is required, escalate to an agent loop only when the signals for an agent fire.
The eval shape for function-calling features
A function-calling feature is graded differently from a single-shot generation feature. The model produced a tool_call — did it call the right tool with the right arguments? Four metrics, all required.
1. Tool-call accuracy. Of the times the model called a tool, what fraction were the right tool with valid arguments? The Berkeley Function-Calling Leaderboard (BFCL v3) is the public benchmark: as of early 2026, frontier models cluster at 85–92% on multi-turn, multi-tool tasks; smaller open-weight models 50–75%.
2. Argument validity. Of the tool calls the model made, what fraction validated against the JSON Schema? With Strict mode and equivalent constrained-decoding modes, this should be ≥99%. If it is not, your schemas are ambiguous or descriptions are weak — fix the schemas, not the model.
3. Latency per call. Function calling adds round-trips: model → tool_call → your code → tool_result → model. Median and 95th-percentile latency end to end matters for any user-facing feature. Common 2026 baselines: 1.5–3 seconds median for a single tool call on frontier models, 4–8 seconds for two-call sequences.
4. Cost per successful task. Every model turn and every tool call costs tokens. Track cost per successful task, not cost per model call.
Without these four on dashboards from day one, the feature is undefended. A vendor whose proposal shows model-quality benchmarks but no tool-call accuracy, no schema-validity rate, and no latency or cost numbers is selling a single-call LLM feature with a function-calling label. The companion piece what is prompt engineering, and how much does it actually matter covers a related primitive — the system prompt that tells the model when and how to use your tools.
3 founder-relevant failure modes
Every founder who ships a function-calling feature in 2026 hits at least one of these in the first two months. Name them in advance and you can put a clause in the SOW.
Failure mode 1 — Wrong tool picked. The user asks “can I get a refund?” and the model calls update_subscription instead of start_refund. This is the most common failure and the hardest to catch without an eval suite. Cause: tool descriptions too similar or vague. Fix: rewrite descriptions to be mutually exclusive and exhaustive; re-eval.
Failure mode 2 — Malformed arguments. The model calls the right tool but with arguments the runtime cannot use — a missing field, a date in the wrong format, a customer ID that does not exist, a refund amount in dollars instead of cents. With Strict mode enabled, syntactic validity is near-100%, but Strict mode does not protect against hallucinated values that happen to be schema-valid. The model can invent a customer ID that conforms to the type but does not exist. Fix: validate semantically in the runtime. Add explicit not_found returns and let the model retry.
Failure mode 3 — Infinite loops. The model calls a tool, gets an error, tries another tool, gets another error, tries again. Without an iteration cap and a budget cap, the loop runs until you notice the bill. Fix: hard cap on iterations per task (5–20), hard cap on dollars per task, and a human-in-the-loop trigger on the third consecutive error.
The decoding “production-ready” in AI agency proposals piece catalogs adjacent procurement-side red flags.
A worked example: the refund-tool agent
Consider an e-commerce business with a support inbox at 200 emails a day, a third of them refund requests. The team wants an agent that reads the email, looks up the customer, checks the order, and — if it qualifies — issues the refund; otherwise escalates.
The four ingredients.
Tool schema — five functions. lookup_customer(email), get_order(order_id), check_refund_policy(order, reason), issue_refund(order_id, amount_cents, reason), escalate_to_human(case_id, reason). Each with a tight JSON Schema; amount_cents is an integer, reason is an enum.
Model decision — Claude Sonnet 4.6 reads the email and picks tools in sequence. Typical path: lookup_customer → get_order → check_refund_policy → either issue_refund or escalate_to_human.
Runtime — Python service. issue_refund is gated by a policy check: amount under $50 and order under 30 days. Anything else, the runtime forces escalate_to_human regardless of what the model decided.
Result handling — every tool returns status, data, error. Errors pass back to the model with explicit not_found and policy_blocked codes so the model can decide whether to retry or escalate.
The eval. Two hundred labeled historical emails. Tool-call accuracy ≥92% (BFCL frontier baseline). Argument-schema validity ≥99% with Strict mode. Median latency under 4 seconds. Cost per successful resolution under $0.05.
The cost model. Median three tool calls per email, 8k input tokens, 1k output tokens. At Claude Sonnet 4.6 pricing, roughly 2.4 cents in model cost per email. At 70 refund emails a day, ~$1.70 a day to deflect an hour of human support time.
The escape hatch. Hard cap of 7 tool calls per email; on the eighth, force escalate. Hard cap of $0.20 per email. Human review of the first 100 production runs before auto-refund is enabled. The eval cost is paid back in week two.
Procurement red flags in 2026 proposals
A non-engineer can audit a function-calling proposal in five minutes with this checklist.
- No JSON Schema in the appendix. The vendor names “function calling” but the schema is not enumerated. Behavior is unspecified.
- No tool-call accuracy benchmark. The eval section talks about model quality but not about whether the model picks the right tool. You are buying a chatbot grade, not a runtime grade.
- No iteration or budget cap. No mention of how the loop is bounded. The first production deploy bills the difference.
- No mention of Strict mode (or equivalent). Schema-enforced output is a one-line API flag in 2026. A vendor that does not turn it on is shipping ~83% argument validity instead of 99%.
- No human-in-the-loop spec for high-stakes tools. Proposal lets the model call
issue_refundordelete_recordwithout a policy gate. Excessive Agency is OWASP LLM06 for a reason. - “We’ll use MCP” with no tool list. MCP is a transport, not your tool registry. If MCP is the answer to what tools does the model have, the answer is non-responsive.
If three or more are missing, the line item is a placeholder for engineering work that has not yet been done.
Frequently Asked Questions
What is the simplest definition of function calling?
A language model emits a structured JSON object describing which function to call and with what arguments. Your code runs the function and returns the result. The model does not run the function; it only requests the call.
Is function calling the same as tool use?
In 2026 the terms are used interchangeably. OpenAI uses “function calling”; Anthropic uses “tool use”; Google uses “function calling.” The mechanism is identical: schema-described functions, model-decided calls, structured arguments.
How does function calling differ from a regular LLM call?
A regular LLM call returns text. A function-calling call returns either text or a structured tool_call object. The model decides based on whether the user’s request needs a side effect.
Do I need an agent to use function calling?
No. Function calling works with a single round-trip: user asks, model calls one function, your code runs it, model formats the answer. An agent is what you get when the loop runs multiple times with the model deciding each next step. Function calling is the primitive; the agent is one pattern built on top.
What is the failure rate of function calling in 2026?
With Strict mode (or equivalent), schema-validity is ≥99%. Tool-call accuracy — picking the right tool with the right arguments — sits at 85–92% on the Berkeley Function-Calling Leaderboard for frontier models on multi-turn tasks.
Should I worry about prompt injection through function calls?
Yes. If a tool returns user-controllable data (an email body, a webpage, a customer note), the model reads it and may follow instructions hidden inside. Treat tool returns as untrusted input: sanitize, scope tool permissions tightly, and add system-prompt instructions to ignore embedded directives.
How much does function calling add to my LLM bill?
The tool schema is in the input on every turn, calls and results add to the conversation, and tasks take 2–5 model turns instead of one. Rule of thumb: 2–5× the token cost of a single-call equivalent. The savings come from the tool doing work the model would otherwise hallucinate or refuse.
What is MCP and do I need it?
The Model Context Protocol, open-sourced by Anthropic in November 2024, is an open standard for exposing tools to LLMs in a portable way. For a single-vendor MVP, the provider’s native function-calling API is enough; MCP earns its keep with multi-provider portability.
What does “structured output” mean vs. function calling?
Structured output forces the model to return JSON conforming to a schema as its final answer — no tool call, no side effect. Function calling is structured output applied to an intermediate step that triggers your runtime. Same underlying capability; different use case.
Closing
Function calling is the contract that lets an LLM cause a side effect: a schema you define, a JSON object the model emits, a runtime your code owns, a result the model reads. Above that, everything is marketing. The decision a non-engineer founder owns is not how to implement function calling — it is whether the feature genuinely needs a real-world side effect, and what evaluation will prove the model picked the right tool with the right arguments before money moves.
The eval-first build playbook covers scoping any AI feature so it ships at quality. The companion pieces what is an LLM agent, actually and what is prompt engineering, and how much does it actually matter cover the two other primitives over-sold to non-engineer founders in the same conversations.
One-sentence summary: function calling is the JSON contract that lets a language model trigger a real-world action through your code — scope it only when a side effect is needed, and audit it on tool-call accuracy and argument validity, not on chat quality.
Dirk Jan van Veen, PhD, is CTO of SFAI Labs, a forward-deployed AI development partner helping non-technical founders ship AI products in 2026.
Dirk Jan van Veen, PhD
