The Tool-Calling Stack, Explained From the Ground Up

Ask most people who use AI agents daily to explain what actually happens between "the agent decided to check the weather" and "the agent told you the weather," and you'll get a shrug. It feels like magic because the interesting part is hidden — a model that seems to reach out, touch the world, and come back with an answer. It isn't magic. It's a fairly small number of well-defined layers stacked on top of each other, and once you've seen all of them once, tool-calling stops looking like a capability the model has and starts looking like what it actually is: a protocol your code and the model agree to follow.

We think that reframing is useful for anyone building on top of agents, not just the people writing the frameworks, because most of the places agents break in practice trace back to one specific layer in this stack — and you can't fix a layer you don't know exists.

Layer one: the schema, which is a contract, not a function

Before a model can "call" anything, someone has to describe what's available to call, and that description is not the function itself — it's a schema, typically JSON, that lists a name, a natural-language description, and a set of parameters with their own types and descriptions. Something like a tool named search_orders that takes a customer_id string and an optional date_range, with a description explaining what it returns and when to use it.

This is worth sitting with, because it means the model never sees your code. It sees a document describing your code, written in prose, and it has to infer from that prose alone when the tool is appropriate and how to fill in the arguments. A vague description — "searches orders" — leaves the model guessing about edge cases your actual function handles very specifically. A precise one — "searches orders for a given customer within an optional date range; returns an empty list rather than an error if none are found" — removes an entire category of misuse before the model ever runs. Teams that treat schema-writing as an afterthought are, without realizing it, writing thin documentation for a very literal-minded new engineer and then being surprised when that engineer misuses the tool.

This gets harder, not easier, as a system grows. An agent with three tools rarely struggles to pick the right one. An agent with thirty tools, several of which do subtly overlapping things, starts making selection mistakes that have nothing to do with reasoning ability and everything to do with description quality — two tools whose descriptions both plausibly apply to a given request, with nothing in the text to break the tie. The fix isn't a smarter model. It's the same fix a competent technical writer would reach for: sharper, more mutually exclusive descriptions, and, past a certain count, actively pruning the tool list a given call can see down to only the ones actually relevant to the task at hand.

Layer two: the model doesn't call anything, it predicts text that looks like a call

Here's the part that demystifies the rest of the stack once it clicks: the model does not reach out and execute your function. It cannot. It has no hands. What it does is predict the next tokens in a sequence, the same way it predicts the next tokens of an ordinary sentence, except that during training it was taught a specific structured format for "I want to invoke this tool with these arguments" — usually something that resolves to a JSON object with a tool name and a set of parameters.

From the model's point of view, generating a structured call to search_orders with a given customer_id is not meaningfully different from generating the next sentence of an email. It's pattern completion, conditioned on the schemas it was shown and the conversation so far. The apparent decision-making — "the model decided to check the order history" — is really the model's learned sense of what a helpful assistant would output next, given a user question and a list of available tools. That's not a knock on the capability; it's genuinely useful pattern completion. But understanding that it is pattern completion, not an internal decision engine reaching out to touch a database, explains a lot of the failure modes you see in practice, like a model confidently calling a tool that doesn't exist because the surrounding context made that shape of output statistically plausible.

Concretely, what the model actually emits looks something like a small, rigid block of structured text — a tool name and a handful of key-value pairs, nothing more expressive than that. There's no hidden channel where the model "thinks about" calling the tool separately from generating that text. The generation is the decision. This is why prompt and schema quality have such an outsized effect on tool-calling reliability: nobody is persuading a decision-maker, they're shaping the statistical landscape that determines which block of structured text is most likely to come next.

Layer three: where the actual work happens, entirely outside the model

The moment the model emits that structured tool-call output, its job is done for this step. Everything that happens next — parsing that output, validating it against the schema, actually executing the function, handling the timeout, catching the exception if the downstream API is down — happens in your code, on infrastructure you own, with no model involved at all.

This is the layer that gets the least attention in tutorials and, in our experience, does the most work in production. It's where you decide what happens if the model hallucinates a parameter that doesn't exist in your schema — do you reject the call and tell the model, or silently coerce it? It's where you decide whether tool execution runs in a sandbox, what permissions it has, whether a destructive action needs a confirmation step before it fires. None of that is model behavior. It's ordinary software engineering, and the fact that a language model triggered the code path doesn't change any of the standard rules about validating input, handling failure, and not trusting a caller — even a caller as sophisticated as a frontier model — to always send exactly what you expect.

A well-built execution layer treats every tool call the model produces the way a well-built API treats every request from a client it doesn't fully control: validate first, execute second. If the arguments don't match the schema — a required field missing, a string where a number was expected — the right move is usually not to guess or silently patch it, but to send a clear error back to the model as though it were a tool result, and let it try again with better information. That retry loop, small and unglamorous as it sounds, is responsible for a meaningful share of the reliability gap between a tool-calling system that feels solid and one that feels flaky.

Layer four: putting the world back into words

Once your code has actually run the function and has a result — a list of orders, a search result, a file's contents — that result has to go back to the model, and it has to go back as text, because text is the only thing the model can read. This sounds trivial and is not. A database might return forty rows with fifteen columns each. Sending all of it back verbatim burns tokens fast, pushes other useful context out of the window, and buries the two rows that actually matter under a pile of ones that don't.

Good tool-response design looks a lot like good API design: return the minimum the model needs to take its next step correctly, formatted so the model can parse it easily, with enough structure that ambiguity is unlikely. Teams that get this wrong tend to notice it as "the model keeps making mistakes after using this tool," when the real problem is that the tool's output was never designed to be read by anything, human or model — it was just whatever the database happened to return.

Layer five: the loop, and how it knows when to stop

The tool result gets appended to the conversation as a new message, and the whole sequence — original question, prior tool calls, prior tool results — goes back to the model for another prediction. The model might generate a final answer for the user. It might generate another tool call. It might generate several tool calls at once, if the platform supports parallel calls and the task allows it, like looking up two independent pieces of information in one round trip instead of two.

Parallel calls are worth a specific mention because they change the shape of the execution layer, not just the model's output. If a model requests two independent lookups in the same turn, the code handling them has to decide whether to run them concurrently, how to handle one succeeding while the other fails, and how to reassemble both results into a single coherent message before the next model call. None of that complexity is visible in the model's output, which just looks like two tool calls sitting next to each other. It lives entirely in the execution layer, which is, again, the layer you're responsible for.

This repeats until the model produces a response that isn't a tool call, or until some limit your code enforces — a maximum number of steps, a time budget, a cost ceiling — cuts it off first. That limit is not optional in any serious system. A model with unclear instructions and access to a tool that returns ambiguous results can, in principle, loop far longer than anyone intended, and the difference between an elegant agent and an expensive runaway process is often nothing more exotic than whether someone remembered to set a hard ceiling.

Seen end to end, tool-calling is five unglamorous layers — describe, predict, execute, translate, repeat — and the model only really owns one and a half of them. The rest is engineering you're responsible for whether you notice it or not, and the agents that hold up under real use are, almost without exception, the ones where someone did the unglamorous layers carefully instead of assuming the model would carry all five on its own.