Agentic Loop Token Costs Are an Architecture Problem

When the API bill for an LLM agent arrives, the reflex is to blame model pricing. The reflex is usually wrong. In production, the largest share of token spend typically comes from the loop itself, from the way each turn re-sends the system prompt, re-attaches tool schemas, and re-reads the entire conversation so far. Agentic loop token costs scale with your architecture, not with the user's question length, and they compound across turns until one successful resolution can cost many times more than the prompt that started it.

The fix is instrumentation, not a cheaper model. Engineers who treat the agent loop like a distributed system trace, logging tokens at every node, find that the majority of spend is duplicated baseline payload and stale history rather than genuine reasoning. This article maps the specific points inside an agentic loop where tokens vanish, then gives you a diagnostic playbook to isolate the waste in your own architecture.

The Architecture Behind Agentic Loop Token Costs

A single user query in an agent does not produce a single API call. It produces a chain. The model reads the prompt, decides to call a tool, the tool runs, the result goes back, the model reads everything again, decides on the next tool, and so on until it emits a final answer. Each link in that chain is a full billable request, and each request charges for everything the model reads, not just the new fragment it produces.

This is where the mental model breaks. Developers trained on chat completions tend to think in terms of one prompt in, one response out. An agent turns that into a fan of calls. If you want to understand how providers account for that, read how OpenAI's token usage accounting works, because every agent request is measured against those input and output rules. Before optimizing anything else, trace a single user query end to end and count the actual API calls it triggers.

The structural insight is that the marginal token cost per step is not constant. Step one pays for the system prompt, the tools, and the user message. Step two pays for the system prompt, the tools, the user message, the model's first reasoning, the first tool call, and the tool result. Step three pays for all of that plus the second reasoning and the second tool result. Within a handful of steps, the static payload from earlier turns dwarfs the new tokens generated, and the user's original question becomes a rounding error in the bill.

Three questions frame the rest of this article. Where does the static payload come from? Why does it grow the way it does? And how do you measure it per node so you can cut it?

System Prompt and Tool Definition Bloat

The first and most consistent source of waste is the payload that never changes. Every turn of a well-formed agent request re-sends the system prompt and the full tool schema set, because the API is stateless from the model's perspective. The model has no memory between calls, so the framework resends everything it needs to behave consistently.

The System Prompt Tax

System prompts are the obvious offender. A prompt that specifies persona, rules, output format, guardrails, and few-shot examples can easily reach into the thousands of tokens, and every single turn pays that tax. The cost is flat per turn but cumulative across a run: a 1,500-token system prompt in a 10-step session costs 15,000 input tokens just for instructions the model already received and cannot retain.

Tool Schema Bloat

The subtler and faster-growing problem is the tool schema set. Each tool you register ships its name, description, parameter JSON schema, and any inline documentation the framework extracts. A modestly documented tool with nested parameters can run into the hundreds of tokens, and an agent with a dozen tools can spend a substantial fraction of every turn just declaring what it is allowed to do.

Before touching code, run your real tool definitions through a function calling token calculator to measure the per-call cost. Most teams are surprised to find that tool declarations alone can rival the size of the user prompt, and that cost is paid on every step, including steps where the tool is never invoked.

The diagnostic question for this section is simple. If you removed a tool from the schema set for a given task, would the model still succeed? Tools that are always present but rarely used are pure overhead, billed once per turn, for the entire run.

Unbounded Context Window Accumulation

If system prompt bloat is a flat tax, context accumulation is compound interest against you. Naive agent architectures append the full message history, including raw tool outputs, to the context window on every subsequent call. The model needs prior context to reason, so frameworks default to passing everything.

The cost curve this produces is the real killer. If each turn's tool output is roughly constant in size, then turn N pays for the sum of all prior turns' outputs. Here is how that plays out across a five-step agent run with a 500-token system prompt, 800 tokens of tool schemas, a 50-token user query, 100 tokens of reasoning per turn, and 400-token tool outputs:

Total input billed across five turns: 11,750 tokens. The static payload (system prompt plus tool schemas) accounts for 6,500 of those, or 55%, even though the model only needed it once. A session that runs ten steps is not paying ten times the per-step cost; it is paying a sum that grows with each step, so total spend climbs faster than linearly as sessions lengthen. This is why long-running agents, support bots, and research assistants blow through budgets even when each individual query looks cheap.

The compounding gets worse when tool outputs are verbose. An untruncated web search result, a full database row dump, or a raw API response can be enormous, and the agent re-reads every byte of it on every later turn. Context length affects both cost and latency in tandem, which is why longer sessions hurt on two axes at once, not just on the pricing line.

The architectural failure here is treating the message log as the agent's memory rather than as a transmission buffer. Everything the model might ever need gets stuffed in, with no pruning, no summarization, and no eviction. Frameworks that manage conversation history for you, like the history management patterns in the OpenAI Assistants tutorial, exist precisely because the default behavior of appending everything is financially unsustainable past a few turns.

Retry Branching and Intermediate Reasoning Overhead

The third drain is the one engineers rarely budget for: failed attempts and the reasoning that produces them. A production agent does not always succeed on the first tool call. A schema mismatch, a missing argument, a rate limit, a malformed response, and the model is asked to try again. Each retry is a full billable turn, complete with the re-sent system prompt, the re-sent tool set, and the accumulated history plus the error message appended to it.

This is where token accounting gets sneaky. The successful resolution is what you measure and report. The three failed attempts that preceded it are often invisible because they did not produce a user-facing result, yet they consumed full turns of input and output tokens. A feature that appears to work can quietly cost several times what a naive per-resolution estimate predicts, because the resolution is the leaf of a tree that includes every dead branch.

Intermediate reasoning compounds this. When models emit planning steps, chain of thought, or structured scratchpad entries between tool calls, those tokens become part of the input on every subsequent turn in the same run. Reasoning that was useful at step two is re-billed at steps three through ten because it sits in the history and gets re-read every time. The model generated it once, but you pay for it repeatedly.

The combined effect of retries and reasoning is that agentic token costs are path-dependent. Two runs of the same agent on the same input can produce wildly different bills depending on which branches died and how much the model chose to reason out loud before answering. Static cost models cannot capture this. Only per-turn tracing can.

Building a Token Diagnostic Playbook

You cannot optimize what you do not measure at the right granularity. Total spend per run, or cost per successful resolution, are ship-or-kill metrics. They tell you whether something is expensive, not why. To find architectural waste you need a token ledger broken down by node, and that means instrumenting the loop.

The minimum viable instrumentation logs four things on every API call: the input token count, the output token count, the role of the call (initial reasoning, tool selection, post-tool reflection, retry), and the cumulative session length at that point. With those four fields you can reconstruct the cost curve and see exactly where the bill spikes. Framework-level helpers make this easier than building it from scratch; log usage with LangChain's callback pattern to capture per-step counts without rewriting your agent code.

For production agents where you need richer observability, a dedicated tracing layer pays off. Tools like MLflow token usage tracing let you attach cost as a first-class span attribute on each node of the agent graph, so you can drill from a slow or expensive run straight into the specific tool call or reasoning step that produced the spike. What matters is that per-turn token logging is the only reliable way to separate architectural waste from genuine reasoning requirements, because anything coarser than per-turn granularity hides the problem.

A practical playbook looks like this:

1. Capture per-turn token deltas for a representative batch of real sessions.
2. Bucket calls by role (reasoning, tool selection, reflection, retry).
3. Compute the ratio of static payload (system prompt plus tools) to dynamic payload (new reasoning and tool outputs) per turn.
4. Track the cumulative context length curve across each session.
5. Flag any session where the static ratio is high and the curve is steep.

Those flagged sessions are where architecture, not model choice, is setting the price.

Architectural Fixes for Token Waste

Rank the four fixes by tokens saved per engineering hour, not by cleverness. The table below uses a 10-step agent run as the reference session.

Cache the Static Payload

Because the system prompt and tool schemas are byte-identical across turns, prompt caching lets the provider bill them at a steep discount on subsequent calls. Caching a 3,000-token static payload across a 10-step run eliminates roughly 27,000 re-billed input tokens per session, savings large enough that a model downgrade is often the wrong first move. Mark your static payload with a cache prefix; the Anthropic prompt caching guide has the API details. The trade-off: any change to the system prompt invalidates the cache, so prompt versioning becomes a release concern. Edit the system prompt mid-session and the next call pays full price.

Prune and Summarize State

Instead of appending every tool output verbatim, summarize older outputs, drop fields the model no longer references, and cap retained history to a rolling window. A 10-step session with 400-token tool outputs accumulates 4,000 tokens of raw history; a summarization layer that compresses each completed subtask to roughly 100 tokens caps that at around 1,000. Frameworks that manage memory as a structured layer, such as the LlamaIndex memory module, make this tractable. The trade-off: aggressive summarization can drop details the model needs to avoid repeating a failed approach, so prune by relevance, not just recency.

Cap and Instrument Retries

A hard retry limit prevents a single malformed tool from turning one resolution into a many-turn tree. Log retries as their own cost bucket so dead branches show up in your ledger. The trade-off: a low retry cap degrades user experience for genuinely recoverable errors, so pair the limit with clean, structured error messages that help the model succeed on the second attempt rather than the fifth.

Load Tools Dynamically

If you have a dozen tools but any given task uses two, gate which tools are registered per turn. A routing step that selects the relevant subset trims the schema payload on every subsequent call. The trade-off: the routing step is itself a billable call, so it only pays off when the schema savings across the remaining turns exceed the routing overhead.

The prioritization rule is simple: cache first, prune second, cap retries third, route tools fourth. The first two reduce the token count itself, which is why they beat a model downgrade. A cheaper model billed on the same bloated payload is still bloated.

Stop Blaming the Model

Agentic loop token costs are an architecture problem dressed up as a pricing problem. The loop re-sends static payload every turn, accumulates raw history that compounds across steps, and silently pays for every dead branch the model explored before answering. None of that shows up in a per-resolution cost number, which is why teams keep blaming the model.

The way out is discipline. Instrument every node. Separate static tax from dynamic reasoning. Cache what is immutable, prune what is stale, and load only the tools the current step needs. Do that before you reach for a smaller model, because the waste you uncover will almost always be larger than the discount a downgrade buys.