LLM Context Window Management With a Token Budget

Somewhere around turn twelve, your agent runs out of room, not intelligence. Tool outputs pile up, an error retry doubles a message, and the model starts dropping the thread: it forgets the entity it defined at the start, truncates its final answer, or re-calls a tool it already used. The tempting fixes, a bigger model or a bigger window, treat the symptom. LLM context window management is mostly accounting, and the teams running capable agents in 8K windows are the ones who count tokens before they compress anything.

A trap also waits for whoever compresses carelessly. Prompt caches match on exact prefixes, so deleting the wrong tokens can raise both your bill and your latency. This guide covers the three tools that prevent that: a token budget ledger that separates fixed costs from variable ones, a compression ladder you climb one rung at a time, and the ordering rules that keep caches warm while you trim.

Why Agent Loops Overflow Small Context Windows

An agent loop is a repeated full re-send. Every call carries the system prompt, every tool schema, the entire conversation so far, and every tool output those turns produced. Nothing expires on its own, so context grows roughly linearly with turns, and a single verbose tool result, say a raw API response or a stack trace, can add hundreds or thousands of tokens in one step.

Two cost classes behave differently, and mixing them up is where the budgeting failure starts:

• Fixed costs sit in every call: the system prompt, tool definitions, long-lived instructions. They set the floor on window usage before your agent does anything. Tool schemas are the sleeper here. Because tool definition token overhead is injected into every request, a realistic toolkit of a dozen functions can quietly consume a four-figure chunk of a small window.
• Variable costs grow with the loop: conversation history, tool outputs, retrieved documents. These drive the overflow, which is why agentic loop token costs explode over long sessions even when each individual call looks small.

If you manage the window as one undifferentiated pile, you will trim blindly and break things. Fitting an agent loop into a small context window starts with a ledger that prices each class separately and protects a reserve the model needs for itself.

The Token Budget Ledger

Token budget allocation for LLM apps comes down to one equation: window size minus fixed costs minus a protected output reserve equals the budget you can actually spend on history. Work that out before writing any compression code.

A worked example for a modest 8,192-token window running a tool-using agent:

The variable budget is the number that should scare you. A single tool round trip, the call plus its result, often runs 300 to 800 tokens, so this agent gets roughly five to fourteen useful turns before it must compress. That is not a defect but the honest price of a small window, and knowing it up front beats discovering it in production logs.

The reserve deserves its own line because generated tokens share the window with input on the major APIs. On reasoning models it is worse: reasoning tokens count too, billed as output and consuming the same context, so an unbudgeted reserve surfaces as silent truncation, completions that stop mid-thought with no error raised. Providers also cap output separately from input, and those output limits versus window size differ enough to matter when you pick a model for agent work.

To fill the ledger with real numbers instead of guesses, count your prompt with the provider's own tokenizer, for example by counting tokens with tiktoken for OpenAI-family models. Tokenizers differ across providers, so re-measure per model. Then encode the ledger as constants and assert the variable budget before every call: refuse to send an over-budget prompt instead of letting the API truncate it for you.

The Compression Ladder

Context compression is not one technique but a ladder, and the rule is to climb only as far as the failure demands. Each rung buys more headroom at a higher risk price, and each has a named failure mode you should be able to spot in your logs.

Rung 1. Trim and cap

Drop the oldest turns, cap tool outputs at their source, strip boilerplate from results. This is the cheapest rung, and it fails quietly. Deleting old turns can destroy an early instruction, an entity definition, or a ticket ID the model still needs, and nothing complains at deletion time; you find out turns later when the agent misbehaves. The debugging cost of that invisibility usually exceeds the tokens saved.

Position matters even for what you keep. The Lost in the Middle study found that models retrieve information from the middle of long contexts markedly less reliably than from the beginning or end, so a critical constraint buried mid-history is half-invisible even when technically present. Keep hard instructions pinned at the top, and follow long-context prompting guidance by placing reference material at the very start or end of the prompt rather than letting it drift into the middle.

Rung 2. Summarize

Rolling conversation summarization compresses old turns into a summary block while keeping recent turns verbatim. It fixes the growth problem, but it introduces two failure modes of its own.

The first is summary drift. Because each new summary is written from the previous one, an early mistake or a fact that went stale gets folded forward into every subsequent summary, persisting long after the original text is gone. Research on error propagation in rolling summaries treats this as the structural weakness of the approach: errors compound rather than occur once. The symptom is an agent that keeps honoring a constraint the user revoked ten turns ago.

The second is specificity loss. Prose summaries reliably preserve gist and reliably lose numbers, IDs, and names, the exact fields agent state depends on. The fix is to split memory in two: a short prose summary for narrative continuity, plus a structured state object, JSON carrying active entity IDs, quantities, decisions, and deadlines, that is passed verbatim and never summarized. Frameworks like LangGraph's summarization memory implement this pattern, but the principle is provider-agnostic: to summarize chat history without losing details, the details must live outside the summary.

Rung 3. Externalize

The top rung moves content out of the window entirely: conversation history to a vector store, working artifacts to a file scratchpad, with retrieval pulling back only what the current step needs. This is external memory for LLM agents in the style of MemGPT's operating system approach, which treats context like RAM and storage like disk, paging data in on demand.

The trade is real. Every retrieval adds a network round trip and a similarity search before the model can act, and a miss means the agent confidently answers without the one fact it needed, a failure that looks like model stupidity but is actually retrieval recall. This is why externalization is the last rung: move conversation history to a vector store only after trimming and summarization have genuinely failed, not as a default architecture.

The Prompt Cache Trade-Off

Most trimming advice skips how caches are matched, yet this is the part of LLM context window management that decides whether your cuts help or hurt. The mechanics are strict: caches match on exact prefixes, covering the longest unchanged run of tokens from the front of your prompt. Edit one token near the front, and every token after that edit becomes a miss.

What cached tokens cost

Cache reads are billed at a steep discount to uncached input, commonly around half price and sometimes far less; check OpenAI's cached-token pricing and Gemini's context caching rates for current numbers. Run the illustrative math with stated assumptions: $3 per million input tokens, a 50% discount on cached tokens, a 40-turn session with a 6,000-token stable prefix. Without cache hits, that prefix re-bills in full every turn, 240,000 tokens or about $0.72 per session. With hits on 39 of 40 calls, the discounted portion saves roughly $0.35 per session, about $3,500 a month at 10,000 sessions. Cached tokens also skip prefill compute, so time-to-first-token typically drops as well.

Why trimming context breaks prompt caching

Now the trap. The natural trim, dropping the oldest turns, edits the prompt right after the tool schemas. The shared cacheable prefix collapses to almost nothing, so every subsequent call re-bills the entire, smaller prompt at full price and re-prefills it from scratch. You removed tokens and increased both cost and latency.

Fewer tokens can cost more money. The cache buys back your stable prefix, and trimming from the front sells it.

The fix is ordering. Put the system prompt, tool schemas, and long-lived instructions first; the summary block next; recent verbatim turns after that; the newest tool output last. Cap tool outputs before they enter history so the past never has to be rewritten, and let the summary block be the only element that changes mid-prompt. Under prefix matching, a change invalidates only the tokens after it, so this layout keeps the expensive fixed head cached no matter how aggressively you compress the tail.

Choosing the Right Rung

With the ledger and the failure modes in hand, escalation becomes signal-driven rather than vibes-driven:

One honest check on that last row: sometimes the right answer is a bigger window. If fixed costs alone consume half a small window, no compression ladder rescues you, and an afternoon of engineering time can cost more than the model upgrade. The ledger tells you which situation you are in; that is its whole job.

An LLM Context Window Management Checklist

Everything above compresses into a short LLM context window management routine you can ship this week:

1. Measure fixed costs once. Count the system prompt and all tool schemas with the provider's tokenizer, then write the numbers into the ledger as constants.
2. Protect the reserve. Budget output tokens, and reasoning tokens on reasoning models, before allocating anything to history. Fail loudly on truncation instead of accepting silence.
3. Assert the variable budget before every call. An over-budget prompt should throw in your code, not degrade in the model.
4. Cap tool outputs at the source. Truncate payloads before they enter history, so the past never needs rewriting.
5. Order stable before volatile. Fixed head, then summary block, then recent turns, newest output last.
6. Split memory into prose plus state. Summaries carry narrative; structured fields carry numbers, IDs, and names verbatim.
7. Watch four metrics. Context utilization per call, cache hit rate, truncation count, and retrieval miss rate at rung 3. Any sustained move is an escalation signal from the table above.
8. Audit summaries against raw logs weekly. Diff what the summary claims against what was actually said; drift caught early is a patch, drift caught late is a rebuild.

Small windows stop being a model problem the moment they become a ledger. Count the fixed costs, defend the reserve, spend the variable budget deliberately, climb the ladder only on signal, and order the prompt so the cache pays you back for everything you refuse to change.