LLM Prompt Caching Can Cut Input Costs Up to 90%

Run an agent for twenty turns and you will pay for the same system prompt and tool schemas twenty times. Hosted APIs are stateless between requests, so every call re-processes your entire prefix and bills it at the full input rate. LLM prompt caching exists to break that cycle: providers keep the computed state of a matching prefix warm and serve repeat occurrences at 50 to 90 percent off. The discount is real, but it is conditional on decisions you make before the loop ever runs. Where your dynamic tokens sit, whether your prefix clears the minimum cacheable size, and whether your turns arrive faster than the cache expires together decide whether you save roughly 85 percent or quietly pay 25 percent more than you did without caching.

This guide assembles the rules for OpenAI, Anthropic, Gemini, and Amazon Bedrock in one place, then runs the loop math including the parts vendors put in footnotes: write surcharges, mid-conversation misses, and the token category no discount touches.

How LLM prompt caching changes your bill

Underneath the API, a transformer computes a key-value tensor for every input token. Caching stores those tensors server-side and reuses them when a later request starts with the identical prefix, which is why a cache hit also cuts time-to-first-token. For a plain-language walkthrough of that mechanic, see this KV cache billing explainer.

The part most engineers miss is that a cache is matched against an exact byte-level prefix. Only the tokens before the first difference between two requests are served from cache. The moment your prompt diverges from what was cached, everything after that point is processed and billed as fresh input, even if the provider still caches the part that matched.

That mechanic produces three bill effects, and you need all three to predict savings:

• Read discount. Cached tokens bill far below the standard input rate, up to 90% on some providers and models.
• Write premium. Anthropic and Amazon Bedrock charge extra, on the order of 25 percent above base input price, for the initial cache write. OpenAI charges no premium.
• An output-token wall. LLM prompt caching only discounts input. Output tokens, often the largest line on an agent bill, are untouched no matter how good your hit rate is.

How OpenAI, Anthropic, Gemini, and Bedrock cache differently

The four providers agree on the prefix-matching mechanic and disagree on almost everything operational. This is why caching is an integration decision, not a checkbox.

OpenAI prompt caching is the zero-integration case: prompts above 1,024 tokens are cached automatically with no code change, per OpenAI's caching guide. The trade-off is control. You cannot place breakpoints, so your only lever is keeping the head of the prompt byte-stable.

Anthropic prompt caching is the opposite bet. You mark stable blocks with cache_control, which lets you cache the system prompt, tools, and progressively the conversation history as separate segments, at the documented costs and TTL options in Anthropic's caching documentation. More control, more integration surface.

Amazon Bedrock prompt caching uses cachePoint markers in the Converse API, scoped to your account and Region, with the thresholds and TTL tiers detailed in the Bedrock prompt caching guide. One operational wrinkle worth knowing: cross-Region inference profiles can occasionally increase cache write frequency because requests route across Regions.

Gemini context caching is a different shape entirely. You create a cached-content object from your stable prompt, set its TTL yourself, and pay a small storage fee per minute while it lives. That model favors long-lived reference material over fast agent loops.

Agent loop savings math, provider by provider

Anthropic launched caching with agent workloads explicitly in mind, per Anthropic's caching announcement. Here is the arithmetic those workloads actually see.

Setup: an agent whose fixed prefix, meaning system prompt plus tool schemas for a few dozen tools, totals 10,000 tokens, re-sent on every one of 20 turns. Define one cost unit as one full-price input token. Ignore the growing history tail and output for now; they bill normally either way.

These use illustrative list-rate discounts; real rates vary by model and change over time, so re-run this with current pricing before you commit. The shape of the result is what matters, and two properties hold across providers:

1. Savings scale with conversation length. At 100 turns, the Anthropic-style column becomes 12,500 + 99,000 = 111,500 units against 1,000,000 uncached, roughly 89 percent. The write amortizes; the reads are nearly free.
2. The write surcharge can flip the sign. If every turn misses, the surcharge-bearing providers charge 20 × 12,500 = 250,000 units, which is 25 percent more than not caching at all.

The failure case is not "no savings." It is paying a premium for a cache you never read.

This is why short loops with few repeat turns can net out worse: a two-turn interaction saves little, and a single-turn request pays the surcharge for nothing.

Prompt ordering and cache breakpoints

Order every prompt by stability

Exact-prefix matching turns your prompt layout into a caching decision. Sort content from most stable to least stable: system prompt, tool schemas, long-lived retrieved documents, conversation history, and finally this turn's dynamic payload. A timestamp, request ID, or user ID placed above the system prompt invalidates the cache on every single turn, because the first differing token ends reuse for everything after it.

Where each provider puts the breakpoint

On OpenAI there is no marker; your layout is the marker, so the discipline is keeping the head byte-identical, including tool-schema key order if your SDK re-serializes JSON. On Anthropic you place cache_control after each stable block, which also lets you cache the accumulating history incrementally as the conversation grows. On Bedrock the cachePoint goes at block boundaries, for example as the last element of the tools array, and Claude models there support a single-checkpoint mode that automatically checks prefixes across prior content blocks. On Gemini the stable prefix lives in the cached-content object itself and each request sends only the delta.

Prompt caching TTL versus turn cadence

Every default TTL is short. OpenAI, Anthropic, and Bedrock all work on the order of five minutes, typically refreshed each time the cache is hit. Gemini is the outlier, letting you set longer TTLs and billing storage per minute.

Map that onto real agent patterns:

• Autonomous loops with seconds between calls never notice the TTL; each hit refreshes it.
• Human-in-the-loop agents with minutes between turns straddle the boundary.
• Batch or cron-driven jobs with hours between runs find the cache dead every time.

Now price a realistic miss pattern: 20 turns with a 10-minute pause every five turns gives 5 writes and 15 reads. On Anthropic-style pricing that is 5 × 12,500 + 15 × 1,000 = 77,500 units, still 61 percent below the uncached 200,000. Occasional misses dent the savings; only systematic misses, where nearly every turn rewrites, push you past break-even into paying extra. That is the precise meaning of "why prompt caching misses mid conversation": TTL expiry converts cheap reads into surcharged writes. For slow-cadence workloads, the 1-hour TTL options (at 2x write cost) or a Gemini cached-content object with a TTL matched to your session length are the fixes worth pricing.

Failure modes that quietly kill the discount

Instrument from turn one. Every provider's usage response reports cache read and write token counts, and those two numbers are the only ground truth about whether your architecture is working.

Design rules, and the limits of the discount

Before writing the loop, qualify the workload:

1. Measure your fixed prefix in tokens. Below roughly 1,024 to 2,048 it may not cache at all.
2. Order the prompt by stability and hash the rendered prefix in CI to catch mutations.
3. Measure real turn gaps and pick a TTL regime that outlasts them, including the 1-hour or explicit-TTL options for slow cadences.
4. Confirm read and write counts in usage metrics on your first test run.
5. Choose provider partly on TTL fit, not just per-token price, because a 90 percent read discount you never hit is worth zero.

Does this make context stuffing affordable again? Partly, and that is a genuine architectural shift. A 50,000-token reference document re-read across hundreds of turns becomes cheap on reads, which changes the old trade-off between always summarizing and including the whole source. But the ceiling is firm: output tokens are untouched, first-turn writes still bill, and unstable or sub-minimum prefixes get nothing. LLM prompt caching rewards loops designed around prefix stability and turn cadence, and quietly taxes loops that ignore them. Treat it as a constraint you architect for, and the discount follows; treat it as a billing toggle, and the surcharge follows instead.