KV Cache Math for Million Token Agent Runs

Every token a long-context agent generates is billed against the entire conversation behind it. Standard attention must stream the full KV cache out of GPU memory before it can decide which few tokens matter to the next word, so the model's attention may land on one paragraph of tool output while the hardware reads hundreds of thousands of ignored tokens to find that paragraph. Peaked attention paying for flat reads is what actually gates long-context agents: the limit is memory bandwidth, not model capability. The builder's ledger has four lines, and this piece works all of them. The formula for KV cache size per token, a worked ledger for million-token agent runs on current hardware, the research that would change the economics, and a scoreboard of ship-today tactics ranked by how much of the theoretical skip each one captures.

Why every decoded token reads the whole KV cache

Generation happens one token at a time. To produce the next token, the model scores a fresh query vector against the key of every cached token, then combines the cached value vectors in proportion to those scores. The arithmetic is trivial; the reads are not. Every key and every value for every earlier token travels from HBM into the compute units on every single step, whether the softmax ends up weighting them at 0.4 or at one part in a million.

Attention cannot decide to skip a read, because the weight that would justify skipping is computed from the read itself. To learn that token 40,000 is irrelevant to this reply, the hardware first has to fetch token 40,000. Dense attention is honest to a fault this way, and it is the core reason LLM decoding slows down with long context even when profiling shows attention mass concentrated on a small slice of it.

Three questions fall out of this, and the rest of the article answers each. How many bytes is the mandatory read (the arithmetic). What it costs at agent scale (the ledger). How much of it you can avoid today (the scoreboard).

The per-token arithmetic behind decoding cost

The KV cache holds a key and a value vector, for every token, for every attention layer. The KV cache size per token formula is:

bytes per token = 2 x layers x KV heads x head dim x bytes per element

The leading 2 counts K and V. The rest is architecture. Apply it to a 70B-class model with grouped-query attention, using the Llama 3.1 70B model card as the reference config: 80 layers, 8 KV heads, 128 head dimension, fp16 storage at 2 bytes per element.

2 x 80 x 8 x 128 x 2 = 327,680 bytes, roughly 320 KB per token.

Notice what GQA is doing there. A 70B-class model runs 64 query heads but only 8 key-value heads, sharing KV across groups of queries. Without that sharing, the same model would carry about 2.6 MB per token. GQA already cut this bill by 8x at design time, and there is no second free factor like it lying around.

Config	Layers	KV heads	Head dim	KV per token, fp16	KV per token, fp8
8B-class GQA	32	8	128	128 KB	64 KB
70B-class GQA	80	8	128	320 KB	160 KB

Why does this make decode memory-bound? Compare arithmetic to bytes. Attending one cached token costs on the order of one FLOP per byte of K and V moved. An H100-class device delivers on the order of 300 FLOPs of fp16 compute for every byte of HBM bandwidth it can move in the same interval. The operation runs hundreds of times below the machine's balance point, so the tensor cores idle while memory streams. This is memory-bound decoding, and it is why every tactic in this article is ultimately a claim about bytes.

The full ledger at one million tokens

Extend the 70B-class row to real agent scale. One million tokens of context, a routine working set for a coding agent carrying repository history and a long tool log, holds on the order of 320 GB of KV cache. The fp16 weights add about 140 GB. An 80 GB H100 holds neither number, so the conversation needs on the order of 6 to 8 GPUs before it physically fits, before any headroom for concurrent requests.

Now the bandwidth line. A single H100-class GPU moves roughly 3.35 TB/s from HBM. Streaming 320 GB takes about 95 ms, call it a tenth of a second, per decoded token, before weight reads, kernel launches, or cross-GPU communication. In the theoretical best case, that one-GPU budget buys about 10 tokens per second at this context length, and a 1,000-token reply carries roughly 95 seconds of pure KV streaming in its critical path.

Sharding helps and hides. Split the cache across 8 GPUs and each streams about 40 GB in parallel, dropping the KV read floor toward 12 ms per token, which is how million-token demos reach interactive latency at all. The bytes do not disappear, though. The fleet still moves 320 GB per token, GPU count scales with context, and per-token cost keeps growing as the conversation appends. Prefix caching rescues prefill by letting later turns reuse cached prompt blocks, but generated tokens still pay full freight on every step.

Continuous batching has the same blind spot. It amortizes weight reads across concurrent sequences, which is why short-prompt serving is cheap. It cannot amortize KV traffic, because each sequence's cache is private and grows with that sequence's own context. One million-token agent in the batch saturates bandwidth and devours capacity that would otherwise hold dozens of short requests. Long agent conversations do not escape batching economics, they poison them.

What attention-control research would change

The clean fix is to make reads proportional to what attention actually uses. If a layer puts nearly all its mass on a small fraction of context, the rest should never leave HBM. A run of papers from 2024 and 2025 attacks exactly this, and their shared empirical finding is that attention mass in long-context tasks concentrates heavily on a small fraction of tokens.

The DuoAttention paper profiles attention heads and splits them into retrieval heads, which genuinely need full context, and streaming heads, which only need recent tokens, then serves that fixed policy. The Native Sparse Attention paper from DeepSeek trains sparse attention directly with hardware-aligned block selection and reports multi-fold decoding speedups at 64k contexts and beyond in its own benchmarks. MoBA routes each query to top-k blocks through a learned router. SeerAttention trains a small gate, distilled during finetuning, that marks which KV blocks a head may skip. Across this line of work, reported cache reductions range from several-fold to over an order of magnitude at comparable benchmark quality.

Then there is the zero-shot end of the spectrum. A recent preprint on declarative attention prompts models to declare global, focus, or local attention modes in their chain of thought, and reports cutting attended tokens during decoding by roughly a third to a half on two off-the-shelf models, with accuracy dropping only in the one-to-three point range. No training, real if modest savings, and a neat existence proof that models hold some self-knowledge about what they need to read.

The honest split: NSA and MoBA require the model to be trained that way, and the gates require a finetune, so none of it drops into your vLLM deployment this quarter. What they establish is direction. Serving stacks built on uniform paged KV blocks can express block-granular skipping without new hardware, so if trained sparse attention lands in open weights, the engine side is closer to ready than the model side.

KV cache eviction methods you can run today

While the training-based work matures, deployed systems approximate the skip by throwing cache away. The SnapKV versus H2O versus StreamingLLM choice reduces to what each keeps, and all three fail agents in the same place.

StreamingLLM and attention sinks

Early tokens gather outsized attention mass, so StreamingLLM keeps a handful of these sink tokens plus a sliding recent window and drops everything between. Cache size becomes constant regardless of stream length, the strongest capacity guarantee of any method here. The cost is total: any question about the evicted middle is unanswerable by construction. The right tool for eternal chat, the wrong tool for an agent whose value is the middle.

H2O and heavy hitters

H2O keeps the tokens with the highest accumulated attention scores alongside the recent window, and its authors report retaining benchmark quality at a small fraction of the cache in their settings. The failure is that accumulated attention is a lagging proxy. A config value mentioned once at turn 3 and never attended again has a near-zero score until the moment the user asks about it, at which point the tokens are already gone. KV cache eviction by historical statistics is a bet that the past predicts the query.

SnapKV and observation windows

The SnapKV paper compresses the prompt once, selecting tokens by where attention lands during an observation window at the end of the prompt, and reports several-fold cache reductions in its evaluation settings. That selection works when the question is visible during compression, which is true for single-shot document QA and false for agent conversations, where the question arrives many turns after compression runs. Selection is also irreversible for the session. Agent traffic violates the method's core assumption.

KV cache quantization and prompt compression

Eviction changes how many tokens you carry. Quantization changes how much each one weighs, and it is the least dangerous lever in the drawer.

Shrinking bytes with fp8 and int4

KV cache quantization to fp8 or int8 halves bytes per token, and int4 quarters them. Read time and capacity improve by the same factor: the 70B-class row drops from 320 KB to 160 KB at 8-bit, halving both the bandwidth cost per decoded token and the GPU count needed to hold the same context. Both major engines support it. To reduce KV cache size in vLLM, set the cache dtype to fp8, a one-line change the vLLM quantized KV cache docs cover in full. TensorRT-LLM exposes the same lever through its fp8 quantization tuning guide.

Quality calibration: public evaluations generally show modest impact at 8-bit, with fp8 KV cache quality trade-offs surfacing mainly in tasks sensitive to small logit shifts, such as long-retrieval precision and numeric answers. At 4-bit, degradation sharpens on some tasks. The sensible deployment order is fp8 first for everything, then int4 only for capacity-desperate cases after you measure recall on your own traffic.

Shortening text with prompt compression

The third lever attacks the input instead of the representation. LLMLingua-style prompt compression uses a small model to delete low-information tokens before they are ever cached, with its authors reporting double-digit compression ratios at modest benchmark loss in favorable settings. It is the only tactic that also cuts prefill compute, and it composes cleanly with quantization. Its failure mode rhymes with eviction's: deleted tokens are chosen by a proxy model's estimate of importance, and agent conversations live on rare specifics. For long agent conversations, compressing at session boundaries, such as summarizing stale tool output, is safer than token-level surgery on live context.

Choosing a tactic and measuring your own sparsity

No deployable tactic makes reads proportional to actual attention usage. Quantization shrinks bytes, eviction drops tokens on a heuristic, prompt compression shortens the text. Each captures a slice of the win a model that controls its own attention could claim. The scoreboard for your KV cache optimization plan:

Tactic	Attacks	Approximate capture	Agent risk
fp8 or int8 KV quantization	bytes per token	about half, always on	low at 8-bit, numeric edge cases
int4 KV quantization	bytes per token	about three-quarters	sharper loss on some tasks
StreamingLLM window	token count	bounded by window size	cannot see evicted history
H2O or SnapKV eviction	token count	several-fold in paper settings	drops rare retrieval details
Prompt compression	input length	bounded by compression ratio	lossy on specifics
Trained sparse attention	the read set itself	the full prize	needs trained weights

Map it to workload. Latency-bound single agents gain most from fp8 immediately, a guaranteed 2x on the bandwidth line with the least quality risk. Capacity-bound fleets serving many concurrent conversations should stack fp8 with eviction, accepting the retrieval risk consciously. Traffic that answers questions about old details, such as audit logs, IDs, and exact numbers, belongs on quantization plus conservative windows, with compression reserved for stale regions.

Before committing, measure. The research above consistently finds attention sparsity, most mass on a small fraction of context, but your traffic's number is the one that pays. Replay a sample of production conversations through the model offline with attention weights captured, and log per layer what share of mass sits in the top 1 percent of KV blocks. Concentrated mass means eviction headroom. Flat mass means your traffic really does read broadly, and the safe money is bytes, not surgery. No major engine exposes this out of the box yet, which is itself a signal about where the field expects the next fight.

The endgame is architectural: a model that skips its own reads. Until weights ship that way, the ledger tells you which tax you are paying, and the audit tells you which discount your traffic can honestly claim.