KV Cache Math for Million Token Agent Runs
KV cache math explains why every decoded token reads the whole context, and what eviction and quantization can cut from million-token agent bills today.
Tag
#kv-cache
Posts tagged with kv-cache
2 posts
Disaggregated GPU Inference Hits the KV Cache Wall
Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.