KV Cache Math for Million Token Agent Runs
KV cache math explains why every decoded token reads the whole context, and what eviction and quantization can cut from million-token agent bills today.
Tag
#inference-optimization
Posts tagged with inference-optimization
2 posts
Why Speculative Decoding Pays Nearly 4x on CPUs
Speculative decoding turns idle CPU cores into 4x faster LLM generation. Learn why it works, when gains collapse, and when CPU beats GPU or API.