Tag

#llm-serving

Posts tagged with llm-serving

3 posts

Engineering • Sep 05, 2026 •12 min read

KV Cache Math for Million Token Agent Runs

KV cache math explains why every decoded token reads the whole context, and what eviction and quantization can cut from million-token agent bills today.

#kv-cache #attention-sparsity +3

A GPU cluster in a data center represents the hardware baseline behind MoE serving cost, where residency, communication, and utilization set the real invoice for a sparse model.

Engineering • Aug 26, 2026 •12 min read

MoE Serving Cost Math for 6 of 125B Active Parameters

MoE serving cost for a 6-of-125B model is not 6B per token. All 125B stay in VRAM, so run the builder math on residency, routing, and break-even.

#moe #inference-cost +4

Prefill decode disaggregation isolates compute phases across separate GPU pools in a datacenter to optimize large language model serving.

Engineering • Aug 03, 2026 •11 min read

Disaggregated GPU Inference Hits the KV Cache Wall

Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.

#llm-serving #gpu-inference +3