MoE Serving Cost Math for 6 of 125B Active Parameters
MoE serving cost for a 6-of-125B model is not 6B per token. All 125B stay in VRAM, so run the builder math on residency, routing, and break-even.
Tag
#llm-serving
Posts tagged with llm-serving
2 posts
Disaggregated GPU Inference Hits the KV Cache Wall
Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.