Disaggregated GPU Inference Hits the KV Cache Wall

The pitch for disaggregated GPU inference is clean and seductive. Split the prefill and decode phases of large language model serving across separate GPU pools, tune each pool for its distinct workload, and watch throughput climb. The production reality is messier. Every request that completes prefill on one node must hand off its full KV cache to a decode node elsewhere in the datacenter, and that handoff moves multi-gigabyte tensors across a network never designed for this traffic pattern. Splitting the work is the easy part. Preventing the network from drowning in cache data is what determines whether the architecture delivers or stalls.

The Compute and Memory Mismatch in Continuous Batching

A prefill GPU capped at batch size 8 when the same silicon could sustain batch 32 wastes most of its compute on every step. That gap is the opening cost of co-locating prefill and decode in one GPU pool, and it compounds under load until the throughput loss overtakes any scheduling gains. Resource phase analysis confirms the structural tension: prefill is compute bound, decode is memory bandwidth bound, and forcing both onto the same silicon starves the math units while hoarding HBM.

The throughput math. A standalone prefill pool running batch 32 keeps its tensor cores saturated on dense matrix multiplications. In a co-located pool, decode sequences already occupying HBM force the scheduler to cap prefill at roughly batch 8, a number chosen for illustration that varies with model size and sequence mix. At batch 8, the GPU's math units sit underfed. Over a sustained prefill spike, that 60 to 75 percent throughput gap translates to tokens per second the cluster will never recover. The exact number depends on architecture and batch composition, but the directionality holds across configurations: co-locating prefill with decode leaves compute stranded at exactly the moments when request volume is highest.

Eviction recompute cost. When HBM is full and a new prefill arrives, the scheduler either queues the request or evicts a running decode sequence. Batching fragmentation patterns make the tradeoff concrete. A decode sequence at 500 generated tokens carries roughly 160 MB of KV cache state (500 tokens at 320 KB per token). Discarding it forces the prefill GPU to reprocess the full prompt plus those 500 tokens. On a 70B model, that recomputation costs roughly 0.5 to 1 second of GPU time per evicted sequence, an illustrative figure that scales with model size. During a traffic spike triggering dozens of simultaneous evictions, those recompute seconds stack into a throughput hole that grows faster than the spike itself. Co-location acts as a compounding tax under exactly the load conditions where throughput matters most.

This is the penalty that disaggregation targets. Physically separating prefill and decode into isolated pools eliminates the batch-size ceiling, the eviction churn, and the SM-slot contention. But the split introduces a new cost: every completed prefill must transfer its full KV cache across the datacenter network to a decode node. Whether that transfer is cheaper than the co-location tax it replaces depends entirely on the physical path the cache data takes, and that question is where production deployments diverge from the academic benchmarks.

How Disaggregated GPU Inference Splits the Workload

The core idea of disaggregation is sound: physically separate prefill and decode into isolated GPU pools, each tuned for its workload. The seminal papers get the architectural split right but systematically underweight the cost of the handoff between pools. DistServe, Splitwise transfer pool design, and Mooncake all acknowledge that KV cache must traverse the network during phase transition, but they treat that transfer as a secondary engineering detail. In production, it is the primary determinant of whether disaggregation pays off at all.

DistServe disaggregation scheduler formalizes the split with a global scheduler that routes incoming requests to prefill workers and migrates completed prefills to decode workers. Splitwise extends the concept with a dedicated KV cache transfer pool, a set of nodes or a shared memory tier that buffers cache data during handoff, decoupling the two pools from strict synchronization. Both designs are elegant in isolation. Prefill nodes batch aggressively without decode interference. Decode nodes pack more concurrent sequences because they control their own memory budget without prefill spikes. On paper, GPU utilization climbs across both pools.

The gap is in evaluation methodology. These papers benchmark under uniform network assumptions that flatten the bandwidth hierarchy of real datacenter fabrics, a production bottleneck the following sections quantify in detail.

Production deployments face a fundamentally harder problem than benchmark conditions. Real traffic is bursty and asymmetric, prefill and decode pools are rarely balanced, and the network fabric must also carry the request and response traffic that existed before disaggregation was introduced. The papers built the right architecture for the wrong network model, and closing that gap is the work that determines whether disaggregated GPU inference delivers in practice.

Calculating the KV Cache Transfer Tax

The numbers get uncomfortable quickly. KV cache memory footprint for a single request is not a few megabytes of metadata. It is the full attention state for every transformer layer, and it scales linearly with context length.

For a 70-billion-parameter model, analytical models of disaggregated serving report approximately 2.6 GB of KV cache per request at production context lengths. The formula behind that figure is straightforward. Each layer stores a separate key tensor and a separate value tensor for every token, so the total storage doubles to account for both. Working through the multiplication explicitly:

Per-token KV cache size = layers × KV heads × head dimension × bytes per element × 2 (key + value)

Example: 80 layers × 8 KV heads × 128 head dim × 2 bytes × 2 (key + value) = 327,680 bytes ≈ 320 KB per token

Grouped-query attention uses fewer KV heads than total query heads, which reduces the per-token footprint relative to multi-head attention. A request with an 8,000-token context generates over 2.5 GB of tensor data that must move from the prefill GPU to the decode GPU.

Aggregate transfer demand. The per-request math scales linearly with request rate. Analytical models of disaggregated serving report sustained transfer demand exceeding 100 GB/s. The extrapolation is direct:

100 requests/s × 2.6 GB per request = 260 GB/s of KV cache traffic

vs. the 50 GB/s ceiling of a single 400-gigabit InfiniBand link

That traffic is roughly five times what one link can carry, and it flows in addition to the normal request and response traffic the network already handles. Links never engineered for synchronized, multi-gigabyte cache bursts now face exactly that pattern.

vAttention paged attention and similar virtualized KV cache layouts improve allocation efficiency within a single GPU. They reduce fragmentation and enable dynamic capacity management. They do nothing to reduce the raw bytes that must traverse the network between physically separate GPU pools. The transfer tax is a physical layer problem, not a software allocation problem.

Why Naive Disaggregation Fails on Standard Networks

The core issue is bandwidth hierarchy. Bandwidth between two GPUs varies by roughly 72x depending on their physical relationship:

These figures represent theoretical peak bandwidths for current-generation hardware. Sustained throughput in production is lower due to protocol overhead, congestion from co-tenant workloads, and the bursty synchronization patterns that cache transfers create.

RDMA over Converged Ethernet provides host-to-host remote memory access without CPU involvement, but it still operates at network-layer bandwidth, not intra-node interconnect bandwidth. NVIDIA GPUDirect RDMA reduces the software overhead of transfers by enabling direct memory access between GPUs across the network, but the physical link speed remains the ceiling.

The problem with DistServe, Splitwise, Mooncake, and similar systems is that they treat the network as uniform. A prefill node sends its KV cache to whatever decode node the scheduler assigns, regardless of whether that node shares an NVLink domain, sits on the same InfiniBand rail, or sits three switch hops away. When a 2.6 GB transfer must traverse a 50 GB/s InfiniBand link instead of a 900 GB/s NVLink path, the transfer alone consumes over 50 milliseconds. At the scale of hundreds of concurrent transfers, the network saturates and the throughput gains from disaggregation evaporate.

Network topology analysis of standard spine-leaf Ethernet fabrics confirms that these topologies are designed for east-west traffic patterns that average out over time. KV cache transfers create synchronized, bursty, many-to-many traffic that concentrates bandwidth demand on specific links, creating congestion hotspots the topology was never engineered to absorb.

Topology-Aware Routing as the Missing Infrastructure Layer

The solution is routing that understands the physical interconnect hierarchy and makes placement decisions based on it, and analytical projections suggest topology-aware placement can reduce transfer latency by 3 to 18x over uniform RDMA.

Topology-aware GPU routing discovers the interconnect topology at cluster initialization and maintains a graph of available bandwidth between every pair of GPU domains. When a prefill request completes, the transfer orchestrator selects a decode node based on three criteria working together.

NVLink Proximity

The orchestrator prefers decode nodes within the same NVLink domain or the same PCIe switch hierarchy as the prefill node. This keeps cache transfers on the fastest available path and avoids consuming scarce InfiniBand or Ethernet bandwidth.

Pipelined Layer-by-Layer Transfer

Rather than waiting for prefill to complete fully and then transferring the entire KV cache as a monolithic block, the orchestrator pipelines the transfer one transformer layer at a time. As each layer finishes computing during prefill, its KV cache slice immediately begins transmitting to the decode node. Analytical projections suggest this technique can hide 60 to 85 percent of the transfer latency behind ongoing computation, effectively folding the network cost into work the GPU is already doing.

Bandwidth-Aware Load Balancing

When no decode node is available within the optimal interconnect tier, the orchestrator falls back to the next tier (InfiniBand, then Ethernet) but tracks the aggregate bandwidth budget to avoid saturating shared links and triggering congestion collapse.

Distributed LLM serving architectures that separate compute from memory tiers are emerging, but production-grade topology-aware routing remains largely the domain of custom infrastructure built by the largest serving operators. In most known production deployments, teams likely rely on uniform RDMA and accept the performance penalty without fully understanding its source.

Implementation Architecture for Disaggregated GPU Inference

For engineering teams evaluating whether disaggregation fits their infrastructure, the decision comes down to three factors: model size, context length distribution, and network topology.

Decision Matrix

KV Cache Compression Before Transfer

KV cache quantization and compression is a necessary tactical step, not an optional optimization. Quantizing the cache from FP16 to INT8 before network transfer halves the data volume with minimal quality impact for most workloads. More aggressive techniques, including eviction-based sparsification that selectively drops low-importance attention entries, can reduce transfer size further while keeping generation quality within acceptable bounds. The compression happens on the prefill GPU before the transfer begins, trading a small amount of local compute for a substantial reduction in network load.

Hardware Tier Considerations

CXL 3.0 memory expanders offer a shared overflow tier for cache data between GPU HBM and network-attached storage. Analytical projections suggest CXL-based pooling could provide roughly 6x the memory capacity of local HBM, with roughly 86x lower access latency than NVMe. That makes CXL a compelling cache tier if the hardware matures.

Deployment status. The hardware remains early in its deployment cycle and is not yet widely available in GPU cloud instances. For Mixture-of-Experts models, the topology-aware orchestrator must also co-optimize expert dispatch routing with KV cache locality, since both expert weights and cache data benefit from NVLink-domain placement.

The bottom line. Disaggregated GPU inference works when the infrastructure respects the physics of data movement. Splitting prefill and decode across separate pools only delivers throughput gains if the network path between those pools can sustain multi-gigabyte cache transfers at low latency. Topology-aware routing is a prerequisite that determines whether the architecture functions at all, not an optimization layered on top of it. Teams that deploy disaggregation on standard Ethernet fabrics without routing intelligence will see GPU utilization charts that look impressive in isolation and end-to-end latency that tells a different story.