What Multi-Node vLLM Serving Really Costs Per GPU

NVIDIA's demo of GLM-5.3 across two DGX stations reports 607.8 tok/s from vLLM with tensor parallelism, and the number is real. Taken as an engineering claim rather than a marketing one, it is a capacity proof, not an efficiency result. Divide the headline by the accelerators actually doing the work, four per station in the illustrative case in Example 1, and it lands near 76 tok/s per GPU. The design quietly assumes two costs: all-reduce traffic over an inter-station link that runs an order of magnitude or more below NVLink, and coordination overhead across two failure domains. Charged for both, the case for multi-node vLLM serving narrows to a single condition: it wins when memory, not bandwidth, is the binding constraint. When bandwidth or budget binds instead, quantizing onto one box is the better trade. The rest of this piece is the arithmetic, and every step is reproducible against your own hardware quote.

What Multi-Node vLLM Serving Actually Proves

State the measurement precisely: two DGX stations, the zai-org GLM-5.3 mixture-of-experts model, vLLM with tensor parallelism spanning both machines, and an aggregate 607.8 tok/s. Treat that headline as a vendor-reported demo figure, not a bankable benchmark. The accompanying GLM-5.3 vLLM recipe documents the deployment's topology, batch settings, and context length, which makes those parameters checkable; the throughput itself deserves a re-run on your own pair before it anchors a design review.

Now run the division the headline skips.

Example 1, normalize all the way to per-GPU. 607.8 tok/s across two stations is 303.9 tok/s per station. Per-GPU throughput is the only figure comparable across deployments, so divide once more by A, the number of accelerators a station contributes: 303.9/A tok/s per GPU. On an illustrative four-accelerator station that is roughly 76 tok/s per GPU; an eight-accelerator station puts it near 38. Swap in the count from your own hardware quote and the headline is fully normalized.

Whether roughly 76 tok/s per GPU counts as good is a comparison you run yourself: benchmark the same model on the single-node vLLM deployment you already operate, divide that result by its accelerator count, and see which side of the demo your hardware lands on. The buying checklist turns this into a standing rule for every vendor claim. What the normalized figure establishes without any comparison is the capacity verdict: GLM-5.3 runs at usable concurrency on this hardware. It says nothing about whether two boxes beat one well-chosen box per dollar, per watt, or per pager alert, which is the question your purchase order is actually funding.

How vLLM Tensor Parallelism Splits GLM-5.3 Across Two Stations

Tensor parallelism cuts each transformer layer sideways rather than stacking whole layers on separate devices. Attention heads divide across ranks; MLP weight matrices slice into column-parallel and row-parallel pieces. In the standard Megatron-style layout, every layer performs two all-reduce collectives per forward pass, one after attention and one after the MLP, so every generated token pays both before the next layer can start. The vLLM parallelism documentation spells out what a builder inherits: Ray forms and supervises the worker group across machines, NCCL transports the collectives over whatever fabric joins them, and startup flags decide how the model is cut.

Two consequences follow. To the serving process, the pair behaves like one large GPU, which is why the demo works at all. To the network, every layer of every token is a synchronization event that must cross the slowest link in the group. Running vLLM tensor parallelism across two machines is also the most communication-hungry of the available cuts; the same distributed inference stack offers pipeline and expert parallelism for MoE models, which changes this profile. Remember that for later.

All-Reduce Traffic on the Inter-Station Link

This is the all-reduce bandwidth cost no vendor slide prices, and you can compute it before signing anything. Two formulas carry the analysis, both grounded in the Megatron-LM communication analysis of tensor-model parallelism:

• Message size per all-reduce: batch_size × hidden_size × bytes_per_element
• Per-GPU traffic in a ring all-reduce: 2(N-1)/N × message_size, which for a two-station group (N = 2) equals the full message

Example 2, sizing the payload. Use 8192 as a stand-in hidden size (the model card lists GLM-5.3's real value; plug that in), a decode batch of 32 concurrent requests, and BF16 activations at two bytes per element. One all-reduce moves 32 × 8192 × 2 = 512 KiB. Two collectives per layer put roughly 1 MiB on the wire per layer, and a model around 60 layers deep moves about 60 MiB per decode step, per machine. Double the batch and the payload doubles with it.

Example 3, time on the link. Sixty MiB across the 900 GB/s of H100 NVLink takes roughly 70 microseconds. The same volume across 200 Gb/s Ethernet, which is 25 GB/s per direction, takes about 2.5 ms, and 400 Gb/s fabric roughly halves that to 1.3 ms. Collectives on messages this size rarely reach line rate, so several milliseconds is the honest planning number. That is an 18 to 36× gap between in-box and cross-station communication, and because tensor-parallel all-reduces are synchronous, the gap sits on the critical path of every layer of every token.

The contrast matters because the two links are not close. Inside a station, collectives ride a coherent NVLink domain (DGX Station architecture). Between stations, they ride whatever NICs the two machines carry: ConnectX-class Ethernet (DGX Spark hardware shows what that class of link delivers) or the InfiniBand or RoCE fabric you procure. In a decode step that commonly runs tens of milliseconds, a few milliseconds of exposed collective is a visible tax, and it grows in relative terms as you shrink the batch to chase latency.

Coordination Overhead per Token and per Restart

The second cost resists a formula but surfaces in every operational review:

• Per token. Every rank must arrive at each collective, so step time is gated by the slowest participant, which in practice means the slowest link or the busiest machine. Jitter on one station taxes both.
• Per restart. You are bringing up a Ray cluster, initializing NCCL communicators across machines, and distributing weights to both boxes. Cold starts can easily run to minutes, against the seconds a single-box vllm serve restart takes.
• Per failure. The failure domain roughly doubles the moment the second station joins: two power systems, two driver stacks, two network paths. Any single rank crashing takes down the whole logical GPU, and recovery is a cluster re-formation rather than a process restart.

None of it appears in a throughput demo, which is exactly why it goes unbudgeted until the first 3 a.m. page.

Fitting GLM-5.3 on One Station With Quantization

Price memory before you price networking. Let P be GLM-5.3's total parameter count in billions, whatever the model card publishes:

FP8 and 4-bit weight quantization cut the footprint roughly two to four times versus BF16, and vLLM supports both serving paths. That one column frequently settles the one-box question before networking enters the conversation: if quantized weights plus KV cache plus headroom fit a single station's memory, running GLM-5.3 on one DGX station with vLLM is the entire deployment, and the second machine is solving a problem you no longer have.

The caveat is quality. FP8 weight-only quantization is often close enough to BF16 for production serving, while 4-bit can move scores on some task families, and MoE routing adds its own sensitivity. Run your own eval on the quantized checkpoint before committing, not after.

A Decision Rule for Scale Up, Scale Out, or Quantize

Run the table once, on the hardest case, rather than restating it in prose. Take archetype B from the buying checklist: memory binds, so 0.5P GB of 4-bit weights plus KV cache at your real context length still overflows a single station, and the budget covers exactly two machines. The quantized-one-station row fails its own win condition and the larger-node row fails on price, so the two-station row wins by elimination, with a condition attached. The condition is the article's own arithmetic: Example 2 puts roughly 60 MiB on the inter-station link per decode step, and Example 3 prices moving it at 1.3 to 2.5 ms per step on 200 to 400 Gb/s class fabric, synchronous, on the critical path of every layer of every token. If that tax, confirmed by the bus-bandwidth run in the buying checklist, fits your latency budget, the second station is justified. If it does not, no row in this table saves the deployment, and you are shopping for a bigger node or a smaller model.

Every other buyer gets three pass or fail questions, asked in cost order:

1. Does the model fit one station after quantization? 0.5P GB of weights (4-bit) plus KV cache at real context length, under the ceiling with headroom. A pass ends the decision: quantize, serve on one box, and multi-node LLM inference leaves the table.
2. Does the budget reach one larger coherent node? Asked only after a fail. A pass buys in-box NVLink instead of an inter-station link.
3. Is the measured link tax acceptable? Asked only after a double fail. This is the scale-out gate, and it is a measurement, not a vendor slide.

That is the whole answer to when to scale out LLM inference versus quantize, arranged in the order the costs actually arrive.

When Scaling Out Genuinely Wins

This audit is not a verdict against multi-node vLLM serving. It wins in three recognizable situations:

1. The model exceeds any single station even quantized. Past some parameter count you double-fail questions 1 and 2 of the decision rule, and cross-node sharding is the only deployment left.
2. The target is aggregate capacity, not per-request latency. Many concurrent users with loose latency tolerance amortize the all-reduce tax across a large batch, which is precisely the regime the 607.8 tok/s demo exercises.
3. BF16 fidelity is non-negotiable. If your eval genuinely fails at FP8, the memory question reopens, and two stations at BF16 beat one station that cannot hold the model at all.

MoE serving has one more lever. vLLM's expert parallelism shards experts across ranks and replaces part of the per-layer tensor-parallel all-reduce with expert dispatch traffic, while pipeline parallelism moves communication to stage boundaries at layer granularity. Both are typically kinder to a slow inter-station link than pure tensor parallelism, and for a MoE model like GLM-5.3 they belong in the experiment matrix before the topology is final.

What to Measure Before Buying the Second Station

Walk the three budget archetypes through the table first:

• A, it fits quantized. Question 1 of the decision rule passes; stop there and keep the CAPEX and the single failure domain.
• B, memory binds hard. Even 4-bit overflows one station and the budget covers exactly two machines. Scale out, but make the purchase order contingent on the checklist below.
• C, one bigger node is reachable. It costs more per GB of memory, keeps every collective in-box, and for latency-sensitive DGX-station LLM serving it is frequently the better dollars-per-token trade.

Then measure before the money moves:

1. Bus bandwidth, not line rate. Run all_reduce_perf from nccl-tests across the exact two-station link and read the busbw figures per the nccl-tests performance notes. That number, not the switch datasheet, is your all-reduce budget, and one run prices the tax.
2. Per-GPU normalization of every benchmark. Apply it to vendor multi-node vLLM serving claims too: ask for concurrency, context length, and accelerator count, then divide. A throughput number without a denominator is marketing.
3. Quantized footprint including KV cache at your real context length. Weights alone are the optimistic bound.
4. A quality eval on FP8 and 4-bit checkpoints. If FP8 passes cleanly, Archetype A may resolve the entire decision.
5. Cold start and crash recovery on a borrowed pair. Time Ray bring-up, NCCL initialization, and a single-rank kill against a single-box restart; those minutes are part of the second station's true price.

The 607.8 tok/s demo proves the two-station deployment works. Proving the second station earns its rack space is four divisions and one bus-bandwidth run away, and that proof is yours to run.