MoE Serving Cost Math for 6 of 125B Active Parameters

Alibaba's Qwen team has posted a Qwen3.8-Flash-Next preview reporting a sparse mixture-of-experts model that activates 6 of 125B parameters per token, trained at one-ninth the cost of the larger rivals it reportedly beats. Both numbers are release claims at the time of writing, and neither is a serving number. MoE serving cost is set by quantities the spec sheet never lists: every one of the 125B parameters has to sit in accelerator memory, expert routing adds all-to-all communication that dense serving never pays, and batch diversity decides how much of the expert pool each forward pass actually reads.

Active parameters price the per-token arithmetic. Residency, communication, and utilization price the invoice. What follows is the math a builder runs before committing GPUs to this release, or to any total-plus-active spec that comes after it.

What 6 of 125B Parameters Actually Means

A "6 active of 125B parameters" spec is really two specs wearing one number.

The compute half. Each token passes through about 6B parameters of multiply-accumulate work, because the router hands every token a small subset of experts in each routed layer. Per token, per layer, this model does dense-6B-class arithmetic. That is the half the release headline prices, and it is real.

The memory half. The router can send any token to any expert, so every expert must be addressable inside a single forward pass. Nothing loads on demand: routing decisions land per token per layer in microseconds, and paging gigabytes of expert weights over PCIe mid-pass would blow the latency budget of the entire step. Total parameters, not active ones, set the memory floor.

This is where active parameters vs total parameters stops being trivia. In mainstream open sparse MoE designs, only the feed-forward blocks are routed. Attention layers, embeddings, and any shared experts stay dense and run for every token, which is why the Qwen3 MoE writeup describes the same pattern at 235B total and 22B active. Sparsity lives in the FFN stack, and the KV cache, which often dominates long-context serving, scales with the dense attention design rather than with the active ratio.

The Residency Bill Every Parameter Pays

MoE VRAM requirements start with one rule: weight bytes ≈ total parameters × bytes per parameter. The active count does not appear in the formula.

Component	FP8 (1 byte)	BF16 (2 bytes)
Weights, 125B parameters	~125 GB	~250 GB
KV cache	Scales with concurrency × context, set by the attention design	Same bytes either way
Activations and framework overhead	A few GB, grows with batch	A few GB, grows with batch

So how much VRAM does a 125B MoE model need? Roughly 125 GB at FP8 before KV cache, roughly 250 GB at BF16. A single 80 GB card cannot hold the weights even at FP8, and that one line item forces the deployment shape:

Setup	Capacity	FP8 (125 GB weights)	BF16 (250 GB weights)
1 × 80 GB	80 GB	No	No
2 × 80 GB	160 GB	Weights only, thin KV room	No
4 × 80 GB	320 GB	Comfortable	Minimum viable
2 × 141 GB	282 GB	Comfortable	Tight
4 × 141 GB	564 GB	Long-context headroom	Comfortable

Card capacities and HBM bandwidth figures come straight from the NVIDIA HPC GPU datasheet. Verify the exact SKU, since the 80 GB and 141 GB classes differ in bandwidth as well as memory.

Aggressive 4-bit quantization would bring weights to roughly 63 GB and put one 80 GB card back in range, at some cost in quality and in how gracefully the router behaves near decision boundaries. For a 125B-class deployment, FP8 vs BF16 is the practical fork: halve residency and accept a little numerical margin, or double it for the comfortable path. Either way, budget 20 to 30 percent of capacity for KV cache once you serve real concurrency with real context lengths.

The All-to-All Routing Tax

To spread 125 GB of experts across multiple GPUs you shard the expert pool, and that is where mixture of experts inference picks up a cost dense models never pay.

With expert parallelism, each routed layer runs two exchanges per step. First a dispatch all-to-all: every token travels to the GPUs holding the experts its router chose. Then, after those experts compute, a combine all-to-all carries the outputs back. Two network round trips per layer, per step, with message sizes that shift token by token as routing changes.

Dense models parallelized with tensor or pipeline parallelism also move data, but their transfers (all-reduce, all-gather) are fixed-size and sit on the most optimized paths in collective libraries. MoE all-to-alls are data-dependent, which makes them harder to schedule, harder to overlap with compute, and more exposed to stragglers. This expert parallelism overhead grows exactly when you least want it:

• Small batches. The per-exchange latency floor does not amortize across many tokens, so all-to-all communication MoE inference overhead can dominate step time for latency-sensitive, low-concurrency traffic.
• High expert-parallel degrees. More peers per exchange means more, smaller messages and more synchronization points.
• Cross-node expert placement. Expert shards that straddle an InfiniBand boundary instead of NVLink commonly see a several-fold jump in effective transfer latency.

The mitigations are known: overlap dispatch with shared-expert compute, fuse the routing kernels, keep the expert-parallel degree as low as the memory math allows, prefer larger batches. Modern serving stacks implement several. They reduce the tax; they do not repeal it.

Batch Diversity Multiplies Effective Weight Reads

Per-token compute stays 6B-class no matter what. Weight reads do not, and this is the subtlety most spec-sheet comparisons miss. MoE throughput and batching are coupled through the union of experts the batch touches.

At batch size 1, a token activates its k selected experts per layer and reads only those weights. Add tokens, and different tokens pick different experts. The expected share of the expert pool read per layer follows 1 − (1 − k/E)^B, with E experts and batch size B. For an illustrative layer with 128 experts and top-8 routing under uniform load:

Batch size	Share of expert pool read per layer
1	~6%
8	~40%
16	~64%
64	~98%

A batch of 16 diverse tokens reads roughly two-thirds of the expert pool to produce 16 tokens. Step-level weight traffic approaches dense-model traffic as the batch diversifies, and the roughly 20× per-token read advantage you get at batch 1 narrows toward parity with a same-total-size dense model. What survives at every batch size is the FLOP advantage, about 20× fewer multiplies per token, which pays off in prefill and in very large batches where serving becomes compute-bound. A fast MoE inference study measured this shape years ago: sparse models pull well ahead of compute-matched dense ones at small batch, and the gap narrows as batch grows.

There is also a ceiling worth knowing. Once batches are diverse enough to sweep the pool, each decode step must move the full expert weights once. Four H100-class cards deliver on the order of 12 TB/s of aggregate HBM bandwidth, so a 125 GB FP8 pool takes roughly 10 ms per sweep and a diverse 64-token step tops out around 6,000 tokens per second before communication costs. Real deployments land below that.

Expert imbalance sets your real throughput

The table above assumes uniform routing. Real token streams are not uniform: hot experts form, and the step waits for the slowest expert replica, so MoE expert imbalance throughput losses appear as stragglers you never budgeted. The DeepSeek-V3 technical report introduces auxiliary-loss-free load balancing precisely because imbalance wastes capacity; it tames the effect at serving time too, but does not eliminate it. Discount uniform-routing projections accordingly.

The Training Bill Is Not the MoE Serving Cost

The one-ninth figure prices training compute, presumably against the rivals the release benchmarks. That is legitimate, because training cost tracks active parameters: training FLOPs are roughly active params × tokens × a forward-and-backward factor, so sparsity is a direct discount on that bill.

Serving is priced by three quantities, none of which scale with the active count. Residency bills total parameters for every hour the weights sit in VRAM. Utilization bills full rate for idle GPUs. Communication bills per all-to-all per step. The one-ninth ratio transfers to your invoice only if your training bill were your serving bill, which it is not.

None of this is new. Conditional computation has been pitched since the Switch Transformers paper as a way to grow capacity at fixed FLOPs, and that framing is a training-compute claim that release notes have been borrowing ever since. The builder's question is different: what do I pay per million tokens at my batch profile, on my hardware, at my utilization? Training arithmetic cannot answer it.

A Spec-to-Deployment Worksheet

Run this on any total-plus-active spec. It fits on one page.

1. Pick precision and compute residency. Weight bytes = total params × bytes per param. FP8 halves the bill versus BF16; decide deliberately.
2. Add the KV budget. KV bytes ≈ 2 × layers × KV heads × head dim × bytes per element × concurrent tokens. At long context this can rival the weight line, so estimate it with your real concurrency, not a default.
3. Get the minimum GPU count. Divide residency plus KV by per-card capacity, then sanity-check against the fit table above.
4. Profile the routing tax. Small median batch plus a high expert-parallel degree is the communication-bound corner. Large diverse batches on few GPUs push you toward the pool-sweep ceiling.
5. Price it per million tokens:

cost_per_million = (gpus × price_per_gpu_hour) / (tokens_per_second × 3600) × 1_000_000

Worked example for this spec at FP8 on 4 × 80 GB cards at an assumed $2.50 per GPU-hour (current cloud GPU pricing comparisons commonly put H100-class rates near that band, provider and commitment depending): at 2,000 sustained tokens per second the fleet costs $1.39 per million; at 4,000 it costs $0.69. Same hardware, same spec, half the price. Qwen3.8-Flash-Next serving cost is not a number, it is a function of your utilization, and utilization is the one variable the spec sheet cannot fix for you.

When Sparse MoE Wins and Loses

Self-host MoE vs hosted API break-even usually gets argued as generic cloud arithmetic: list prices, committed-use discounts, ops overhead. For a sparse model the deciding fact is blunter: the residency floor does not care about volume. Four GPUs at $2.50 per hour burn about $7,300 a month at zero tokens, and the same $7,300 whether traffic is heavy or absent. A dense checkpoint of comparable quality, call it 30B-class at FP8, is roughly 30 GB of weights, so it idles on a single 80 GB card at a quarter of that floor. That asymmetry, not provider pricing, is the MoE-specific half of the decision.

Then the API math. Against an API charging on the order of $1 per million blended tokens (the DeepSeek API pricing guide tracks rates among the cheapest at frontier quality), the fleet ties out near 7.3 billion tokens per month, roughly 2,800 tokens per second sustained around the clock. That clause is the trap. Product traffic is peaky, a fleet sized for peak runs at a fraction of capacity off-peak, and the price per million scales with the shortfall:

Effective utilization	Tokens per month	Break-even $ per million
100 percent	7.3B	$1.00, ties the cheapest APIs
50 percent	~3.7B	$2.00
20 percent	~1.5B	$5.00, five times the API price

At 20 percent effective utilization the break-even has quintupled to $5, and the API wins on price while you still carry operations. The 40 to 50 percent band only turns winnable against providers charging $2 and up per million, which is why the call rides on traffic shape and the residency floor, not list prices:

Your situation	Call
Steady, high, predictable volume	Self-host once sustained utilization clears 40 to 50 percent, because the floor finally gets paid down
Bursty or low volume with spiky peaks	Hosted API, almost always: you would fund the full floor for a sliver of utilization
Hard latency SLOs, data residency, model control	Self-host or dedicated capacity; the floor is the price of control
Mid volume with peaks	Hybrid: reserved GPUs for the base load that justifies the floor, API for spikes

The other half of the decision is when the compute dividend pays, and it pays on a schedule. Prefill-heavy traffic and large sustained batches are the regimes where the 6B active count earns its keep, because compute-bound serving buys FLOPs and sparsity is a direct discount on them. Spiky, low-volume traffic never reaches that regime. Sparse vs dense LLM serving is the same coin from the other side: the sparse MoE buys capability per FLOP and pays for it in a multi-GPU footprint plus an all-to-all profile. If your volume never runs big, diverse batches, you fund the residency bill and never collect the dividend.

What Changes If the Final Specs Change

The 6-of-125B and one-ninth figures are release claims reported at time of writing, and the shipped model may differ. The worksheet does not care. Swap the inputs and every output updates:

• Expert count and top-k change routing granularity and message structure, so they move the communication line, not the residency line.
• Shipped precision, native FP8 versus BF16, moves the residency line and the GPU count with it.
• Attention design, GQA head counts or MLA-style KV compression, rescales the KV budget.
• Load-balancing scheme sets the discount you take on throughput projections for imbalance.

Not all four move the bill equally. Precision is the lever most likely to swing it, because it halves or doubles the residency line outright. Attention and KV design come second, rescaling the line that rivals the weights at long context. Pin those two down before anything else.

Rerun the worksheet on the final numbers, for this release or the next one. The spec sheet still will not do this math for you.