The MLOps Playbook: Choosing Cloud GPU Providers for LLM Inference

Is $2.50 per hour actually cheaper than $0.004 per 1,000 tokens? If you have a specific workload, a strict latency target, and a defined budget—yet still can't answer that question from a vendor's pricing page—you're not alone. Cloud providers tend not to surface cross-comparisons between APIs and raw compute, a structure that advantages their managed ecosystems. A quick search for 'cloud gpu providers llm inference' surfaces dozens of vendor listicles but almost no mathematical frameworks for determining when $/hr actually beats $/token.

Teams evaluating cloud GPU providers for LLM inference all start from the same realization: comparing hourly rates alone is a trap. The true cost is dictated by the interplay between throughput, model architecture, and pricing models. When engineers attempt a straightforward cloud GPU comparison using only hourly rates, they ignore the critical variable of token generation speed. A "cheaper" instance that generates tokens 50% slower than a premium instance is actually far more expensive per unit of intelligence produced.

To optimize inference costs, MLOps practitioners must abandon spreadsheet-only evaluations and adopt a mathematical framework that evaluates both $/hr and $/token structures simultaneously. An LLM inference cost calculator can bridge this initial gap, but understanding the underlying mechanics is what truly empowers an engineering team to make the right architectural bet.

Deconstructing the Compute vs API Pricing Models

The choice between GPU compute and API access is, at its core, a question of risk distribution. Each model offloads a different cost onto a different party—and understanding who bears that cost determines which structure wins for your workload.

The API Model: Paying per Token

When you choose an API ($/token), the cloud provider assumes the risk of idle hardware. They provision massive clusters, manage the infrastructure, and absorb the cost of unused compute cycles. You pay a premium per token but gain infinite elasticity: no warm-up time, no minimum commitment, no idle-hour penalty.

This structure is especially attractive early in a product's lifecycle. Enterprise cloud LLM pricing trends show how aggressively providers are competing on per-token costs—sometimes subsidizing frontier model access to capture market share. The premium you pay per token buys flexibility that dedicated hardware simply cannot match.

Dedicated Compute: Paying per Hour

When you choose raw compute ($/hr), you assume the risk. A quiet three-hour afternoon still costs the full GPU rate. A traffic spike beyond your provisioned capacity means failed requests. The tradeoff: once your workload stabilizes, dedicated hardware can cost a fraction of the API equivalent.

A side-by-side comparison of AWS SageMaker (hourly compute) and Bedrock (per-token API) shows how dramatically the economics shift at scale. Consider a 70B-parameter model serving roughly 500 requests per minute with an average 500-token output. On Bedrock's per-token pricing (assuming Llama 3 70B Instruct, US-East region, output tokens only at 2024 published rates), that workload might cost roughly $8,000–$12,000 per month. On a dedicated SageMaker instance running the same model at consistent utilization, the same throughput could drop to approximately $3,000–$5,000 monthly. An AWS LLM pricing comparison indicates that once you cross a sustained volume threshold, the managed API premium tends to outpace the cost of dedicated hardware by 2–3x.

Hardware choice also inverts the obvious calculus. An older A100 GPU carries a lower hourly rate than an H100, but the H100's superior memory bandwidth translates to significantly higher tokens per second. The H100 vs A100 cost efficiency suggests that paying a higher hourly rate for newer silicon often drops your effective cost per token enough to outweigh the raw rate difference—upending traditional infrastructure procurement logic.

The Break-Even Math: When to Self-Host vs Use an API

To find your exact tipping point, calculate where self-hosting becomes cheaper than the API. The formula requires accurate benchmarking of your specific model on your target hardware:

Effective Cost per 1M Tokens = (Hourly GPU Rate / Tokens Generated per Hour) × 1,000,000

If a cloud GPU costs $3.00 per hour and your optimized model generates 2,000,000 tokens per hour under peak load, the math is straightforward: ($3.00 / 2,000,000) × 1,000,000 = $1.50 per 1M tokens. Against an API charging $5.00 per 1M for a comparable model, self-hosting wins by 3.3x—assuming sustained peak utilization.

But here is the insight most break-even analyses miss: that number has an expiration date. Model-layer economics depreciate faster than hardware commitments. Three forces continuously shift the break-even point:

• Quantization advances. INT8 and INT4 inference can double or triple throughput on the same GPU, shifting your effective cost per 1M tokens downward by 2–3x within a single optimization cycle. A break-even calculation run before quantization is obsolete the day your team enables it.
• Speculative decoding adoption. Draft-model serving can cut inference latency by 40–60% for certain model families, fundamentally altering the tokens-per-hour input to your formula.
• Context-length changes. Expanding context from 4K to 32K tokens increases KV cache memory pressure, which can reduce achievable throughput by 30–50% on the same hardware—raising your effective cost per token even though the hourly rate never moved.

Any one of these can shift your break-even by 3–5x within months. A calculation that showed self-hosting as 2x cheaper in January may flip to favoring the API by April after a quantization migration and a context-window expansion. This is why the break-even math is not a one-time exercise—it must be re-run quarterly, or whenever your team changes model architecture, serving framework, or token context limits.

A self-hosted vs API cost calculator lets you adjust these variables—hourly rates, concurrent requests, average sequence lengths, and throughput assumptions—as they shift, so you can re-run the comparison without rebuilding the model from scratch each quarter.

Why Throughput Matters More Than Hourly GPU Rates

Maximum achievable batch size—not raw FLOPS, not memory bandwidth, not clock speed—is the single most underestimated variable in any cloud GPU comparison. Two providers renting the same A100 at the same hourly rate can deliver vastly different $/token economics, and the reason is almost always batch size.

Inference throughput does not scale linearly until you hit a hardware wall. It scales with how many concurrent requests the serving framework can pack into GPU memory simultaneously. The vLLM optimization documentation details how PagedAttention manages the KV cache like an operating system's virtual memory, enabling continuous batching to keep dozens of requests in flight without fragmentation. But the theoretical maximum batch size that vLLM can achieve is gated by how much VRAM your provider actually allocates to your container—and that number is rarely the full 80GB advertised on the spec sheet.

The Batch-Size Cost Cascade

Consider two illustrative scenarios on identical A100 hardware serving Llama 3 70B with continuous batching. At a batch size of 32, you might see roughly 2,000 tokens per second. At a batch size of 128—achievable when the provider gives you full VRAM headroom—that same GPU might deliver 6,000+ tokens per second. At $2.50 per hour, the first scenario costs approximately $0.35 per million tokens. The second costs roughly $0.12. Same GPU, same hourly rate, same model—a 3x difference in effective cost per token driven entirely by configuration headroom.

The Silent Cap Problem

Many providers cap effective batch size without documenting it. Multi-tenant platforms, in particular, often limit individual containers to a fraction of total GPU VRAM to prevent noisy-neighbor contention—protecting their overall cluster stability at the direct expense of your throughput. A provider advertising "A100 80GB" may allocate only 60GB to your workload after container overhead, model weights, and system reservations, quietly cutting your achievable batch size by 25% or more. You will not find this policy on any pricing page.

How to Discover the Real Ceiling

Before committing to any provider, run this benchmark: deploy your production model, ramp concurrent requests from 1 to 256 in increments, and log tokens-per-second at each step. The throughput curve will plateau where VRAM or provider-imposed limits bind—and that plateau point is the only number that matters for your $/token calculations. The LLM benchmarking fundamentals from NVIDIA provide the methodology for measuring this rigorously, including first-token latency and inter-token latency alongside raw throughput. If the plateau arrives well before where your model's memory profile predicts, you have found a silent cap—and that provider is more expensive than their hourly rate suggests.

How to Evaluate Cloud GPU Providers for Reliability

The most expensive GPU failure is not a crash—it is silent degradation. Consider a scenario that plays out routinely: your tensor-parallel inference cluster meets latency SLOs during off-peak hours, but during afternoon contention the provider silently throttles InfiniBand bandwidth to accommodate other tenants. Inter-GPU communication latency doubles, time-to-first-token balloons from 400ms to 1.5 seconds, and users start churning. No error is logged. No instance is preempted. The provider's status page stays green. You discover the problem weeks later when a downstream product team reports latency spikes that correlate perfectly with peak hours.

This is why reliability testing must go beyond checking SLA percentages. Here are three protocols that surface real-world failure modes before they surface in production:

Sustained-Load Preemption Testing

Run your full inference stack at production concurrency for 72 continuous hours. Log every instance preemption, container restart, and OOM kill. A top cloud GPU providers roundup helps you shortlist vetted candidates, but only sustained testing reveals which ones actually hold up under load. Acceptable threshold: preemption events affecting fewer than 5% of your testing window. Anything higher means your effective hourly rate is inflated by boot times, model loading, and recovery overhead that you pay for without serving a single token.

Inter-GPU Bandwidth Verification

Before committing to multi-GPU deployments, run nccl-tests to measure actual AllReduce bandwidth between GPUs and compare it to the advertised interconnect spec. A provider listing "NVLink at 600 GB/s" may deliver 450 GB/s in practice after topology constraints, NUMA effects, or shared-fabric overhead. For tensor-parallel inference serving 70B+ models, that gap directly inflates per-token latency and undermines the cost-per-token savings from continuous batching.

Cold-Start Benchmarking

Time how long it takes to load model weights from checkpoint storage into GPU memory and serve the first inference request. Five minutes is acceptable for a 70B model with optimized checkpoint loading. Twenty minutes means your autoscaling strategy is broken—by the time new capacity comes online during a traffic spike, the spike is already over. Test cold-starts at different times of day; some providers route checkpoint loading through shared storage that saturates during peak hours.

The Scheduling Deprioritization Risk

Most teams overlook one reliability factor entirely: provider scheduling algorithms. Cloud GPU platforms do not treat all instance classes equally during contention. Spot instances, lower-tier commitments, and newer account workloads are often deprioritized in the scheduler—meaning your inference jobs are the first to be reclaimed or throttled when the cluster fills up. Ask your provider explicitly which instance classes are subject to preemption priority, and test by running identical workloads on different commitment tiers simultaneously.

Decision Matrix: Choosing Cloud GPU Providers for LLM Inference

The preceding sections built the economic logic. Here's how to apply it at architecture-review time—before you commit to a provider or write a line of integration code.

Three diagnostic questions before you commit:

1. What's your peak-to-mean traffic ratio? A 10:1 ratio means dedicated hardware sits idle 90% of the time. API pricing absorbs that risk. The break-even math from the previous section only favors dedicated compute when your utilization curve is flat enough to sustain it.

2. Does your model fit on a single GPU? Multi-GPU tensor parallelism adds interconnect latency, networking overhead, and operational complexity. If a 70B model forces a 4-GPU setup, your break-even volume must be high enough to amortize that infrastructure—or route through an API that already handles sharding.

3. What's the real preemption cost? If your provider reclaims instances every few hours, model reloading can eat 15–20% of effective throughput. Factor that into your cost-per-token calculation before signing anything. Stress-test shortlisted providers for silent preemptions and network throttling under sustained load.

Start with your traffic profile. Find your row. Let the economics dictate the architecture—not the other way around.