Megakernels in LLM Inference When Fusion Actually Wins

The debate over megakernels in LLM inference has taken on an almost religious tone. Some practitioners insist hand-fused kernels are essential for competitive latency. Others, citing modern compiler stacks and upcoming hardware features, argue they are a research indulgence. The truth lives in the workload. A megakernel that shaves milliseconds off a single-stream agentic decode can stall a high-throughput serving stack, and the compiler path that wins for batch-32 chat can choke on a 128K-token agent context.

The right question is not whether megakernels are dead or alive, but which workload makes them necessary. This article walks the specific hardware tension that decides the answer: the tradeoff between CPU launch overhead and Streaming Multiprocessor occupancy, and how it shifts with batch size and sequence length. The recent megakernels debate on Latent Space captured the split well, but stayed at the level of strong opinions. We are going to ground it in the physics.

The Microarchitecture Bottleneck Behind Every Inference Bill

Latency and token cost are macro symptoms. The disease is almost always at the Streaming Multiprocessor level. When you profile a serving stack with a tool like Nsight Systems, what you usually see is not compute saturation. You see gaps. Long stretches where the GPU is idle, waiting for the host CPU to enqueue the next kernel, waiting for a memory fence, waiting for a tensor core block to drain.

Most inference practitioners have internalized that LLM decoding is memory-bandwidth bound. The activations are tiny per token; the weight and KV cache reads dominate. What gets less attention is that the memory subsystem is only the second bottleneck at low batch sizes. The first is the host. Each kernel launch carries a CPU-side cost, often on the order of single-digit microseconds and sometimes more, and a forward pass for a transformer layer can require dozens of distinct kernels. Stack those launches across attention, KV updates, FFN, normalization, and activation, then multiply by layer count, and the host becomes the gate.

This is why two stacks serving the same model at the same batch can post very different latencies. The number is not just about kernel code quality. It is about how densely packed the launch schedule is. NVIDIA's launch overhead guidance makes the point explicitly: launch overhead is a measurable, addressable component of end-to-end latency, not a fixed tax you have to accept.

What Defines a Megakernel in Modern Inference

A megakernel is a single fused kernel that absorbs what would otherwise be a sequence of separate kernel launches into one persistent execution context. Where a modular CUDA implementation would call attention, then softmax, then dropout, then a projection, then a normalization, each as a distinct kernel with a distinct launch boundary, a megakernel holds the thread grid resident on the SMs and threads the intermediate tensors through registers and shared memory without returning control to the host.

The technique has roots in older GPU programming research. NVIDIA's persistent threads model described this pattern more than a decade ago, and the same underlying logic now appears in modern fused attention implementations like FlashAttention, which fuses the attention computation with online softmax to avoid materializing the full attention matrix in HBM.

The defining property is not size. It is that the kernel crosses what would normally be a launch boundary. That crossing is what eliminates host overhead and the intermediate round-trips through global memory. It is also what makes megakernels painful to write. You lose the modularity of separate kernels, the ability to independently tune each one, and the freedom to overlap execution of independent kernels with streams. A fused forward pass that took months to write can break the moment the attention variant, quantization scheme, or hardware generation changes.

The Core Tradeoff: Launch Overhead vs Occupancy

This is where the hardware physics gets interesting. Every GPU kernel launch has two costs, and they pull in opposite directions.

The first is CPU launch overhead. Every separate kernel costs some number of microseconds on the host. At high batch sizes, where each kernel runs long enough, this overhead is amortized into noise. At low batch sizes, where kernels complete in tens of microseconds, the launch overhead can rival or exceed the compute time. At batch 1, a megakernel that eliminates 30 kernel launches from a layer can meaningfully reduce latency even if the per-kernel code is slightly less tuned.

The second is SM occupancy. A GPU like the H100 has a fixed number of Streaming Multiprocessors, each with a finite warp slot budget and a finite register file. The LSU SM occupancy notes describe this constraint in detail. A well-tuned modular kernel can be configured to exactly fill the SMs and keep the tensor cores fed. A megakernel that holds its grid resident has to balance all of its sub-computations against the same register and shared memory budget, and it is easy to end up under-occupied on some phases while over-budget on others.

So the real tension is:

• Launch overhead dominates when batch is small, kernels are short, and the host is the bottleneck. Megakernels win.
• Occupancy dominates when batch is large, kernels are long, and the GPU is already saturated. Modular kernels with independent tuning win, because the host is no longer the gate and you can squeeze each kernel for maximum throughput.

The crossover point depends heavily on the model and the specific kernel implementations, so treat the batch 8 to 32 range as a diagnostic heuristic rather than a fixed threshold. Below it, fusion tends to pay. Above it, you are paying complexity for diminishing returns. Architectural shifts in modern hardware only sharpen the tension. The H200's physical specifications, with more SMs and faster tensor cores, make each individual kernel shorter, which means the fixed per-launch cost looms larger relative to compute time and the regime where fusion helps stretches across a wider band of batch sizes.

When Megakernels Win: Agentic Workflows and Long Context

There is one regime where megakernels are not just useful but essential. That is the regime where decoding is sequential, batch size is small (often 1), context is long, and latency is the user-facing metric.

Agentic workloads fit this profile precisely. A coding agent iterating on a single problem, a tool-using assistant waiting between steps, a research agent streaming a long reasoning chain. These workloads generate tokens one at a time, with the full KV cache resident, and the user (or the agent loop) is waiting on each token. The decode is deeply memory-bandwidth bound, the batch is tiny, and every microsecond of launch overhead shows up in time-to-first-token and time-per-output-token.

In this regime, a hand-tuned megakernel that fuses attention, KV updates, normalization, and projection can eliminate tens of kernel launches per layer per token. The same fusion that would be wasted at batch 64 becomes the dominant latency lever at batch 1.

There is a subtlety here that often gets lost. The benefit is not just about removing launches. It is about removing intermediate writes to HBM. In a low-batch decode, every intermediate tensor that gets written to global memory and read back is pure overhead, because the tensor is small and the memory transaction cost dominates the math. Fusing those intermediates out of existence, by keeping them in registers and shared memory, is where the real latency savings come from. This is the same insight that made FlashAttention decisive for training, applied to a different bottleneck at decode time.

When Compilers and CUDA Graphs Win: High Throughput Serving

For high-throughput serving, the calculus flips. At batch 32 or 64, each kernel runs long enough that launch overhead is amortized, the GPU is genuinely busy, and the bottleneck moves from the host to the memory subsystem and the tensor cores. Here, what you want is not one giant kernel. You want each individual kernel tuned to maximum occupancy, and you want them to overlap.

This is what modern compiler stacks are built to do. Triton lets you write fused kernels in Python that compile to efficient PTX, and it has effectively closed the gap between hand-written CUDA and compiler-generated code for a wide range of operations. MLIR provides the compiler infrastructure underneath many of these stacks, and TensorRT-LLM's reference design shows the full strategy: aggressively fuse where it helps, then expose the rest as tuned modular kernels that the runtime can schedule independently.

The other half of the story is CUDA graphs. A CUDA graph lets you capture an entire sequence of kernel launches once, then replay it with near-zero host overhead on every subsequent invocation. This is the cleanest answer to launch overhead without writing a megakernel. Capture the forward pass as a graph, replay it per token, and the host is no longer the bottleneck. Most modern inference engines, including those built on vLLM's PagedAttention, lean heavily on graph capture and kernel-level tuning rather than hand-fused megakernels.

The practical implication is sharp. For general high-throughput serving, hand-written megakernels are usually premature optimization. The compiler and graph path gets you most of the way there with a fraction of the engineering cost, and the kernel code remains readable and individually tunable. Teams that ship megakernels in production for this regime often report that the modular, compiler-tuned path is actually faster, because each sub-kernel can be optimized independently and scheduled with overlap.

Practical Engineering Guide for Execution Strategy

The decision framework is workload-driven. Ask these questions in order.

1. What is your median batch size at peak load? If it is 1 to 8, especially for agentic or interactive workloads, fusion matters. If it is 32 or above, launch overhead is already amortized and the compiler path wins.

2. What is your typical sequence length? Long contexts (32K and above) sharpen the decode bottleneck. Memory bandwidth dominates, and intermediate HBM writes are pure tax. Fusion helps more here.

3. What is your latency budget? If time-to-first-token or per-token latency is user-facing and tight, the host is in the critical path and graph capture or megakernels are worth evaluating. If you are optimizing for throughput (tokens per second per GPU) and the user is not waiting on each token, modular kernels with good scheduling are the simpler answer.

4. Are you on a stack that already captures graphs? If your engine already uses CUDA graphs for the decode loop, the marginal value of a hand-fused megakernel drops sharply. Check first. Many of the launch overhead wins you would chase with fusion have already been captured for you.

5. Do you have the engineering capacity to maintain a megakernel? A fused forward-pass kernel is not write-once code. It breaks when the model architecture changes, when the attention variant changes, when the quantization scheme changes, when the hardware generation changes. Most teams underestimate this cost. The broader inference discussion captures this honestly: even teams that have written megakernels often do not run them in production, because the modular path is easier to keep competitive.

A reasonable default for most teams: start with a compiler-and-graphs stack, profile with Nsight Systems to see where the real gaps are, and only invest in hand-fused megakernels for the specific decode path where profiling shows host overhead is the bottleneck and where the workload is latency-bound at low batch. For everything else, the compiler is your friend.

The Verdict on Megakernels in LLM Inference

The framing of megakernels as dead or alive misses the point. They are a tool with a narrow, well-defined region of applicability. That region is latency-bound, low-batch, long-context, sequential decoding. Outside it, modern compilers and CUDA graphs do the same work with less engineering pain.

The real lesson is more general. Inference performance is not determined by chasing whichever technique is currently trending. It is determined by understanding which bottleneck, host or SM, dominates for your specific workload, and choosing the execution strategy that addresses that bottleneck. Megakernels in LLM inference are the right answer when the host is the gate. They are an expensive distraction when the GPU already is.