Mixture-of-Experts kernel engineering on Amazon SageMaker AI

AWS published details of three kernel engineering optimizations it built for fused Mixture-of-Experts (MoE) inference, work done with the vLLM community and applied in the Large Model Inference (LMI) Deep Learning Container on Amazon SageMaker AI. Combined, the company reports up to 10 percent higher throughput and up to 9 percent faster Time to First Token (TTFT) on NVIDIA H200 hardware. Two of the three changes were upstreamed to vLLM.
MoE is a sparsely activated architecture: each token is routed to only a subset of the model's expert subnetworks rather than the full parameter set, which lets a model carry far more capacity than a dense model of equivalent per-token compute. Fused MoE takes that further at the systems level by merging token routing, data shuffling and expert computation into a single batched GPU operation instead of running routing and each expert as separate sequential loops.
The first optimization targets the operand layout Hopper's Tensor Cores expect. In the WGMMA instruction protocol, which executes across a cooperative warpgroup of 128 threads, the hardware fixes the M (row) dimension of the first operand at 64, and that operand slot is not interchangeable with the second operand's variable N dimension. Peak Tensor Core utilization therefore requires M to reach at least 64. During decode, MoE models process one token per sequence at a time, so M is 1 — or at least well under 64 — while the expert weight block stays large. AWS's example shapes were a 1 x 4096 activation against 4096 x 8192 expert weights. Under that layout much of the Tensor Core capacity sits idle.
In a fused MoE kernel the N dimension is not a single hidden size but the number of selected experts multiplied by the expert intermediate size. With a top-k of 8 the effective N grows into the tens or hundreds of thousands, making the asymmetry between tiny M and huge N extreme. AWS's fix is to swap the operands: compute B transposed times A transposed, giving C transposed. Because transposition is algebraically valid, model math is unchanged, but the large, contiguous expert dimension now occupies the leading operand slot. That both fills the hardware's mandatory 64-row layout and lets Hopper's asynchronous copy engines prefetch weights from shared memory into registers ahead of execution. The reported result is a 2 to 4 percent end-to-end output throughput gain for large-scale MoE models on H200. The change was upstreamed to vLLM and is available from version 0.24.0 onward.
The second optimization is a routing decision rather than a kernel rewrite. The vLLM MoE oracle dispatches different quantization schemes to Triton, FlashInfer plus CUTLASS, or other backends. Profiling showed that for FP8 tensor-quantized models on Hopper, Triton consistently beats the default FLASHINFER_CUTLASS path. In practice this is a low-concurrency phenomenon: CUTLASS treats FP8 tensor scaling, matrix multiplication and expert aggregation as separate stages built from standard C++ templates, so data makes repeated round trips to global memory between stages. Triton fuses the pipeline into a single pass, reading from high-bandwidth memory once and doing the intermediate work in on-chip registers, then streaming results out. Its dynamic scheduling also suits the variable, fragmentary token batches of live decode, where CUTLASS's block tiling and thread mapping — designed for large, predictable matrices — give way to warp under-utilization or padding overhead. Changing the routing policy required no kernel code and produced a 10 percent end-to-end throughput improvement for FP8 quantized fused MoE on H200. It ships in the AWS LMI v28 container.
The third fix addresses a less visible cost: unnecessary just-in-time recompilation. Triton specializes integer arguments by default, so any call with a previously unseen value for a specialized parameter triggers a fresh compile. Under serving workloads where sequence lengths and batch counts vary continuously, that overhead accumulates in latency, and TTFT is where it shows first. The affected kernel, _fused_inv_rope_fp8_quant_per_head, which handles fused rotary position embedding and FP8 quantization, was recompiling on every new num_tokens value. Adding do_not_specialize annotations to the variable-length parameters eliminated the recompiles, improving TTFT by 9 percent on H200 at 8-way tensor parallelism (PR #42849). That gain has a cluster dimension too: at 8-way tensor parallelism all eight GPUs must stay in microsecond-level lockstep, so if one GPU pauses to recompile while its local token count changes, the other seven idle. Removing recompiles across the cluster smoothed that coordination cost. This was upstreamed to vLLM and is available from version 0.22.0 onward.
A useful way to read this set of results is that only one of the three changes is a genuinely new kernel. The other two — a backend routing switch and a compiler annotation — are configuration-level and small in code terms, which suggests that in MoE serving a large share of available performance sits in dispatch and compilation policy rather than in hand-written math. The measurements are also tightly scoped: they are H200 numbers, on FP8 tensor-quantized MoE workloads, and the low-batch decode regime is exactly where CUTLASS-style dense tiling and 64-row Tensor Core layouts struggle most. A reasonable inference is that the backend preference may not carry over to high-concurrency, prefill-heavy serving or to non-FP8 quantization, and the optimizations are described for Hopper specifically, so behavior on other GPU generations is not established by these figures. Whether the throughput and TTFT gains are additive or overlapping is also not stated.
Why it matters: teams serving open-weight MoE models such as the DeepSeek or Qwen families on SageMaker or self-managed vLLM have concrete, low-effort levers here — a container upgrade or version bump, a backend routing change, and compiler annotations — rather than a rewrite. Since the vLLM pieces are open source, the benefit is not limited to AWS: any Hopper-based deployment on vLLM 0.24.0 or later can pick up the operand-swap and recompilation fixes. The broader lesson for product teams is that decode-phase latency in MoE serving is often an artifact of how work is scheduled and compiled, not of how much compute the model needs.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
AWS engineers share kernel optimizations for fused Mixture-of-Experts (MoE) models on Amazon SageMaker AI: Swap-AB for WGMMA utilization, Triton backend routing for FP8 quantization, and eliminating Triton JIT recompilation. Together they deliver up to 10% higher throughput and up to 9% faster Time to First Token on H200 GPUs.