AivexaNewsSearch
AI news for builders and product teamsChecked every hour

3 Ways to Boost MoE Inference on NVIDIA DGX Spark

Collected Oct 27, 2026

NVIDIA published a developer blog post detailing three techniques to speed up mixture-of-experts (MoE) inference on the DGX Spark desktop AI computer, using Inclusion AI's Ling-3.0-flash model as the implementation example. Profiling at batch size 1 showed decoding was memory-bound: reading model weights dominated execution time, with the MoE computation and the output-layer projection (LM head) standing out as the main costs. The post walks through each optimization, measures its effect, and flags an integration pitfall that can make the combined configuration slower than the unoptimized baseline.

The evaluation used Ling-3.0-flash-fp4, a MoE model with 124 billion total parameters and 5.1 billion active per token. It has 42 layers: 35 Kimi Delta Attention (KDA) linear-attention layers and seven multi-head latent attention (MLA) layers. Each MoE layer routes each token to eight of 512 routed experts, plus one shared expert. The model offers a native 256K-token context window, expandable to 1M tokens, and supports enabling or disabling reasoning depending on latency needs. The baseline followed the official cookbook's W4A8 configuration: FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM, and the --enable-fp32-lm-head option. The LM head projects a hidden dimension of 2,560 to a vocabulary of 157,184 tokens.

The first optimization replaces the flashinfer_mxfp4 MoE backend with Humming, an independent kernel library available via the humming-kernels pip package. Unlike the FlashInfer CUTLASS path, Humming uses NVIDIA CUDA Runtime Compilation (NVRTC) to generate and compile CUBINs at runtime for the actual matrix dimensions, data types, and GPU architecture, caching binaries in ~/.humming/cache. This specializes kernels for Ling-3.0-flash's shapes, including its relatively narrow moe_intermediate_size of 768. It also compiles at a finer granularity: an individual kernel completes in seconds, whereas the FlashInfer CUTLASS path uses ninja to compile 97 targets and takes on the order of tens of minutes. Performance is reported as time per output token (TPOT) and decode throughput.

The second optimization applies online FP8 quantization to the LM head. In the checkpoint, the LM head is stored in BF16 and listed in modules_to_not_convert, so it is not quantized during loading. Its vocabulary GEMM is 157,184 × 2,560, and the entire weight block must be read for each generated token, making it a high-value target under bandwidth pressure. The change applies a dynamic FP8 quantization configuration only when constructing the LM head, while the rest of the model continues to load with the checkpoint's FP8/MXFP4 configuration. Moving the LM-head weights from BF16 to FP8 halves the read volume for this layer. It is enabled with SGLANG_ENABLE_FP8_LM_HEAD=1, which is mutually exclusive with the baseline --enable-fp32-lm-head option.

The third optimization uses multi-token prediction (MTP) speculative decoding with NEXTN. While the first two reduce the cost of each decode step, speculative decoding reduces the number of steps. Ling-3.0-flash includes a NEXTN prediction layer (num_nextn_predict_layers: 1) that can serve as a draft model without additional training or downloads. The configuration uses three draft steps and a four-token verification window: each round, the draft predicts three tokens in sequence, and the target verifies four positions at once, accepting the correct prefix. The benefit depends on accept length (average accepted tokens per round) and the verification-cost ratio (cost of one verification step relative to a normal decode step), measured at approximately 2.26. The main verification cost was not attention computation—at batch size 1, KV reads were small relative to weight reads—but MoE expert activation: a normal decode activates eight experts for one token, while verification processes four tokens and can activate up to 32 experts.

The post documents a critical integration issue: when online FP8 LM-head quantization and MTP are enabled together, the draft model must share the entire LM-head module with the target, not just its weight tensor. The original Ling NEXTN implementation didn't expose this interface. Without it, the framework silently falls back: the draft receives FP8 weights but interprets them as BF16, accept length drops to 1.00, and throughput becomes lower than a configuration with no optimizations. Sharing the full LM-head module through this interface restores normal token acceptance; the Ling cookbook provides instructions for enabling both together.

The component-level measurements used a synthetic workload with 32K input tokens and 2K output tokens at batch size 1. Adding FP8 LM-head quantization increased decode throughput by 11.4% over Humming alone. With all three optimizations, MTP accept length reached 3.90 tokens per round. NVIDIA cautions that synthetic workloads overestimate speculative-decoding benefits due to repetitive outputs and simple sampling. The end-to-end evaluation on all 1,319 GSM8K questions used five-shot prompting, temperature 0.6, top_p 0.95, top_k 29, max_tokens 32,768, and chain-of-thought enabled, with other variables aligned. Mean output lengths were 567 and 607 tokens for baseline and optimized runs; median output length was approximately 310 tokens, and P95 output lengths ranged from about 1,900 to 2,060 tokens. On this realistic workload, accept length was 2.68 versus 3.90 on synthetic, and the synthetic test's 185.1% throughput gain does not establish gains for other workloads.

Across the full GSM8K evaluation, the three optimizations increased decode throughput by 42.5%, reduced mean end-to-end latency by 24.5%, and produced no statistically detectable accuracy change. Their mechanisms do not overlap—Humming and the FP8 LM head reduce the cost of each step, while MTP reduces the number of steps—so they can be combined. However, stacking is not automatic: each optimization worked independently, but enabling all three required the draft model to share the full LM-head module. Otherwise, the system silently degrades to a configuration slower than the unoptimized baseline. NVIDIA recommends validating that every optimization is active at startup rather than inferring configuration state from later performance numbers.

This suggests that developers deploying MoE models on constrained hardware like the DGX Spark should treat kernel selection, output-layer quantization, and speculative decoding as complementary levers, but must verify each is active. The silent fallback with FP8 and MTP is a reminder that performance tuning on edge devices can be fragile: a misconfiguration may not surface as an error but as slower-than-baseline throughput. The Humming backend's runtime compilation approach may also reduce iteration time compared with the FlashInfer CUTLASS path, though it introduces a dependency on NVRTC and caching. A likely trade-off is that FP8 quantization of the LM head could affect output quality, but the GSM8K results showed no statistically detectable accuracy change for this model. The synthetic-to-real gap in accept length underscores that speculative decoding gains are workload-dependent, so teams should benchmark on representative traffic before committing to a configuration.

Why it matters: Developers and product teams working with large MoE models on desktops or edge devices now have a documented recipe that yields double-digit throughput and latency improvements on a real benchmark, but they must follow the LM-head sharing interface exactly to avoid a silent performance regression. Inference engineers should incorporate startup validation of each optimization into their deployment pipelines, as inference suggests the difference between a 42.5% speedup and a slowdown can hinge on a single configuration flag.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA...