AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each

Collected Sep 30, 2026

NVIDIA published a developer blog post comparing dense and Mixture-of-Experts (MoE) model architectures, focused on active parameters, throughput and deployment tradeoffs. The post states that Nemotron 3.5 Lightning uses an MoE architecture that activates only 3B of its 30B total parameters per token.

The post describes dense models as activating all parameters for every token through a single shared feed-forward network block per decoder layer. In an MoE model, that block is replaced by multiple expert FFN blocks, and a router network assigns each token to the top k scoring experts, skipping the rest for that layer. Routing decisions are made separately at each decoder layer. The post also notes most modern MoEs, including Mistral Small 4, run one shared expert that every token routes to.

According to the post, MoE decouples memory cost from compute cost: all experts load into VRAM, while compute scales only with the experts that fire per token. In dense models, hosting and inference costs scale together. The post says MoE holds a throughput advantage across batch sizes at equal total parameter counts, but its latency margin versus a well-optimized dense model narrows at high concurrency. It also states all experts must reside in GPU memory simultaneously, leaving less room for the KV cache than a comparable dense model.

The post cites a comparison in which Gemma 4 31B and Nemotron 3.5 Lightning are both roughly 30B total parameters, with output-speed ranges across NVIDIA GPU providers that do not overlap. It attributes this in part to activating 3B parameters per token, and says Lightning's Mamba-2 layers and speculative decoding contribute independently of the hybrid MoE architecture. The post also says Lightning delivers four to five times Qwen3.8-27B's output speed at a fourteenth of the price, and scores less than half as well on general capability.

On selection, the post lists memory budget, concurrency, fine-tuning plans and quantization as factors. It states a 30B MoE and a 30B dense model both need roughly 60 GB. It warns that a full fine-tune on an MoE can unbalance the router, and says LoRA/PEFT approaches or freezing the router help avoid this. On quantization, it says MoE routers and, in hybrid-attention models, recurrent projections are the modules that quantize badly, and notes Mistral Small 4 ships natively in FP8.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the...