Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory

NVIDIA's developer blog lays out a five-part argument that AI inference performance now depends less on how much HBM a GPU holds than on how much useful work each stored byte does. The headline example: between GB300 NVL72 and Vera Rubin NVL72, HBM capacity per GPU stays at 288 GB while per-GPU memory bandwidth rises to 19.2 TB/s, yet throughput per megawatt grows far faster than the capacity curve would suggest. NVIDIA cites SemiAnalysis AgentX, which replays recorded coding-agent sessions with real tool-call gaps and growing context, as showing Vera Rubin NVL72 delivering up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user.
The five levers target five distinct kinds of waste: precision bits, duplicated weight copies, recomputed prefill, idle KV cache during tool calls, and context stranded on individual nodes. NVIDIA frames them as one co-designed memory system spanning GPU architecture, the scale-up interconnect, serving software, CPU and storage, such that gains stack: a 4-bit KV block, for instance, saves space in HBM, host memory and shared storage alike.
Lever one is NVFP4, a 4-bit format pairing an E2M1 element with two scaling levels: an FP8 (E4M3) scale factor shared by every 16-value micro-block, plus a per-tensor FP32 scale on top. NVIDIA says that granularity preserves accuracy where naive 4-bit formats fall short. On weights, NVFP4 cuts model footprint roughly 3.5x versus FP16 and 1.8x versus FP8, freeing memory for KV cache, larger batches and concurrency. NVIDIA states NVFP4 is the default precision for open agentic benchmarks with strict accuracy requirements, naming AA-AgentPerf, SemiAnalysis AgentX and MLPerf Endpoints, and that it delivers up to 4x the throughput of an FP8 baseline at the same interactivity for DeepSeek R1 on GB300. The same format applied to the KV cache cuts footprint by up to 50% versus FP8, keeping roughly 2x more context resident and reducing decode bandwidth, while retaining over 99% of BF16 baseline scores across 10 DeepSeek-v4-Pro benchmarks in NVIDIA's testing. In practice, the trade-off with any 4-bit scheme is sensitivity: accuracy depends on how well per-block scales track outliers, which is why the scaling design carries as much weight as the bit width.
Lever two is wide expert parallelism (Wide-EP) in TensorRT-LLM. Mixture-of-experts models activate only some parameters per token, but every expert's weights must reside somewhere in HBM. Spreading experts across eight or more GPUs reduces per-GPU weight memory, leaving room for KV cache and concurrency, and lets GroupGEMM kernels reuse each loaded weight across more tokens. NVIDIA gives DeepSeek-R1 at EP64 as an example: four experts per layer per GPU, or 232 expert blocks across 58 MoE layers out of 14,848 in the full model. An Expert Parallel Load Balancer (EPLB) shifts hot and cold experts between forward passes. This leans on fast collectives, since every MoE layer triggers all-to-all dispatch and combine traffic; on GB200 NVL72, 130 TB/s of aggregate NVLink bandwidth absorbs it, and EP32 delivers up to 1.8x more output tokens per second per GPU than EP8 at 100 tokens per second per user. Vera Rubin NVL72 carries up to 288 GB of HBM4 and 3.0 TB/s of NVLink 6 per GPU, roughly 20.7 TB of HBM4 per rack, and adds inline TMA descriptor updates so one shared descriptor can serve experts with a common layout, trimming per-expert metadata.
Lever three is NVIDIA Dynamo, the serving layer that treats KV cache as a fleet resource. Its router tracks where KV blocks live and weighs cached-prefix overlap against worker load. Disaggregated serving splits prefill and decode onto separate GPU pools so decode nodes can spend HBM on cache and batch size. The KV Cache Runner (KVCR) tracks blocks across GPU, pinned CPU memory, local SSDs and mounted network filesystems, moving data via the NVIDIA Inference Xfer Library (NIXL); a KVCR-Guard option preserves committed host-memory cache across an engine failure while the guard service and host remain up. NVIDIA reports separate vendor validation: VAST Data reached 35 GB/s to one H100 GPU using GPUDirect Storage, and WEKA showed up to 270 GB/s aggregate read across eight H100 GPUs through a custom NIXL plugin, neither acting as a transfer bottleneck. A separate VAST test with vLLM and LMCache on DGX H100 cut time to first token for Qwen3-32B with a 130K-token prompt by loading stored cache rather than recomputing. The economics are simple: offloading only pays when retrieval beats a repeat prefill.
Lever four is the Vera CPU. Agentic sessions alternate GPU steps with CPU-side tool calls, code execution, retrieval and queries; during those gaps, retained KV cache occupies HBM without generating tokens, and under full subscription the GPU takes new work, so long gaps push the paused session's cache toward eviction and a full rebuild. Vera brings 88 Olympus cores, 1.8x faster single-threaded performance, 40% lower peak loaded latency than x86 CPUs and up to 14 GB/s of memory bandwidth per core, plus up to 1.5 TB of LPDDR5X, more than 5x a single Rubin GPU's HBM4, linked coherently over NVLink-C2C at 1.8 TB/s (roughly 7x an x16 PCIe Gen 6 link's 256 GB/s). Dynamo's KV Block Manager can park a paused session's cache in Vera memory and pull it back when the agent resumes.
Lever five is NVIDIA CMX, a pod-level storage tier. Local SSDs are node-bound, while shared enterprise storage carries replication and metadata overhead ephemeral KV cache does not need. CMX adds an Ethernet-attached flash layer built on the STX reference design, combining BlueField-4 STX storage processors, Spectrum-X RDMA networking, DOCA Memos KV-aware services and Dynamo/NIXL integration. NVIDIA claims up to 5x higher tokens per second and up to 5x better power efficiency than traditional storage for long-context inference, with prefixes reused across compatible model instances in a pod.
Why it matters: teams running agentic inference at scale are affected, because the levers turn context management, quantization and CPU choice into throughput decisions rather than fixed hardware properties, and the benchmark names suggest independent validation is emerging. A reasonable inference is that these techniques arrive bundled with NVIDIA's stack, so their value depends on adopting TensorRT-LLM, Dynamo and the Rubin generation together. A likely trade-off is operational complexity: routing, tiering and guard services create more components to run and tune.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
In an AI factory, high bandwidth memory (HBM)is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an...