More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

NVIDIA published an explainer arguing that the answer to rising GPU memory pressure is not simply packing more high-bandwidth memory onto each accelerator but designing GPUs, CPUs, networking, storage and software together so that every byte does more useful work. The company says it was first to adopt HBM3e and HBM4 into an accelerated computing stack, and it frames the shift as a response to larger models, longer contexts and agentic applications that are straining GPU memory while supply tightens and prices rise.
The headline numbers span two platform generations. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, NVIDIA says GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform at 100 tokens per second per user, with HBM capacity per GPU roughly doubling. From Blackwell Ultra to Vera Rubin, HBM capacity per GPU rose only about 3%, yet at 125 tokens per second per user Vera Rubin NVL72 reaches up to 30x higher throughput per dollar than GB300 NVL72. In other words, the per-dollar gains are growing much faster than capacity.
NVIDIA's explanation rests on what it calls the memory wall. HBM is the fastest and most expensive tier, and during inference it holds both model weights and the KV cache, the per-request working memory that stores prior context so the model does not reprocess the same tokens. Frontier models now carry trillions of parameters, and agentic trajectories in some cases process up to 3 million tokens as context accumulates across repeated model and tool calls. RAG pipelines add more context still. The company likens HBM to a warehouse and memory bandwidth to its loading docks: once the docks are saturated, extra storage brings cost and power draw without improving response times, which matters when users expect fast replies.
Three technique families address different sources of waste. The first is shrinking weights: NVFP4, a 4-bit format, cuts the space weights need compared with FP8, described as the lowest precision available on the Hopper platform. NVIDIA lists FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark from Meta, Step 5 Preview from StepFun and NVIDIA's own Nemotron 3.5 Super VL as recent models shipping with NVFP4 checkpoints. Within an NVL72 system, weights can also be spread across up to 72 GPUs, so each holds a smaller share and leaves more HBM for cache and concurrent users.
The second is shrinking and relocating KV cache. NVFP4 applied to the cache reduces its footprint versus FP8. NVIDIA's Vera CPU plays two roles: faster tool execution shortens the window in which cache sits idle, and its large LPDDR memory, linked to Rubin GPUs over NVLink-C2C, gives paused context a nearby place to wait. When context is not in use, Dynamo inference software moves it to CMX context memory storage, a capacity-oriented tier with a much lower cost per bit than HBM. CMX differs from local SSDs in being shared across servers, and it omits the long-term data-protection overhead of enterprise storage because context is short-lived. Partners named include Cloudian, DDN, Dell Technologies, Everpure (previously Pure Storage), HPE, IBM, NetApp, VAST and WEKA. Dynamo's KV Cache Reuse treats cache across GPU memory, CPU memory and storage as one system-wide pool.
The third targets retrieval. Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy, per NVIDIA, and the cuVS library builds and serves the index across CPU and storage so the knowledge base never occupies HBM. Only retrieved passages enter the context window.
For developers, the practical thread is that memory efficiency is increasingly a software and system-design problem as much as a hardware one. Techniques like 4-bit weight and cache quantization, weight sharding across a large GPU domain, and tiered cache offload with reuse change how inference services are deployed and tuned.
Why it matters: Teams running large models or long agentic workloads are squeezed by HBM cost and availability, and NVIDIA is arguing that throughput per dollar, not raw capacity, is the metric to plan around. That suggests orchestration software such as Dynamo, quantization formats such as NVFP4 and storage tiers for idle context will become standard parts of an inference stack, and that model builders shipping low-precision checkpoints widen the set of models that can take advantage of them. Whether the quoted throughput-per-dollar multiples hold outside SemiAnalysis's AgentX workload is the open question operators will want to test on their own traffic.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise — a trend expected to continue through the near term. NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack. […]