NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0

NVIDIA has published provisional MLPerf Endpoints v1.0 results for its Vera Rubin NVL72 system, submitted against the Kimi K3, DeepSeek-V4.1-Flash and Qwen3.6-35B-A3B models. The headline claim is more than 10x the throughput per kilowatt of provisioned power compared with NVIDIA's GB300 NVL72, measured at a shared minimum end-to-end interactivity of 50 tokens per second per user. The results were published on November 9, 2026, for Kimi K3 and Qwen and on November 10, 2026, for DeepSeek. NVIDIA's Vera Rubin numbers remain provisional and subject to change pending MLCommons Association verification and peer review. The GB300 figures cited alongside them have already been verified by MLCommons.
MLPerf Endpoints v1.0 is a benchmark aimed specifically at agentic serving, where an assistant holds a growing conversation across many turns and interleaves reasoning, tool calls and final responses. The suite runs coding and enterprise workflows through API endpoints and checks generated tool calls and workflow intent against reference values during performance runs. A separate SWE-bench Verified evaluation is used to check end-to-end task-solving capability, so the benchmark is designed to test whether faster serving keeps model quality intact rather than only how many tokens a system can emit. This suggests the benchmark is meant to reflect production agent traffic rather than single-shot prompt completion, which is a meaningful shift in what vendors are asked to optimize for.
The detailed Kimi K3 record illustrates how the comparison works. At a concurrency of 1,024 queries, Vera Rubin NVL72 with 32 accelerators reported 41,414.2 system tokens per second, a P90 time to first token of 4,673.9 ms, average end-to-end interactivity of 50.4 tokens per second per user, and operating power of 100.89 kW. The GB300 NVL72 record at the same concurrency used 48 accelerators and reported 13,905.8 tokens per second, a P90 TTFT of 15,223.0 ms, 16.8 tokens per second per user and 60 kW. On NVIDIA's figures that is roughly a 3x interactivity advantage at matched concurrency for Vera Rubin, with more than twice the absolute throughput.
The DeepSeek-V4.1-Flash records show the per-kilowatt case more directly. At a minimum of 280 tokens per second per user, Vera Rubin with 24 GPUs reported 272,711.7 tokens per second, 294.49 tokens per second per user and 75.67 kW of operating power, against a GB300 configuration of 4 GPUs at 3,054.0 tokens per second, 288.65 tokens per second per user and 7.50 kW. At a matched concurrency of 1,024, Vera Rubin with 20 GPUs reported 204,207.7 tokens per second and 307.36 tokens per second per user at 63.06 kW, while the 4-GPU GB300 entry reported 49,701.8 tokens per second and 56.65 tokens per second per user at 7.50 kW. The Qwen3.6-35B-A3B entries follow a similar pattern, with Vera Rubin at 12.61 kW reaching 47,785.5 tokens per second at a minimum of 410 tokens per second per user versus GB300 at 7.50 kW and 5,960.4 tokens per second, and 501.0 versus 154.5 tokens per second per user at matched concurrency of 256. NVIDIA describes the aggregate as 5x to 10x higher throughput per kilowatt and 3x to 5x higher end-to-end interactivity at matched concurrency across the three models, depending on which configuration and comparison point is used.
One methodological detail worth noting is that the throughput-per-kilowatt curves are plotted as step functions across at least seven concurrency levels per curve, with no interpolation between measured points. In practice that means a reader comparing systems can only compare at operating points both platforms actually demonstrated, which tends to make the comparisons more conservative than a smoothed curve would. The trade-off a reader should keep in mind is that these are efficiency ratios at specific operating points: the absolute configurations differ in GPU count and total power, so per-kilowatt gains do not automatically translate into a fixed-fleet capacity gain without accounting for the hardware involved.
On the hardware side, NVIDIA attributes the results to Rubin GPUs combining HBM4 memory, Tensor Cores and sixth-generation NVLink. HBM4 supplies bandwidth for model weights and conversation state, Tensor Cores handle matrix math, and NVLink carries inter-GPU communication. The software stack is arguably as important for agentic traffic. Prefix caching and state reuse avoid recomputing context that has already been processed, Dynamo routing distributes requests across workers, and disaggregated serving lets prompt processing (prefill) and token generation (decode) be tuned separately instead of sharing one compromise configuration. Model-specific work is also cited: a MegaMoE kernel for DeepSeek and multi-token prediction with Tensor Core verification for Qwen, both aimed at reducing data movement and the number of sequential decoding steps.
This is a sensible direction for the workload. Agentic sessions repeatedly resend a long, mostly unchanged history plus a small amount of new material, so caching and routing can matter more than raw per-token speed. A likely trade-off is that such gains depend heavily on how much context is genuinely reusable; workloads with short or highly varied prompts would see less benefit from prefix caching, though the benchmark's dataset is built around multi-turn coding and shared enterprise instructions, which is where cache hits are most likely.
Why it matters: Teams building agent products are increasingly constrained by cost per served user at a fixed responsiveness target, which is exactly the axis these results emphasize. If the provisional figures hold through MLCommons verification (an inference, not a confirmed outcome), operators could serve substantially more concurrent agent sessions within the same power envelope on Vera Rubin than on GB300. Developers evaluating infrastructure should watch for the verified results and for how the caching and disaggregated serving techniques perform on their own prompt distributions rather than the benchmark's.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Agentic workloads are demanding more from inference systems. AI agents carry growing conversation histories across turns and generate a mix of reasoning, tool...