AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Making Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One

Collected Nov 2, 2026

NVIDIA introduced a Preview recipe family for the Vera Rubin NVL72 platform inside its AI Cluster Runtime (AICR), a component of the NVIDIA DSX AI Factory Platform. The recipes capture a validated Kubernetes configuration for the new rack-scale hardware, generate an ordered deployment bundle for standard tooling, and validate a running cluster against the expected state. The stated aim is to have Kubernetes environments ready for Vera Rubin NVL72 compute trays from day one, and NVIDIA reports that the training and inference variants passed deployment and conformance checks on reference clusters, with training also clearing an initial performance gate.

The recipe encodes specific versions rather than general guidance. On the driver side it assumes a host-installed R615.23 open-kernel-module driver, so the GPU Operator runs with driver installation disabled while the container toolkit stays operator-managed. The DRA driver is v0.5.0 with ComputeDomains enabled for IMEX-backed multi-node NVLink, while whole-GPU allocation still goes through the device plugin. The base OS is Ubuntu 26.04 on a Vera-optimized arm64 64k-page kernel, and Kubernetes must be at least 1.34.1: 1.34 for the GA resource.k8s.io/v1 DRA API that ComputeDomains rely on, and the patch release because the RKE2 CDI/NRI toolkit path needs containerd 2.1, shipped as v1.34.1+rke2r1. Smaller sharp edges are baked in too, such as raising the memory limit on the NFD topology updater, whose 60Mi default gets OOM-killed on multi-terabyte NVL72 workers.

Validation ran with the official GPU Operator v26.7.0 and official DRA driver v0.5.0 releases on Vera Rubin clusters. Deployment conformance covered four validators in the deployment phase and eight in conformance, all passing. The performance phase ran one validator: a cross-node NCCL all-reduce over the NVLink fabric through an IMEX ComputeDomain, gated on a bus-bandwidth floor, which achieved 744.71 GB/s bus bandwidth at 16 GiB. The inference variant, which adds the Dynamo platform, an inference gateway, gang scheduling and DCGM metrics, passed the same deployment and conformance matrix on the same cluster. NVIDIA notes the first training recipe now carries this NCCL performance gate but that inference serving performance is deferred.

One detail from the validation is instructive for anyone tracking early hardware bring-up. The first validation rounds were blocked by a hardware or fabric issue on an early cluster, unrelated to the GPU Operator or DRA driver, that kept one compute tray out of the NVLink fabric; after the hardware was reseated, all GPUs on both nodes joined a single clean fabric clique. Separately, an unmapped Rubin architecture value in go-nvlib crash-looped the DRA driver's kubelet plugin; the fix propagated through the stack and shipped in the official DRA driver v0.5.0 release.

For teams running GB200 NVL72 today, several pieces change. Vera Rubin NVL72 is the third generation of the MGX rack-scale architecture, and it replaces ConnectX-7 NICs and BlueField-3 DPUs with ConnectX-9 SuperNICs and BlueField-4 DPUs. DOCA 3.3.0 is the first GA release supporting ConnectX-9, and the validated system image ships DOCA-OFED 3.3.0, though NVIDIA frames that as the testbed version rather than a requirement. Network Operator compatibility is described as a separate qualification step. The device topology also differs: each tray has two CPUs and four GPUs as before, with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on node 1, but Vera Rubin trays expose eight ConnectX-9 PCI device groups with 32 functions versus four ConnectX-7 adapters. That means GPU-to-NIC affinity and NUMA placement need rechecking for I/O-heavy work such as RDMA networking and GPUDirect Storage. Rubin GPUs also advertise new MIG profiles under R615.23, though the AICR recipes use whole-GPU allocation and do not configure MIG; that path is outside this validation.

The workflow has four steps. Generating a recipe is an offline operation that resolves criteria against a catalog and takes seconds; the command shown uses rke2 as the service, vr200 as the accelerator, ubuntu as the OS and training as the intent, producing a recipe with 14 components and five overlays. Rendering the bundle emits 61 files, organized as numbered per-component Helm chart directories plus a single deploy script and checksums. Snapshotting captures the cluster's actual state, after which validation runs in phases as in-cluster jobs: readiness constraints are checked against the snapshot, then deployment, conformance and performance validators run in turn. The whole sequence of recipe, bundle, snapshot and the deployment and conformance phases completes in about five minutes on a live cluster, with the NCCL job adding its own runtime.

A few operational caveats come with the setup. Host-managed IMEX and DRA-managed ComputeDomains are mutually exclusive, so on images with host-managed IMEX the static nvidia-imex service must be stopped, disabled and masked. On fresh installs with GPU Operator v26.7.0 or later, the chart ships its own stricter ComputeDomain CRD requiring numNodes with no default, so manifests must set it explicitly; existing clusters upgrading the operator are unaffected because Helm will not overwrite an installed CRD. The apply-to-running-pods test with a two-node ComputeDomain, each pod getting its own IMEX channel injected, took about 12 seconds.

Inference timings are not yet gated for a concrete reason. vLLM merged Rubin support into v0.27.0, but the kernel path is gated at build time on CUDA 13.4 or newer, and vLLM's published images through v0.28.0 still build against CUDA 13.0.3. ai-dynamo's latest release pins vLLM 0.26.0, which predates the code, and its unreleased main branch pins v0.28.0, which is not built against 13.4. NVIDIA's conclusion is that a Vera Rubin inference performance floor needs a published runtime image built against CUDA 13.4 or newer and verified on real hardware. This suggests inference users should expect to wait for rebuilt container images before trusting out-of-the-box throughput numbers on Rubin.

My reading is that the most transferable idea here is the recipe as a versioned artifact: pinning the driver mode, kernel page size, DRA API floor and component set together, then verifying a live cluster against it, turns bring-up knowledge into something two clusters can reproduce a day apart. That matters because mismatches between drivers, kernels, runtimes and schedulers are exactly where new hardware stalls. The Preview status is a real limit, since it does not carry production support or lifecycle qualification while the platform is pre-release, and MIG, RDMA data-path performance over ConnectX-9 and inference serving performance all sit outside the validated boundary.

Why it matters: Platform engineers planning Vera Rubin NVL72 clusters get an explicit, reproducible baseline instead of rediscovering driver, kernel and DRA combinations, and teams on GB200 NVL72 can see which NIC, topology and DOCA changes to budget for. The Preview label, the deferred inference gate and the unpublished CUDA 13.4-built runtimes suggest developers should treat this as an early-adopter path for evaluation rather than a production commitment, an inference drawn from the stated status and version pins. The public signed validation evidence lets teams compare their own clusters against NVIDIA's results before committing.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Configuring and optimizing Kubernetes for new GPU hardware such as NVIDIA Vera Rubin is complex and error-prone. Drivers, kernels, runtimes, networking,...