Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

NVIDIA's developer blog published a step-by-step guide to configuring, certifying and operating Kubernetes GPU clusters using three open-source components that sit under the company's DSX OS layer, part of the DSX AI Factory Platform. All three — NVIDIA AI Cluster Runtime (AICR), NVIDIA Cluster Readiness Engine (NVCRE) and NVSentinel — are released under Apache 2.0 and accept external contributions. The premise is a familiar operational failure: a single underperforming GPU can drag a synchronous distributed training job down to its slowest rank, even while the node passes basic checks and reports normal utilization. The guide follows one cluster across its lifecycle, from recipe-driven build to pre-production certification, live monitoring and recertification after repair.
AICR is the configuration piece. Operators describe a target environment — the example uses Amazon EKS, H100 accelerators, Ubuntu, an inference intent and the Dynamo platform — and AICR resolves it against a library of validated overlays to emit a recipe file. Each component entry pins a version, names a Helm chart and source, and declares dependencyRefs, an explicit dependency graph. AICR resolves that graph at render time, so ordering is baked into the artifacts a deployer applies; there is no hand-numbered install sequence to update when components are added. Entries also carry health assertions that run after installation. The guide points out that Kubernetes can report a pod stuck in CrashLoopBackOff as Running, so shipped assertions filter on container waiting reason rather than pod phase alone.
A second command renders the recipe for existing tooling: --deployer accepts helm, argocd, argocd-helm, flux or helmfile. Because the baseline is a file, teams can apply code review, version history and drift detection to it. Two commands support that: aicr validate checks a live cluster against recipe constraints, and aicr diff compares a golden baseline against a live cluster or config map snapshot, with a --fail-on-drift flag that yields a non-zero exit and turns drift detection into a CI gate.
NVCRE handles certification. Basic health checks confirm services are up; they do not confirm that a multinode job will run across the interconnect without stalling. NVCRE runs training jobs and NCCL tests, driven by a Certification resource that selects target nodes (the example uses a nodeSelector on nvidia.com/gpu.present) and names test categories. A report shows per-category status, runtime, measured bandwidth and failure reasons. The catalog covers NCCL communication variants, DCGM level 4 diagnostics, and pretraining workloads built on NeMo Framework with 8-billion- and 56-billion-parameter models; a WorkloadRun resource lets teams turn their own containerized test into a check.
Thresholds are the notable design choice. None are configured by default: without them a run reports measurements and fails only when tests cannot start, so a node performing below target can still pass. The example adds a bus bandwidth threshold of 300 GB/s. This is what catches degradation rather than outright breakage — a GPU running just under its target may produce no Xid errors, breach no monitoring threshold and look normal in telemetry. NVCRE also bisects failures: when a 64-node all-reduce shows poor bandwidth, it splits failing groups and reruns until a minimum group size, then names suspect nodes with reasons.
NVSentinel is the continuous layer, running alongside workloads. It watches signals the cluster already produces — DCGM health checks, Xid errors, NIC states and cloud maintenance events — and the guide states it consumes no GPU time. It installs in monitoring mode by default, reporting faults as node conditions without cordoning, evicting or remediating, so operators can evaluate its reports on a live cluster before enabling quarantine. Behavior is governed by Fault Quarantine rulesets written in CEL expressions in the values file; a matching rule can cordon a node and apply a taint such as nvidia.com/gpu-error with NoSchedule effect. A circuit breaker caps how many GPU nodes can be quarantined in a sliding window — an important safeguard, since an aggressive ruleset on a shared fabric could otherwise take out much of a cluster at once. A health events analyzer looks at a node's accumulated failure history so repeatedly failing nodes become eligible for investigation, and events marked CONTACT_SUPPORT require manual or vendor intervention.
The closing piece is the loop between NVCRE and NVSentinel. Two lifecycle events can trigger revalidation automatically when configured: a node joining the cluster, and a node returning after remediation. The lifecycle controller creates a ValidationRequest, gates scheduling with a cordon or taint, runs NVCRE tests and removes the gate only if they pass, so a node that still fails cannot silently rejoin. In-place changes — driver upgrades, operator upgrades, fabric changes — do not trigger validation automatically; there the recommended pattern is to let aicr diff --fail-on-drift fail a pipeline stage and then raise a ValidationRequest for the named nodes. The division of responsibility is deliberate: NVCRE stops at attribution, recording which nodes failed and why, while NVSentinel owns quarantine and repair, to avoid two systems double-cordoning the same node. That boundary also lets platform teams substitute their own node lifecycle tooling against the same certification results.
Why it matters: teams running large GPU fleets on Kubernetes tend to assemble this kind of configuration pinning, burn-in testing and fault handling themselves, and the guide's own framing suggests the expensive failures are silent performance regressions rather than crashes. Adopting these tools means picking up declarative recipes and CEL-based remediation policies, plus the operational discipline of writing thresholds and enabling quarantine only once reports look trustworthy. The open-source, Apache 2.0 licensing lowers the adoption barrier, though it is a fair inference that the value depends on how closely a team's stack matches the supported deployers, runtimes and test catalog.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to...