AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Validating GPU Platforms with Runtime AI Inventory via the NVIDIA DSX AI Factory

Collected Nov 11, 2026

NVIDIA's AI Cluster Runtime (AICR), an open source modular component of the NVIDIA DSX AI Factory Platform, now includes a qualified integration of k8s-aibom, an AI bill of materials (AIBOM) project originally created by Google, the company said in a developer blog post. Two stock AICR recipes select the component by default: h100-gke-cos-inference and h100-gke-cos-inference-dynamo. Other GKE recipes can opt in at generation time with --runtime-inventory enabled, and recipes that include it can opt back out with --runtime-inventory disabled.

The integration addresses a gap between platform configuration and what is actually running. AICR itself converts known-good GPU platform configurations into version-locked recipes and reproducible deployment bundles. Each recipe records the desired setup for one combination of hardware, cloud, OS and intent: its components, version pins, values, constraints, deployment ordering and validation expectations. Bundles carry checksums, and signing is optional through aicr bundle --attest. But as the post explains, application teams define and update workloads independently of the platform recipe, so a reproducible bundle alone cannot establish which workload artifacts are running on each node. Mutable image tags such as :latest compound the problem, since two nodes can run different code under the same tag.

k8s-aibom works higher in the stack and later in the cluster lifecycle. It watches Kubernetes workload and pod APIs and emits a CycloneDX 1.6 AIBOM for each recognized AI workload. Coverage is deliberately bounded: only supported workload kinds carrying a recognized AI signal are inventoried. That includes native Dynamo graphs, NIMService, LeaderWorkerSet, DisaggregatedSet and CronJob resources, and tracked child workloads are folded into their owner's AIBOM. The stated purposes include finding unregistered or shadow AI workloads in opted-in namespaces, identifying models, inference runtimes, agents and RAG components, tracking external AI APIs and vector databases, detecting drift in AI dependencies, and supporting security reviews, incident response and compliance audits.

The blog frames the two projects as handling opposite ends of one governance problem. AICR records qualified deployment intent and checks a cluster against it at a point in time; k8s-aibom observes supported workloads and records what it found, including how confidently each attribute was determined. According to the post, neither view is sufficient on its own.

The qualification is presented as a specific, tested claim rather than a formality. AICR binds the exact upstream source, chart, CRDs and digest-pinned controller image as one tested set, verifies provenance against Google's expected build identity, reviews rendered RBAC and security defaults, and exercises installation, upgrade, health and uninstall on the deployment paths it tested.

The worked examples target AICR v1.2.0 and its catalog, which pins k8s-aibom 1.6.2; the post advises using a CLI build matching that catalog. The starting point is a GKE cluster on Kubernetes 1.32 or later with H100 nodes running Container-Optimized OS and GKE-managed drivers and device plugins. The workflow is four steps: generate a recipe (aicr recipe with --service gke --accelerator h100 --os cos --intent inference), render a bundle (aicr bundle, with node selectors), verify and deploy (aicr verify followed by the generated deploy.sh), then validate (aicr snapshot, then aicr validate with the deployment and conformance phases). Resolved values can be inspected without writing a file; the post shows aicr query returning 1.6.2 for the k8s-aibom version. Bundles are one directory per component, ordered by dependency graph, each with values, checksums and an install script. Deployers include helmfile, flux and argocd, selected with --deployer.

After deployment the controller runs in the k8s-aibom-system namespace. AICR uses the main chart's CRDs rather than the companion k8s-aibom-crds chart; Helm bundles include a CRD apply step, Flux and Argo CD handle updates through their controllers, while Helmfile, Terraform and Fleet require the documented manual CRD update. The post shows a kubectl query returning v1beta1 as the storage version, and notes that a matching storage version does not prove the full CRD schema is current; during upgrades teams should also check the target schema and watch for Degraded=True with reason SchemaPredatesController. The deployment phase's health check requires the controller deployment to be available and the cluster-scoped AIBOMControllerConfig to report Ready=True.

Inventory is opt-in per namespace via the label aibom.k8saibom.dev/enabled=true. Notably, the label controls which namespaces emit AIBOM records but does not narrow what the controller reads: informers still cache workload and pod specs cluster-wide, including container arguments and inline environment values. No sink and no secret access are configured by default, so nothing leaves the cluster without an explicit decision. This is a sensible caution for platform teams weighing the integration, since cluster-wide caching of pod specs is a different exposure profile from per-namespace reporting.

The demonstration uses a vLLM deployment serving Qwen/Qwen2.5-0.5B-Instruct, pulling the mutable tag vllm/vllm-openai:latest. After applying it, kubectl get aiboms lists a record named apps-deployment-vllm-qwen with category inference, runtime vllm and Ready=True, with the controller classifying category and runtime automatically. One boundary the post stresses: the record is built from the workload spec, so an AIBOM can appear and report Ready=True while the pod is still pulling its image. It describes what was declared, and once a pod exists what it resolved to, but does not prove scheduling or execution.

The document is stored inline on the resource when small enough, with a published SHA-256 over the canonical bytes, which can be recomputed and compared with shasum. The decoded CycloneDX document in the example lists components including the application vllm, the container image vllm/vllm-openai and the machine-learning-model Qwen/Qwen2.5-0.5B-Instruct. Attributes carry one of four confidence tiers (declared, inferred, unresolved or verified) plus an evidence locator. Put plainly, an SBOM tells you which image is running while an AIBOM says which model and runtime are serving it, with a record of how each claim was reached.

Resource impact figures come from upstream measurements of k8s-aibom v1.6.1 on GKE with NVIDIA Dynamo and AICR's configuration, run on CPU-only nodes with workload pods left pending, so they describe controller inventory overhead: roughly four API-server requests per minute during a 15-minute steady-state window, average controller CPU of 2.2 millicores, a memory working set of 28 to 34 MiB, and one AIBOM regeneration when the tested workload's image tag changed. Those numbers are small, which suggests the integration is cheap to run alongside existing platform components, though the pending-pod setup means the figures do not capture behavior under real GPU scheduling churn.

The post explicitly flags what the integration does not do. Runtime observations are unsigned, so they are not tamper-evident; signing would add that but still would not prove every declaration true or establish successful execution. k8s-aibom can verify Sigstore signatures on model claims against operator-configured trust roots, enforce signer-identity constraints, check Rekor inclusion and assign the verified tier, but AICR leaves this verification disabled by default. Connecting AICR's transparency-logged bundle attestations to runtime AI inventory, and correlating recipe intent with observed workloads via shared artifact identities, freshness rules and infrastructure-to-workload mapping, are described as future work. An AIBOM also does not establish successful execution or complete cluster conformance.

Both projects are open source: AICR at NVIDIA/aicr and k8s-aibom at GoogleCloudPlatform/k8s-aibom.

Why it matters: Platform teams running GPU workloads on Kubernetes gain a way to reconcile a reproducible, validated platform configuration with an actual inventory of recognized AI workloads, including models and runtimes, at low measured overhead. For developers, the practical changes are namespace opt-in labels, a new AIBOM resource to query, and evidence tiers that show how each attribute was determined. The remaining gaps, notably disabled-by-default signature verification and unsigned observations, mean teams should treat AIBOM output as inventory rather than proof of execution, in my reading, and plan upgrades around CRD schema checks.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

As teams deploy and scale AI workloads using NVIDIA Dynamo, Kubeflow, or Slurm, tracking what actually runs requires visibility beyond the platform...