Build a Validated Kubernetes Runtime for NVIDIA Dynamo

NVIDIA's developer blog lays out a validated approach for running its Dynamo distributed inference framework on Kubernetes, built around NVIDIA AI Cluster Runtime (AICR), a component of the NVIDIA DSX AI Factory Platform. The post walks through the runtime layer that sits beneath a DynamoGraphDeployment — the resource that specifies how vLLM, SGLang or NVIDIA TensorRT LLM run in pods — and explains how AICR packages Dynamo plus its dependencies into recipes with pinned component versions, compatibility constraints and validation checks. AICR version 1.0.0 is referenced for the Amazon EKS Dynamo inference recipe targeting NVIDIA GB200 NVL72 and GB300 NVL72 systems.
The stated problem is that a model graph is only the top of the stack. A production inference cluster also needs a GPU Operator and driver stack matching the nodes, labeled and discoverable GPU nodes, GPUs exposed through the expected Dynamic Resource Allocation driver, a scheduler able to place GPU workers with required topology and gang scheduling semantics, node health signals before workloads land on degraded nodes, reachable Prometheus metrics for Dynamo and accelerators, and a configured Gateway API / Endpoint Picker path when ingress should be Kubernetes-native. AICR encodes these requirements in its recipes rather than leaving them as tribal knowledge.
The recipe ships the dynamo-platform chart pinned to version 1.4.2, which bundles the Dynamo operator, runtime images and the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition. That v1beta1 version replaces the earlier v1alpha1 API, so teams with existing manifests should expect an apiVersion change to work through. Recipes also record configuration values, node selectors, tolerations, checksums and validation expectations.
A notable architectural change: with Dynamo 1.4, frontends, workers, optional Gateway API Inference Extension / Endpoint Picker components and other services discover each other through Kubernetes resources. AICR's default path no longer uses NATS, and the dynamo-platform chart no longer installs it by default; NATS remains an opt-in for deployments that need it.
Dynamo's central claim to efficiency is cache-aware placement. Agentic workloads keep extending long token prefixes — system prompts, tool definitions, conversation history and tool results — and a turn that lands on a worker without that prefix in GPU memory forces reprocessing of intermediate tokens. Dynamo instead routes to workers already holding the relevant KV blocks. It also supports disaggregated serving, splitting prompt processing (prefill) from token generation (decode) so each pool can be scaled and tuned independently; on a cache miss, prefill must reprocess all or part of the prompt before decode takes over.
In Dynamo 1.4.2 on the AICR Kubernetes path, requests travel over TCP while KV-cache events travel over ZMQ. Each vLLM worker engine emits raw cache events through a local ZMQ publisher, configured roughly via --enable-prefix-caching with a kv-transfer-config enabling kv_cache_events and a zmq publisher endpoint on tcp://*:5557. Dynamo's worker runtime subscribes, normalizes those events and republishes them on the Dynamo event plane. Both routing paths consume that Dynamo-managed state.
Path 1 is the Dynamo Frontend router: a Service exposes the Frontend, which with DYN_ROUTER_MODE=kv tokenizes each request, scores eligible workers on prefix overlap and current load, picks one and dispatches over the request plane. The post calls this the smallest topology and the most production-proven serving option, suited to model-specific endpoints or when an existing edge proxy simply forwards to the Frontend Service.
Path 2 integrates the Gateway API and Gateway API Inference Extension. An agentgateway resolves an HTTPRoute to an InferencePool that the Dynamo operator creates from the DynamoGraphDeployment, with a selector for worker pods and an endpointPickerRef for the Dynamo EPP Service. The gateway calls the EPP, which tokenizes and uses Dynamo's KV router to choose an endpoint, then returns the selection and routing metadata to the gateway, which proxies the request to a Frontend sidecar on the chosen worker pod running --router-mode direct. This suits platform teams that manage shared ingress and want timeouts, retries and fallback handled by Gateway API mechanisms. In 1.4, this path uses GAIE's Go EPP framework with Dynamo's Rust KV router embedded through CGO and configured via eppConfig.
Looking ahead, the post states that starting with Dynamo 1.5 a native Rust EPP replaces the Go/CGO bridge, implementing GAIE's Lightweight Endpoint Picker (ext_proc) interface directly, running the full router in-process, and using DYN_* environment variables instead of the plugin and scheduling profiles. Aggregated and disaggregated routing are handled from worker roles registered through Dynamo discovery, the Kubernetes topology stays the same, and removing the Go framework and CGO boundary should cut per-request EPP overhead. My reading is that this is the more consequential change for platform teams: it simplifies the plugin surface but shifts configuration conventions.
The post also contrasts live cache events with prediction. Dynamo can be configured to predict KV state from where it sent requests, but prediction cannot see evictions, so the router's view can drift and routing quality can degrade under memory pressure. The AICR configuration here enables live events for both paths.
The five-step flow separates runtime setup from application deployment. First, aicr snapshot captures cluster state to snapshot.yaml; without placement flags it targets nodes labeled nvidia.com/gpu.present=true and tolerates GPU-node taints. Second, aicr recipe with --intent inference --platform dynamo reads the snapshot to identify the Kubernetes distribution, accelerator and GPU-node OS, then emits pinned versions and checks. Third, aicr bundle renders Helm values and manifests with node selectors, tolerations, a deployer choice (helmfile, helm, argocd, argocd-helm or flux) and optional settings such as agentgateway allowedSourceRanges; a --storage-class is needed when no default StorageClass exists. Fourth, deploy: helmfile diff then helmfile apply processes dependency tiers in order and applies independent components concurrently, with reruns reconciling releases, while the default Helm deployer generates a deploy.sh fallback. Fifth, aicr validate runs deployment and conformance phases against the live cluster before the application DynamoGraphDeployment is installed; performance validation is separate because it spins up a temporary serving workload and consumes the GPUs under test. An offline aicr validate --no-cluster run checks declarative constraints against a different snapshot and reports behavioral checks as skipped.
Why it matters: Platform teams running multi-node inference on Kubernetes are the immediate audience, since AICR turns driver, DRA, scheduling, networking and metrics prerequisites into reproducible, version-pinned artifacts that can be validated before workloads land. For developers, the payoff is fewer lost days debugging cluster prerequisites and a clearer choice between the leaner Dynamo Frontend router and a Gateway API/EPP ingress path — with the Router mode and EPP wiring as the key knobs to design around. Teams should also plan for the v1alpha1-to-v1beta1 CRD move and, per the stated roadmap, the Dynamo 1.5 shift to a native Rust EPP if they build on the eppConfig plugin today — both inferences from the published plans rather than guarantees.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference...