Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS

AWS has published a reference implementation for serving NVIDIA Nemotron 3.5 Super VL, an open multimodal model, on Amazon Elastic Kubernetes Service, Amazon EKS Auto Mode and Amazon SageMaker HyperPod EKS. The model joins the Nemotron 3 Nano, Nano Omni, Super, Ultra and Nemotron 3.5 Lightning models already offered on AWS, and the walkthrough lives in the aws-do-eks project, covering prerequisites, cluster creation, weight download, deployment, testing, benchmarking and cleanup.
Nemotron 3.5 Super VL is a 120B-parameter mixture-of-experts model that activates 12B parameters per token. It accepts text, images, documents and video, and produces text only. It is distilled from Nemotron 3 Ultra, and NVIDIA publishes the weights, training datasets and post-training recipes, so teams can fine-tune it for domain work and run it on infrastructure they control. Context length reaches up to 1M tokens, and checkpoints ship in BF16 and NVFP4 precision. NVIDIA reports 63.44 percent on Hermes Bench, 87.05 percent on LangChain agent evaluations and 80.8 percent on the CharXiv chart-reasoning benchmark.
The architecture interleaves Mamba-2, attention and expert layers, with a RADIO vision encoder for visual input. Sparse routing means compute per token resembles a far smaller model, but GPU memory must still hold the full weight set, which is what drives the GPU counts below. Mamba-2 layers keep a constant-size state during generation rather than a KV cache that grows with sequence length, a design choice that makes long contexts cheaper to serve. A recursively shared Multi-Token Prediction head enables native speculative decoding.
Intended workloads are agentic and visual: validating code and diagnosing errors, verifying sources returned by search tools, extracting summaries from tables, charts and page layouts, vulnerability triage, and video search and incident review. NVIDIA provides reference implementations, cookbooks and its Agent Toolkit to connect the model to open agent frameworks and orchestration systems.
Minimum GPU configurations reported for vLLM are: 8 B200 or GB200 GPUs in BF16 and 2 in NVFP4; 4 B300 or GB300 in BF16 and 1 in NVFP4; 8 H200 in BF16 and 4 in NVFP4; and 16 H100 in BF16 and 8 in NVFP4. NVFP4 roughly halves or quarters the hardware footprint, which is the practical lever for cost per replica.
The reference stack runs the same Kubernetes manifests across all three orchestrator options, so switching between plain EKS, Auto Mode and HyperPod EKS does not require rewriting deployment specs. Each cluster pairs a control plane running the Kubernetes API with a data plane of CPU and GPU nodes; CPU nodes host system and test pods while GPU nodes serve the model. Container images sit in Amazon Elastic Container Registry, model weights on a shared Amazon FSx for Lustre volume, and Elastic Fabric Adapter links the GPU nodes for fast interconnect.
NVIDIA Dynamo handles serving, either aggregated on a single node or disaggregated with prefill and decode on separate nodes. For a vision-language model the image and video encoding stage can also be split onto its own node, which lets each stage scale independently. This reflects a broader industry pattern in which prefill, compute-bound and latency-sensitive to prompt length, is separated from decode, memory-bandwidth-bound and sensitive to batch size.
The setup has been validated on p6-b200.48xlarge instances with eight B200 GPUs and on p5en.48xlarge with eight H200 GPUs. An NCCL all-reduce micro-benchmark checks the cluster and its network: seeing utilization on every GPU and traffic on every EFA adapter during the test confirms the fabric is healthy. Performance is measured with NVIDIA's aiperf tool, which reports time to first token, inter-token latency and throughput in tokens per second across concurrency levels.
A published concurrency sweep shows lower time to first token and roughly equal inter-token latency for disaggregated versus aggregated deployment on two nodes. The suggested method is to draw a latency budget as horizontal lines, for example TTFT under 600 ms and ITL under 100 ms per token, then find the rightmost point below both lines. In the example that point is concurrency 32, about 320 tokens per second per replica; dividing a target throughput by that figure gives the number of prefill-decode pairs to deploy. Because the two metrics are plotted separately, the chart also indicates which side to grow: rising inter-token latency points to adding decode capacity, rising time to first token to adding prefill capacity. Re-running the sweep after changing precision, tensor or pipeline parallelism, aggregation mode, or a Dynamo or vLLM release makes each trade-off visible.
In practice, the 120B total weight set combined with a 1M-token context window makes memory, not compute, the binding constraint for most teams, which is why the NVFP4 checkpoints matter so much. A likely trade-off is that lower-precision serving buys smaller clusters and lower cost at some risk to answer quality, so teams should benchmark on their own tasks rather than assume parity with BF16.
Why it matters: Teams building agents that must read screenshots, PDFs, charts or video can now self-host a single open multimodal model on AWS Kubernetes infrastructure instead of stitching together a separate vision model and text model. The published weights, datasets and recipes mean domain tuning and data residency stay in the customer's account, which matters for regulated workloads such as security triage. The reference manifests, validation and benchmarking steps lower the effort to get from cluster creation to a measured throughput number, though operators still own capacity planning and SLO tuning. Whether disaggregated serving beats aggregated deployment at a given scale remains something each team should confirm with its own sweep.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter mixture-of-experts model that pairs reasoning and tool use with multimodal understanding. This post explores its architecture and use cases, then shows how to deploy it on Amazon EKS, EKS Auto Mode, and Amazon SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM.