When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

NVIDIA's developer blog published a walkthrough for serving GLM-5.3 NVFP4, a quantized build of Z.ai's mixture-of-experts model, on two GB300 DGX Stations tied together with ConnectX-8 RDMA. The setup exposes one OpenAI-compatible endpoint from Station A while both machines share the work. The core problem the post addresses is memory: the checkpoint needs about 464 GB, while a single GB300 GPU carries 252 GB of HBM3e.
The model itself has 753 billion parameters with 40 billion activated per token. Sparse activation cuts compute, but because each token can be routed to different experts, every weight has to stay resident. That is why the recipe disables CPU weight offload, and it is the reason a single machine no longer suffices for a frontier-scale model. The quantized checkpoint uses NVFP4 with a mixed-precision recipe: the tutorial notes that quantization targets MoE-block linear weights and activations rather than every tensor, and a validation step asserts that the config declares MIXED_PRECISION and the GlmMoeDsaForCausalLM architecture.
The mechanism is tensor parallelism with TP=2. vLLM splits the model's tensors across the two GB300 GPUs, so each rank holds and computes roughly half the weights; the startup log reports about 218 GiB of weights per GPU. The two ranks communicate over a chosen RDMA rail, in the example cx8r1 with device mlx5_1, rather than falling back to TCP. Because neither GPU has a complete copy, every token requires an exchange of partial results across the rail — a failure there is a failure of the service.
Prerequisites are specific. Both Stations run Linux and should be wired and validated first with NVIDIA's Connect Two DGX Stations playbook, including its GPU-memory RDMA test on both rails. Users need SSH and sudo with an identical Linux username on each box; Docker and the NVIDIA Container Toolkit ship with the DGX Station stack. Each Station needs at least 464 GB of free disk for its own copy of the checkpoint, plus space for the container image. Licensing is two-layered: acceptance of the NVIDIA Open Model Agreement and the original GLM-5.3 terms, plus the NVIDIA Evaluation Dataset License Agreement for SPEED-Bench. Downloading evaluation prompts also requires a Hugging Face account and gated access to the cais/hle dataset.
The instructions pin the software tightly: container image vllm/vllm-openai:v0.30.0-aarch64 with a SHA-256 digest, and the model at revision e3b8f9f2fad29727dfa95e766c5cac18ea1414bf. Configuration lives in a shared ~/glm53.env file that sets rank (0 on Station A, 1 on B), head and worker IPs, rail interface and RDMA device, and the local model path. Preflight checks look for a 400000 Mb/s link, MTU 9000, successful jumbo-frame pings over the rail, synchronized time, and no compute processes on the GPU — the post frames these as ways to catch cabling and addressing mistakes before they surface as NCCL timeouts.
Station A downloads the checkpoint through hf download, then rsyncs it to B over SSH, which requires passwordless key access from A to B. The tutorial gives transfer estimates of roughly an hour at 1 Gb/s and about six minutes at 10 Gb/s for the 464 GB payload. A verification block compares file lists and sizes via sha256 on both sides, checks that 50 safetensors shards are present, and prints MODEL_READY on success. Notably, each disk keeps the full checkpoint; vLLM partitions weights at load time and no further quantization happens on the host.
The launch script runs on both machines and starts one vLLM rank per Station, with rank 1 flagged headless and no API server. Key flags include --max-model-len 98304, which caps input plus output tokens, --max-num-batched-tokens 8192, --block-size 128, --max-num-seqs 16, and --gpu-memory-utilization 0.93, leaving 93 percent of GPU memory to vLLM. The service enables multi-token prediction as speculative decoding: the model's MTP layer drafts four tokens per step and the full model verifies them, so a step can yield several tokens; the benefit depends on how often drafts are accepted and therefore varies by prompt. FP8 KV-cache storage halves cache memory versus 16-bit, making room for more tokens, and prefix caching avoids reprocessing prompt prefixes already seen. Kernel configuration selects flashinfer_cutlass as the MoE backend, and compilation config captures CUDA graphs at sizes 5, 10, 20, 40 and 80.
Environment variables tune NCCL for RoCE — GID index, HCA, timeouts of 22 and seven retries, GDR level 5, and debug logging to /tmp/nccl logs. The container comments that NCCL 2.30.7 from the image is required because older versions can deadlock on cross-node CUDA graph replays over RoCE, referencing a vLLM issue. The procedure starts B first, then A, and warns that weight loading plus kernel preparation and graph capture takes several minutes. Health is confirmed with the /health endpoint returning HEALTH_OK, a log grep confirms GPU Direct RDMA is in use with no NET/Socket or NCCL warnings, and nvidia-smi should show VLLM::Worker_TP0 on A and VLLM::Worker_TP1 on B.
The benchmark numbers are the part worth scrutinizing. On AIPerf with the SPEED-Bench Coding dataset, overall output throughput peaked at 607.8 tokens per second at concurrency 16, then dropped to 385.0 tokens per second at concurrency 32, where mean request latency reached 453.5 seconds. That shape — better aggregate throughput at moderate concurrency, worse at higher concurrency — is worth reading carefully: higher concurrency did not simply scale, and the reported configuration also sets max-num-seqs to 16. In practice, operators should expect to retune for their own prompt mix rather than assume the peak transfers. The tutorial's own next steps suggest exactly this: rerun AIPerf with custom prompts to find the right concurrency.
Why it matters: teams that want to run a frontier-scale model on their own hardware now have a concrete, version-pinned recipe instead of a conceptual claim, and it shows the real costs — roughly 464 GB of checkpoint per machine, two high-end Nodes, and careful RDMA and NCCL tuning. The latency figures at higher concurrency suggest this class of deployment suits throughput-oriented or long-running workloads more than snappy interactive chat, though that is an inference from the stated benchmark rather than a stated conclusion. Developers evaluating local serving versus hosted APIs should weigh the multi-machine operational burden against data control and per-token economics.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...