Distributed training with Ray Train DLCs and EFA on EKS

AWS published a walkthrough of distributed PyTorch training on Amazon EKS that uses its Ray Train Deep Learning Container exactly as shipped, with no image build at any point. The sample fine-tunes Qwen2.5-1.5B with PyTorch Fully Sharded Data Parallel (FSDP) across eight NVIDIA L4 GPUs on two g6.12xlarge instances, and the whole solution lives in a sample repository that reuses the same KubeRay-on-EKS foundation as an earlier multi-node Ray Serve post.
The motivating problem is packaging, not model code. A training container has to align the CUDA runtime, a PyTorch build compiled against it, NCCL, the Elastic Fabric Adapter userspace libraries, the AWS OFI NCCL plugin that connects NCCL to EFA, and a Ray version the cluster operator tolerates. AWS assembles that stack, tests it as a unit, publishes it to Amazon ECR Public and keeps it patched.
The image referenced is public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1. Inside: Ray 2.58.0 with Train, Tune and Data; PyTorch 2.13.0 and torchvision 0.28.0; CUDA 13.0.2 with cuDNN and NCCL 2.29.7; the EFA installer, libfabric and aws-ofi-nccl built against that NCCL, plus GDRCopy; DeepSpeed, flash-attn, Transformer Engine, PyTorch Lightning and the Hugging Face libraries; and all_reduce_perf already on the path for diagnostics.
The networking layer is the one AWS frames as riskiest to own. A mismatch there produces no error: NCCL quietly falls back to TCP sockets and the job finishes at a fraction of the intended speed. AWS says each release is gated on multi-node EFA and multi-node KubeRay training tests, which are the same paths the sample exercises.
Several conveniences come for free. The image is public, so nodes pull it without credentials. On startup the entrypoint compares the host's NVIDIA driver with the image's CUDA and, if the driver is older, enables bundled CUDA forward-compatibility libraries. One tag serves both head and worker because the container has no fixed role; whatever starts it supplies the ray start command. Ray Train shares the ray ECR repository with the Ray Serve and Ray LLM containers under the train-ml prefix.
Version handling deserves care. Tags like v1.1 move as security rebuilds land. Fully qualified vX.Y.Z tags stay frozen at the original release build, so pinning a digest is the way to guarantee a byte-identical run. The same image is also reachable through private ECR URIs.
Kubernetes has no native concept of a Ray cluster, so KubeRay, an open source operator, adds a RayCluster resource. One manifest describes a head group and a worker group, and KubeRay creates the pods, starts Ray in the right role in each, points workers at the head address and replaces failed pods. The Ray head holds the global control store, the dashboard and the job server; workers supply GPUs and run training processes.
In the sample's layout, a single m7i.xlarge system node hosts kube-system add-ons, the KubeRay operator and the head, which advertises no GPUs and serves the job server on port 8265 and the control store on 6379. Jobs are submitted from inside the head pod via kubectl exec through the EKS API, so 8265 never leaves the cluster. Two g6.12xlarge instances, each with four L4 GPUs, 48 vCPUs, 192 GiB of memory, 40 Gbps networking and one EFA interface, run one worker pod apiece claiming all four GPUs and the EFA device; ranks 0–3 land on one instance and 4–7 on the other. Because EFA traffic does not cross Availability Zones, the scripts pin both instances to a single zone verified to offer the instance type. All nodes sit in private subnets and reach the internet only through a NAT Gateway, which is how they pull the image and download weights from Hugging Face.
Configuration is deliberately external. KubeRay's rayStartParams gives each pod group its role, with the head set to num-gpus: "0". GPU and EFA counts come from Kubernetes resource requests, here nvidia.com/gpu: 4 and vpc.amazonaws.com/efa: 1; requesting the EFA resource is what makes the device plugin mount the interface, without which NCCL never sees EFA. Transport is set by four environment variables (FI_PROVIDER=efa, NCCL_NET_PLUGIN=ofi, NCCL_SOCKET_IFNAME=eth0 and NCCL_DEBUG), which a job can override through Ray's runtime_env. Extra Python packages also arrive via runtime_env at submit time, so no new tag is ever needed for a library the image lacks.
Ray Train removes launcher code. Instead of starting a process per GPU per host with hand-assigned ranks, world size and rendezvous addresses, you write a training function and hand it to TorchTrainer with a ScalingConfig. Ray starts one worker per GPU, forms the torch.distributed process group and runs the function on each. The sample's function asks for FSDP through prepare_model with a size-based auto-wrap policy that makes any submodule above 10 million parameters its own FSDP unit, keeping the script model-agnostic.
FSDP shards weights, gradients and optimizer state across all eight GPUs instead of replicating them. Qwen2.5-1.5B in bfloat16 needs roughly 12 GB for those tensors, so eight-way sharding puts about 1.5 GB on each L4. The cost is communication: every step gathers each layer's weights and reduce-scatters gradients, so the full model crosses between ranks, and that inter-instance traffic is what the EFA stack carries. The driver joins the cluster with ray.init(address="auto"), reads the GPU count from cluster resources and sizes the trainer accordingly, so the same script runs on one node or many. Batches are random token IDs, making the run a pure test of whether real FSDP collectives crossed both nodes and the job completed; a real fine-tune swaps in a Ray Data or DataLoader pipeline without changing the function.
Prerequisites are AWS CLI v2 signed in, eksctl, kubectl, Helm and envsubst, plus 96 vCPUs of G-family On-Demand quota in a Region where some zone offers g6.12xlarge, or an existing Capacity Reservation via CAPACITY_RESERVATION_ID. The stated budget is about $9.50 an hour while the GPU instances are up, covering both instances, the EKS control plane and the NAT Gateway. This suggests idle clusters are expensive to leave running.
Bringing it up starts with scripts/env.sh, holding cluster name, Region, Kubernetes version, node types and counts, the image, pinned KubeRay and device plugin versions, and training parameters. The five scripts are idempotent. deploy_all.sh begins with find_gpu_capacity.sh, which checks zone availability, quota headroom and Spot placement scores per candidate Region and, with the default probe flag, confirms capacity by creating and immediately canceling a real Capacity Reservation. It shows a plan and asks once before chaining the cluster, node group, GPU plugin and KubeRay installs, which take 20 to 30 minutes on a fresh Region. Chosen Region and zone are saved to scripts/.placement.env for later commands. The GPU node group gets labels role=gpu-worker and nvidia.com/gpu.present=true; AWS notes that omitting the second means the NVIDIA device plugin DaemonSet runs on zero nodes with no error pointing at why.
Training runs separately unless you pass --train. deploy_ray_train_job.sh renders the manifest with envsubst, re-reads each GPU node's allocatable resources before waiting on pods so a missing device plugin fails clearly instead of hanging, copies code/ to the head and submits ray job submit from inside, streaming logs from all eight ranks. It then checks which transport NCCL actually used and exits nonzero if the job failed or EFA was bypassed, which makes it usable as a CI gate.
Two manifest details keep multi-node runs stable. Setting requests equal to limits puts workers in Kubernetes' Guaranteed QoS class, which matters because one throttled or evicted rank stalls the other seven. And num-gpus is derived from the same variable as the GPU request, so Ray's view of capacity matches Kubernetes'.
Why it matters: teams that previously spent weeks maintaining a compatible CUDA, PyTorch, NCCL and EFA image now inherit one from a public registry and express differences as environment variables, resource requests and runtime packages. The check that NCCL actually used EFA is the detail to steal even if you never adopt the container, since silent fallback to sockets is the most common way multi-node training underperforms without anyone noticing. My inference is that the split between infrastructure deployment and repeated job submission is the pattern most teams will copy, and that the roughly $9.50 hourly cost means ephemeral clusters rather than persistent ones are the sensible default.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
The Ray Train Deep Learning Container ships the full CUDA-through-EFA stack, so you build no image. This post walks through a reproducible sample that fine-tunes Qwen2.5-1.5B with PyTorch FSDP across eight GPUs on two EFA-enabled g6.12xlarge instances on Amazon EKS, using the container exactly as published.