Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

AWS's machine learning blog published a walkthrough of a recovery loop for silent data corruption (SDC) in distributed PyTorch training on Amazon SageMaker HyperPod. The premise: a GPU can return a wrong but plausible number with no error raised, the bad value flows through gradients into every replica's weights, and training loss can look normal for hours while compute is wasted. The post cites Meta's Llama 3 405B pre-training over a 54-day window on up to 16K H100 GPUs, where 6 of 419 unexpected job interruptions were attributed to silent data corruption, counting only cases that interrupted the job and were traced to SDC.
HyperPod's existing protections, the health-monitoring agent, deep health checks and automatic node recovery, all key off hardware reporting a fault. SDC by definition produces no report, so detection has to come from inside the training job. The proposed design adds three checks to a DistributedDataParallel loop, then uses Kubernetes and HyperPod to map an alarm to an instance and swap it out.
The checks are arranged around where in a step a fault can land. In data parallelism each replica sees a different data slice, so local gradients legitimately differ, but after the all-reduce every replica applies the same averaged gradient, and after optimizer.step() all weights must match bit for bit. A wrong local gradient gets smeared across all replicas by the all-reduce and leaves them in agreement, so it must be caught before the update. Corruption during the update or in stored weights breaks that agreement and can be caught after it.
Check one is a non-finite gradient test using a DDP communication hook. Before the all-reduce, it records each rank's squared gradient norm and flags ranks whose norm is NaN or Inf, catching overflow in a GPU's forward or backward pass on the step it happens. After the all-reduce, the reduced gradient is tested again, because a NaN produced inside the reduction reaches every replica identically and would otherwise pass the weight comparison; clip_grad_norm_ already computes that norm, so the extra test costs one single-element all-reduce per step. Teams training in fp16 with a gradient scaler should exclude overflow steps the scaler already skips; bf16 has none.
Check two is a ring replay audit. On audit steps, between backward and the optimizer step, each rank recomputes a peer's micro-batch, eight positions ahead (one node away on p5 instances), with the unwrapped module so nothing is all-reduced, restoring gradients, buffers and RNG state afterwards. It compares the loss bit for bit and, when deterministic kernels are enabled, per-parameter gradient fingerprints. Because a replay blocks the whole job regardless of how many ranks participate, auditing every rank costs the same wall clock as auditing one. Each rank then shows up in two comparisons; a mismatch in both points to that rank. A single mismatch is ambiguous, settled by a re-replay on a third node that can also distinguish a persistent from a transient fault. Topology checks verify the three GPUs sit on three different nodes before training starts.
Two audit modes exist. With default kernels only the loss is compared, which caught forward-pass faults and reproduced healthy losses exactly in AWS's tests; gradients cannot be compared because attention's default backward pass is not deterministic. With deterministic kernels and the math attention kernel, gradient fingerprints match too and backward-only faults are caught, at a cost of 2.4 times a normal step on the test model.
Check three compares weight fingerprints after the optimizer step and before every checkpoint. Each tensor is reduced to two 64-bit sums of its raw bit patterns, one plain and one weighted by position, and ranks are gathered and compared. A rank is named only when its copy differs from a strict majority; with two copies, or a split with no strict majority, the tool reports disagreement without naming anyone. AWS notes that a plain sum alone missed a test case where two weight values were swapped and two others changed to compensate, which the position-weighted sum caught. For tensors under 2^32 elements of up to 4 bytes, any change to one or two elements alters the fingerprint, but large coordinated changes could in principle cancel, so equal fingerprints are strong evidence rather than proof.
The post also documents a failed first attempt: flagging ranks whose local gradient norm deviates from the step median by k median absolute deviations. Recording three fault-free runs of 22,800 steps on 32 H100s training a 953M-parameter model, healthy local norms reached 224 MADs in the first 800 steps and 1,240 overall; a threshold of 336 produced 11 false alarms in 22,000 healthy steps, and the worst faulty rank scored at most 6.9 even with one activation multiplied by 10,000. Comparing finite wrong values requires recomputation, hence the replay.
The operational side needs an EKS-orchestrated HyperPod cluster with at least three GPU nodes, since both the ring audit and the majority vote need three copies. Checkpoints belong on shared storage such as FSx for Lustre or S3, because node replacement destroys root and secondary volumes including local NVMe. The Kubernetes node name contains the instance ID, making rank-to-node mapping a lookup; the job then stops, the node is cordoned, GPU metrics are reviewed in the HyperPod observability dashboards, and the node is replaced via the BatchReplaceClusterNodes permission, with StartClusterHealthCheck available for on-demand diagnostics. Advanced DCGM metrics add ECC, NVLink and throttling counters. Correlated telemetry corroborates an alarm but clean telemetry does not clear a node, and repeated alarms can be tracked as a signal of hardware worth retiring early. Resuming from a checkpoint saved before the suspect window avoids restarting from poisoned weights.
In practice, the three checks are the only layers that see truly silent faults; the hardware-facing layers either confirm, triage or localize. The cost is periodic replay overhead, an all-gather per audit and per checkpoint fingerprint pass, and a design that assumes identical replicas, so tensor, pipeline or context parallelism needs separate handling. AWS points to its GitHub repository for the full code.
Why it matters: Teams running long GPU training jobs on HyperPod now have a concrete pattern, with code, for naming a suspect GPU and replacing its node rather than debugging an unexplained loss curve. The trade-offs are real: deterministic audit steps ran 2.4 times slower on the test model, and the weight check gives no verdict when only two replicas disagree. For platform teams, the implication, and this is inference, is that a small slice of audit-step overhead plus checkpoint hygiene may be cheaper than restarting a multi-day run from a poisoned checkpoint.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.