Netflix Treats GPU Reliability as a Distributed Systems Problem

Netflix has published an account of how it approaches reliability for large-scale GPU training and inference, describing a platform built on Kubernetes that detects failing hardware, removes it from service automatically and lets long-running jobs recover when machines are lost. The write-up, by Craig Risi for InfoQ, reports that the effort has produced more than 99.95% GPU-node availability and raised job success from approximately 98% to more than 99.5%.
The stakes are set by how synchronous training works. A job can run for weeks across hundreds of GPUs with every worker advancing in lockstep, so one failed accelerator, a degraded network adapter or a single slow machine can stall or throttle the whole run. Netflix's framing is that this turns GPU reliability into a distributed systems problem rather than a matter of keeping individual servers powered on.
A core observation is that ordinary cloud health checks are insufficient. A virtual machine can be running and reachable while its GPUs are unfit for distributed work. Netflix found accelerators that answered status queries normally but failed once memory was allocated or sustained computation started. Network interfaces show a similar pattern: an adapter can be present yet carry fewer links than expected, so collective operations still complete but at heavily reduced bandwidth.
To catch this, device-level checks use NVIDIA DCGM to watch for GPU errors, memory problems, interconnect state and thermal conditions, while networking checks inspect the state and count of high-speed Elastic Fabric Adapter interfaces on AWS. A node agent inside Kubernetes translates those results into node conditions that controllers can act on. The checks are built to fail closed: if a diagnostic tool hangs, a timeout marks the machine unhealthy rather than letting a broken check report success, and kernel logs serve as a fallback when the normal GPU diagnostic path itself fails. That design choice is notable in practice, because an unreliable health check that defaults to healthy is worse than none at all in a system that schedules work automatically.
Remediation is staged rather than immediate. An initial, possibly transient problem can block new work from landing on a node while existing workloads keep running. If the condition persists, the machine is drained and its workloads moved elsewhere. Depending on the fault, the response can escalate from killing a problematic job or resetting a GPU to rebooting or replacing the whole machine. This hooks into Karpenter's node auto-repair, with Netflix adding policies that set how long different failures are tolerated and how much of the cluster may be disrupted at once, which is essentially a blast-radius control for automated repair.
Netflix also assumes some failures will always get through, and focuses on making them cheap to recover from. Large jobs use checkpointing and elastic workers so training can resume from the last healthy checkpoint after a machine disappears, and the platform retries failed work automatically. Application errors are classified separately from infrastructure errors so that jobs with genuine software bugs are not restarted indefinitely.
The job itself is monitored as a distributed system too. Beyond looking for dead machines, the platform can identify a single slow rank that is holding back the rest of a synchronous job, which matters because throughput can collapse without any GPU actually failing. For inference, Netflix uses NVIDIA Multi-Instance GPU to let smaller workloads share a physical GPU with memory-level isolation between tenants, improving utilisation compared with dedicating a whole device to each workload. It is also working toward Kubernetes Dynamic Resource Allocation, which would allow more expressive resource requests and topology-aware placement, including positioning GPUs, network adapters and CPUs sensibly relative to one another inside a machine.
The comparison drawn is with Meta, which also pairs hardware diagnostics, automated remediation, checkpointing and fast recovery at large scale, with recent work centred on detecting hardware faults including silent data corruption across very large AI clusters. Meta has reported that hardware failures accounted for more than 66% of training interruptions in its Llama 3 experience. The distinction offered is one of emphasis: Netflix concentrates on closing the operational loop from pre-scheduling validation through in-flight monitoring, fault-domain identification, automatic remediation and measurement of resulting goodput, while Meta has invested heavily in hardware-level fault detection and cluster maintenance.
Why it matters: teams running multi-node training or inference on rented GPUs face the same failure modes, and the reported jump from roughly 98% to over 99.5% job success suggests that treating health checks, remediation policy and checkpointing as one system pays off in usable capacity rather than just uptime figures. The specific stack here is Kubernetes, DCGM, Karpenter and Elastic Fabric Adapter on AWS, so portability to other schedulers and clouds is an open question in practice. A likely inference is that smaller clusters gain less from elaborate tiered remediation, but any job long enough to span a hardware fault will need the checkpoint-and-retry path regardless.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Netflix has detailed how it is building a reliability platform for large-scale GPU training and inference, using layered health checks, automated remediation, and fault-tolerant training to keep jobs running despite hardware failures. By Craig Risi