AivexaNewsSearch
AI news for builders and product teamsChecked every hour

GKE Pod Snapshots Cut Model Load Times, and Move the Work to Snapshot Lifecycle Management

Collected Sep 30, 2026

Google has published benchmark results for GKE Pod snapshots, reporting startup latency reductions of as much as 89%. According to the benchmarks, a 70B parameter model loads in 37 seconds and an 8B model in 15 seconds. The feature saves the running state of a workload, including CPU and GPU memory, and restores it on demand. It reached general availability in May on clusters running version 1.35.3-gke.1234000 or later.

Pods must run in GKE Sandbox because that is where the gVisor runtime lives. Autopilot clusters already have it; standard clusters need a node pool with gVisor enabled. An agent on each node handles the snapshot lifecycle, a control plane controller clears obsolete snapshots, and Cloud Storage holds the data. Configuration uses two custom resources: PodSnapshotStorageConfig points at the bucket, while PodSnapshotPolicy selects Pods by label, sets the trigger to workload or manual, and sets retention with a lastAccessTimeout and a cap on snapshots per group.

Google's customer example is Codeway. Its Retake platform had a custom caching layer for compiled artifacts that got startup down to a minute. Lead DevOps engineer Ahmet Furkan Çomak said Pod snapshots cut that to "just 8 seconds."

Practitioner reaction focused on what happens after capture. Responding to a LinkedIn analysis by Suresh Rajashekaraiah, senior DevOps and MLOps engineer Mohana Narasimha G. wrote that the restore path is compelling but that snapshot invalidation will likely be the harder platform problem than capture itself.

Google's documentation states that GKE builds a hash from the Pod's essential runtime fields, the distilled Pod spec, embeds it in the snapshot, and requires a restoring Pod to produce an identical hash. The target node must have an identical machine series and CPU architecture, and the gVisor kernel version and GPU driver version must match those captured. Where no compatible snapshot exists, the Pod starts normally. A rootfs-only scope skips the hash comparison and can cross machine families, including to E2.

The documentation also states that encryption keys and certificates created before a snapshot must be re-created afterward, that environment variables are read from /proc/gvisor/spec_environ, that external connections are terminated on restore, that persistent volumes are not checkpointed, and that user-added iptables or nftables rules and routes are not restored. Whole-pod snapshots do not work on E2 machine types, multi-GPU Pods are supported only on L4 GPUs, and Multi-Instance GPU sharing is not supported.

Read at InfoQ · AI, ML & Data Engineering

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Google has published benchmarks for GKE Pod snapshots, reporting up to 89% lower startup latency and a 70B model loading in 37 seconds. The feature checkpoints CPU and GPU memory through gVisor into Cloud Storage. Practitioners have asked whether invalidation is the harder problem, since snapshots match on a spec hash, machine series, and kernel and driver versions. By Steef-Jan Wiggers