AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

Collected Oct 27, 2026

NVIDIA published a developer blog post describing memory-enforced GPU sharing on Kubernetes, built around a new open-source Kubernetes operator called gpu-fractioning. The operator is hosted under the KAI Scheduler organization on GitHub and runs alongside the NVIDIA GPU Operator, which supplies the driver and container runtime setup. The enforcement capability itself comes from CUDA 13.4, and the first scheduler integrated with the operator is KAI Scheduler, starting at version v0.18.0.

The underlying problem is one of utilization. Many AI workloads only need a fraction of a GPU, so handing each one a whole device leaves capacity idle. NVIDIA GPUs already offer several sharing approaches. Multi-instance GPU (MIG) partitions a GPU into hardware-isolated instances that are transparent to the application. Time-slicing lets several processes take turns without memory enforcement. Multi-Process Service (MPS) lets processes run concurrently in separate address spaces, a technique the post calls sm-sharing, after the streaming multiprocessors it divides.

CUDA 13.4 adds a memory boundary that the application does not have to know about, and pairs it with a new time-sharing mode alongside the existing sm-sharing mode. A workload can ask for an exact memory amount, in which case an allocation above that size fails just as it would on a physically smaller GPU. Alternatively it can request a range: a guaranteed minimum plus a maximum it may grow into when memory is free. Each workload sees only its fraction of GPU memory, and each can choose a compute mode rather than accepting the default.

CUDA alone does not make this usable from Kubernetes. Something on every GPU node has to run and supervise the dependent daemons, apply each Pod container's memory limit and compute mode before the process starts, and report consumption per fraction rather than per whole GPU. That node-level role belongs to gpu-fractioning regardless of which scheduler placed the workload. The operator does three things: it sets up and supervises MPS on every GPU node, it applies each workload's memory limits and compute mode, and it measures memory and compute use per fraction.

The division of labor is straightforward. A scheduler picks the GPUs for a workload and records the memory allocation and compute-sharing mode in per-container Pod annotations. The operator reads those annotations and configures the assigned devices accordingly. Because the contract is just annotations, any Kubernetes scheduler able to write them can enforce GPU fractions with CUDA. The scheduler keeps its own responsibilities: choosing a GPU with enough free memory and never mixing compute modes on a single device.

KAI Scheduler's new sharing mode is called NvFractions, and it can be turned on at install time with a Helm flag: helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler, with global.nvFractions.set=true. Doing so also installs gpu-fractioning as a dependency. KAI Scheduler's other sharing modes remain available, and administrators select one per cluster.

Workloads then describe their fractional needs through four annotations, which answer how much memory is needed, how far the workload may grow, how compute is shared, and how many GPUs it spans. Memory can be requested as an absolute size using the per-container NVIDIA annotation and standard Kubernetes units; the container name is part of the key, so the setting targets the right container. Alternatively it can be a portion of a GPU through KAI Scheduler's existing gpu-fraction annotation, for example 0.5.

Burstable limits are new in KAI Scheduler. A workload declares a guaranteed minimum as its request and a maximum it may burst into when memory is free. The limit annotation depends on how the request was written, and it is optional: if omitted, the limit equals the request and the workload is capped at exactly what it asked for. A portion limit must be greater than the gpu-fraction value and less than 1.

Compute mode is now explicit per container, set to either sm-sharing or time-slicing. With n workloads sharing a GPU, each gets roughly 1/n of the compute either way; the difference is whether they take turns or run together. Time-slicing is serial and is the default: workloads take turns on the whole GPU, unused time is shared equally among the others, and each workload gets its own error domain, meaning full error isolation. SM-sharing is parallel: the streaming multiprocessors are split between workloads running simultaneously, with nothing carved out in advance. The hardware allocates and releases SMs as needed, so a quiet workload's SMs go to its neighbors and a single active workload can use them all. The cost is that SM-sharing workloads share one error domain.

Because a shared GPU runs only one compute mode at a time, KAI Scheduler never places a time-slicing workload and an sm-sharing workload on the same device. It tracks this per GPU rather than per node, so a single node can host both modes on different GPUs. Multi-device fractions are requested with gpu-fraction-num-devices; the request applies to each device, so a 40Gi request with four devices yields 40Gi on each of four GPUs.

NVIDIA's examples illustrate the trade-offs. An inference server spanning four GPUs might ask for a guaranteed 40Gi on each, a burst ceiling of 60Gi, and sm-sharing so it runs alongside neighbors instead of waiting its turn; the post cautions that sm-sharing puts workloads in one error domain, so it suits workloads trusted to run together. A development or batch job such as notebooks or small fine-tuning runs might instead take a guaranteed half of two GPUs, a limit of 0.8, and the default time-slicing so a crash in one job does not affect its neighbors; running nvidia-smi inside the container should show two GPUs, each with that workload's fractional share.

Why it matters: platform teams running Kubernetes clusters can now bin-pack workloads onto GPUs with enforced memory ceilings rather than trust, which in practice should mean more workloads per device without admission-by-convention. The annotation-based contract keeps the operator scheduler-agnostic, so teams invested in schedulers other than KAI could adopt the enforcement layer without switching, though that integration would be their own work. The error-domain distinction between time-slicing and sm-sharing is the key design decision to weigh: isolation versus concurrency. Both projects are open source and available now on GitHub, with admin and user guides covering all three sharing mechanisms.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads...