AivexaNewsSearch
AI news for builders and product teamsChecked every hour

SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Collected Nov 6, 2026

Amazon SageMaker Inference now supports GPU partitioning built on NVIDIA Multi-Instance GPU (MIG), AWS announced in a post on its machine learning blog. MIG carves a single physical accelerator into independent slices, each with a fixed share of compute units and a dedicated block of memory, with the isolation enforced in hardware rather than by software scheduling. AWS frames this as removing the long-standing rule that the smallest unit of GPU you can request is one whole accelerator.

The mechanics are worth understanding. An A100 or H100 divides into sevenths on compute and eighths on memory; the G7 family carrying an NVIDIA RTX PRO 4500 Blackwell (32 GB) divides into halves, and G7e with an RTX PRO 6000 Blackwell Server Edition (96 GB) into quarters. Profile names read as compute units then memory, so mig-3g.40gb takes three compute units and 40 GB. No accelerator exceeds seven partitions. A quirk to note: the 3g profile takes three of seven compute units but four of eight memory units, so two of them fill a device and nothing else fits alongside.

The feature rides on inference components, the SageMaker construct for hosting multiple models on one endpoint with per-model compute settings. Two optional fields are added to existing API calls, with no new operations, resource types or IAM actions. AcceleratorPartitionLayouts sits on a production variant in CreateEndpointConfig and declares how every accelerator in that variant is divided; the same layout applies uniformly across all instances, and the list is not positional. AcceleratorPartitionType sits on an inference component's ComputeResourceRequirements and picks which partition a model lands in. With a partition type set, NumberOfAcceleratorDevicesRequired at 1 means one partition rather than one whole device, while the CPU core and host memory fields keep their usual meaning.

Raising CopyCount places additional replicas of a component on free partitions of the requested type, which suits a small model under high concurrency where a single process leaves the GPU idle between requests. Invocation is unchanged: InvokeEndpoint still addresses a model by inference component name, and the partition never appears in requests. Carving happens during provisioning, adding roughly 17 seconds in AWS's testing; at InService the accelerators are already split and partitions start empty. Single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned, and models needing more than one GPU still require whole devices. Layouts are fixed at endpoint creation, and the compute and memory units in a layout are validated against the selected GPU, with violations rejected outright rather than falling back to a whole accelerator.

There is a P4d driver caveat. The mig-1g.10gb profile needs an R525 or newer NVIDIA driver, while the default P4d image ships 470.256.02; requesting that profile there fails without an error naming it. Setting InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 on the variant moves to the 580 driver line.

AWS also offers guidance on choosing a profile. Compute units govern prefill, the compute-heavy phase that reads the prompt, so they matter more as input length and arrival rate grow. Memory units govern decode, which is bounded by memory bandwidth, and capacity, since memory caps KV cache and therefore context length and concurrent sequences. The practical advice is to budget memory for three things, not one: weights (about 16 GB for an 8B model at FP16), the KV cache sized for intended context and concurrency, and 1–2 GB of runtime overhead for the CUDA context and serving process. AWS cites an 8B FP16 model comfortably serving an 8,000-token context on a 40 GB partition.

Salesforce helped define requirements and validated the initial implementation, according to the post, and is preparing to validate it against a representative production workload. Richard Chang, a software engineering architect at Salesforce, is quoted saying many inference workloads do not need a whole GPU and that MIG would let the company consolidate workloads safely, improve utilization, reduce inference costs and keep performance predictable.

AWS ran a benchmark comparing one mig-3g.40gb slice of an H100 against a whole uncarved A100-40GB, chosen because their memory is comparable. Both served Llama-3.1-8B-Instruct on real-time endpoints with identical container, tokenizer and client configuration, driven by genai-bench across five traffic shapes and concurrency from 2 to 512. The partition produced 53–89 percent more output tokens per second in every scenario at every concurrency, with 8.9–42.1 percent lower inter-token latency and 5.6–47.5 percent lower time to first token. Cost per million output tokens was 18–34 percent lower above a crossover that arrives earlier as prompts lengthen: concurrency 64 for 100-token prompts, 4 for 8,000-token prompts. Below that point the whole A100 was cheaper. At the top of the ladder on 6,000- and 8,000-token traffic, the A100 failed a third to half of requests against a serving-path deadline while the partition failed none. AWS flags these as on-demand prices in US West (Oregon) at the time of analysis: $25.251286 per hour for an ml.p4d.24xlarge and $63.296000 for an ml.p5.48xlarge.

My reading is that the most useful framing AWS offers is to treat each profile as a GPU type in its own right, sitting alongside whole accelerators in a cost comparison. That reframes instance selection: moving to a newer, larger instance stops being all-or-nothing, but it also multiplies the sizing decisions, and a layout sized only for weights will load and then degrade under concurrency. The isolation claim deserves care too. AWS itself notes that models on one instance still share host CPU, system memory, storage, networking and endpoint routing, so this is a GPU-level boundary, not complete tenant separation. That distinction will matter to teams reading it as a compliance control.

Why it matters: Teams running portfolios of fine-tuned variants, embedding models, rerankers or per-customer models can now bin-pack them onto fewer accelerators with hardware isolation and no cold-start load penalty, without operating a Kubernetes MIG stack themselves. Inference is the likely first beneficiary, since the benchmark shows a slice of a newer GPU beating a whole older one on long-input, high-concurrency traffic while losing on short or low-concurrency work. Developers should expect to redo capacity sizing around KV cache and concurrency rather than weights alone, and to check profile compatibility before moving a layout between instance families.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG), which divides one GPU into hardware-isolated partitions so you can run a different model in each or take a right-sized fraction of a newer accelerator. This post shows how two optional API fields place one model per partition, and what we measured against a whole older GPU.