AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Autoscaling endpoints for LLM inference

Collected Oct 29, 2026

Together AI introduced autoscaling for Dedicated Model Inference on its platform. The feature lets a deployment adjust its replica count based on metrics that the inference engine exports, with users setting replica bounds, a scaling metric and target, and two timing windows that govern how quickly capacity is added or removed. Configuration is done through a single PATCH to the endpoints API, and deployments can be set to a fixed size or explicitly stopped by setting both minimum and maximum replicas to zero.

Dedicated inference bills per replica-minute, so the company frames capacity planning as a balance between two costly failure modes. Over-provisioning leaves GPUs idling at low utilization, which it calls almost never viable given GPU scarcity. Under-provisioning is worse for users: LLM serving degrades nonlinearly, and a replica at its concurrency limit begins queueing rather than slowing gracefully. Time to first token can jump from around 200 milliseconds to 15 seconds, according to the post.

Classic autoscaling assumptions break for LLM serving in two ways, the post argues. GPU utilization can read 60 percent while the request queue is already backing up, because utilization measures arithmetic intensity rather than pressure. And cold starts can take several minutes: a new replica must be placed on a GPU node, pull tens of gigabytes of weights, load them into VRAM and warm up. Scaling in response to a spike therefore arrives too late, which puts a premium on leading signals.

The control loop is proportional. Observed load is divided by the target and multiplied by the current replica count, rounded up, then dampened by the timing windows and clamped to bounds before GPU placement. For example, a target of 8 in-flight requests per replica with 16 observed implies twice the replicas, within bounds. Traffic splitting follows capacity automatically because weights are applied per ready replica.

The two windows are asymmetric by design. The scale-up window controls how long pressure must persist before replicas are added and should be kept short, since a false scale-up costs only a few replica-minutes while a missed one costs user-facing latency. The scale-down window defaults to 5 minutes and should exceed the natural rhythm of traffic, because scaling down just before a peak forces a cold start. A sample command sets bounds of 1 to 6 replicas, a 60-second up window, a 300-second down window, and a TTFT p95 target of 500.

Eight metrics are available. They fall into three families. Concurrency-driven scaling uses inflight_requests, described as the safe default because it rises before latency visibly degrades and maps directly onto how engines batch; a target of 8 means roughly eight concurrent requests per replica, and can be raised for short-prompt chat traffic or lowered for long-context coding agents. SLO-driven metrics are ttft and e2e_latency; these protect a latency promise but are trailing, so the post recommends pairing them with headroom in min_replicas. Efficiency-driven metrics are gpu_utilization and token_utilization, which optimize cost but leave little room for arrival bursts, and the post advises watching p95 closely when using them. Per-token metrics including ttft, decoding_speed and throughput_per_replica are emitted only on the streaming path, so non-streaming clients produce no data for them; e2e_latency is the exception. When multiple metrics are attached, the first in the list is used.

There is no scale-to-zero with automatic wake. Setting min_replicas to 0 is legal only alongside max_replicas 0, which stops the deployment, halts billing and returns errors to requests until an explicit restart. The post breaks a cold start into GPU placement, weight download, engine load, and warmup, each timestamped in the event feed, and notes that larger models have proportionally longer phases. It recommends this auto-stop pattern for development and staging endpoints, while keeping min_replicas at 1 for anything with a p95 SLO or unattended callers.

Two edge cases are addressed. If traffic spikes faster than a cold start, requests queue on existing replicas, TTFT rises first, and error or timeout risk follows while new capacity comes up; the only defenses are minimum-replica headroom or a higher-target policy that keeps slack, which the post calls a capacity-shape question rather than a tuning one. During a rollout, the platform pins autoscaling bounds on both deployments and restores configured bounds when the rollout completes or aborts. A sawtooth replica count is the signature of a down-window shorter than the traffic's rhythm, and widening it trades a few replica-minutes for fewer repeated cold starts.

The post reports an experiment on a Qwen3.5-9B deployment with one H100 per replica and bounds of 1 to 3, reset to a single replica before each round. The same load was replayed three times: a sine wave between roughly 12 and 48 requests per second with two spikes to 80 rps, under policies of inflight_requests target 8, ttft p95 target 300 milliseconds, and gpu_utilization target 75 percent. Only the concurrency policy scaled. Client p95 ran at 3 to 5 seconds and the system was clearly saturated, but the TTFT policy never fired because continuous batching absorbed queue pressure into total latency rather than first-token latency, and the utilization policy never fired because short bursty requests left the GPU below 75 percent. During the stretch where the in-flight policy held two replicas, its p95 sat visibly below the single-replica runs on identical load. One replica handled 12 to 48 rps at 3 to 5 seconds p95 without falling over, which the post notes would fail a 500-millisecond SLO.

Why it matters: Teams running dedicated LLM endpoints now have to choose a scaling signal rather than accept a default, and the reported experiment suggests the intuitive choices, latency thresholds and GPU utilization, can stay silent while users wait. Starting with inflight_requests at 8 and real bounds, then tuning from observed traffic, is the path the post recommends; my reading is that teams with streaming latency SLOs will still need minimum-replica headroom because no autoscaler can outrun a multi-minute cold start.

Read at Together AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

GPU utilization can read healthy while your queue backs up, and a new replica takes minutes to warm. Here's how to pick autoscaling metrics, tune scale-up/down windows, and budget for cold starts on dedicated inference.