AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

Collected Oct 19, 2026

AWS engineers have contributed a push-based KV transfer connector to vLLM, the open-source LLM inference and serving engine, and published benchmark results showing lower latency for disaggregated prefill/decode deployments. The work was built and validated on AWS Trainium clusters connected with Elastic Fabric Adapter (EFA) and then contributed to vLLM as an accelerator-agnostic feature; the reported numbers come from a GPU cluster using Amazon EC2 P5en instances with EFA. On a one-prefill-one-decode (1P1D) deployment serving Qwen3-32B, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30 percent better time-per-output-token (TPOT) than the default pull mode. The authors note results depend on workload and configuration.

Prefill-decode disaggregation splits inference into two stages run on separate engine instances: prefill processes the prompt and builds the KV cache, and decode uses that cache to generate tokens one at a time. Splitting them lets each stage scale independently, but the KV cache must move from prefill to decode before generation starts, so transfer efficiency sits directly on the TTFT critical path. vLLM moves these blocks through the NVIDIA Inference Xfer Library (NIXL) connector over RDMA-capable interconnects such as AWS EFA, which supports GPUDirect RDMA and Neuron Direct RDMA on supported instance types. That lets KV blocks move directly between GPU or Trainium device memory without staging through host memory.

The default behaviour is pull-based. After prefill finishes, the router relays source metadata, including prefill block IDs, to the decode instance; the decode side then allocates destination blocks, prepares the NIXL transfer and issues an NIXL READ. Those steps run sequentially after prefill and before decoding, lengthening TTFT. Push mode inverts the coordination. When the decode instance allocates local blocks for a request that needs remote prefill, it registers those destinations with the prefill instance as a NIXL notification within the same engine step, so the destination is known before prefill completes. The prefill side then pairs the registration with its own prefill completion, whichever arrives second, and issues an NIXL WRITE directly into the decode instance's registered memory. Request matching is by ID and accounts for reissued, preempted or retried requests.

A dedicated background transfer thread on each prefill worker handles connection setup or reuse, maps block IDs to memory descriptors, merges contiguous blocks into larger transfer regions and submits the write, keeping that per-request work off the main forward-pass loop. The producer and consumer can use different internal block layouts and block sizes, since each side maps locations to its own physical layout at transfer time. Each write carries a completion notification, one per producer tensor-parallel rank writing into the decoder; decoding begins only after all expected notifications arrive.

In the benchmark, prefix caching was disabled with --no-enable-prefix-caching so every request pulled a full KV cache from the prefill node rather than reusing local cache. Input lengths swept from 1k to 16k and request rates from 1 to 8 queries per second, with 128 output tokens. At low load and short prompts the two modes were nearly identical. Under load, pull mode degraded sharply, reaching about 1,744 ms TTFT at 8 QPS with 16k inputs, while push mode stayed near 540 ms, roughly a 3x reduction. TPOT followed the same shape: around 5.6–5.8 ms for both at light load, pull climbing to about 10.4 ms at 4–8 QPS and 16k inputs, push holding near 7 ms. The stated reason is that pull mode posts NIXL descriptors inline and synchronously on the decode worker, stalling token generation, while push moves that work to the background thread.

The routing trade-off is worth flagging for anyone planning a deployment. Pull mode lets the router pick a decode instance after prefill finishes, which gives more room for load-aware routing and quality-of-service management. Push mode commits to the decode instance earlier so destination setup can overlap with prefill; in exchange, the prefill instance can release KV blocks sooner once both the prompt KV and the registration are ready. In practice, teams optimising purely for latency should test push; teams with highly variable load or bidirectional KV needs may still prefer pull.

To enable it, configure both the prefill and decode servers to use NixlPushConnector instead of NixlConnector. Both connectors ship in vLLM, so the choice is per deployment. Reproduction instructions for the 1P1D setup are published in a separate guide covering environment setup, launching the prefill and decode servers, running the benchmark and collecting results. The team also says it is exploring bringing the background NIXL threading model to the pull-mode connector so pull workloads could gain similar off-critical-path setup, and has opened a pull request to investigate.

Why it matters: Teams running disaggregated vLLM serving at high request rates or long contexts are the most affected, since that is where the reported gap between push and pull widens. The change is a configuration swap rather than a fork, so it can be evaluated without architectural change, though the gains are unverified outside the authors' 1P1D P5en benchmark and may not transfer to other hardware or topologies. A likely trade-off, inferred from the routing discussion, is reduced flexibility in placing decode instances, which matters for schedulers that depend on post-prefill load information.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.