When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving

NVIDIA published a developer blog post on encode-prefill-decode (EPD) disaggregation, an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill and decode stages. According to the post, EPD disaggregation with NVIDIA Dynamo delivers up to 5x faster time to first token and 7x faster end-to-end response time, and is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models.

Dynamo, described as an open source inference framework for serving AI models in distributed environments, implements EPD by separating encoder and PD worker roles without fixing hardware placement. Encoder workers produce vision embeddings, and PD workers consume them and run the LLM. The post lists three placement topologies: aggregated serving, colocated encoder workers sharing GPUs with prefill-decode workers, and disaggregated encoder workers on a separate lower-cost GPU tier connected through the NVIDIA Inference Transfer Library (NIXL). In the test environment, two NVIDIA RTX 6000D GPUs ran encoder workers while four NVIDIA GB200 GPUs ran PD workers.

The post states benefits depend on input media load, output sequence length, model size and precision, and traffic mix, and that gains shrink when decode dominates latency or when large dense models reduce the vision encoder's share of compute. In one image-heavy test of ten images per request at OSL 1024, TTFT dropped 58% with colocated encoder and 50% with heterogeneous placement, while the heterogeneous tier served 70% more traffic at the same latency SLO. In mixed text and multimodal traffic, EPD reduced mean TTFT for text requests by 42.2% and for image requests by 30.8%. Quantizing LLM weights to NVFP4 while keeping the vision encoder in BF16 increased colocated EPD goodput from 1.78x to 2.64x over aggregated serving.

Suggested next steps include reproducing the experiments via the ai-dynamo/dynamo GitHub guide, enabling parallel media decoding, using embedding cache, and applying multimodal KV routing. The post notes vLLM and SGLang have roadmaps for further development of their EPD stacks.