Build real-time voice applications with vLLM-Omni on SageMaker AI – Part 1

AWS published Part 1 of a tutorial series showing how to deploy the Qwen3-TTS text-to-speech model on Amazon SageMaker AI using the AWS vLLM-Omni Deep Learning Container (DLC). The workflow streams text into the model and receives 24 kHz PCM audio chunks back over a single persistent bidirectional connection, exposed through a Gradio application.
SageMaker bidirectional streaming provides a full-duplex WebSocket transported over HTTP/2. The client connects to the SageMaker Runtime endpoint on port 8443, and a SageMaker inference sidecar forwards the connection to the vLLM-Omni native WebSocket route inside the container. The sample uses the v1/audio/speech/stream route. The vLLM-Omni v1.5 DLC adds the bidirectional streaming capability label and the routing middleware required by SageMaker AI.
The sample lives in 03-features/bidirectional-streaming-vLLM-Omni in the AWS sagemaker-genai-hosting-examples repository. Deployment uses an endpoint configuration with SageMaker instance pools: SageMaker first tries ml.g6.xlarge, then falls back to ml.g6e.xlarge, ml.g5.xlarge, or ml.g4dn.xlarge. It provisions one instance, not one per pool, and validates quota for every configured pool entry when creating the endpoint, so quota is needed for each included type. The --instance-types flag restricts the pool.
Prerequisites include an AWS account with configured CLI or SDK credentials, Git, Python 3.12 or newer, a SageMaker AI execution role, permission to create and invoke SageMaker AI endpoints, endpoint quota for at least one pooled instance type, Boto3 1.40.0 or newer, and the SageMaker Runtime HTTP/2 Python client 0.4.0. The walkthrough uses the US East (N. Virginia) Region.
vLLM-Omni extends vLLM beyond text generation to models that process or generate text, audio, images, and video, using a heterogeneous pipeline abstraction that coordinates multi-stage workflows including autoregressive and diffusion stages, plus streaming outputs and OpenAI-compatible APIs. Supported models include unified omni models, ASR, TTS, audio generation, image generation, and video generation.
Part 1 covers streamed speech; AWS says Part 2 will apply the vLLM-Omni DLC to image and video generation. The post follows an earlier one that streamed microphone audio to the Voxtral-Mini-4B Realtime speech-to-text model, covering the input side of a voice pipeline; orchestration between the two endpoints is outside the walkthrough.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Deploy a text-to-speech model on Amazon SageMaker AI with the AWS vLLM-Omni Deep Learning Container and stream generated speech over a persistent bidirectional connection. This Part 1 tutorial deploys Qwen3-TTS and streams speech through a Gradio application.