AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Stream low-latency speech from AI agents using Kokoro TTS on Amazon SageMaker AI

Collected Oct 22, 2026

AWS published a technical walkthrough showing how to reduce the silence a caller hears from a voice agent by streaming each completed sentence to a hosted Kokoro-82M text-to-speech model on an Amazon SageMaker AI real-time endpoint as soon as the agent produces it, rather than waiting for the whole response. The post frames the problem with concrete numbers: a one-sentence reply might generate in 600 milliseconds while a full multi-paragraph answer takes around five seconds, so a batch integration leaves roughly five seconds of dead air before synthesis begins.

Kokoro-82M has 82 million parameters, which the post describes as small enough to run cheaply on one mid-tier GPU. It runs in a custom container on an ml.g5.xlarge instance and exposes two routes: POST /invocations, which streams a response for one complete input, and WS /invocations-bidirectional-stream, which holds a single connection open across many messages in both directions. Deployment is split into two CLI tools wrapping separate AWS CDK stacks. speech-infra deploy kokoro-82m provisions the endpoint, including foundation and model-cache stacks that create IAM roles and sync model weights into Amazon S3 before building the container image and creating the SageMaker model, endpoint configuration and endpoint. agent-infra deploy then provisions an Amazon Bedrock AgentCore runtime hosting a Strands agent that calls the endpoint, so the endpoint must go first.

Prerequisites include Python 3.12 or later with uv and Docker, because the underlying HTTP/2 client only ships pre-built wheels for Python 3.12 and newer; CDK v2 bootstrapped in the target account and Region; a container declaring the com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true Docker label; and a Boto3 version that supports invoke_endpoint_with_bidirectional_stream. The post also flags that the sample container pins an older PyTorch carrying a known advisory, PYSEC-2026-2286, fixed in torch 2.10.0, and says the pin should be reviewed before any non-proof-of-value use.

The first implementation trap is framing. SageMaker AI delivers every part of a client message to the container as a binary WebSocket frame, so a Starlette handler calling receive_text() raises KeyError on the missing text field for every request. The post notes this is easy to misdiagnose: a generic handler that forwards the exception string turns it into an error frame reading 'text', which looks like a missing field, while the endpoint continues to report InService. The fix is to call receive() and decode the bytes as UTF-8 when no text key is present. Because the failure produces well-formed errors and a healthy endpoint status, the post recommends testing by sending a binary frame.

The second trap is session lifetime. After sending a message's audio plus a synthesis_complete control frame, the loop should return to receive() on the same open session, closing only when the client sends an explicit close message. Closing after each response would force a new connection per sentence and undo most of the latency gain.

On the client side, an IncrementalSentenceChunker buffers incoming text fragments and splits on whitespace following a period, exclamation or question mark, returning newly completed sentences and holding back the trailing partial. A flush() call at end of input releases any leftover text that arrived without final punctuation. The post is explicit that this is not concurrent synthesis: each bidirectional handler is a strictly sequential loop, so sentence two's audio is not generated while sentence one transmits. The gain comes entirely from starting synthesis on sentence one as soon as it exists.

A live check against the endpoint with a three-sentence input over one session returned three audio chunks matching their text, measuring 73,200 bytes for 1.525 seconds, 108,000 bytes for 2.250 seconds and 98,400 bytes for 2.050 seconds.

Getting audio back to a browser is a separate decision. Most chat frontends already stream text over Server-Sent Events, a one-directional text protocol that cannot carry raw binary, so the natural move is base64-encoding audio into a JSON event. Across 24 clips, with Amazon Transcribe confirming no quality difference between encodings, base64 inflated payloads by close to 33 percent: raw PCM went from 2,396,400 to 3,195,200 bytes, MP3 at 48k from 308,304 to 411,072, and AAC 32k fMP4 from 258,039 to 344,084. The subtler cost is frame alignment: in one measured response four logical MP3 frames arrived as six delivery chunks, two ending mid-frame, so a consumer must buffer and act only on a blank-line delimiter.

The alternative multiplexes raw binary audio and JSON control frames over the same bidirectional session already carrying text input. A chunk that fails to parse as a JSON object is treated as audio and forwarded untouched. The edge case worth knowing is that raw PCM can legitimately begin with byte 0x7b, the opening brace of a JSON object, which the rule avoids misclassifying because it requires the whole chunk to parse, not just the first byte.

In the end-to-end Strands agent on AgentCore runtime, three legs connect caller, Bedrock and the Kokoro endpoint, with a mode field choosing bidi or batch. In bidi, two concurrent tasks drain the Bedrock text stream into the chunker and drain the Kokoro session into one shared queue, so events reach the caller in true arrival order. The agent-facing leg back to the caller still uses base64 because BedrockAgentCoreApp's entrypoint must yield JSON-serializable dictionaries; both modes pay that identical cost, so the measured difference is purely when the first audio chunk is emitted.

Isolating the transport, the same 267-character, four-sentence block sent as one message versus only its first sentence, five trials each, cut mean time to first audio byte from 329.3 ms to 156.7 ms, a 52.4 percent relative improvement. With a live Bedrock model behind the identical prompt, bidi mode was 2.4 times faster to first audio than batch against a local container and 3.5 times faster against the deployed AgentCore runtime. The gap widens with a real network hop, the post argues, because bidi hides that hop inside the generation-synthesis overlap while batch's serial wait absorbs all of it. A demo command, uv run agent-infra demo, reproduces the comparison. Teardown runs agent-infra destroy then speech-infra destroy; neither empties the S3 weight cache nor deletes the ECR repository, both of which keep incurring small storage charges until removed by hand.

Why it matters: teams adding voice to agents that already stream text can adopt the base64-in-SSE route with about a third more payload and a buffering step, while teams building a voice path fresh get a cheaper unified transport; the 2.4 to 3.5 times gains quoted here are measured on the author's own stack and will vary with model and network, so treating them as an upper bound is the cautious reading. The operational warnings, a healthy-looking endpoint serving only errors and a stale PyTorch pin, are the parts most likely to bite during a first deployment.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Voice agents often wait for an agent's full response before starting speech synthesis, adding latency users feel directly. This post shows how to stream each sentence to a Kokoro-82M text-to-speech model on Amazon SageMaker AI as the agent produces it, and how to get that audio back to a browser over Server-Sent Events or a unified binary transport.