How to stream Kokoro TTS on SageMaker AI

To make a Kokoro voice agent speak before it finishes generating, send each completed sentence to Kokoro-82M on an Amazon SageMaker AI real-time endpoint while the agent continues producing text. AWS’s October 22, 2026 example pairs that endpoint with a Strands agent running in Amazon Bedrock AgentCore, making this a practical pattern for developers building streaming text-to-speech (TTS) agents.

What you need before deploying

The example uses a custom container for Kokoro-82M, a model AWS identifies as having 82 million parameters, on an ml.g5.xlarge GPU instance. Its endpoint exposes GET /ping, POST /invocations for response streaming, and WS /invocations-bidirectional-stream for a persistent two-way session.

Prepare an AWS account with the required permissions, bootstrapped AWS CDK v2, Python 3.12 or later, uv, Docker, and a compatible Boto3 version. The container also needs the label for SageMaker bidirectional streaming. The implementation uses Starlette or another ASGI-compatible approach, so familiarity with WebSockets and ASGI is useful.

Deploy the endpoint, then connect the agent

1. Deploy the Kokoro endpoint first. Build the custom container and deploy the SageMaker AI real-time endpoint on the instance type used in the example, ml.g5.xlarge. The agent needs this endpoint to synthesize its text.
2. Make the WebSocket handler accept binary frames. SageMaker’s bidirectional transport delivers client messages as binary WebSocket frames. A handler that assumes text frames by calling receive_text() can raise KeyError('text') even when the endpoint reports InService. Accept bytes and decode their payload as UTF-8 where appropriate.
3. Keep one bidirectional session open for successive messages. For each synthesis request, the endpoint sends control and audio frames. The session can carry another request without reconnecting; close it only after the client sends an explicit close message.
4. Deploy the Strands agent after the endpoint. The example runs the agent in Amazon Bedrock AgentCore and uses Amazon Bedrock to generate text. As complete sentences arrive, send them to Kokoro for synthesis while the agent continues generating the rest of its response.

Start audio early without synthesizing sentences concurrently

The client-side chunker emits a sentence once it is complete and flushes any remaining text when generation ends. That lets synthesis of the first sentence start before the full response is ready. The endpoint still processes synthesis messages sequentially; later sentences are not synthesized concurrently with the first.

Check binary handling and session behavior

An InService status alone does not exercise the bidirectional message path. Test the handler with a binary frame, then send another synthesis message on the same session and close it explicitly. The expected message sequence for each request is a synthesis_start control frame, raw PCM audio, and a synthesis_complete control frame.

Choose how to send audio to the browser

If a frontend already consumes Server-Sent Events (SSE), base64-encoded audio can fit that pipeline. Base64 expands payloads by about one-third, and transport chunks can split audio frames, so the client needs to buffer and reassemble them. In the AWS example, the browser-facing AgentCore leg uses base64 because the runtime returns JSON-serializable dictionaries. On the bidirectional model leg, raw audio can be multiplexed with JSON control frames. Treat a chunk as a control frame only when the entire chunk parses as a JSON object.

AWS measured these payload sizes across 24 clips:

Other ways to serve Kokoro

Trelis Research demonstrates Kokoro-ONNX running locally on a CPU and a separate Kokoro-FastAPI server on an H100 instance hosted by Runpod. Trelis reports processing about 55–78 hours of audio per wall-clock hour under concurrency for that server setup. Those examples are distinct from the SageMaker and AgentCore design; the reported throughput measures a different workload from time to first audio.

What AWS’s latency results measure

In an isolated test using the same 267-character, four-sentence text block, AWS reported a mean time to first audio byte of 329.3 ms in batch mode and 156.7 ms when it initially sent only the first sentence. AWS ran five trials per mode and reported a 52.4% relative reduction for that test.

For its end-to-end comparison, AWS reported that bidirectional mode reached time to first audio 2.4 times faster in a local-container test and 3.5 times faster on the deployed AgentCore runtime than batch mode. AWS says the comparison used the same prompt, Bedrock model, and endpoint, with the timing of synthesis start as the changed variable.

Account for the sample’s security and cleanup details

AWS flags the sample’s older PyTorch pin as associated with security advisory PYSEC-2026-2286 and says the advisory is fixed in PyTorch 2.10.0. Review the pinned version before using the sample beyond a proof of value.

For cleanup, remove the agent runtime before the SageMaker endpoint. The listed commands leave the S3 model-cache bucket and ECR repository in place; those resources may continue to incur storage charges until separately removed.