Measure SageMaker TTS auto scaling instead of guessing

To set auto scaling for a real-time text-to-speech (TTS) endpoint, measure the highest per-instance concurrency that meets your latency target, then time a forced scale-out. In a tutorial published October 22, 2026, AWS applies that method to Kokoro-82M on an ml.g5.xlarge: its reported Q_max is 50 concurrent requests under a 3,000 ms p95 time-to-first-audio-byte (TTFAB) objective. These are example results, not SageMaker defaults.

Measure the concurrency limit

Q_max is the highest tested number of concurrent requests per instance that meets a chosen service-level objective (SLO), the latency target for the endpoint. AWS uses SageMaker’s ConcurrentRequestsPerModel metric, reported as a Maximum over a 10-second window. For its tested concurrency levels, AWS reports a 1.0 ratio between the benchmark client’s concurrency readings and CloudWatch’s metric.

AWS reports a p95 TTFAB of 2,805.7 ms at 50 concurrent requests and 3,374.8 ms at 60. The 3,000 ms SLO is met at 50 and exceeded at 60, making 50 the reported Q_max for this Kokoro-82M deployment. At concurrency one, AWS reports an average service time of 61.3 ms and a p95 of 66.1 ms.

Prepare the example deployment

AWS lists permissions for IAM, Amazon S3, Amazon ECR, and SageMaker resources; AWS CDK v2 bootstrapped in the target account and Region; Python 3.11 or later; uv; Docker; and spare ml.g5.xlarge quota. A live qmax run takes about 36 minutes, and ttotal can take up to about 28 minutes.

The sample deploys three CDK stacks in order: SpeechFoundationStack creates IAM roles and an S3 model-weights bucket; SpeechModelCacheStack synchronizes model weights; and SpeechEndpointStack creates the endpoint resources. The endpoint stack synthesizes auto-scaling and observability resources only when scaling is enabled and a feasible measured plan artifact exists.

AWS Samples describes the repository as a proof of value, not a production-ready implementation. It pins torch==2.6.0 and identifies advisory PYSEC-2026-2286; the repository names torch 2.10.0 as a fixed version.

Run the measurement workflow in order

The workflow proceeds from configuration checks to live measurements, then uses those measurements to plan and validate the policy:

1. Run drift to compare deployed scaling resources with the configuration. AWS describes this as read-only and taking seconds.
2. Run qmax to hold one instance and test a closed-loop concurrency ladder against the SLO. This affects live endpoint state and takes about 36 minutes. Its output is needed by the next measurement.
3. Run ttotal to force a scale-out and time recovery. This also affects live endpoint state and can take up to about 28 minutes; it depends on the qmax artifact.
4. Run plan to calculate policy settings from both measurement artifacts. AWS describes this as read-only and taking seconds.
5. Run validate to compare the deployed policy with the plan artifact. This is also read-only and takes seconds.

If a qmax or ttotal run is stopped or canceled, AWS says it can leave auto scaling frozen. The tutorial’s recovery sequence uses a drift check and the thaw command before further measurements.

Set thresholds from the measurements

For Q_max=50 and an operator-selected 1.25× surge ratio, AWS calculates a scale-out threshold of 37.5 concurrent requests and a scale-in threshold of 25: (1−0.25)×50=37.5 and (1−0.50)×50=25. The example configuration uses an ml.g5.xlarge, one minimum and nine maximum instances, a 30-second scale-out cooldown, and a 960-second scale-in cooldown. Those settings belong to this example workload.

AWS reports a forced scale-out measurement of 260.2 seconds. Its planning figure is 320.2 seconds, adding a bounded 60-second policy-detection allowance that was not measured in that run.

The eight stage durations AWS lists—30.0, 180.8, 3.7, 0.0, 6.5, 1.0, 20.7, and 47.4 seconds—sum to 290.1 seconds, which is 29.9 seconds more than the reported 260.2-second measurement. The published stage figures therefore do not add up to that total.

Account for scale-in and queued requests

AWS warns that scaling a two-instance fleet down to one can leave the remaining instance at Q_max, creating renewed pressure to scale out. In the tutorial’s calculation, a minimum fleet of three avoids that specific condition but keeps more capacity provisioned.

The example’s separate admission gate rejects requests when the queue exceeds its configured depth of 50, returning HTTP 503 queue_saturated. That gate rejects work; it does not add endpoint capacity.

AWS Samples also says the sample pins the Kokoro model to an exact Hugging Face commit. The repository characterizes the PyTorch advisory as memory corruption in the weights_only unpickler triggered by a malicious checkpoint, and says the described attack path would require compromise of that pinned upstream revision. It recommends reviewing and upgrading the dependency before using the sample beyond proof-of-value work.