How AWS Batch quota shares manage SageMaker Training capacity

AWS Batch quota management divides SageMaker Training capacity into shares within a compatible job queue. Choose whether each share keeps its idle capacity, lends it to other shares, or can both lend and borrow. The feature applies to queues connected to a SAGEMAKER_TRAINING service environment, so the queue and its supporting resources need to fit that design before you assign shares. (AWS Batch quota management)

Choose how each share handles idle capacity

The sharing strategy determines what happens when a share has capacity its jobs are not using. The three options are RESERVE, LEND, and LEND_AND_BORROW. (AWS Batch quota shares)

Use RESERVE when keeping capacity available for a share matters more than letting others use it while idle. Choose LEND to make idle capacity available elsewhere. LEND_AND_BORROW allows both lending and borrowing, so it suits a share intended to participate in both sides of capacity sharing.

These shares allocate configured capacity within a job queue. They do not increase an AWS account’s service quotas. (AWS Batch quota management)

Check the queue and share limits

Quota management has a defined queue boundary and fixed share limits. AWS Batch supports it only for job queues connected to a SAGEMAKER_TRAINING service environment. A queue can have up to 20 quota shares, and each share must have one to five capacity limits, expressed as limits for SageMaker Training-supported instance types. (AWS Batch quota management; AWS Batch quota shares)

Set up the resources in order

Before creating the resources, make sure your IAM permissions allow you to create and manage AWS Batch queues, scheduling policies, and service environments. (Create AWS Batch quota-management resources)

1. Create a quota-compatible Scheduling Policy. The job queue needs a policy configured for quota management.
2. Configure a SAGEMAKER_TRAINING Service Environment. Set its capacity limits using instance types accepted by SageMaker Training, rather than NUM_INSTANCES.
3. Create the Job Queue. Connect it to the compatible policy and service environment. That environment can connect to only one quota-management-enabled queue.
4. Add quota shares. Give each share one to five supported instance-type capacity limits, then select its strategy. Keep the total at 20 shares or fewer per queue.
5. Check the queue state. AWS’s resource-creation guidance identifies ENABLED and VALID, with the reason JobQueue Healthy, as the queue’s healthy status. (Create AWS Batch quota-management resources)

AWS illustrates a three-project setup with one ml.g5.xlarge capacity limit per share and a service-environment maximum capacity of 3. In that example, one share uses LEND_AND_BORROW, another uses LEND, and the third uses RESERVE.

Plan for capacity recovery and preemption

When a share needs capacity it previously lent, AWS Batch can reclaim it by preempting jobs that borrowed capacity. Its selection heuristic considers the number and type of borrowed instances, job priorities, and job duration. (AWS Batch quota shares)

Preemption can also occur within a share: configurable in-share preemption lets higher-priority jobs displace lower-priority jobs in SCHEDULED, STARTING, or RUNNING states. Lowering a share’s capacity limit can also make its jobs eligible for preemption when their use exceeds the new limit, even if the share uses LEND or RESERVE. (AWS Batch quota shares)

Treat a limit change as a possible interruption point. A share set to RESERVE or LEND is not necessarily protected from preemption after its capacity limit is reduced.