To start running NVIDIA Nemotron 3.5 Super VL on AWS, first make sure your account can use one of the sample’s large GPU instances and that your workstation has Git and Docker. The AWS sample walkthrough then takes you through the opening steps for a standard Amazon EKS cluster, from setting the region to checking its nodes.

What you need before starting

The AWS sample calls for:

  • An AWS account with access to P5en.48xlarge or P6-b200.48xlarge EC2 instances. A quota increase or capacity reservation may be needed.
  • A terminal with Git and Docker installed.
  • AWS credentials permitted to manage EC2, EKS, and FSx for Lustre.

These are prerequisites for this sample’s route. Its opening cluster-creation sequence may take up to 30 minutes.

Start the standard EKS setup

Follow the sample’s steps in order. This sequence creates and checks the cluster; the sample continues afterward with drivers, model weights, deployment, testing, benchmarking, and resource cleanup.

  1. Clone the AWS sample repository. Run git clone https://github.com/aws-samples/aws-do-eks, then enter the new aws-do-eks directory. This gives you the helper scripts used in the walkthrough.
  2. Set the AWS region. Run ./config.sh, then set AWS_DEFAULT_REGION in the .env file to the region for your cluster.
  3. Build the helper image. Run ./build.sh to create the local aws-do-eks Docker image.
  4. Start the helper container. Run ./run.sh from the repository directory.
  5. Check that the container is running. Run ./status.sh; the sample’s expected result is that the container is up.
  6. Open a shell in the container. Run ./exec.sh. The container includes AWS CLI, eksctl, kubectl, and helper utilities.
  7. Configure AWS credentials. In the container shell, run aws configure and follow its prompts.
  8. Choose the cluster manifest. Run ./env-config.sh and set CONF to the manifest for the cluster configuration you intend to use.
  9. Edit the cluster settings. Run ./eks-config.sh to open the manifest and configure settings such as cluster name, region, instance types, and volume size.
  10. Create the cluster. Run ./eks-create.sh. In the sample’s standard EKS route, this invokes eksctl create cluster with the configured manifest. AWS says this operation may take up to 30 minutes.
  11. Check the cluster nodes. After creation finishes, run kubectl get nodes to inspect the nodes registered with the cluster.

AWS’s reference architecture uses CPU and GPU nodes, Amazon ECR for container images, shared Amazon FSx for Lustre storage for model weights, and EFA networking between GPU nodes. After the node check, the sample moves on to Kubernetes drivers and operators, cluster validation, weight downloads, model deployment, inference tests, benchmarking, and cleanup.

Choose an AWS setup and GPU precision

AWS describes three orchestration options for this deployment: standard Amazon EKS, Amazon EKS Auto Mode, and SageMaker HyperPod EKS. Auto Mode manages scaling, GPU, and EFA support. The sample lists storage with EBS plus FSx or FSx alone, and supports BF16 or NVFP4 checkpoints. It also includes aggregated and disaggregated serving options through NVIDIA Dynamo with a vLLM backend.

AWS lists these minimum GPU counts for serving Nemotron with vLLM, by accelerator and checkpoint precision:

GPU modelBF16 minimum GPUsNVFP4 minimum GPUs
B200/GB20082
B300/GB30041
H20084
H100168

These are model-serving minimums, not a complete cluster design. The AWS deployment overview gives the counts and the deployment architecture.

Read AWS’s latency result in context

AWS reports lower time to first token (TTFT), the wait for the first generated token, and about the same inter-token latency (ITL), the gap between generated tokens, in its two-node disaggregated test compared with its aggregated test.

The plotted test used Nemotron 3.5 Super VL 120B-A12B in BF16 on H200 GPUs, with 1,024 input tokens and 512 output tokens. AWS labeled the aggregated setup as 8 GPUs and the disaggregated setup as 16 GPUs; its concurrency sweep ran from 1 to 128, with one launch per point. Because the GPU counts differ, the result describes those configurations rather than isolating the effect of serving topology.

Validate the cluster and measure your workload

AWS describes an NCCL all-reduce micro-benchmark for checking that the cluster and network are operating, including activity across GPUs and EFA adapters. For model performance, AWS uses NVIDIA aiperf to measure TTFT, ITL, and output throughput across different request-concurrency levels. Running a concurrency sweep gives you measurements for your own cluster and serving configuration.