Skip to content

vLLM inf2 ec2 examples #276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
joozero merged 3 commits into from
Oct 28, 2025
Merged

vLLM inf2 ec2 examples #276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cc48afc
Remove custom docker image and change model to TinyLlama
Oct 3, 2025
9c47c75
add compiled model cache and update docs
Oct 20, 2025
abdeaa4
Merge branch 'main' into vllm-inf2-ec2-examples
joozero Oct 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 0 additions & 19 deletions terraform/ec2-examples/vllm-inferentia/Dockerfile
View file
Open in desktop

This file was deleted.

165 changes: 24 additions & 141 deletions terraform/ec2-examples/vllm-inferentia/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,11 @@
# ECS machine learning distributed training
# ECS Inference using vLLM with inf2

This solution blueprint creates the infrastructure needed to run GenAI inference using [vLLM](https://docs.vllm.ai/en/latest/index.html) with [AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) and Inferentia 2 instances. This solution is based on similar examples for running inference using vLLM on [EKS](https://aws.amazon.com/blogs/machine-learning/deploy-meta-llama-3-1-8b-on-aws-inferentia-using-amazon-eks-and-vllm/) and [EC2](https://aws.amazon.com/blogs/machine-learning/serving-llms-using-vllm-and-amazon-ec2-instances-with-aws-ai-chips/) using Inferentia-based instances.
This solution blueprint creates the infrastructure to run models in multiple nueron cores using tensor parallelism within a single task with vLLM. By default, it uses one inf2.8xlarge instance.

> Insert Diagram here

By default, this blueprint deploys inf2.8xlarge instances optimized for GenAI inference workloads. The setup is tailored for running vLLM with pre-compiled Neuron-compatible models. You can modify the instance type and resource allocation by changing the variables in the Terraform configuration.

## Components

* ECS Cluster:
* Uses an autoscaling group to provision inf2 instances for the ECS cluster.
* Allows dynamic scaling of GenAI workloads.
* ECS Cluster
* ECS Service Definition:
* vLLM Service: Configured to serve requests for GenAI inference using vLLM.
* Application Load Balancer:
Expand All @@ -20,129 +15,6 @@ By default, this blueprint deploys inf2.8xlarge instances optimized for GenAI in
* Logs from ECS tasks and services are collected in CloudWatch for monitoring and debugging.


## Prequequisites

### Hugging Face Account and API Key

To use the meta-llama/Llama-3.2-1B model within the blueprint, you’ll need a Hugging Face account and and an API key to access to the model. Follow these steps to set these up:

1. [Sign up for a Hugging Face account](https://huggingface.co/join) if you don’t already have one.
2. Go to the [meta-llama/Llama-3.2-1B model card](https://huggingface.co/meta-llama/Llama-3.2-1B) on Hugging Face.
3. Agree to the model license to gain access.
4. Generate your Hugging Face API key:
* Navigate to your [Hugging Face Account Settings](https://huggingface.co/settings/tokens).
* Under the Access Tokens section, click New Token.
* Provide a name for your token and set the role to write or read.
* Copy the token when prompted (as shown in the following figure). The token will not be displayed again.

## Preparing the Docker Image

To run the model, you’ll need to build and push a Docker image with the required dependencies to Amazon Elastic Container Registry (Amazon ECR). While [you can use docker buildx](https://docs.docker.com/build/building/multi-platform/) to do this, if you dont have your local machine configured for this, you can use an Inf2-based EC2 instance as a build environment to build your container for the arm64 architecture.

### Steps to launch an Inf2-based Build Environment:

#### 1. Launch an Inf2-based EC2 Instance
1. Open the AWS Management Console and launch an Inf2-based EC2 instance (e.g., inf2.8xlarge). You can use a [guide like this](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-instance-wizard.html). If this is your first time using inf/trn instances, you will need to [request a quota increase](https://repost.aws/articles/ARgmEMvbR6Re200FQs8rTduA/inferentia-and-trainium-service-quotas).
2. Ensure the instance has:
* Access to your [Amazon ECR repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-push-iam.html).
* Permissions for Docker and AWS CLI operations.
* Can be accessed via Session Manager or is [configured for SSH access](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connection-prereqs-general.html)
3. Access the instance through Session manager or SSH into the EC2 instance using the following command:

```bash
ssh -i your-key.pem ec2-user@<ec2-public-ip>
```

#### 2. Setup Environmental Variables

```bash
export ECR_REPO_NAME=vllm-neuron
export AWS_REGION=us-west-2
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
```

#### 3. Create an ECR Repository
Run the following command to create an ECR repository:

```bash
aws ecr create-repository --repository-name $ECR_REPO_NAME --region $AWS_REGION
```

#### 4. Create the Dockerfile

> If you're using your local development machine, you can skip this step as a Dockerfile already exists in this project.

Create the Dockerfile for the VLLM model:
```bash
cat > Dockerfile <<EOF
# default base image
FROM public.ecr.aws/neuron/pytorch-inference-neuronx:2.1.2-neuronx-py310-sdk2.20.0-ubuntu20.04
# Clone the vllm repository
RUN git clone https://github.com/vllm-project/vllm.git
# Set the working directory
WORKDIR /vllm
RUN git checkout v0.6.0
# Set the environment variable
ENV VLLM_TARGET_DEVICE=neuron
# Install the dependencies
RUN python3 -m pip install -U -r requirements-neuron.txt
RUN python3 -m pip install .
# Modify the arg_utils.py file to support larger block_size option
RUN sed -i "/parser.add_argument('--block-size',/ {N;N;N;N;N;s/\[8, 16, 32\]/[8, 16, 32, 128, 256, 512, 1024, 2048, 4096, 8192]/}" vllm/engine/arg_utils.py
# Install ray
RUN python3 -m pip install ray
RUN pip install -U triton>=3.0.0
# Set the entry point
ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]
EOF
```

#### 5. Build and Push the Docker Image

Run the following commands to build and push the Docker image:

1. Authenticate Docker to your ECR registry:

```bash
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
```

2. Build the Docker image:

```bash
docker build -t ${ECR_REPO_NAME}:latest .
```

3. Tag the image

```bash
docker tag ${ECR_REPO_NAME}:latest $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/${ECR_REPO_NAME}:latest
```

4. Push the image to ECR

```bash
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/${ECR_REPO_NAME}:latest
```

5. Copy the ECR image URI for your use in the main.tf file within this project.

```bash
echo "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/${ECR_REPO_NAME}:latest"
```

## Deployment Prerequisites

1. Modify the local variables at line 6 of `main.tf`
```nano
name = "ecs-demo-vllm-inferentia" # Defaul name of the project
region = "us-west-2" # Default region
instance_type = "inf2.8xlarge" # Default instance size
vllm_container_image = "<ECR IMAGE URI>" # ECR Image URI you created when building and pushing your image
hugging_face_api_key = "<YOUR HUGGIN FACE API KEY>" # Your Hugging Face API Key
```


## Deployment

1. Deploy core-infra resources
Expand All @@ -168,10 +40,13 @@ Once the cluster and services are deployed, you can use the load balancer DNS na

Send a POST request to the vLLM OpenAI-compatible endpoint:
```bash
curl -X POST http://<ALB_DNS_NAME>:8000/v1/completions \

ALB_DNS_NAME=$(terraform output -raw load_balancer_dns_name)

curl -X POST http://$ALB_DNS_NAME:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.2-1B",
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"prompt": "Write a short poem about technology",
"max_tokens": 100,
"temperature": 0.7
Expand All @@ -184,7 +59,7 @@ Example Response:
"id": "cmpl-6ze...",
"object": "text_completion",
"created": 1680307267,
"model": "meta-llama/Llama-3.2-1B",
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"choices": [
{
"text": "\n\nTechnology, a wondrous art,\nA force that shapes the world's heart.\nIn circuits small and data vast,\nIt links the future to the past.",
Expand All @@ -195,6 +70,7 @@ Example Response:
]
}
```

## What do you do next?

Congratulations on successfully deploying your vLLM inference solution on ECS with AWS Inferentia! Here are some ideas to take your implementation to the next level:
Expand Down Expand Up @@ -223,27 +99,34 @@ Congratulations on successfully deploying your vLLM inference solution on ECS wi
* Experiment with autoscaling policies to dynamically adjust the number of running tasks based on request volume.

6. Learn from Amazon’s Approach
* Discover how Amazon’s engineering team scaled generative AI for Amazon Rufus, powering conversational shopping experiences during Prime Day.
* Discover how Amazon’s engineering team [scaled generative AI for Amazon Rufus](https://aws.amazon.com/blogs/machine-learning/scaling-rufus-the-amazon-generative-ai-powered-conversational-shopping-assistant-with-over-80000-aws-inferentia-and-aws-trainium-chips-for-prime-day/), powering conversational shopping experiences during Prime Day.
* Adapt lessons learned from their implementation to improve scalability, reliability, and cost-efficiency in your use case.

## Clean up

1. Destroy this blueprint
1. Stop ECS tasks and wait for the status to be propagated with ECS.

```shell
terraform destroy
aws ecs update-service --service neuronx-vllm-service \
--desired-count 0 --cluster ecs-demo-vllm-inferentia \
--region us-west-2 --query 'service.serviceName'

sleep 30s
```

1. Destroy core-infra resources
2. Destroy this blueprint

```shell
cd ../core-infra
terraform destroy

```

## Troubleshooting
3. Destroy core-infra resources

```shell
cd ../core-infra
terraform destroy

```

## Support

Expand Down
56 changes: 32 additions & 24 deletions terraform/ec2-examples/vllm-inferentia/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@ provider "aws" {
region = local.region
}
locals {
name = "ecs-demo-vllm-inferentia" # Defaul name of the project
region = "us-west-2" # Default region
instance_type = "inf2.8xlarge" # Default instance size - if you change this - you will need to modify the cpu/memory details in the task definition
vllm_container_image = "<ECR IMAGE URI>" # ECR Image URI you created when building and pushing your image
hugging_face_api_key = "<YOUR HUGGIN FACE API KEY>" # Your Hugging Face API Key
name = "ecs-demo-vllm-inferentia" # Defaul name of the project
region = "us-west-2" # Default region
instance_type = "inf2.8xlarge" # Default instance size - if you change this - you will need to modify the cpu/memory details in the task definition
vllm_container_image = "public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.9.1-neuronx-py311-sdk2.26.0-ubuntu22.04"
user_data = <<-EOT
#!/bin/bash
cat <<'EOF' >> /etc/ecs/ecs.config
Expand Down Expand Up @@ -80,6 +79,10 @@ module "ecs_cluster" {
source = "terraform-aws-modules/ecs/aws//modules/cluster"
version = "~> 5.0"
cluster_name = local.name
cluster_settings = [{
name = "containerInsights",
value = "enhanced"
}]
# Capacity provider - autoscaling group
default_capacity_provider_use_fargate = false
autoscaling_capacity_providers = {
Expand Down Expand Up @@ -163,30 +166,31 @@ resource "aws_ecs_task_definition" "neuronx_vllm" {
]
environment = [
{
name = "HF_TOKEN"
value = local.hugging_face_api_key
name = "VLLM_NEURON_FRAMEWORK"
value = "neuronx-distributed-inference"
},
{
name = "FI_EFA_FORK_SAFE"
value = "1"
},
{
name = "VLLM_TARGET_DEVICE"
value = "neuron"
},
{
name = "NEURON_CC_FLAGS"
value = "--target=inf2"
name = "NEURON_COMPILE_CACHE_URL"
value = "/neuron-compile-cache"
}
]
command = [
"--model",
"meta-llama/Llama-3.2-1B",
"--device", "neuron",
"python",
"-m",
"vllm.entrypoints.openai.api_server",
"--model", "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"--max-num-seqs", "4",
"--max-model-len", "128",
"--tensor-parallel-size", "2",
"--block-size", "8",
"--max-model-len", "4096",
"--max-num-seqs", "32",
"--port", "8000",
"--device", "neuron"
]
mountPoints = [
{
sourceVolume = "neuron-compile-cache-volume"
containerPath = "/neuron-compile-cache"
readOnly = false
}
]
linuxParameters = {
devices = [
Expand All @@ -197,7 +201,7 @@ resource "aws_ecs_task_definition" "neuronx_vllm" {
}
]
capabilities = {
add = ["IPC_LOCK"]
add = ["IPC_LOCK", "SYS_ADMIN"]
}
}
logConfiguration = {
Expand All @@ -211,6 +215,10 @@ resource "aws_ecs_task_definition" "neuronx_vllm" {
essential = true
}
])
volume {
name = "neuron-compile-cache-volume"
host_path = "/opt/cache/neuron-compile"
}
tags = {
app = "neuronx-vllm"
}
Expand Down