Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
337 changes: 331 additions & 6 deletions platform-cloud/docs/compute-envs/aws-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,225 @@ The following permissions enable Seqera to populate values for dropdown fields.
}
```

## Create the IAM policy
## Seqera Intelligent Compute

Seqera Intelligent Compute is an optional capability that executes Nextflow tasks on a Seqera-managed Amazon ECS cluster instead of running them entirely on the head EC2 instance. The AWS Cloud compute environment scales beyond the resources of a single instance while preserving its fast startup behavior.

When you enable Seqera Intelligent Compute, Seqera provisions and manages all ECS infrastructure on your behalf, including clusters, capacity providers, task definitions, IAM roles, and (optionally) Auto Scaling Groups for spot and on-demand capacity. All managed resources use the `seqera-sched-` prefix and are torn down automatically when no longer needed.

### Additional IAM permissions

To enable Seqera Intelligent Compute, attach an additional IAM policy (beyond the [Required Platform IAM permissions](#required-platform-iam-permissions)) to the same IAM user or role that Seqera uses to access your AWS account.

The policy scopes every ARN-eligible action to the `seqera-sched-*` prefix. The remaining `Resource: "*"` entries correspond to AWS APIs that do not support resource-level permissions, such as EC2 `Describe*`, ECR authorization tokens, and Cost Explorer.

<details>
<summary>Seqera Intelligent Compute policy</summary>

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECSScopedOperations",
"Effect": "Allow",
"Action": [
"ecs:CreateCluster",
"ecs:DeleteCluster",
"ecs:DescribeClusters",
"ecs:PutClusterCapacityProviders",
"ecs:CreateCapacityProvider",
"ecs:DeleteCapacityProvider",
"ecs:DescribeCapacityProviders",
"ecs:RunTask",
"ecs:StopTask",
"ecs:DescribeTasks",
"ecs:DescribeContainerInstances",
"ecs:TagResource"
],
"Resource": "arn:aws:ecs:*:*:*/seqera-sched-*"
},
{
"Sid": "ECSUnscopedOperations",
"Effect": "Allow",
"Action": [
"ecs:RegisterTaskDefinition",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeTaskDefinition",
"ecs:ListTaskDefinitions",
"ecs:ListTaskDefinitionFamilies",
"ecs:ListTasks"
],
"Resource": "*"
},
{
"Sid": "IAMRoleManagement",
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:GetRole",
"iam:DeleteRole",
"iam:PutRolePolicy",
"iam:DeleteRolePolicy",
"iam:ListRolePolicies",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:CreateInstanceProfile",
"iam:GetInstanceProfile",
"iam:AddRoleToInstanceProfile",
"iam:ListInstanceProfilesForRole",
"iam:RemoveRoleFromInstanceProfile",
"iam:DeleteInstanceProfile"
],
"Resource": [
"arn:aws:iam::*:role/seqera-sched-*",
"arn:aws:iam::*:instance-profile/seqera-sched-*"
]
},
{
"Sid": "PassRoleToECS",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/seqera-sched-*",
"Condition": {
"StringEquals": {
"iam:PassedToService": [
"ecs-tasks.amazonaws.com",
"ecs.amazonaws.com",
"ec2.amazonaws.com"
]
}
}
},
{
"Sid": "ServiceLinkedRoles",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": [
"ecs.amazonaws.com",
"ecs-compute.amazonaws.com",
"autoscaling.amazonaws.com",
"spot.amazonaws.com"
]
}
}
},
{
"Sid": "CloudWatchLogs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:DeleteLogGroup",
"logs:PutRetentionPolicy",
"logs:DescribeLogStreams",
"logs:GetLogEvents",
"logs:TagResource"
],
"Resource": "arn:aws:logs:*:*:log-group:/seqera/sched*"
},
{
"Sid": "EC2NetworkDiscovery",
"Effect": "Allow",
"Action": [
"ec2:DescribeVpcs",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeRouteTables",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeInstances",
"ec2:CreateSecurityGroup",
"ec2:CreateVpcEndpoint",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:CreateTags"
],
"Resource": "*"
},
{
"Sid": "ECRAccess",
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Resource": "*"
},
{
"Sid": "S3Access",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Sid": "ASGEC2Operations",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceTypes",
"ec2:CreateLaunchTemplate",
"ec2:DeleteLaunchTemplate",
"ec2:RunInstances"
],
"Resource": "*"
},
{
"Sid": "ASGManagement",
"Effect": "Allow",
"Action": [
"autoscaling:CreateAutoScalingGroup",
"autoscaling:UpdateAutoScalingGroup",
"autoscaling:DeleteAutoScalingGroup",
"autoscaling:CreateOrUpdateTags"
],
"Resource": "arn:aws:autoscaling:*:*:*/seqera-sched-*"
},
{
"Sid": "ASGDescribe",
"Effect": "Allow",
"Action": "autoscaling:DescribeAutoScalingGroups",
"Resource": "*"
},
{
"Sid": "SSMECSOptimizedAmi",
"Effect": "Allow",
"Action": "ssm:GetParameter",
"Resource": "arn:aws:ssm:*:*:parameter/aws/service/ecs/optimized-ami/*"
},
{
"Sid": "CostExplorer",
"Effect": "Allow",
"Action": "ce:GetCostAndUsage",
"Resource": "*"
}
]
}
```

</details>

:::note
- The `ASGEC2Operations` and `ASGManagement` statements are required only if you enable Auto Scaling Group-backed clusters (managed instances). Omit them for Fargate-only deployments.
- The `CreateECSServiceLinkedRole` is required only if the Service Role is not already created.
- The `CostExplorer` statement is required only if you enable Cost Analysis.
:::

Like the base AWS Cloud policy, you can attach this policy directly to the IAM user or to an IAM role that the user assumes. See [IAM resource provisioning](#iam-resource-provisioning) for setup instructions.

## IAM resource provisioning

You can provision the required IAM resources manually via the AWS Console or automatically using a CloudFormation template.

### Manual provisioning

#### Create the IAM policy

The policy above must be created in the AWS account where the AWS Batch resources need to be created.

Expand All @@ -306,15 +524,15 @@ The policy above must be created in the AWS account where the AWS Batch resource
1. Following the instructions detailed in the [IAM permissions breakdown section](#required-platform-iam-permissions) replace the default text in the policy editor area under the **JSON** tab with a policy adapted to your use case, then select **Next**.
1. Enter a name and description for the policy on the **Review and create** page, then select **Create policy**.

## IAM user creation
#### IAM user creation

Seqera requires an Identity and Access Management (IAM) User to create and manage AWS Batch resources in your AWS account. We recommend creating a separate IAM policy rather than an IAM User inline policy, as the latter only allows 2048 characters, which may not be sufficient for all the required permissions.

In certain scenarios, for example when multiple users need to access the same AWS account and provision AWS Batch resources, an IAM role with the required permissions can be created instead, and the IAM user can assume that role when accessing AWS resources, as detailed in the [IAM role creation (optional)](#iam-role-creation-optional) section.

Depending whether you choose to let Seqera automatically create the required AWS Batch resources in your account, or prefer to set them up manually, the IAM user must have specific permissions as detailed in the [Required Platform IAM permissions](#required-platform-iam-permissions) section. Alternatively, you can create an IAM role with the required permissions and allow the IAM user to assume that role when accessing AWS resources, as detailed in the [IAM role creation (optional)](#iam-role-creation-optional) section.

### Create an IAM user
##### Create an IAM user

1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Users** in the left navigation menu, then select **Create User** at the top right of the page.
1. Enter a name for your user (e.g., _seqera_) and select **Next**.
Expand All @@ -338,7 +556,7 @@ Depending whether you choose to let Seqera automatically create the required AWS

The user has now been created. The most up-to-date instructions for creating an IAM user can be found in the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html).

### Obtain IAM user credentials
##### Obtain IAM user credentials

To get the credentials needed to connect Seqera to your AWS account, follow these steps:

Expand All @@ -348,7 +566,7 @@ To get the credentials needed to connect Seqera to your AWS account, follow thes
1. Optionally provide a description for the access key, like the reason for creating it, then select **Create access key**.
1. Save the **Access key** and **Secret access key** in a secure location as you will need to provide them when creating credentials in Seqera.

## IAM role creation (optional)
#### IAM role creation (optional)

Rather than attaching permissions directly to the IAM user, you can create an IAM role with the required permissions and allow the IAM user to assume that role when accessing AWS resources. This is useful when multiple IAM users are used to access the same AWS account: this way the actual permissions to operate on the resources are only granted to a single centralized role.

Expand Down Expand Up @@ -384,7 +602,7 @@ Multiple users can be specified in the trust policy by adding more ARNs to the `
Seqera Platform generates the `External ID` value during AWS credential creation. For role-based credentials, use this exact value in your IAM trust policy (`sts:ExternalId`).
:::

### Role-based trust policy example (Seqera Cloud)
##### Role-based trust policy example (Seqera Cloud)

For role-based AWS credentials in Seqera Cloud, allow the Seqera Cloud access role `arn:aws:iam::161471496260:role/SeqeraPlatformCloudAccessRole` in your trust policy and enforce the `External ID` generated during credential creation:

Expand Down Expand Up @@ -415,6 +633,113 @@ For role-based AWS credentials in Seqera Cloud, allow the Seqera Cloud access ro
}
```

### Programmatic provisioning (CloudFormation)

As an alternative to manual setup, you can use the provided AWS CloudFormation template to create all required IAM resources in a single operation. The template creates the IAM user, the access key, and the role with the appropriate policies attached.

<details>
<summary>CloudFormation template</summary>

```yaml
{% include '../compute-envs/template.yaml' %}
```

</details>

#### Template parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `IAMUserName` | Name of the IAM user created for Seqera Platform | `SeqeraPlatform` |
| `RoleName` | Name of the IAM role assumed by the Seqera Platform IAM user | `SeqeraPlatformRole` |
| `ForgePrefix` | Prefix used for resources created by Seqera Platform (Forge) | `TowerForge` |
| `SeqeraIntelligentComputeEnabled` | Set to `true` to attach the additional permissions required for Seqera Intelligent Compute | `false` |

#### Template outputs

| Output | Description |
|--------|-------------|
| `SeqeraPlatoformUserAccessKeyId` | Access key ID to use when creating AWS credentials in Seqera Platform |
| `SeqeraPlatoformUserSecretAccessKey` | Secret access key to use when creating AWS credentials in Seqera Platform |
| `SeqeraPlatoformRole` | ARN of the IAM role to provide in the **Assume role** field when creating AWS credentials in Seqera Platform |

#### Deploy the stack

The `--capabilities CAPABILITY_NAMED_IAM` flag is required because the template creates IAM resources with custom names. CloudFormation requires explicit acknowledgement before creating named IAM resources as a safeguard against unintended privilege escalation.

**Create a stack with default parameters:**

```bash
aws cloudformation create-stack \
--stack-name seqera-platform-iam \
--template-body file://template.yaml \
--capabilities CAPABILITY_NAMED_IAM
```

**Create a stack with custom parameters:**

```bash
aws cloudformation create-stack \
--stack-name seqera-platform-iam \
--template-body file://template.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameters \
ParameterKey=IAMUserName,ParameterValue=my-seqera-user \
ParameterKey=RoleName,ParameterValue=my-seqera-role \
ParameterKey=ForgePrefix,ParameterValue=TowerForge \
ParameterKey=SeqeraIntelligentComputeEnabled,ParameterValue=true
```

**Update an existing stack:**

```bash
aws cloudformation update-stack \
--stack-name seqera-platform-iam \
--template-body file://template.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameters \
ParameterKey=SeqeraIntelligentComputeEnabled,ParameterValue=true
```

**Delete a stack:**

```bash
aws cloudformation delete-stack \
--stack-name seqera-platform-iam
```

#### Retrieve stack outputs

After the stack reaches `CREATE_COMPLETE`, retrieve the access key and role ARN to use when creating credentials in Seqera Platform:

```bash
aws cloudformation describe-stacks \
--stack-name seqera-platform-iam \
--query "Stacks[0].Outputs"
```

Example output:

```json
[
{
"OutputKey": "SeqeraPlatoformUserAccessKeyId",
"OutputValue": "AKIAIOSFODNN7EXAMPLE",
"Description": "The User access key to upload to Seqera Platform as credentials"
},
{
"OutputKey": "SeqeraPlatoformUserSecretAccessKey",
"OutputValue": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Description": "The User secret access key to upload to Seqera Platform as credentials"
},
{
"OutputKey": "SeqeraPlatoformRole",
"OutputValue": "arn:aws:iam::123456789012:role/SeqeraPlatformRole",
"Description": "The role to be assumed by Seqera Platform to create and use the AWS Cloud credentials"
}
]
```

## AWS credential options

AWS credentials can be configured in two ways:
Expand Down
Loading
Loading