Update EKS basic components (cloudposse/terraform-aws-components#509)

Author: Nuru

src/README.md

@@ -1,11 +1,16 @@
-# Component: `karpenter`
+# Component: `eks/karpenter`
 
 This component provisions [Karpenter](https://karpenter.sh) on an EKS cluster.
 
 ## Usage
 
 **Stack Level**: Regional
 
+These instructions assume you are provisioning 2 EKS clusters in the same account
+and region, named "blue" and "green", and alternating between them.
+If you are only using a single cluster, you can ignore the "blue" and "green"
+references and remove the `metadata` block from the `karpenter` module.
+
 ```yaml
 components:
   terraform:
@@ -27,7 +32,7 @@ components:
         name: "karpenter"
         chart: "karpenter"
         chart_repository: "https://charts.karpenter.sh"
-        chart_version: "v0.10.1"
+        chart_version: "v0.16.3"
         create_namespace: true
         kubernetes_namespace: "karpenter"
         resources:
@@ -59,51 +64,29 @@ We will be using the `plat-ue2-dev` stack as an example.
 
 ### Provision Service-Linked Roles for EC2 Spot and EC2 Spot Fleet
 
-__Note:__ If you want to use EC2 Spot or Spot Fleet for the instances launched by Karpenter,
-you will need to provision the following Service-Linked Roles for EC2 Spot and Spot Fleet before provisioning Karpenter:
+__Note:__ If you want to use EC2 Spot for the instances launched by Karpenter,
+you may need to provision the following Service-Linked Role for EC2 Spot:
 
 - Service-Linked Role for EC2 Spot
-- Service-Linked Role for EC2 Spot Fleet
-
-This is only necessary if this is the first time you're using EC2 Spot and Spot Fleet in the account.
 
-```yaml
-components:
-  terraform:
-    iam-service-linked-roles:
-      settings:
-        spacelift:
-          workspace_enabled: true
-      vars:
-        enabled: true
-        root_account_tenant_name: core
-        service_linked_roles:
-          spot_amazonaws_com:
-            aws_service_name: "spot.amazonaws.com"
-            description: "AWSServiceRoleForEC2Spot Service-Linked Role for EC2 Spot"
-          spotfleet_amazonaws_com:
-            aws_service_name: "spotfleet.amazonaws.com"
-            description: "AWSServiceRoleForEC2SpotFleet Service-Linked Role for EC2 Spot Fleet"
-```
+This is only necessary if this is the first time you're using EC2 Spot in the account.
+Since this is a one-time operation, we recommend you do this manually via
+the AWS CLI:
 
 ```bash
-atmos terraform plan iam-service-linked-roles -s plat-gbl-dev
-atmos terraform apply iam-service-linked-roles -s plat-gbl-dev
+aws --profile <namespace>-<tenamt>-gbl-<stage>-admin iam create-service-linked-role --aws-service-name spot.amazonaws.com
 ```
 
-Note that if the Service-Linked Roles already exist in the AWS account (if you used EC2 Spot or Spot Fleet before), 
+Note that if the Service-Linked Roles already exist in the AWS account (if you used EC2 Spot or Spot Fleet before),
 and you try to provision them again, you will see the following errors:
 
 ```text
 An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation:
 Service role name AWSServiceRoleForEC2Spot has been taken in this account, please try a different suffix
-
-An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation:
-Service role name AWSServiceRoleForEC2SpotFleet has been taken in this account, please try a different suffix
 ```
 
 For more details, see:
- - https://karpenter.sh/v0.10.1/getting-started/getting-started-with-terraform/
+ - https://karpenter.sh/v0.18.0/getting-started/getting-started-with-terraform/
  - https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html
  - https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html
 
@@ -145,26 +128,26 @@ __Notes__:
 We use EKS Fargate Profile for Karpenter because It is recommended to run Karpenter on an EKS Fargate Profile.
 
 ```text
-Karpenter is installed using a Helm chart. The Helm chart installs the Karpenter controller and 
-a webhook pod as a Deployment that needs to run before the controller can be used for scaling your cluster. 
-We recommend a minimum of one small node group with at least one worker node. 
+Karpenter is installed using a Helm chart. The Helm chart installs the Karpenter controller and
+a webhook pod as a Deployment that needs to run before the controller can be used for scaling your cluster.
+We recommend a minimum of one small node group with at least one worker node.
 
 As an alternative, you can run these pods on EKS Fargate by creating a Fargate profile for the
-karpenter namespace. Doing so will cause all pods deployed into this namespace to run on EKS Fargate. 
+karpenter namespace. Doing so will cause all pods deployed into this namespace to run on EKS Fargate.
 Do not run Karpenter on a node that is managed by Karpenter.
 ```
 
 See [Run Karpenter Controller on EKS Fargate](https://aws.github.io/aws-eks-best-practices/karpenter/#run-the-karpenter-controller-on-eks-fargate-or-on-a-worker-node-that-belongs-to-a-node-group)
 for more details.
 
-We provision IAM Role for Nodes launched by Karpenter because they must run with an Instance Profile that grants 
+We provision IAM Role for Nodes launched by Karpenter because they must run with an Instance Profile that grants
 permissions necessary to run containers and configure networking.
 
 We define the IAM role for the Instance Profile in `components/terraform/eks/cluster/karpenter.tf`.
 
 Note that we provision the EC2 Instance Profile for the Karpenter IAM role in the `components/terraform/eks/karpenter` component (see the next step).
 
-Run the following commands to provision the EKS Fargate Profile for Karpenter and the IAM role for instances launched by Karpenter 
+Run the following commands to provision the EKS Fargate Profile for Karpenter and the IAM role for instances launched by Karpenter
 on the blue EKS cluster and add the role ARNs to the `aws-auth` ConfigMap:
 
 ```bash
@@ -174,8 +157,8 @@ atmos terraform apply eks/cluster-blue -s plat-ue2-dev
 
 For more details, refer to:
 
-- https://karpenter.sh/v0.10.1/getting-started/getting-started-with-terraform
-- https://karpenter.sh/v0.10.1/getting-started/getting-started-with-eksctl
+- https://karpenter.sh/v0.18.0/getting-started/getting-started-with-terraform
+- https://karpenter.sh/v0.18.0/getting-started/getting-started-with-eksctl
 
 
 ### 2. Provision `karpenter` component
@@ -207,10 +190,10 @@ Note that the stack config for the blue Karpenter component is defined in `stack
 
 ### 3. Provision `karpenter-provisioner` component
 
-In this step, we provision the `components/terraform/eks/karpenter-provisioner` component, which deploys Karpenter [Provisioners](https://karpenter.sh/v0.10.1/aws/provisioning) 
+In this step, we provision the `components/terraform/eks/karpenter-provisioner` component, which deploys Karpenter [Provisioners](https://karpenter.sh/v0.18.0/aws/provisioning)
 using the `kubernetes_manifest` resource.
 
-__NOTE:__ We deploy the provisioners in a separate step as a separate component since it uses `kind: Provisioner` CRD which itself is created by 
+__NOTE:__ We deploy the provisioners in a separate step as a separate component since it uses `kind: Provisioner` CRD which itself is created by
 the `karpenter` component in the previous step.
 
 Run the following commands to deploy the Karpenter provisioners on the blue EKS cluster:
@@ -238,7 +221,7 @@ You can override the default values from the `eks/karpenter-provisioner` base co
 
 For your cluster, you will need to review the following configurations for the Karpenter provisioners and update it according to your requirements:
 
-  - [requirements](https://karpenter.sh/v0.10.1/provisioner/#specrequirements):
+  - [requirements](https://karpenter.sh/v0.18.0/provisioner/#specrequirements):
 
     ```yaml
         requirements:
@@ -297,8 +280,8 @@ For your cluster, you will need to review the following configurations for the K
 
 For more details, refer to:
 
- - https://karpenter.sh/v0.10.1/provisioner/#specrequirements
- - https://karpenter.sh/v0.10.1/aws/provisioning
+ - https://karpenter.sh/v0.18.0/provisioner/#specrequirements
+ - https://karpenter.sh/v0.18.0/aws/provisioning
  - https://aws.github.io/aws-eks-best-practices/karpenter/#creating-provisioners
  - https://aws.github.io/aws-eks-best-practices/karpenter
  - https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html
@@ -310,31 +293,30 @@ For more details, refer to:
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | ~> 4.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.9.0 |
 | <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.0 |
+| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.7.1 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | ~> 4.0 |
-| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | n/a |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.9.0 |
 
 ## Modules
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_eks"></a> [eks](#module\_eks) | cloudposse/stack-config/yaml//modules/remote-state | 0.22.4 |
+| <a name="module_eks"></a> [eks](#module\_eks) | cloudposse/stack-config/yaml//modules/remote-state | 1.3.1 |
 | <a name="module_iam_roles"></a> [iam\_roles](#module\_iam\_roles) | ../../account-map/modules/iam-roles | n/a |
-| <a name="module_karpenter"></a> [karpenter](#module\_karpenter) | cloudposse/helm-release/aws | 0.5.0 |
+| <a name="module_karpenter"></a> [karpenter](#module\_karpenter) | cloudposse/helm-release/aws | 0.7.0 |
 | <a name="module_this"></a> [this](#module\_this) | cloudposse/label/null | 0.25.0 |
 
 ## Resources
 
 | Name | Type |
 |------|------|
 | [aws_iam_instance_profile.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource |
-| [kubernetes_namespace.default](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace) | resource |
 | [aws_eks_cluster_auth.eks](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/eks_cluster_auth) | data source |
 
 ## Inputs
@@ -400,14 +382,14 @@ For more details, refer to:
 
 - https://karpenter.sh
 - https://aws.github.io/aws-eks-best-practices/karpenter
-- https://karpenter.sh/v0.10.1/getting-started/getting-started-with-terraform
+- https://karpenter.sh/v0.18.0/getting-started/getting-started-with-terraform
 - https://aws.amazon.com/blogs/aws/introducing-karpenter-an-open-source-high-performance-kubernetes-cluster-autoscaler
 - https://github.com/aws/karpenter
 - https://www.eksworkshop.com/beginner/085_scaling_karpenter
 - https://ec2spotworkshops.com/karpenter.html
 - https://www.eksworkshop.com/beginner/085_scaling_karpenter/install_karpenter
-- https://karpenter.sh/v0.10.1/development-guide
-- https://karpenter.sh/v0.10.1/aws/provisioning
+- https://karpenter.sh/v0.18.0/development-guide
+- https://karpenter.sh/v0.18.0/aws/provisioning
 - https://docs.aws.amazon.com/eks/latest/userguide/pod-execution-role.html
 - https://aws.amazon.com/premiumsupport/knowledge-center/fargate-troubleshoot-profile-creation
 - https://learn.hashicorp.com/tutorials/terraform/kubernetes-crd-faas

src/default.auto.tfvars

@@ -9,79 +9,89 @@
 # https://catalog.us-east-1.prod.workshops.aws/workshops/76a5dd80-3249-4101-8726-9be3eeee09b2/en-US/autoscaling/karpenter
 
 locals {
-  enabled                          = module.this.enabled
-  create_namespace                 = local.enabled && var.create_namespace
-  eks_cluster_identity_oidc_issuer = module.eks.outputs.eks_cluster_identity_oidc_issuer
-  karpenter_iam_role_name          = module.eks.outputs.karpenter_iam_role_name
+  enabled = module.this.enabled
+
+  eks_cluster_identity_oidc_issuer = try(module.eks.outputs.eks_cluster_identity_oidc_issuer, "")
+  karpenter_iam_role_name          = try(module.eks.outputs.karpenter_iam_role_name, "")
+  karpenter_role_enabled           = local.enabled && length(local.karpenter_iam_role_name) > 0
 }
 
 resource "aws_iam_instance_profile" "default" {
-  count = local.enabled ? 1 : 0
+  count = local.karpenter_role_enabled ? 1 : 0
 
   name = local.karpenter_iam_role_name
   role = local.karpenter_iam_role_name
   tags = module.this.tags
 }
 
-# Create Karpenter Kubernetes namespace
-resource "kubernetes_namespace" "default" {
-  count = local.create_namespace ? 1 : 0
-
-  metadata {
-    name        = var.kubernetes_namespace
-    annotations = {}
-    labels      = module.this.tags
-  }
-}
-
 # Deploy Karpenter helm chart
 module "karpenter" {
   source  = "cloudposse/helm-release/aws"
-  version = "0.5.0"
+  version = "0.7.0"
 
-  chart                = var.chart
-  repository           = var.chart_repository
-  description          = var.chart_description
-  chart_version        = var.chart_version
-  kubernetes_namespace = local.create_namespace ? one(kubernetes_namespace.default[*].id) : var.kubernetes_namespace
-  create_namespace     = false
-  wait                 = var.wait
-  atomic               = var.atomic
-  cleanup_on_fail      = var.cleanup_on_fail
-  timeout              = var.timeout
+  chart           = var.chart
+  repository      = var.chart_repository
+  description     = var.chart_description
+  chart_version   = var.chart_version
+  wait            = var.wait
+  atomic          = var.atomic
+  cleanup_on_fail = var.cleanup_on_fail
+  timeout         = var.timeout
 
-  eks_cluster_oidc_issuer_url = replace(local.eks_cluster_identity_oidc_issuer, "https://", "")
+  create_namespace_with_kubernetes = var.create_namespace
+  kubernetes_namespace             = var.kubernetes_namespace
+  kubernetes_namespace_labels      = merge(module.this.tags, { name = var.kubernetes_namespace })
+
+  eks_cluster_oidc_issuer_url = coalesce(replace(local.eks_cluster_identity_oidc_issuer, "https://", ""), "deleted")
 
   service_account_name      = module.this.name
   service_account_namespace = var.kubernetes_namespace
 
-  iam_role_enabled = true
+  iam_role_enabled = local.karpenter_role_enabled
 
   # https://karpenter.sh/v0.6.1/getting-started/cloudformation.yaml
   # https://karpenter.sh/v0.10.1/getting-started/getting-started-with-terraform
+  # https://github.com/aws/karpenter/issues/2649
+  # Apparently the source of truth for the best IAM policy is the `data.aws_iam_policy_document.karpenter_controller` in
+  # https://github.com/terraform-aws-modules/terraform-aws-iam/blob/master/modules/iam-role-for-service-accounts-eks/policies.tf
   iam_policy_statements = [
     {
       sid       = "KarpenterController"
       effect    = "Allow"
       resources = ["*"]
 
       actions = [
+        # https://github.com/terraform-aws-modules/terraform-aws-iam/blob/99c69ad54d985f67acf211885aa214a3a6cc931c/modules/iam-role-for-service-accounts-eks/policies.tf#L511-L581
+        # The reference policy is broken up into multiple statements with different resource restrictions based on tags.
+        # This list has breaks where statements are separated in the reference policy for easier comparison and maintenance.
         "ec2:CreateLaunchTemplate",
         "ec2:CreateFleet",
-        "ec2:RunInstances",
         "ec2:CreateTags",
-        "ec2:TerminateInstances",
-        "ec2:DeleteLaunchTemplate",
         "ec2:DescribeLaunchTemplates",
+        "ec2:DescribeImages",
         "ec2:DescribeInstances",
         "ec2:DescribeSecurityGroups",
         "ec2:DescribeSubnets",
         "ec2:DescribeInstanceTypes",
         "ec2:DescribeInstanceTypeOfferings",
         "ec2:DescribeAvailabilityZones",
-        "ssm:GetParameter",
-        "iam:PassRole"
+        "ec2:DescribeSpotPriceHistory",
+        "pricing:GetProducts",
+
+        "ec2:TerminateInstances",
+        "ec2:DeleteLaunchTemplate",
+
+        "ec2:RunInstances",
+
+        "iam:PassRole",
       ]
+    },
+    {
+      sid    = "KarpenterControllerSSM"
+      effect = "Allow"
+      # Allow Karpenter to read AMI IDs from SSM
+      actions   = ["ssm:GetParameter"]
+      resources = ["arn:aws:ssm:*:*:parameter/aws/service/*"]
     }
   ]
 
@@ -110,4 +120,6 @@ module "karpenter" {
   ])
 
   context = module.this.context
+
+  depends_on = [aws_iam_instance_profile.default]
 }