Skip to content

Commit e9f908c

Browse files
NuruNuru
authored
Add support for AL2023 (#186)
1 parent ec07111 commit e9f908c

23 files changed

+956
-483
lines changed

‎README.md‎

Lines changed: 99 additions & 64 deletions
Large diffs are not rendered by default.

‎README.yaml‎

Lines changed: 48 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -61,18 +61,59 @@ related:
6161
url: "https://github.com/cloudposse/terraform-aws-ec2-instance-group"
6262
# Short description of this project
6363
description: |-
64-
Terraform module to provision an EKS Node Group for [Elastic Container Service for Kubernetes](https://aws.amazon.com/eks/).
64+
Terraform module to provision an EKS Managed Node Group for [Elastic Kubernetes Service](https://aws.amazon.com/eks/).
6565
66-
Instantiate it multiple times to create many EKS node groups with specific settings such as GPUs, EC2 instance types, or autoscale parameters.
66+
Instantiate it multiple times to create EKS Managed Node Groups with specific settings such as GPUs, EC2 instance types, or autoscale parameters.
67+
68+
**IMPORTANT:** When SSH access is enabled without specifying a source security group, this module provisions `EKS Node Group` nodes that are globally accessible by SSH (22) port. Normally, AWS recommends that no security group allows unrestricted ingress access to port 22 .
69+
70+
introduction: |-
71+
This module creates an [EKS Managed Node Group](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)
72+
for an [EKS](https://aws.amazon.com/eks/) cluster.
73+
It assumes you have already created an EKS cluster, but you can create the cluster and the node group in the
74+
same Terraform configuration. See our
75+
[full-featured root module (a.k.a. component) `eks/cluster`](https://github.com/cloudposse/terraform-aws-components/tree/main/modules/eks/cluster)
76+
for an example of how to do that.
77+
78+
### Launch Templates
79+
80+
This module always uses a [launch template](https://docs.aws.amazon.com/autoscaling/ec2/userguide/launch-templates.html)
81+
to create the node group. You can create your own launch template and
82+
pass in its ID, or else this module will create one for you.
83+
84+
The AWS default for EKS is that if the launch template is updated, the existing nodes will not be affected. Only
85+
new instances added to the node group would get the changes specified in the new launch template. In contrast,
86+
when the launch template changes, this module can immediately create a new node group from the new launch template
87+
to replace the old one.
88+
89+
See the inputs `create_before_destroy` and `immediately_apply_lt_changes` for details about how to control this behavior.
90+
91+
### Operating system differences
92+
93+
Currently, EKS supports 4 Operating Systems: Amazon Linux 2, Amazon Linux 2023, Bottlerocket, and Windows Server.
94+
This module supports all 4 OSes, but support for detailed configuration of the nodes varies by OS. The 4 inputs:
95+
96+
1. `before_cluster_joining_userdata`
97+
2. `kubelet_additional_options`
98+
3. `bootstrap_additional_options`
99+
4. `after_cluster_joining_userdata`
100+
101+
are fully supported for Amazon Linux 2 and Windows, and take advantage of the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh)
102+
supplied on those AMIs. **NONE** of these inputs are supported on Bottlerocket. On AL2023, only the first 2 are supported.
103+
104+
Note that for all OSes, you can supply the complete `userdata` contents, which will be untouched by this module, via `userdata_override_base64`.
67105
68-
**IMPORTANT:** This module provisions an `EKS Node Group` nodes globally accessible by SSH (22) port. Normally, AWS recommends that no security group allows unrestricted ingress access to port 22 .
69106
70-
introduction: ""
71107
# How to use this project
72-
usage: |2-
108+
usage: |-
73109
74110
### Major Changes (breaking and otherwise)
75111
112+
With the v3.0.0 release of this module, support for Amazon Linux 2023 (AL2023) has
113+
been added, and some breaking changes have been made. Please see the
114+
[release notes](https://github.com/cloudposse/terraform-aws-eks-node-group/releases/tag/3.0.0)
115+
for details.
116+
76117
With the v2.0.0 (a.k.a. v0.25.0) release of this module, it has undergone major breaking
77118
changes and added new features. Please see the [migration](docs/migration-v1-v2.md)
78119
document for details.
@@ -83,13 +124,6 @@ usage: |2-
83124
For automated tests of the complete example using [bats](https://github.com/bats-core/bats-core) and [Terratest](https://github.com/gruntwork-io/terratest) (which tests and deploys the example on AWS),
84125
see [test](test).
85126
86-
### Terraform Version
87-
88-
Terraform version 1.0 is out. Before that, there was Terraform version 0.15, 0.14, 0.13 and so on.
89-
The v2.0.0 release of this module drops support for Terraform 0.13. That version is old and has lots of known issues.
90-
There are hardly any breaking changes between Terraform 0.13 and 1.0, so please upgrade to
91-
the latest Terraform version before raising any issues about this module.
92-
93127
### Sources of Information
94128
95129
- The code examples below are manually updated and have a tendency to fall out of sync with actual code,
@@ -159,7 +193,7 @@ usage: |2-
159193
module "eks_cluster" {
160194
source = "cloudposse/eks-cluster/aws"
161195
# Cloud Posse recommends pinning every module to a specific version
162-
# version = "2.x.x"
196+
# version = "4.x.x"
163197
164198
vpc_id = module.vpc.vpc_id
165199
subnet_ids = module.subnets.public_subnet_ids
@@ -173,7 +207,7 @@ usage: |2-
173207
module "eks_node_group" {
174208
source = "cloudposse/eks-node-group/aws"
175209
# Cloud Posse recommends pinning every module to a specific version
176-
# version = "2.x.x"
210+
# version = "3.x.x"
177211
178212
instance_types = [var.instance_type]
179213
subnet_ids = module.subnets.public_subnet_ids

‎ami.tf‎

Lines changed: 104 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,80 +1,116 @@
11

2+
# Previously, we found AMIs by using the aws_ami data source with a name_regex filter
3+
# and `most_recent = true`. Unfortunately, `most_recent` means most recently created,
4+
# and may not be the most recent Kubernetes version if, for example, a previous version
5+
# had a new `eksbuild`. So instead, we now use the AMI IDs published in SSM.
6+
# See https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html
7+
# https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id-bottlerocket.html
8+
9+
# Amazon Linux: https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html
10+
# aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.30/amazon-linux-2/recommended/image_id \
11+
# --query "Parameter.Value" --output text
12+
# Bottlerocket https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART-EKS.md#finding-an-ami
13+
# aws ssm get-parameter --name /aws/service/bottlerocket/aws-k8s-1.30/x86_64/latest/image_id \
14+
# --query "Parameter.Value" --output text
15+
# Windows: https://docs.aws.amazon.com/eks/latest/userguide/retrieve-windows-ami-id.html
16+
# aws ssm get-parameter --name /aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-1.30/image_id \
17+
# --region region-code --query "Parameter.Value" --output text
18+
19+
220
locals {
3-
# "amazon-eks-gpu-node-",
4-
arch_label_map = {
5-
AL2_x86_64 = "",
6-
AL2_x86_64_GPU = "-gpu",
7-
AL2_ARM_64 = "-arm64",
8-
BOTTLEROCKET_x86_64 = "x86_64",
9-
BOTTLEROCKET_ARM_64 = "aarch64"
10-
BOTTLEROCKET_ARM_64_NVIDIA = "-gpu"
11-
BOTTLEROCKET_x86_64_NVIDIA = "-gpu"
12-
WINDOWS_CORE_2019_x86_64 = ""
13-
WINDOWS_FULL_2019_x86_64 = ""
14-
WINDOWS_CORE_2022_x86_64 = ""
15-
WINDOWS_FULL_2022_x86_64 = ""
21+
given_ami_id = length(var.ami_image_id) > 0
22+
23+
# Public SSM parameters all start with /aws/service/
24+
25+
ami_os = split("_", var.ami_type)[0]
26+
27+
# format string that makes
28+
# format(fmt, specifier, k8s_version) the SSM parameter name to retrieve
29+
30+
ami_ssm_format = {
31+
AL2_x86_64 = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2/%[1]v/image_id"
32+
AL2_x86_64_GPU = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2-gpu/%[1]v/image_id"
33+
AL2_ARM_64 = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2-arm64/%[1]v/image_id"
34+
AL2023_x86_64_STANDARD = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2023/x86_64/standard/%[1]v/image_id"
35+
AL2023_ARM_64_STANDARD = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2023/arm64/standard/%[1]v/image_id"
36+
BOTTLEROCKET_x86_64 = "/aws/service/bottlerocket/aws-k8s-%[2]v/x86_64/%[1]v/image_id"
37+
BOTTLEROCKET_ARM_64 = "/aws/service/bottlerocket/aws-k8s-%[2]v/arm64/%[1]v/image_id"
38+
BOTTLEROCKET_x86_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-%[2]v-nvidia/x86_64/%[1]v/image_id"
39+
BOTTLEROCKET_ARM_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-%[2]v-nvidia/arm64/%[1]v/image_id"
40+
WINDOWS_CORE_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-%[2]v/image_id"
41+
WINDOWS_FULL_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Full-EKS_Optimized-%[2]v/image_id"
42+
WINDOWS_CORE_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-%[2]v/image_id"
43+
WINDOWS_FULL_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-EKS_Optimized-%[2]v/image_id"
44+
}
45+
46+
release_version_parts = concat(split("-", try(var.ami_release_version[0], "")), ["", ""])
47+
amazon_linux_ami_name_release_part = try(join(".", slice(split(".", local.release_version_parts[0]), 0, 2)), "")
48+
# AMI Public SSM Parameter specifiers?
49+
# Release versions for AL2 and AL2023 are from https://github.com/awslabs/amazon-eks-ami/releases
50+
# Amazon Linux Release Version: 1.29.0-20240213
51+
# AL2
52+
# AMI name: amazon-eks-node-1.29-v20240117
53+
# AMI SSM param: /aws/service/eks/optimized-ami/1.29/amazon-linux-2/amazon-eks-node-1.29-v20240117/image_id
54+
# AL2023
55+
# AMI name: amazon-eks-node-al2023-x86_64-standard-1.29-v20240213
56+
# AMI SSM param: /aws/service/eks/optimized-ami/1.29/amazon-linux-2023/x86_64/standard/amazon-eks-node-al2023-x86_64-standard-1.29-v20240213/image_id
57+
# Specifiers for Bottlerocket are the bare release version (e.g. `1.18.0`) or
58+
# the release version and the first 8 characters of the commit hash (e.g. `1.18.0-7452c37e`). NOTE: GitHub commit hash abbreviations are only 7 characters.
59+
# From:
60+
# Bottlerocket:
61+
# AMI name: bottlerocket-aws-k8s-1.29-nvidia-x86_64-v1.18.0-7452c37e
62+
# AMI SSM param: /aws/service/bottlerocket/aws-k8s-1.26-nvidia/x86_64/1.18.0/image_id # No "v"
63+
# /aws/service/bottlerocket/aws-k8s-1.26-nvidia/x86_64/1.18.0-7452c37e/image_id
64+
# Windows does not allow a specifier for SSM parameters, they only have the latest AMI ID
65+
ami_specifier_amazon_linux = {
66+
AL2_x86_64 = format("amazon-eks-node-%v-v%v", local.amazon_linux_ami_name_release_part, local.release_version_parts[1])
67+
AL2_x86_64_GPU = format("amazon-eks-gpu-node-%v-v%v", local.amazon_linux_ami_name_release_part, local.release_version_parts[1])
68+
AL2_ARM_64 = format("amazon-eks-arm64-node-%v-v%v", local.amazon_linux_ami_name_release_part, local.release_version_parts[1])
69+
AL2023_x86_64_STANDARD = format("amazon-eks-node-al2023-x86_64-standard-%v-v%v", local.amazon_linux_ami_name_release_part, local.release_version_parts[1])
70+
AL2023_ARM_64_STANDARD = format("amazon-eks-node-al2023-arm64-standard-%v-v%v", local.amazon_linux_ami_name_release_part, local.release_version_parts[1])
1671
}
1772

18-
ami_kind = split("_", var.ami_type)[0] != "WINDOWS" ? split("_", var.ami_type)[0] : format("WINDOWS_%s_%s", split("_", var.ami_type)[1], split("_", var.ami_type)[2])
19-
20-
ami_format = {
21-
# amazon-eks{arch_label}-node-{ami_kubernetes_version}-v{ami_version}
22-
# e.g. amazon-eks-arm64-node-1.21-v20211013
23-
AL2 = "amazon-eks%s-node-%s"
24-
# bottlerocket-aws-k8s-{ami_kubernetes_version}-{arch_label}-v{ami_version}
25-
# e.g. bottlerocket-aws-k8s-1.21-x86_64-v1[2].0-ccf1b754
26-
BOTTLEROCKET = "bottlerocket-aws-k8s-%s-%s-%s"
27-
# Windows_Server-2019-English-Core-EKS_Optimized-{ami_kubernetes_version}-{ami_version}
28-
# e.g. Windows_Server-2019-English-Core-EKS_Optimized-1.23-2022.11.08
29-
WINDOWS_CORE_2019 = "Windows_Server-2019-English-Core-EKS_Optimized-%s-%s"
30-
WINDOWS_FULL_2019 = "Windows_Server-2019-English-Full-EKS_Optimized-%s-%s"
31-
WINDOWS_CORE_2022 = "Windows_Server-2022-English-Core-EKS_Optimized-%s-%s"
32-
WINDOWS_FULL_2022 = "Windows_Server-2022-English-Full-EKS_Optimized-%s-%s"
73+
ami_specifier = length(var.ami_release_version) == 0 ? (local.ami_os == "BOTTLEROCKET" ? "latest" : "recommended") : (
74+
lookup(local.ami_specifier_amazon_linux, var.ami_type, var.ami_release_version[0])
75+
)
76+
77+
# As usual, Windows is difficult.
78+
is_window_version = local.ami_os == "WINDOWS" && local.ami_specifier != "recommended"
79+
80+
windows_name_base = {
81+
WINDOWS_CORE_2019_x86_64 = "Windows_Server-2019-English-Core-EKS_Optimized"
82+
WINDOWS_FULL_2019_x86_64 = "Windows_Server-2019-English-Full-EKS_Optimized"
83+
WINDOWS_CORE_2022_x86_64 = "Windows_Server-2022-English-Core-EKS_Optimized"
84+
WINDOWS_FULL_2022_x86_64 = "Windows_Server-2022-English-Full-EKS_Optimized"
3385
}
3486

35-
# Kubernetes version priority (first one to be set wins)
36-
# 1. prefix of var.ami_release_version
37-
# 2. var.kubernetes_version
38-
# 3. data.eks_cluster.this.kubernetes_version
39-
need_cluster_kubernetes_version = local.enabled ? local.need_ami_id && length(var.kubernetes_version) == 0 : false
40-
41-
use_cluster_kubernetes_version = local.need_cluster_kubernetes_version && (local.ami_kind == "BOTTLEROCKET" || length(var.ami_release_version) == 0)
42-
43-
ami_kubernetes_version = local.need_ami_id ? (local.use_cluster_kubernetes_version ? data.aws_eks_cluster.this[0].version :
44-
regex("^(\\d+\\.\\d+)", coalesce(local.ami_kind == "AL2" ? try(var.ami_release_version[0], null) : null, try(var.kubernetes_version[0], null)))[0]
45-
) : ""
46-
47-
# if ami_release_version is provided
48-
ami_version_regex = local.need_ami_id ? {
49-
# if ami_release_version = "1.21-20211013"
50-
# insert the letter v prior to the ami_version so it becomes 1.21-v20211013
51-
# if not, use the kubernetes version
52-
AL2 = (length(var.ami_release_version) == 1 ? replace(var.ami_release_version[0], "/^(\\d+\\.\\d+)\\.\\d+-(\\d+)$/", "$1-v$2") : "${local.ami_kubernetes_version}-*"),
53-
# if ami_release_version = "1.2.0-ccf1b754"
54-
# prefix the ami release version with the letter v
55-
# if not, use an asterisk
56-
BOTTLEROCKET = (length(var.ami_release_version) == 1 ? format("v%s", var.ami_release_version[0]) : "*"),
57-
WINDOWS_CORE_2019 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
58-
WINDOWS_FULL_2019 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
59-
WINDOWS_CORE_2022 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
60-
WINDOWS_FULL_2022 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
61-
} : {}
62-
63-
ami_regex = local.need_ami_id ? {
64-
AL2 = format(local.ami_format["AL2"], local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
65-
BOTTLEROCKET = format(local.ami_format["BOTTLEROCKET"], local.ami_kubernetes_version, local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
66-
WINDOWS_CORE_2019 = format(local.ami_format["WINDOWS_CORE_2019"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
67-
WINDOWS_FULL_2019 = format(local.ami_format["WINDOWS_FULL_2019"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
68-
WINDOWS_CORE_2022 = format(local.ami_format["WINDOWS_CORE_2022"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
69-
WINDOWS_FULL_2022 = format(local.ami_format["WINDOWS_FULL_2022"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
70-
} : {}
87+
# We do not really need to compute all the names, but it makes debugging easier if we do.
88+
ami_name_windows = { for k, v in local.windows_name_base : k => format("%s-%s", v, try(var.ami_release_version[0], "")) }
89+
90+
fetched_ami_id = try(local.is_window_version ? data.aws_ami.windows_ami[0].image_id : data.aws_ssm_parameter.ami_id[0].insecure_value, "")
91+
ami_id = local.given_ami_id ? var.ami_image_id[0] : local.fetched_ami_id
7192
}
7293

73-
data "aws_ami" "selected" {
74-
count = local.enabled && local.need_ami_id ? 1 : 0
94+
data "aws_ssm_parameter" "ami_id" {
95+
count = local.need_to_get_ami_id && !local.is_window_version ? 1 : 0
7596

76-
most_recent = true
77-
name_regex = local.ami_regex[local.ami_kind]
97+
name = format(local.ami_ssm_format[var.ami_type], local.ami_specifier, local.resolved_kubernetes_version)
98+
99+
lifecycle {
100+
precondition {
101+
condition = var.ami_type != "CUSTOM"
102+
error_message = "The AMI ID must be supplied when AMI type is \"CUSTOM\"."
103+
}
104+
}
105+
}
106+
107+
data "aws_ami" "windows_ami" {
108+
count = local.need_to_get_ami_id && local.is_window_version ? 1 : 0
78109

79110
owners = ["amazon"]
111+
filter {
112+
name = "name"
113+
values = [local.ami_name_windows[var.ami_type]]
114+
}
80115
}
116+

0 commit comments

Comments
 (0)