Windows node support (#139)

* Add initial changes for windows managed node support
Add taint example to windows doc

* Update iam.tf

Co-authored-by: Andriy Knysh <[email protected]>

* Update Changes to allow windows autotaint label, fixes

* Add the windows taint by default to windows nodes
* Correct ami_kind filter
* Alter userdata nt to match current userscript
* Guard userdata against errors from userscript stopping node-join.
* Remove disk_size
* Add windnows_coredns_service_address because EKS Windows networking isn't fab.

* Remove extraneous statement, fix output

* Add drive mapping to windows userdata; put bootstrap extra args in correct place and remove extraneous params (based on eksctl code base and aws generated script)

* Convert ami.tf maps to map format ("":"" to key = "value")

* Revert min provider; remove enclosure on ipv6 policy

* Uncomment userdata/label versions in example

* Remove leading whitespace in markdown

* Change `ami_release_version` regex to support windows ami format

Tested here: https://regex101.com/r/xb7q2f/2

* Auto Format

* Update versions.tf

Ref version change related to change log https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md#4480-december-19-2022

* Revert renaming of linux worker node group in example/complete

* Auto Format

---------

Co-authored-by: Andriy Knysh <[email protected]>
Co-authored-by: cloudpossebot <[email protected]>

Author: 3 people

.gitignore

@@ -13,3 +13,7 @@
 
 # vim editor
 *.swp
+
+/.terraform.lock.hcl
+/.vscode
+/examples/complete/.terraform.lock.hcl

README.md

@@ -243,19 +243,56 @@ Available targets:
 ```
 <!-- markdownlint-restore -->
 <!-- markdownlint-disable -->
+## Windows Managed Node groups
+
+Windows managed node-groups have a few pre-requisites.
+
+* Your cluster must contain at least one linux based worker node
+* Your EKS Cluster must have the `AmazonEKSVPCResourceController` and `AmazonEKSClusterPolicy` policies attached
+* Your cluster must have a config-map called amazon-vpc-cni with the following content
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+name: amazon-vpc-cni
+namespace: kube-system
+data:
+enable-windows-ipam: "true"
+```
+
+* Windows nodes will automatically be tainted
+
+```yaml
+kubernetes_taints = [{
+  key    = "WINDOWS"
+  value  = "true"
+  effect = "NO_SCHEDULE"
+}]
+```
+* Any pods that target Windows will need to have the following attributes set in their manifest
+
+```yaml
+  nodeSelector:
+    kubernetes.io/os: windows
+    kubernetes.io/arch: amd64
+```
+
+https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html
+<!-- markdownlint-disable -->
 ## Requirements
 
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.14.11 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.56 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.48 |
 | <a name="requirement_random"></a> [random](#requirement\_random) | >= 2.0 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.56 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.48 |
 | <a name="provider_random"></a> [random](#provider\_random) | >= 2.0 |
 
 ## Modules
@@ -294,8 +331,8 @@ Available targets:
 | <a name="input_additional_tag_map"></a> [additional\_tag\_map](#input\_additional\_tag\_map) | Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`.<br>This is for some rare cases where resources want additional configuration of tags<br>and therefore take a list of maps with tag key, value, and additional configuration. | `map(string)` | `{}` | no |
 | <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node after joining the EKS cluster (after executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
 | <a name="input_ami_image_id"></a> [ami\_image\_id](#input\_ami\_image\_id) | AMI to use. Ignored if `launch_template_id` is supplied. | `list(string)` | `[]` | no |
-| <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | EKS AMI version to use, e.g. For AL2 "1.16.13-20200821" or for bottlerocket "1.2.0-ccf1b754" (no "v"). For AL2 and bottlerocket, it defaults to latest version for Kubernetes version. | `list(string)` | `[]` | no |
-| <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64`, `AL2_x86_64_GPU`, `AL2_ARM_64`, `BOTTLEROCKET_x86_64`, and `BOTTLEROCKET_ARM_64`. | `string` | `"AL2_x86_64"` | no |
+| <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | EKS AMI version to use, e.g. For AL2 "1.16.13-20200821" or for bottlerocket "1.2.0-ccf1b754" (no "v") or  for Windows "2023.02.14". For AL2, bottlerocket and Windows, it defaults to latest version for Kubernetes version. | `list(string)` | `[]` | no |
+| <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64_NVIDIA, BOTTLEROCKET_x86_64_NVIDIA, WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2019_x86_64, WINDOWS_CORE_2022_x86_64, WINDOWS_FULL_2022_x86_64`. | `string` | `"AL2_x86_64"` | no |
 | <a name="input_associate_cluster_security_group"></a> [associate\_cluster\_security\_group](#input\_associate\_cluster\_security\_group) | When true, associate the default cluster security group to the nodes. If disabled the EKS managed security group will not<br>be associated to the nodes, therefore the communications between pods and nodes will not work. Be aware that if no `associated_security_group_ids`<br>nor `ssh_access_security_group_ids` are provided then the nodes will have no inbound or outbound rules. | `bool` | `true` | no |
 | <a name="input_associated_security_group_ids"></a> [associated\_security\_group\_ids](#input\_associated\_security\_group\_ids) | A list of IDs of Security Groups to associate the node group with, in addition to the EKS' created security group.<br>These security groups will not be modified. | `list(string)` | `[]` | no |
 | <a name="input_attributes"></a> [attributes](#input\_attributes) | ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`,<br>in the order they appear in the list. New attributes are appended to the<br>end of the list. The elements of the list are joined by the `delimiter`<br>and treated as a single ID element. | `list(string)` | `[]` | no |
@@ -367,6 +404,7 @@ Available targets:
 | <a name="output_eks_node_group_role_name"></a> [eks\_node\_group\_role\_name](#output\_eks\_node\_group\_role\_name) | Name of the worker nodes IAM role |
 | <a name="output_eks_node_group_status"></a> [eks\_node\_group\_status](#output\_eks\_node\_group\_status) | Status of the EKS Node Group |
 | <a name="output_eks_node_group_tags_all"></a> [eks\_node\_group\_tags\_all](#output\_eks\_node\_group\_tags\_all) | A map of tags assigned to the resource, including those inherited from the provider default\_tags configuration block. |
+| <a name="output_eks_node_group_windows_note"></a> [eks\_node\_group\_windows\_note](#output\_eks\_node\_group\_windows\_note) | Instructions on changes a user needs to follow or script for a windows node group in the event of a custom ami |
 <!-- markdownlint-restore -->
 
 

README.yaml

@@ -191,6 +191,7 @@ usage: |2-
 
 include:
   - "docs/targets.md"
+  - "docs/windows.md"
   - "docs/terraform.md"
 
 # Contributors to this project

ami.tf

@@ -2,22 +2,34 @@
 locals {
   # "amazon-eks-gpu-node-",
   arch_label_map = {
-    "AL2_x86_64" : "",
-    "AL2_x86_64_GPU" : "-gpu",
-    "AL2_ARM_64" : "-arm64",
-    "BOTTLEROCKET_x86_64" : "x86_64",
-    "BOTTLEROCKET_ARM_64" : "aarch64"
+    AL2_x86_64                 = "",
+    AL2_x86_64_GPU             = "-gpu",
+    AL2_ARM_64                 = "-arm64",
+    BOTTLEROCKET_x86_64        = "x86_64",
+    BOTTLEROCKET_ARM_64        = "aarch64"
+    BOTTLEROCKET_ARM_64_NVIDIA = "-gpu"
+    BOTTLEROCKET_x86_64_NVIDIA = "-gpu"
+    WINDOWS_CORE_2019_x86_64   = ""
+    WINDOWS_FULL_2019_x86_64   = ""
+    WINDOWS_CORE_2022_x86_64   = ""
+    WINDOWS_FULL_2022_x86_64   = ""
   }
 
-  ami_kind = split("_", var.ami_type)[0]
+  ami_kind = split("_", var.ami_type)[0] != "WINDOWS" ? split("_", var.ami_type)[0] : format("WINDOWS_%s_%s", split("_", var.ami_type)[1], split("_", var.ami_type)[2])
 
   ami_format = {
     # amazon-eks{arch_label}-node-{ami_kubernetes_version}-v{ami_version}
     # e.g. amazon-eks-arm64-node-1.21-v20211013
-    "AL2" : "amazon-eks%s-node-%s"
+    AL2 = "amazon-eks%s-node-%s"
     # bottlerocket-aws-k8s-{ami_kubernetes_version}-{arch_label}-v{ami_version}
     # e.g. bottlerocket-aws-k8s-1.21-x86_64-v1.2.0-ccf1b754
-    "BOTTLEROCKET" : "bottlerocket-aws-k8s-%s-%s-%s"
+    BOTTLEROCKET = "bottlerocket-aws-k8s-%s-%s-%s"
+    # Windows_Server-2019-English-Core-EKS_Optimized-{ami_kubernetes_version}-{ami_version}
+    # e.g. Windows_Server-2019-English-Core-EKS_Optimized-1.23-2022.11.08
+    WINDOWS_CORE_2019 = "Windows_Server-2019-English-Core-EKS_Optimized-%s-%s"
+    WINDOWS_FULL_2019 = "Windows_Server-2019-English-Full-EKS_Optimized-%s-%s"
+    WINDOWS_CORE_2022 = "Windows_Server-2022-English-Core-EKS_Optimized-%s-%s"
+    WINDOWS_FULL_2022 = "Windows_Server-2022-English-Full-EKS_Optimized-%s-%s"
   }
 
   # Kubernetes version priority (first one to be set wins)
@@ -37,19 +49,24 @@ locals {
     # if ami_release_version = "1.21-20211013"
     #   insert the letter v prior to the ami_version so it becomes 1.21-v20211013
     # if not, use the kubernetes version
-    "AL2" : (length(var.ami_release_version) == 1 ?
-      replace(var.ami_release_version[0], "/^(\\d+\\.\\d+)\\.\\d+-(\\d+)$/", "$1-v$2") :
-    "${local.ami_kubernetes_version}-*"),
+    AL2 = (length(var.ami_release_version) == 1 ? replace(var.ami_release_version[0], "/^(\\d+\\.\\d+)\\.\\d+-(\\d+)$/", "$1-v$2") : "${local.ami_kubernetes_version}-*"),
     # if ami_release_version = "1.2.0-ccf1b754"
-    #   prefex the ami release version with the letter v
+    #   prefix the ami release version with the letter v
     # if not, use an asterisk
-    "BOTTLEROCKET" : (length(var.ami_release_version) == 1 ?
-    format("v%s", var.ami_release_version[0]) : "*"),
+    BOTTLEROCKET      = (length(var.ami_release_version) == 1 ? format("v%s", var.ami_release_version[0]) : "*"),
+    WINDOWS_CORE_2019 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
+    WINDOWS_FULL_2019 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
+    WINDOWS_CORE_2022 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
+    WINDOWS_FULL_2022 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
   } : {}
 
   ami_regex = local.need_ami_id ? {
-    "AL2" : format(local.ami_format["AL2"], local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
-    "BOTTLEROCKET" : format(local.ami_format["BOTTLEROCKET"], local.ami_kubernetes_version, local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
+    AL2               = format(local.ami_format["AL2"], local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
+    BOTTLEROCKET      = format(local.ami_format["BOTTLEROCKET"], local.ami_kubernetes_version, local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
+    WINDOWS_CORE_2019 = format(local.ami_format["WINDOWS_CORE_2019"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
+    WINDOWS_FULL_2019 = format(local.ami_format["WINDOWS_FULL_2019"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
+    WINDOWS_CORE_2022 = format(local.ami_format["WINDOWS_CORE_2022"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
+    WINDOWS_FULL_2022 = format(local.ami_format["WINDOWS_FULL_2022"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
   } : {}
 }
 

docs/terraform.md

@@ -4,14 +4,14 @@
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.14.11 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.56 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.48 |
 | <a name="requirement_random"></a> [random](#requirement\_random) | >= 2.0 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.56 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.48 |
 | <a name="provider_random"></a> [random](#provider\_random) | >= 2.0 |
 
 ## Modules
@@ -50,8 +50,8 @@
 | <a name="input_additional_tag_map"></a> [additional\_tag\_map](#input\_additional\_tag\_map) | Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`.<br>This is for some rare cases where resources want additional configuration of tags<br>and therefore take a list of maps with tag key, value, and additional configuration. | `map(string)` | `{}` | no |
 | <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node after joining the EKS cluster (after executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
 | <a name="input_ami_image_id"></a> [ami\_image\_id](#input\_ami\_image\_id) | AMI to use. Ignored if `launch_template_id` is supplied. | `list(string)` | `[]` | no |
-| <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | EKS AMI version to use, e.g. For AL2 "1.16.13-20200821" or for bottlerocket "1.2.0-ccf1b754" (no "v"). For AL2 and bottlerocket, it defaults to latest version for Kubernetes version. | `list(string)` | `[]` | no |
-| <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64`, `AL2_x86_64_GPU`, `AL2_ARM_64`, `BOTTLEROCKET_x86_64`, and `BOTTLEROCKET_ARM_64`. | `string` | `"AL2_x86_64"` | no |
+| <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | EKS AMI version to use, e.g. For AL2 "1.16.13-20200821" or for bottlerocket "1.2.0-ccf1b754" (no "v") or  for Windows "2023.02.14". For AL2, bottlerocket and Windows, it defaults to latest version for Kubernetes version. | `list(string)` | `[]` | no |
+| <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64_NVIDIA, BOTTLEROCKET_x86_64_NVIDIA, WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2019_x86_64, WINDOWS_CORE_2022_x86_64, WINDOWS_FULL_2022_x86_64`. | `string` | `"AL2_x86_64"` | no |
 | <a name="input_associate_cluster_security_group"></a> [associate\_cluster\_security\_group](#input\_associate\_cluster\_security\_group) | When true, associate the default cluster security group to the nodes. If disabled the EKS managed security group will not<br>be associated to the nodes, therefore the communications between pods and nodes will not work. Be aware that if no `associated_security_group_ids`<br>nor `ssh_access_security_group_ids` are provided then the nodes will have no inbound or outbound rules. | `bool` | `true` | no |
 | <a name="input_associated_security_group_ids"></a> [associated\_security\_group\_ids](#input\_associated\_security\_group\_ids) | A list of IDs of Security Groups to associate the node group with, in addition to the EKS' created security group.<br>These security groups will not be modified. | `list(string)` | `[]` | no |
 | <a name="input_attributes"></a> [attributes](#input\_attributes) | ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`,<br>in the order they appear in the list. New attributes are appended to the<br>end of the list. The elements of the list are joined by the `delimiter`<br>and treated as a single ID element. | `list(string)` | `[]` | no |
@@ -123,4 +123,5 @@
 | <a name="output_eks_node_group_role_name"></a> [eks\_node\_group\_role\_name](#output\_eks\_node\_group\_role\_name) | Name of the worker nodes IAM role |
 | <a name="output_eks_node_group_status"></a> [eks\_node\_group\_status](#output\_eks\_node\_group\_status) | Status of the EKS Node Group |
 | <a name="output_eks_node_group_tags_all"></a> [eks\_node\_group\_tags\_all](#output\_eks\_node\_group\_tags\_all) | A map of tags assigned to the resource, including those inherited from the provider default\_tags configuration block. |
+| <a name="output_eks_node_group_windows_note"></a> [eks\_node\_group\_windows\_note](#output\_eks\_node\_group\_windows\_note) | Instructions on changes a user needs to follow or script for a windows node group in the event of a custom ami |
 <!-- markdownlint-restore -->

docs/windows.md

@@ -0,0 +1,37 @@
+<!-- markdownlint-disable -->
+## Windows Managed Node groups
+
+Windows managed node-groups have a few pre-requisites.
+
+* Your cluster must contain at least one linux based worker node
+* Your EKS Cluster must have the `AmazonEKSVPCResourceController` and `AmazonEKSClusterPolicy` policies attached
+* Your cluster must have a config-map called amazon-vpc-cni with the following content
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+name: amazon-vpc-cni
+namespace: kube-system
+data:
+enable-windows-ipam: "true"
+```
+
+* Windows nodes will automatically be tainted
+
+```yaml
+kubernetes_taints = [{
+  key    = "WINDOWS"
+  value  = "true"
+  effect = "NO_SCHEDULE"
+}]
+```
+* Any pods that target Windows will need to have the following attributes set in their manifest
+
+```yaml
+  nodeSelector:
+    kubernetes.io/os: windows
+    kubernetes.io/arch: amd64
+```
+
+https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html

examples/complete/fixtures.us-east-2.tfvars

@@ -26,8 +26,6 @@ max_size = 3
 
 min_size = 2
 
-disk_size = 20
-
 kubernetes_labels = {
   terratest = "true"
 }

examples/complete/main.tf

@@ -114,11 +114,9 @@ module "https_sg" {
   context = module.label.context
 }
 
-
 module "eks_cluster" {
-  source  = "cloudposse/eks-cluster/aws"
-  version = "2.4.0"
-
+  source                       = "cloudposse/eks-cluster/aws"
+  version                      = "2.4.0"
   region                       = var.region
   vpc_id                       = module.vpc.vpc_id
   subnet_ids                   = module.subnets.public_subnet_ids
@@ -147,7 +145,7 @@ module "eks_node_group" {
   kubernetes_version = [var.kubernetes_version]
   kubernetes_labels  = merge(var.kubernetes_labels, { attributes = coalesce(join(module.this.delimiter, module.this.attributes), "none") })
   kubernetes_taints  = var.kubernetes_taints
-  # disk_size          = var.disk_size
+
   ec2_ssh_key_name              = var.ec2_ssh_key_name
   ssh_access_security_group_ids = [module.ssh_source_access.id]
   associated_security_group_ids = [module.ssh_source_access.id, module.https_sg.id]

examples/complete/outputs.tf

@@ -93,7 +93,6 @@ output "eks_node_group_cbd_pet_name" {
   value       = module.eks_node_group.eks_node_group_cbd_pet_name
 }
 
-
 output "eks_node_group_launch_template_id" {
   value = module.eks_node_group.eks_node_group_launch_template_id
 }