feat: Introduce support for managing the default worker pool (#499)

Co-authored-by: Vincent Burckhardt <[email protected]>

Author: Aashiq-J

README.md

@@ -105,9 +105,42 @@ module "ocp_base" {
         }
     ]
   }
+  worker_pools         = [
+    {
+      subnet_prefix    = "default"
+      pool_name        = "default"
+      machine_type     = "bx2.4x16"
+      workers_per_zone = 2
+    }
+  ]
 }
 ```
 
+### Default Worker Pool
+
+You can manage the default worker pool using Terraform, and make changes to it through this module. This option is enabled by default. Under the hood, the default worker pool is imported as a `ibm_container_vpc_worker_pool` resource. Advanced users may opt-out of this option by setting `import_default_worker_pool_on_create` parameter to `false`. For most use cases it is recommended to keep this variable to `true`.
+
+- **Important**: If the default worker pool is handled as a stand-alone ibm_container_vpc_worker_pool (which is the default behavior), you must manually remove all worker pools from the Terraform state prior to running a `terraform destroy` command on the module to avoid an error due to this [limitation](https://cloud.ibm.com/docs/containers?topic=containers-faqs#smallest_cluster).
+    - Terraform CLI. Example for a cluster with 2 worker pools: one named 'default' and the other named 'secondarypool'.
+    ```sh
+        $ terraform state list | grep ibm_container_vpc_worker_pool
+          > module.ocp_base.data.ibm_container_vpc_worker_pool.all_pools["default"]
+          > module.ocp_base.data.ibm_container_vpc_worker_pool.all_pools["secondarypool"]
+          > ...
+
+        $ terraform state rm "module.ocp_base.ibm_container_vpc_worker_pool.all_pools[\"default\"]"
+        $ terraform state rm "module.ocp_base.ibm_container_vpc_worker_pool.all_pools[\"secondarypool\"]"
+        $ ...
+    ```
+    - Schematics. Example for a cluster with 2 worker pools: one named 'default' and the other named 'secondarypool'.
+    ```sh
+        $ ibmcloud schematics workspace state rm --id <workspace_id> --address "module.ocp_base.ibm_container_vpc_worker_pool.all_pools[\"default\"]"
+        $ ibmcloud schematics workspace state rm --id <workspace_id> --address "module.ocp_base.ibm_container_vpc_worker_pool.all_pools[\"secondarypool\"]"
+        $ ...
+    ```
+
+- **Important**: If the default worker pool is handled as a stand-alone ibm_container_vpc_worker_pool (which is the default behavior), if you wish to make any change to the default worker pool which requires the re-creation of the default pool (for example a change to the worker node `operating_system`), then set the variable `allow_default_worker_pool_replacement` to true, perform the apply, and set it back for false. This is ONLY needed for changes that require the recreation of existing worker nodes in the default pool, and is NOT needed for scenario such as changing the number of workers in the default worker pool. This approach is due to a limitation in the terraform provider that may be lifted in the future.
+
 ### Advanced security group options
 
 The Terraform module provides options to attach additional security groups to the worker nodes, VPE, and load balancer associated with the cluster.
@@ -195,7 +228,7 @@ Optionally, you need the following permissions to attach Access Management tags
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
-| <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.67.0, < 2.0.0 |
+| <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.68.0, < 2.0.0 |
 | <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.16.1, < 3.0.0 |
 | <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.2.1, < 4.0.0 |
 
@@ -245,6 +278,7 @@ Optionally, you need the following permissions to attach Access Management tags
 | <a name="input_additional_lb_security_group_ids"></a> [additional\_lb\_security\_group\_ids](#input\_additional\_lb\_security\_group\_ids) | Additional security groups to add to the load balancers associated with the cluster. Ensure that the number\_of\_lbs is set to the number of LBs associated with the cluster. This comes in addition to the IBM maintained security group. | `list(string)` | `[]` | no |
 | <a name="input_additional_vpe_security_group_ids"></a> [additional\_vpe\_security\_group\_ids](#input\_additional\_vpe\_security\_group\_ids) | Additional security groups to add to all existing load balancers. This comes in addition to the IBM maintained security group. | <pre>object({<br>    master   = optional(list(string), [])<br>    registry = optional(list(string), [])<br>    api      = optional(list(string), [])<br>  })</pre> | `{}` | no |
 | <a name="input_addons"></a> [addons](#input\_addons) | Map of OCP cluster add-on versions to install (NOTE: The 'vpc-block-csi-driver' add-on is installed by default for VPC clusters and 'ibm-storage-operator' is installed by default in OCP 4.15 and later, however you can explicitly specify it here if you wish to choose a later version than the default one). For full list of all supported add-ons and versions, see https://cloud.ibm.com/docs/containers?topic=containers-supported-cluster-addon-versions | <pre>object({<br>    debug-tool                = optional(string)<br>    image-key-synchronizer    = optional(string)<br>    openshift-data-foundation = optional(string)<br>    vpc-file-csi-driver       = optional(string)<br>    static-route              = optional(string)<br>    cluster-autoscaler        = optional(string)<br>    vpc-block-csi-driver      = optional(string)<br>    ibm-storage-operator      = optional(string)<br>  })</pre> | `{}` | no |
+| <a name="input_allow_default_worker_pool_replacement"></a> [allow\_default\_worker\_pool\_replacement](#input\_allow\_default\_worker\_pool\_replacement) | (Advanced users) Set to true to allow the module to recreate a default worker pool. Only use in the case where you are getting an error indicating that the default worker pool cannot be replaced on apply. Once the default worker pool is handled as a stand-alone ibm\_container\_vpc\_worker\_pool, if you wish to make any change to the default worker pool which requires the re-creation of the default pool set this variable to true. | `bool` | `false` | no |
 | <a name="input_attach_ibm_managed_security_group"></a> [attach\_ibm\_managed\_security\_group](#input\_attach\_ibm\_managed\_security\_group) | Specify whether to attach the IBM-defined default security group (whose name is kube-<clusterid>) to all worker nodes. Only applicable if custom\_security\_group\_ids is set. | `bool` | `true` | no |
 | <a name="input_cluster_config_endpoint_type"></a> [cluster\_config\_endpoint\_type](#input\_cluster\_config\_endpoint\_type) | Specify which type of endpoint to use for for cluster config access: 'default', 'private', 'vpe', 'link'. 'default' value will use the default endpoint of the cluster. | `string` | `"default"` | no |
 | <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | The name that will be assigned to the provisioned cluster | `string` | n/a | yes |
@@ -257,6 +291,7 @@ Optionally, you need the following permissions to attach Access Management tags
 | <a name="input_existing_cos_id"></a> [existing\_cos\_id](#input\_existing\_cos\_id) | The COS id of an already existing COS instance to use for OpenShift internal registry storage. Only required if 'enable\_registry\_storage' and 'use\_existing\_cos' are true | `string` | `null` | no |
 | <a name="input_force_delete_storage"></a> [force\_delete\_storage](#input\_force\_delete\_storage) | Flag indicating whether or not to delete attached storage when destroying the cluster - Default: false | `bool` | `false` | no |
 | <a name="input_ignore_worker_pool_size_changes"></a> [ignore\_worker\_pool\_size\_changes](#input\_ignore\_worker\_pool\_size\_changes) | Enable if using worker autoscaling. Stops Terraform managing worker count | `bool` | `false` | no |
+| <a name="input_import_default_worker_pool_on_create"></a> [import\_default\_worker\_pool\_on\_create](#input\_import\_default\_worker\_pool\_on\_create) | (Advanced users) Whether to handle the default worker pool as a stand-alone ibm\_container\_vpc\_worker\_pool resource on cluster creation. Only set to false if you understand the implications of managing the default worker pool as part of the cluster resource. Set to true to import the default worker pool as a separate resource. Set to false to manage the default worker pool as part of the cluster resource. | `bool` | `true` | no |
 | <a name="input_kms_config"></a> [kms\_config](#input\_kms\_config) | Use to attach a KMS instance to the cluster. If account\_id is not provided, defaults to the account in use. | <pre>object({<br>    crk_id           = string<br>    instance_id      = string<br>    private_endpoint = optional(bool, true) # defaults to true<br>    account_id       = optional(string)     # To attach KMS instance from another account<br>    wait_for_apply   = optional(bool, true) # defaults to true so terraform will wait until the KMS is applied to the master, ready and deployed<br>  })</pre> | `null` | no |
 | <a name="input_manage_all_addons"></a> [manage\_all\_addons](#input\_manage\_all\_addons) | Instructs Terraform to manage all cluster addons, even if addons were installed outside of the module. If set to 'true' this module will destroy any addons that were installed by other sources. | `bool` | `false` | no |
 | <a name="input_number_of_lbs"></a> [number\_of\_lbs](#input\_number\_of\_lbs) | The number of LBs to associated the additional\_lb\_security\_group\_names security group with. | `number` | `1` | no |

common-dev-assets

@@ -6,7 +6,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = "1.67.0"
+      version = "1.68.0"
     }
   }
 }

examples/add_rules_to_sg/version.tf

@@ -109,7 +109,7 @@ locals {
     {
       subnet_prefix                     = "zone-1"
       pool_name                         = "default" # ibm_container_vpc_cluster automatically names default pool "default" (See https://github.com/IBM-Cloud/terraform-provider-ibm/issues/2849)
-      machine_type                      = "bx2.4x16"
+      machine_type                      = "mx2.4x32"
       workers_per_zone                  = 1
       enableAutoscaling                 = true
       minSize                           = 1

examples/advanced/main.tf

@@ -6,7 +6,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = ">= 1.67.0"
+      version = ">= 1.68.0"
     }
     kubernetes = {
       source  = "hashicorp/kubernetes"

examples/advanced/version.tf

@@ -6,7 +6,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = "1.67.0"
+      version = "1.68.0"
     }
   }
 }

examples/basic/version.tf

@@ -6,7 +6,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = ">= 1.67.0"
+      version = ">= 1.68.0"
     }
   }
 }

examples/cross_kms_support/version.tf

@@ -6,7 +6,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "IBM-Cloud/ibm"
-      version = ">= 1.67.0"
+      version = ">= 1.68.0"
     }
   }
 }

examples/custom_sg/version.tf

@@ -251,21 +251,22 @@ module "custom_sg" {
 }
 
 module "ocp_fscloud" {
-  source                           = "../../modules/fscloud"
-  cluster_name                     = var.prefix
-  resource_group_id                = module.resource_group.resource_group_id
-  region                           = var.region
-  force_delete_storage             = true
-  vpc_id                           = module.vpc.vpc_id
-  vpc_subnets                      = local.cluster_vpc_subnets
-  existing_cos_id                  = module.cos_fscloud.cos_instance_id
-  worker_pools                     = local.worker_pools
-  tags                             = var.resource_tags
-  access_tags                      = var.access_tags
-  ocp_version                      = var.ocp_version
-  additional_lb_security_group_ids = [module.custom_sg["custom-lb-sg"].security_group_id]
-  use_private_endpoint             = true
-  ocp_entitlement                  = var.ocp_entitlement
+  source                               = "../../modules/fscloud"
+  cluster_name                         = var.prefix
+  resource_group_id                    = module.resource_group.resource_group_id
+  region                               = var.region
+  force_delete_storage                 = true
+  vpc_id                               = module.vpc.vpc_id
+  vpc_subnets                          = local.cluster_vpc_subnets
+  existing_cos_id                      = module.cos_fscloud.cos_instance_id
+  worker_pools                         = local.worker_pools
+  tags                                 = var.resource_tags
+  access_tags                          = var.access_tags
+  ocp_version                          = var.ocp_version
+  import_default_worker_pool_on_create = false
+  additional_lb_security_group_ids     = [module.custom_sg["custom-lb-sg"].security_group_id]
+  use_private_endpoint                 = true
+  ocp_entitlement                      = var.ocp_entitlement
   kms_config = {
     instance_id      = var.hpcs_instance_guid
     crk_id           = local.cluster_hpcs_cluster_key_id

examples/fscloud/main.tf

@@ -6,7 +6,7 @@ terraform {
   required_providers {
     ibm = {
       source  = "ibm-cloud/ibm"
-      version = ">= 1.67.0"
+      version = ">= 1.68.0"
     }
     logdna = {
       source  = "logdna/logdna"