Replace references to extra_kustomize_* with the usage of user_kustomizations[]

Author: vsalomaki

MIGRATION.md

@@ -0,0 +1,31 @@
+# Migration advice when updating the module
+
+# 2.18.3 -> 2.19.0
+
+## Extra_kustomization
+The extra_kustomization-feature has been moved to a module so that multiple extra_kustomizations can be run in sequential steps.
+A new variable `user_kustomizations` is now in use, which contains the previous extra_kustomize_* vars.
+
+### Affects
+If you are using helms from `extra-manifests`-folder or if you are using any of the following variables: `extra_kustomize_deployment_commands`, `extra_kustomize_parameters` or `extra_kustomize_folder`.
+
+### Steps
+
+1. Create a new variable `user_kustomizations`, see below and the kube.tf.example.
+
+```
+user_kustomizations = {
+    "1" = {
+    source_folder        = "extra-manifests" # Place here the source-folder defined previously in `var.extra_kustomize_folder`. If `var.extra_kustomize_folder` was previously undefined, leave as "extra-manifests".
+
+    kustomize_parameters = {} # Replace with contents of `var.extra_kustomize_parameters`. If `var.extra_kustomize_parameters` was previously undefined, remove the line or keep the default {}.
+
+    pre_commands         = ""
+
+    post_commands        = "" # Replace with contents of `var.extra_kustomize_deployment_commands`. If `var.extra_kustomize_deployment_commands` was previously undefined, remove the line or keep the default "".
+
+    }
+}
+```
+
+2. After placing the variables, remove the variables `extra_kustomize_deployment_commands`, `extra_kustomize_parameters` and `extra_kustomize_folder` from kube.tf.

README.md

@@ -318,11 +318,11 @@ See the [guide on adding robot servers](docs/add-robot-server.md)
 
 If you need to install additional Helm charts or Kubernetes manifests that are not provided by default, you can easily do so by using [Kustomize](https://kustomize.io). This is done by creating one or more `extra-manifests/kustomization.yaml.tpl` files beside your `kube.tf`.
 
-These files need to be valid `Kustomization` manifests, additionally supporting terraform templating! (The templating parameters can be passed via the `extra_kustomize_parameters` variable (via a map) to the module).
+These files need to be valid `Kustomization` manifests, additionally supporting terraform templating! (The templating parameters can be passed via the `user_kustomizations` variable (via a map) to the module).
 
 All files in the `extra-manifests` directory and its subdirectories including the rendered versions of the `*.yaml.tpl` will be applied to k3s with `kubectl apply -k` (which will be executed after and independently of the basic cluster configuration).
 
-If you'd like to use a different folder name or apply the kustomizations in multiple steps due to CRD installation etc, you can override the values using `user_kustomizations`-variable. Note! This will override `extra_kustomize_deployment_commands` and `extra_kustomize_parameters`, so you will need to move them into the `user_kustomizations`-variable.
+If you'd like to use a different folder name, specify the template parameters or apply the kustomizations in multiple steps due to CRD installation etc, you can override the values using `user_kustomizations`-variable, see below:
 
 Example of using user_kustomizations in `kube.tf`:
 ```
@@ -333,7 +333,7 @@ Example of using user_kustomizations in `kube.tf`:
     "2" = {
       source_folder        = "extra-manifests"
       kustomize_parameters = {key: "LowSecretKey"}
-      post_commands        = var.extra_kustomize_deployment_commands
+      post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
     }
   }
 ```
@@ -356,8 +356,6 @@ However, some applications that e.g. provide custom CRDs (e.g. [ArgoCD](https://
 
 To support these scenarios, you can use the `user_kustomizations` variable in your `kube.tf`. This allows you to define multiple "sets" of kustomizations that are applied sequentially. Each set can have its own `source_folder`, `kustomize_parameters`, and `pre_commands`/`post_commands` for running scripts before or after the `kubectl apply -k`. 
 
-For backward compatibility, if `user_kustomizations` is not defined, the module uses the `extra_kustomize_parameters` and `extra_kustomize_deployment_commands` with `extra-manifests` being the source folder.
-
 ### Example: external-secrets repo and Helm
 You can install Helm repos and CRDs before the main Kustomization scripts by adding the Helm charts a different folder, e.g. `extra-manifests-preinstall` and specifying the folder in `user_kustomizations`-settings.
 

docs/llms.md

@@ -2201,33 +2201,34 @@ Locked and loaded! Let's continue the detailed exploration.
 **Section 2.21: Kustomize and Post-Deployment Operations**
 
 ```terraform
-  # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).
-  # extra_kustomize_deployment_commands=""
+  # You can add user kustomizations to be deployed in sequence by setting the `user_kustomizations` variable. 
+  # The Kustomization "sets" are run in sequential order (by numeric key) so that you can for example install a CRD and wait for it to be deployed.
+  #
+  # Properties of each value:
+  # - source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
+  # - kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set". Defaults to {}.
+  # - pre_commands: Commands to be executed before applying the Kustomization ("kubectl apply -k"). Defaults to "".
+  # - post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc. Defaults to "".
+  # -- An example to wait for deployments in all namespaces: `kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true` (The `|| true` is necessary to prevent the script from exiting on a timeout if you want the sequence to continue.)
+  # -- It is recommended to use more specific `kubectl wait` commands depending on the case, for example filtering for a certain deployment or pod.
+  # -- You can pass full bash-compatible scripts into the `post_commands`-variable with EOT
+  # 
+  # An example:
+  # user_kustomizations = {
+  #   "1" = {
+  #     source_folder        = "extra-manifests"
+  #     kustomize_parameters = { myvar = "myvalue" }
+  #     pre_commands         = ""
+  #     post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
+  #   }
+  # }
 ```
 
-* **`extra_kustomize_deployment_commands` (String or List of Strings, Optional):**
-  * **Purpose:** Allows you to specify shell commands that will be executed *after* the module has run its main Kustomize deployment (which applies manifests for core components like CCM, CSI, Ingress, etc., based on your selections).
-  * **Mechanism:** The module likely uses a `local-exec` or `remote-exec` provisioner (if commands need to run on a node) to execute these. If they are `kubectl` commands, they'd run from where Terraform is executed, using the generated kubeconfig.
+* **`user_kustomizations` (Map of Objects, Optional):**
+  * **Purpose:** Allows you to specify Kustomization sets that are run sequantially, with each set containing its own source_folder, pre_commands, post_commands and kustomize_parameters
   * **Use Cases:**
-    * **Waiting for CRDs:** Some applications deployed via Helm or Kustomize install CustomResourceDefinitions (CRDs) first, and then CustomResources (CRs) that depend on those CRDs. There can be a race condition if the CRs are applied before the CRDs are fully registered. You could add a command here to wait for CRDs to become available (e.g., `kubectl wait --for condition=established crd/mycrd.example.com --timeout=120s`).
-    * Applying additional Kubernetes manifests that depend on the core setup.
-    * Running post-install scripts or triggering initial application setup jobs.
-  * **Format:** Can be a single string with commands separated by `&&` or `\n`, or a list of individual command strings.
-
-```terraform
-  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.
-  # extra_kustomize_parameters={}
-```
-
-* **`extra_kustomize_parameters` (Map of Strings, Optional):**
-  * **Purpose:** If you are using the module's "extra manifests" feature (where you can provide your own Kustomize setup in an `extra-manifests` directory), this map allows you to pass key-value parameters into a `kustomization.yaml.tpl` template file within that directory.
-  * **Mechanism:** The module would process `extra-manifests/kustomization.yaml.tpl` as a template, substituting placeholders with values from this map, and then run `kustomize build` on the result.
-  * **Use Case:** Parameterizing your custom Kustomize deployments based on Terraform inputs or computed values from the `kube-hetzner` module (e.g., passing in the cluster name, node IPs, etc., to your custom manifests).
-  * **Reference:** The comment points to examples in the module's repository for how to use this feature.
-
-```terraform
-  # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md
-```
+    * Some applications deployed via Helm or Kustomize install CustomResourceDefinitions (CRDs) first, and then CustomResources (CRs) that depend on those CRDs. There can be a race condition if the CRs are applied before the CRDs are fully registered. You could add a command here to wait for CRDs to become available (e.g., `kubectl wait --for condition=established crd/mycrd.example.com --timeout=120s`).
+    * The `user_kustomzizations`-map allows you to define steps of install where e.g. the first step installs CRDs, checks for their proper existence and then second step that install further CRs.
 
 * **Documentation Pointer:** This directs users to example usage of the "extra manifests" feature, which is crucial for extending the module's capabilities with custom deployments.
 

examples/kustomization_user_deploy/README.md

@@ -4,6 +4,8 @@ Kube-Hetzner allows you to provide user-defined resources after the initial setu
 
 When you execute terraform apply, the manifests in the extra-manifests directory, including the rendered versions of the `*.yaml.tpl` files, will be automatically deployed to the cluster.
 
+Note: If you would like to use a different folder, define the `user_kustomizations` in kube.tf.
+
 ## Examples
 
 Here are some examples of common use cases for deploying additional resources:
@@ -12,14 +14,18 @@ Here are some examples of common use cases for deploying additional resources:
 
 ### Deploying Simple Resources
 
-The easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `extra_kustomize_parameters` map of the `kube.tf` file.
+The easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `kustomize_parameters` map of the `user_kustomizations`.
 
 #### `kube.tf`
 
 ```
 ...
-extra_kustomize_parameters = {
-  my_config_key = "somestring"
+user_kustomizations = {
+  ...
+  kustomize_parameters = {
+    my_config_key = "somestring"
+  }
+  ...
 }
 ...
 ```

kube.tf.example

@@ -953,35 +953,27 @@ module "kube-hetzner" {
   # More information about the registration can be found here https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/
   # rancher_registration_manifest_url = "https://rancher.xyz.dev/v3/import/xxxxxxxxxxxxxxxxxxYYYYYYYYYYYYYYYYYYYzzzzzzzzzzzzzzzzzzzzz.yaml"
 
-  # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).
-  # extra_kustomize_deployment_commands=""
-
-  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl`.
-  # extra_kustomize_parameters={}
-
-  # You can add extra kustomizations to be deployed in sequence by setting the `user_kustomizations` variable. 
-  # This will override the use of `extra_kustomize_deployment_commands` and `extra_kustomize_parameters` for extra-manifests, hence they will need to be reset, see below.
-  #
+  # You can add user kustomizations to be deployed in sequence by setting the `user_kustomizations` variable. 
   # The Kustomization "sets" are run in sequential order (by numeric key) so that you can for example install a CRD and wait for it to be deployed.
   #
-  # source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
-  # kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set". Defaults to {}.
-  # pre_commands: Commands to be executed before applying the Kustomization ("kubectl apply -k"). Defaults to "".
-  # post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc. Defaults to "".
-  # - An example to wait for deployments in all namespaces: `kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true` (The `|| true` is necessary to prevent the script from exiting on a timeout if you want the sequence to continue.)
-  # - It is recommended to use more specific `kubectl wait` commands depending on the case, for example filtering for a certain deployment or pod.
-  # - You can pass full bash-compatible scripts into the `post_commands`-variable with EOT
+  # Properties of each value:
+  # - source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
+  # - kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set". Defaults to {}.
+  # - pre_commands: Commands to be executed before applying the Kustomization ("kubectl apply -k"). Defaults to "".
+  # - post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc. Defaults to "".
+  # -- An example to wait for deployments in all namespaces: `kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true` (The `|| true` is necessary to prevent the script from exiting on a timeout if you want the sequence to continue.)
+  # -- It is recommended to use more specific `kubectl wait` commands depending on the case, for example filtering for a certain deployment or pod.
+  # -- You can pass full bash-compatible scripts into the `post_commands`-variable with EOT
   # 
-  # An example from the default values:
+  # An example:
   # user_kustomizations = {
   #   "1" = {
   #     source_folder        = "extra-manifests"
-  #     kustomize_parameters = var.extra_kustomize_parameters # Same as `extra_kustomize_parameters` in kube.tf, replace with actual values if uncommenting
+  #     kustomize_parameters = { myvar = "myvalue" }
   #     pre_commands         = ""
-  #     post_commands        = var.extra_kustomize_deployment_commands # Same as `extra_kustomize_deployment_commands` in kube.tf, replace with actual values if uncommenting
+  #     post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
   #   }
   # }
-
   # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md
 
   # It is best practice to turn this off, but for backwards compatibility it is set to "true" by default.

kustomization_user.tf

@@ -1,10 +1,10 @@
 locals {
   default_user_kustomize = {
     "1" = {
-      source_folder        = var.extra_kustomize_folder
-      kustomize_parameters = var.extra_kustomize_parameters
+      source_folder        = "extra-manifests"
+      kustomize_parameters = {}
       pre_commands         = ""
-      post_commands        = var.extra_kustomize_deployment_commands
+      post_commands        = ""
     }
   }
 

variables.tf

@@ -1068,26 +1068,6 @@ variable "postinstall_exec" {
   description = "Additional to execute after the install calls, for example restoring a backup."
 }
 
-variable "extra_kustomize_deployment_commands" {
-  type        = string
-  default     = ""
-  description = "Commands to be executed after the `kubectl apply -k <dir>` step."
-  sensitive   = true
-}
-
-variable "extra_kustomize_parameters" {
-  type        = any
-  default     = {}
-  description = "All values will be passed to the `kustomization.tmp.yml` template."
-  sensitive   = true
-}
-
-variable "extra_kustomize_folder" {
-  type        = string
-  default     = "extra-manifests"
-  description = "Folder from where to upload extra manifests"
-}
-
 variable "user_kustomizations" {
   type = map(object({
     source_folder        = string