Add user_kustomizations-parameter and supporting modules, which can be used to define multiple Kustomization-sets that are run sequentially.

This allows to e.g. install CRD in one step, wait for their deployment and then proceed with the next set.

Author: vsalomaki

MIGRATION.md

@@ -0,0 +1,31 @@
+# Migration advice when updating the module
+
+# 2.18.3 -> 2.19.0
+
+## User Kustomization
+The extra_kustomization-feature has been moved to a module so that multiple extra_kustomizations can be run in sequential steps.
+A new variable `user_kustomizations` is now in use, which contains the previous extra_kustomize_* vars.
+
+### Affects
+If you are using Helm charts from the `extra-manifests` folder or if you are using any of the following variables: `extra_kustomize_deployment_commands`, `extra_kustomize_parameters` or `extra_kustomize_folder`.
+
+### Steps
+
+1. Create a new variable `user_kustomizations`, see below and the kube.tf.example.
+
+```
+user_kustomizations = {
+    "1" = {
+    source_folder        = "extra-manifests" # Place here the source-folder defined previously in `var.extra_kustomize_folder`. If `var.extra_kustomize_folder` was previously undefined, leave as "extra-manifests".
+
+    kustomize_parameters = {} # Replace with contents of `var.extra_kustomize_parameters`. If `var.extra_kustomize_parameters` was previously undefined, remove the line or keep the default {}.
+
+    pre_commands         = ""
+
+    post_commands        = "" # Replace with contents of `var.extra_kustomize_deployment_commands`. If `var.extra_kustomize_deployment_commands` was previously undefined, remove the line or keep the default "".
+
+    }
+}
+```
+
+2. After placing the variables, remove the variables `extra_kustomize_deployment_commands`, `extra_kustomize_parameters` and `extra_kustomize_folder` from kube.tf.

README.md

@@ -318,12 +318,26 @@ See the [guide on adding robot servers](docs/add-robot-server.md)
 
 If you need to install additional Helm charts or Kubernetes manifests that are not provided by default, you can easily do so by using [Kustomize](https://kustomize.io). This is done by creating one or more `extra-manifests/kustomization.yaml.tpl` files beside your `kube.tf`.
 
-If you'd like to use a different folder name, you can configure it using the `extra_kustomize_folder` variable. By default, it is set to `extra-manifests`. This can be useful when working with multiple environments, allowing you to deploy different manifests for each one.
-
-These files need to be valid `Kustomization` manifests, additionally supporting terraform templating! (The templating parameters can be passed via the `extra_kustomize_parameters` variable (via a map) to the module).
+These files need to be valid `Kustomization` manifests, additionally supporting terraform templating! (The templating parameters can be passed via the `user_kustomizations` variable (via a map) to the module).
 
 All files in the `extra-manifests` directory and its subdirectories including the rendered versions of the `*.yaml.tpl` will be applied to k3s with `kubectl apply -k` (which will be executed after and independently of the basic cluster configuration).
 
+If you'd like to use a different folder name, specify the template parameters or apply the kustomizations in multiple steps due to CRD installation etc, you can override the values using `user_kustomizations`-variable, see below:
+
+Example of using user_kustomizations in `kube.tf`:
+```
+  user_kustomizations = {
+    "1" = {
+      source_folder        = "extra-manifests-pre"
+    },
+    "2" = {
+      source_folder        = "extra-manifests"
+      kustomize_parameters = {key: "LowSecretKey"}
+      post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
+    }
+  }
+```
+
 See a working example in [examples/kustomization_user_deploy](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/master/examples/kustomization_user_deploy).
 
 _You can use the above to pass all kinds of Kubernetes YAML configs, including HelmChart and/or HelmChartConfig definitions (see the previous section if you do not know what those are in the context of k3s)._
@@ -334,20 +348,126 @@ _That said, you can also use pure Terraform and import the kube-hetzner module a
 
 <details>
 
-<summary>Custom post-install actions</summary>
+<summary>Adding applications with Helm, custom pre- and post-install actions</summary>
 
 After the initial bootstrapping of your Kubernetes cluster, you might want to deploy applications using the same terraform mechanism. For many scenarios it is sufficient to create a `kustomization.yaml.tpl` file (see [Adding Extras](#adding-extras)). All applied kustomizations will be applied at once by executing a single `kubectl apply -k` command.
 
-However, some applications that e.g. provide custom CRDs (e.g. [ArgoCD](https://argoproj.github.io/cd/)) need a different deployment strategy: one has to deploy CRDs first, then wait for the deployment, before being able to install the actual application. In the ArgoCD case, not waiting for the CRD setup to finish will cause failures. Therefore, an additional mechanism is available to support these kind of deployments. Specify `extra_kustomize_deployment_commands` in your `kube.tf` file containing a series of commands to be executed, after the `Kustomization` step finished:
+However, some applications that e.g. provide custom CRDs (e.g. [ArgoCD](https://argoproj.github.io/cd/)) need a different deployment strategy: one has to deploy CRDs first, then wait for the deployment, before being able to install the actual application. In the ArgoCD case, not waiting for the CRD setup to finish will cause failures.
 
-```tf
-  extra_kustomize_deployment_commands = <<-EOT
-    kubectl -n argocd wait --for condition=established --timeout=120s crd/appprojects.argoproj.io
-    kubectl -n argocd wait --for condition=established --timeout=120s crd/applications.argoproj.io
-    kubectl apply -f /var/user_kustomize/argocd-projects.yaml
-    kubectl apply -f /var/user_kustomize/argocd-application-argocd.yaml
+To support these scenarios, you can use the `user_kustomizations` variable in your `kube.tf`. This allows you to define multiple "sets" of kustomizations that are applied sequentially. Each set can have its own `source_folder`, `kustomize_parameters`, and `pre_commands`/`post_commands` for running scripts before or after the `kubectl apply -k`. 
+
+### Example: external-secrets repo and Helm
+You can install Helm repos and CRDs before the main Kustomization scripts by adding the Helm charts a different folder, e.g. `extra-manifests-preinstall` and specifying the folder in `user_kustomizations`-settings.
+
+For example, to add `external-secrets` you can first create the CRDs:
+```yaml
+# extra-manifests-1/eso-crd.yaml.tpl
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: external-secrets
+  namespace: kube-system
+spec:
+  chart: external-secrets
+  repo: https://charts.external-secrets.io
+  targetNamespace: external-secrets
+  createNamespace: true
+```
+```yaml
+# extra-manifests-1/kustomization.yaml.tpl
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - eso-crd.yaml
+```
+
+Then create the objects that use the previously created CRDs:
+
+```yaml
+# extra-manifests-2/eso-secrets.yaml.tpl
+apiVersion: v1
+kind: Secret
+metadata:
+  name: vault-creds
+  namespace: external-secrets
+type: Opaque
+data:
+  username: ${base64encode(eso_access_username)}
+  password: ${base64encode(eso_access_password)}
+
+---
+apiVersion: external-secrets.io/v1
+kind: SecretStore
+metadata:
+  name: vault-sm-store
+  namespace: external-secrets
+spec:
+  provider:
+    aws:
+      service: SecretsManager
+      region: us-east-1
+
+      auth:
+        secretRef:
+          accessKeyIDSecretRef:
+            name: vault-creds
+            key: username
+            namespace: external-secrets
+          secretAccessKeySecretRef:
+            name: vault-creds
+            key: password
+            namespace: external-secrets
+
+```
+```yaml
+# extra-manifests-2/kustomization.yaml.tpl
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - eso-secrets.yaml
+```
+
+In `kube.tf`, specify the folders in `user_kustomizations`.
+
+```
+  # kube.tf
+  ...
+  user_kustomizations = {
+    "1" = {
+      source_folder = "extra-manifests-1"
+      post_commands = "kubectl -n external-secrets wait --for=condition=Available deployment --all --timeout=120s || true"
+    },
+    "2" = {
+      source_folder = "extra-manifests-2"
+      kustomize_parameters = {
+        eso_access_username = "..."
+        eso_access_password = "..."
+      }
+    },
     ...
-  EOT
+  }
+```
+
+### Example: ArgoCD with Post-install actions
+
+See examples from `examples/kustomization_user_deploy/helm-chart`, place the chart-files along with Kustomization.yaml.tpl into folder `argocd`.
+Then specify additional project-helms in `argocd-projects`.
+
+```
+  user_kustomizations = {
+    "1" = {
+      source_folder = "argocd"
+      post_commands = <<-EOT
+        kubectl -n argocd wait --for condition=established --timeout=120s crd/appprojects.argoproj.io
+        kubectl -n argocd wait --for condition=established --timeout=120s crd/applications.argoproj.io
+      EOT    
+    },
+    "2" = {
+      source_folder = "argocd-projects"
+    }
+  }
 ```
 
 </details>

docs/llms.md

@@ -2201,33 +2201,34 @@ Locked and loaded! Let's continue the detailed exploration.
 **Section 2.21: Kustomize and Post-Deployment Operations**
 
 ```terraform
-  # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).
-  # extra_kustomize_deployment_commands=""
+  # You can add user kustomizations to be deployed in sequence by setting the `user_kustomizations` variable. 
+  # The Kustomization "sets" are run in sequential order (by numeric key) so that you can for example install a CRD and wait for it to be deployed.
+  #
+  # Properties of each value:
+  # - source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
+  # - kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set". Defaults to {}.
+  # - pre_commands: Commands to be executed before applying the Kustomization ("kubectl apply -k"). Defaults to "".
+  # - post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc. Defaults to "".
+  # -- An example to wait for deployments in all namespaces: `kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true` (The `|| true` is necessary to prevent the script from exiting on a timeout if you want the sequence to continue.)
+  # -- It is recommended to use more specific `kubectl wait` commands depending on the case, for example filtering for a certain deployment or pod.
+  # -- You can pass full bash-compatible scripts into the `post_commands`-variable with EOT
+  # 
+  # An example:
+  # user_kustomizations = {
+  #   "1" = {
+  #     source_folder        = "extra-manifests"
+  #     kustomize_parameters = { myvar = "myvalue" }
+  #     pre_commands         = ""
+  #     post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
+  #   }
+  # }
 ```
 
-* **`extra_kustomize_deployment_commands` (String or List of Strings, Optional):**
-  * **Purpose:** Allows you to specify shell commands that will be executed *after* the module has run its main Kustomize deployment (which applies manifests for core components like CCM, CSI, Ingress, etc., based on your selections).
-  * **Mechanism:** The module likely uses a `local-exec` or `remote-exec` provisioner (if commands need to run on a node) to execute these. If they are `kubectl` commands, they'd run from where Terraform is executed, using the generated kubeconfig.
+* **`user_kustomizations` (Map of Objects, Optional):**
+  * **Purpose:** Allows you to specify Kustomization sets that are run sequentially, with each set containing its own source_folder, pre_commands, post_commands and kustomize_parameters
   * **Use Cases:**
-    * **Waiting for CRDs:** Some applications deployed via Helm or Kustomize install CustomResourceDefinitions (CRDs) first, and then CustomResources (CRs) that depend on those CRDs. There can be a race condition if the CRs are applied before the CRDs are fully registered. You could add a command here to wait for CRDs to become available (e.g., `kubectl wait --for condition=established crd/mycrd.example.com --timeout=120s`).
-    * Applying additional Kubernetes manifests that depend on the core setup.
-    * Running post-install scripts or triggering initial application setup jobs.
-  * **Format:** Can be a single string with commands separated by `&&` or `\n`, or a list of individual command strings.
-
-```terraform
-  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.
-  # extra_kustomize_parameters={}
-```
-
-* **`extra_kustomize_parameters` (Map of Strings, Optional):**
-  * **Purpose:** If you are using the module's "extra manifests" feature (where you can provide your own Kustomize setup in an `extra-manifests` directory), this map allows you to pass key-value parameters into a `kustomization.yaml.tpl` template file within that directory.
-  * **Mechanism:** The module would process `extra-manifests/kustomization.yaml.tpl` as a template, substituting placeholders with values from this map, and then run `kustomize build` on the result.
-  * **Use Case:** Parameterizing your custom Kustomize deployments based on Terraform inputs or computed values from the `kube-hetzner` module (e.g., passing in the cluster name, node IPs, etc., to your custom manifests).
-  * **Reference:** The comment points to examples in the module's repository for how to use this feature.
-
-```terraform
-  # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md
-```
+    * Some applications deployed via Helm or Kustomize install CustomResourceDefinitions (CRDs) first, and then CustomResources (CRs) that depend on those CRDs. There can be a race condition if the CRs are applied before the CRDs are fully registered. You could add a command here to wait for CRDs to become available (e.g., `kubectl wait --for condition=established crd/mycrd.example.com --timeout=120s`).
+    * The `user_kustomizations`-map allows you to define steps of install where e.g. the first step installs CRDs, checks for their proper existence and then second step that install further CRs.
 
 * **Documentation Pointer:** This directs users to example usage of the "extra manifests" feature, which is crucial for extending the module's capabilities with custom deployments.
 

examples/kustomization_user_deploy/README.md

@@ -4,6 +4,8 @@ Kube-Hetzner allows you to provide user-defined resources after the initial setu
 
 When you execute terraform apply, the manifests in the extra-manifests directory, including the rendered versions of the `*.yaml.tpl` files, will be automatically deployed to the cluster.
 
+Note: If you would like to use a different folder, define the `user_kustomizations` in kube.tf.
+
 ## Examples
 
 Here are some examples of common use cases for deploying additional resources:
@@ -12,14 +14,18 @@ Here are some examples of common use cases for deploying additional resources:
 
 ### Deploying Simple Resources
 
-The easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `extra_kustomize_parameters` map of the `kube.tf` file.
+The easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `kustomize_parameters` map of the `user_kustomizations`.
 
 #### `kube.tf`
 
 ```
 ...
-extra_kustomize_parameters = {
-  my_config_key = "somestring"
+user_kustomizations = {
+  ...
+  kustomize_parameters = {
+    my_config_key = "somestring"
+  }
+  ...
 }
 ...
 ```

examples/kustomization_user_deploy/letsencrypt/kustomization.yaml.tpl

@@ -3,4 +3,3 @@ kind: Kustomization
 
 resources:
   - letsencrypt.yaml
-

kube.tf.example

@@ -961,12 +961,27 @@ module "kube-hetzner" {
   # More information about the registration can be found here https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/
   # rancher_registration_manifest_url = "https://rancher.xyz.dev/v3/import/xxxxxxxxxxxxxxxxxxYYYYYYYYYYYYYYYYYYYzzzzzzzzzzzzzzzzzzzzz.yaml"
 
-  # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).
-  # extra_kustomize_deployment_commands=""
-
-  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.
-  # extra_kustomize_parameters={}
-
+  # You can add user kustomizations to be deployed in sequence by setting the `user_kustomizations` variable. 
+  # The Kustomization "sets" are run in sequential order (by numeric key) so that you can for example install a CRD and wait for it to be deployed.
+  #
+  # Properties of each value:
+  # - source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
+  # - kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set". Defaults to {}.
+  # - pre_commands: Commands to be executed before applying the Kustomization ("kubectl apply -k"). Defaults to "".
+  # - post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc. Defaults to "".
+  # -- An example to wait for deployments in all namespaces: `kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true` (The `|| true` is necessary to prevent the script from exiting on a timeout if you want the sequence to continue.)
+  # -- It is recommended to use more specific `kubectl wait` commands depending on the case, for example filtering for a certain deployment or pod.
+  # -- You can pass full bash-compatible scripts into the `post_commands`-variable with EOT
+  # 
+  # An example:
+  # user_kustomizations = {
+  #   "1" = {
+  #     source_folder        = "extra-manifests"
+  #     kustomize_parameters = { myvar = "myvalue" }
+  #     pre_commands         = ""
+  #     post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
+  #   }
+  # }
   # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md
 
   # It is best practice to turn this off, but for backwards compatibility it is set to "true" by default.