Move user Kustomization to module, allow sequential user Kustomizations

Author: vsalomaki

README.md

@@ -334,11 +334,40 @@ _That said, you can also use pure Terraform and import the kube-hetzner module a
 
 <details>
 
-<summary>Custom post-install actions</summary>
+<summary>Custom pre- and post-install actions</summary>
 
 After the initial bootstrapping of your Kubernetes cluster, you might want to deploy applications using the same terraform mechanism. For many scenarios it is sufficient to create a `kustomization.yaml.tpl` file (see [Adding Extras](#adding-extras)). All applied kustomizations will be applied at once by executing a single `kubectl apply -k` command.
 
-However, some applications that e.g. provide custom CRDs (e.g. [ArgoCD](https://argoproj.github.io/cd/)) need a different deployment strategy: one has to deploy CRDs first, then wait for the deployment, before being able to install the actual application. In the ArgoCD case, not waiting for the CRD setup to finish will cause failures. Therefore, an additional mechanism is available to support these kind of deployments. Specify `extra_kustomize_deployment_commands` in your `kube.tf` file containing a series of commands to be executed, after the `Kustomization` step finished:
+However, some applications that e.g. provide custom CRDs (e.g. [ArgoCD](https://argoproj.github.io/cd/)) need a different deployment strategy: one has to deploy CRDs first, then wait for the deployment, before being able to install the actual application. In the ArgoCD case, not waiting for the CRD setup to finish will cause failures. Therefore, an additional mechanism is available to support these kind of deployments. 
+
+### Pre-install Actions, Example: external-secrets repo and Helm
+You can install Helm repos and CRDs before the main Kustomization scripts by adding the helm charts to the `extra-manifests-preinstall` folder and specifying the Helm chart in `extra-manifests-preinstall/kustomization.yaml.tpl`, just like with `extra-manifests`.
+
+For example, to add `external-secrets` so that it can be referenced later on, create files:
+1. extra-manifests-preinstall/eso.yaml.tpl
+```yaml
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: external-secrets
+  namespace: kube-system
+spec:
+  chart: external-secrets
+  repo: https://charts.external-secrets.io
+  targetNamespace: external-secrets
+  createNamespace: true
+```
+2. extra-manifests-preinstall/kustomization.yaml.tpl
+```yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - eso.yaml
+```
+
+### Post-install actions, ArgoCD-example
+Specify `extra_kustomize_deployment_commands` in your `kube.tf` file containing a series of commands to be executed, after the `Kustomization` step has finished:
 
 ```tf
   extra_kustomize_deployment_commands = <<-EOT

kube.tf.example

@@ -955,9 +955,34 @@ module "kube-hetzner" {
   # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).
   # extra_kustomize_deployment_commands=""
 
-  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.
+  # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` and `extra-manifests-preinstall/kustomization.yaml.tpl` when present.
   # extra_kustomize_parameters={}
 
+  # You can add extra kustomizations to be deployed in sequence by setting the `user_kustomizations` variable. 
+  # This will override the use of `extra_kustomize_deployment_commands` and `extra_kustomize_parameters` for extra-manifests, hence they will need to be reset, see below.
+  #
+  # The Kustomization "sets" are run in sequential order (by numeric key) so that you can for example install a CRD and wait for it to be deployed.
+  #
+  # source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
+  # kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set".
+  # post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc.
+  # - An example to wait for deployments in all namespaces: `kubectl wait --for=condition=Available deployment --all -A --timeout=120s || exit 0` (The `|| exit 0` is not "required", it just makes the deployment to continue even if a deployment hasn't started up properly)
+  # - You can pass full bash-compatible scripts into the `post_commands`-variable with EOT
+  # 
+  # An example from the default values: (uncomment or copy your own)
+  # user_kustomizations = {
+  #   "1" = {
+  #     source_folder        = "extra-manifests-preinstall"
+  #     kustomize_parameters = {}
+  #     post_commands        = ""
+  #   },
+  #   "2" = {
+  #     source_folder        = "extra-manifests" # Uses `var.extra_kustomize_folder` internally, defaults to "extra-manifests"
+  #     kustomize_parameters = var.extra_kustomize_parameters # Same as `extra_kustomize_parameters` in kube.tf, replace with actual values if uncommenting
+  #     post_commands        = var.extra_kustomize_deployment_commands  # Same as `extra_kustomize_deployment_commands` in kube.tf, replace with actual values if uncommenting
+  #   }
+  # }
+
   # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md
 
   # It is best practice to turn this off, but for backwards compatibility it is set to "true" by default.

kustomization_user.tf

@@ -1,48 +1,34 @@
 locals {
-  user_kustomization_templates = try(fileset(var.extra_kustomize_folder, "**/*.yaml.tpl"), toset([]))
-}
-
-resource "null_resource" "kustomization_user" {
-  for_each = local.user_kustomization_templates
-
-  connection {
-    user           = "root"
-    private_key    = var.ssh_private_key
-    agent_identity = local.ssh_agent_identity
-    host           = local.first_control_plane_ip
-    port           = var.ssh_port
-
-    bastion_host        = local.ssh_bastion.bastion_host
-    bastion_port        = local.ssh_bastion.bastion_port
-    bastion_user        = local.ssh_bastion.bastion_user
-    bastion_private_key = local.ssh_bastion.bastion_private_key
-
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      "mkdir -p $(dirname /var/user_kustomize/${each.key})"
-    ]
+  default_user_kustomize = {
+    "1" = {
+      source_folder        = "extra-manifests-preinstall"
+      kustomize_parameters = {}
+      post_commands        = ""
+    },
+    "2" = {
+      source_folder        = var.extra_kustomize_folder
+      kustomize_parameters = var.extra_kustomize_parameters
+      post_commands        = var.extra_kustomize_deployment_commands
+    }
   }
 
-  provisioner "file" {
-    content     = templatefile("${var.extra_kustomize_folder}/${each.key}", var.extra_kustomize_parameters)
-    destination = replace("/var/user_kustomize/${each.key}", ".yaml.tpl", ".yaml")
-  }
+  user_kustomize_defaulted = length(var.user_kustomizations) > 0 ? var.user_kustomizations : local.default_user_kustomize
 
-  triggers = {
-    manifest_sha1 = "${sha1(templatefile("${var.extra_kustomize_folder}/${each.key}", var.extra_kustomize_parameters))}"
+  processed_kustomizes = {
+    for key, config in local.user_kustomize_defaulted : key => {
+      # kustomize_parameters may contain secrets
+      kustomize_parameters = sensitive(config.kustomize_parameters)
+      source_folder        = config.source_folder
+      post_commands        = config.post_commands
+    }
   }
-
-  depends_on = [
-    null_resource.kustomization
-  ]
 }
 
-resource "null_resource" "kustomization_user_deploy" {
-  count = length(local.user_kustomization_templates) > 0 ? 1 : 0
+module "user_kustomizations" {
+
+  source = "./modules/user_kustomizations"
 
-  connection {
+  ssh_connection = {
     user           = "root"
     private_key    = var.ssh_private_key
     agent_identity = local.ssh_agent_identity
@@ -53,27 +39,11 @@ resource "null_resource" "kustomization_user_deploy" {
     bastion_port        = local.ssh_bastion.bastion_port
     bastion_user        = local.ssh_bastion.bastion_user
     bastion_private_key = local.ssh_bastion.bastion_private_key
-
   }
 
-  # Remove templates after rendering, and apply changes.
-  provisioner "remote-exec" {
-    # Debugging: "sh -c 'for file in $(find /var/user_kustomize -type f -name \"*.yaml\" | sort -n); do echo \"\n### Template $${file}.tpl after rendering:\" && cat $${file}; done'",
-    inline = compact([
-      "rm -f /var/user_kustomize/**/*.yaml.tpl",
-      "echo 'Applying user kustomization...'",
-      "kubectl apply -k /var/user_kustomize/ --wait=true",
-      var.extra_kustomize_deployment_commands
-    ])
-  }
-
-  lifecycle {
-    replace_triggered_by = [
-      null_resource.kustomization_user
-    ]
-  }
+  kustomizations_map = local.processed_kustomizes
 
   depends_on = [
-    null_resource.kustomization_user
+    null_resource.kustomization,
   ]
 }

modules/user_kustomization_set/locals.tf

@@ -0,0 +1,9 @@
+locals {
+  source_folder_files = try(fileset(var.source_folder, "**/*.yaml.tpl"), toset([]))
+  source_files_sha = join("", [
+    for file_path in local.source_folder_files :
+    filesha1("${var.source_folder}/${file_path}")
+  ])
+  parameters_sha           = sha1(jsonencode(var.template_parameters))
+  post_commands_string_sha = sha1(var.post_commands_string)
+}

modules/user_kustomization_set/main.tf

@@ -0,0 +1,66 @@
+
+# Purpose of this module is to copy a single user kustomization "set" to control plane.
+# The set contains the yaml-files for Kustomization and the postinstall.sh script.
+
+resource "null_resource" "create_target_directory" {
+
+  triggers = {
+    source_files_sha         = local.source_files_sha
+    parameters_sha           = local.parameters_sha
+    post_commands_string_sha = local.post_commands_string_sha
+  }
+
+  connection {
+    user           = var.ssh_connection.user
+    private_key    = var.ssh_connection.private_key
+    agent_identity = var.ssh_connection.agent_identity
+    host           = var.ssh_connection.host
+    port           = var.ssh_connection.port
+
+    bastion_host        = var.ssh_connection.bastion_host
+    bastion_port        = var.ssh_connection.bastion_port
+    bastion_user        = var.ssh_connection.bastion_user
+    bastion_private_key = var.ssh_connection.bastion_private_key
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "mkdir -p ${var.destination_folder}"
+    ]
+  }
+
+  provisioner "file" {
+    content     = templatefile("${path.module}/templates/bash.sh.tpl", { commands = var.post_commands_string })
+    destination = "${var.destination_folder}/postinstall.sh"
+  }
+}
+
+resource "null_resource" "user_kustomization_template_files" {
+  for_each = local.source_folder_files
+
+  lifecycle {
+    replace_triggered_by = [
+      null_resource.create_target_directory
+    ]
+  }
+
+  connection {
+    user           = var.ssh_connection.user
+    private_key    = var.ssh_connection.private_key
+    agent_identity = var.ssh_connection.agent_identity
+    host           = var.ssh_connection.host
+    port           = var.ssh_connection.port
+
+    bastion_host        = var.ssh_connection.bastion_host
+    bastion_port        = var.ssh_connection.bastion_port
+    bastion_user        = var.ssh_connection.bastion_user
+    bastion_private_key = var.ssh_connection.bastion_private_key
+  }
+
+  provisioner "file" {
+    content     = templatefile("${var.source_folder}/${each.key}", var.template_parameters)
+    destination = replace("${var.destination_folder}/${each.key}", ".yaml.tpl", ".yaml")
+  }
+
+  depends_on = [null_resource.create_target_directory]
+}

modules/user_kustomization_set/out.tf

@@ -0,0 +1,26 @@
+output "destination_folder" {
+  value = var.destination_folder
+}
+
+output "source_files_sha" {
+  value = local.source_files_sha
+}
+
+output "parameters_sha" {
+  value = local.parameters_sha
+}
+
+output "post_commands_string_sha" {
+  value = local.post_commands_string_sha
+}
+
+output "files_count" {
+  description = "Number of template files found in the source folder."
+  value       = length(local.source_folder_files)
+}
+
+output "changes_sha" {
+  value = sha1(join("", [
+    local.source_files_sha, local.parameters_sha, local.post_commands_string_sha
+  ]))
+}

modules/user_kustomization_set/templates/bash.sh.tpl

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+${commands}

modules/user_kustomization_set/variables.tf

@@ -0,0 +1,40 @@
+variable "ssh_connection" {
+  type = object({
+    user           = string
+    private_key    = string
+    agent_identity = string
+    host           = string
+    port           = string
+
+    bastion_host        = string
+    bastion_port        = number
+    bastion_user        = string
+    bastion_private_key = string
+  })
+  sensitive = true
+}
+
+variable "source_folder" {
+  type = string
+}
+
+variable "destination_folder" {
+  type    = string
+  default = "/var/user_kustomize"
+
+  validation {
+    condition     = startswith(var.destination_folder, "/") && can(regex("^[^\\s]*$", var.destination_folder))
+    error_message = "destination_folder must start with '/' and must not contain spaces."
+  }
+}
+
+variable "template_parameters" {
+  type      = map(any)
+  default   = {}
+  sensitive = true
+}
+
+variable "post_commands_string" {
+  type    = string
+  default = ""
+}

modules/user_kustomizations/locals.tf

@@ -0,0 +1,9 @@
+locals {
+  sorted_kustomization_destination_folders = [
+    for idx in sort([
+      for key, mod in module.user_kustomization_set : tonumber(key) if mod.files_count > 0
+    ]) :
+    module.user_kustomization_set[tostring(idx)].destination_folder
+  ]
+  base_destination_folder = "/var/user_kustomize"
+}