Creating some namespaces after cluster have been deployed

[gigo1980]

Hi, we try to create some namespaces and do other "helm", "kubectl" Stuff.

How can I wait until the cluster is up and running? I know there is a "depends_on" feature, but on what should it depends on?

provider "kubernetes" {
  host                   = module.kube-hetzner-hub-cluster.kubeconfig_data.host
  client_certificate     = module.kube-hetzner-hub-cluster.kubeconfig_data.client_certificate
  client_key             = module.kube-hetzner-hub-cluster.kubeconfig_data.client_key
  cluster_ca_certificate = module.kube-hetzner-hub-cluster.kubeconfig_data.cluster_ca_certificate
}

And this is the kubernetes namespace, that will deploy?

resource "kubernetes_namespace" "argocd" {
  depends_on = [
    module.kube-hetzner-hub-cluster
  ]
  metadata {
    name = var.namespace_argocd
  }
}

[mysticaltech]

@gigo1980 Just depend on "null_resource.kustomization" this is the last step.

depends_on = [null_resource.kustomization]

[jr-dimedis]

@mysticaltech After my chat with Chat GPT I learned that a resource needs to be exported in Terraform in order to be used in depends_on in another file.

I had to add the following snippet to my local .terraform/modules/kube-hetzner/output.tf to make this work for me:

output "kustomization" {
  description = "K8s customization YAML merely to use as a dependency on cluster creation"
  value       = null_resource.kustomization
  sensitive   = true
}

I then use this as follows:

data "kubectl_file_documents" "lets_encrypt_install" {
  content = file("yaml/lets-encrypt.yml")
}

resource "kubectl_manifest" "kubectl_lets_encrypt" {
  depends_on = [module.kube-hetzner.kustomization]
  for_each  = data.kubectl_file_documents.lets_encrypt_install.manifests
  yaml_body = each.value
}

If this looks correct to you I would be glad to create a PR for this :).

[mysticaltech]

@jr-dimedis I honestly did not know this was possible. Very interesting! @kube-hetzner/core Any thoughts on this?

[valkenburg-prevue-ch]

I think that the kustomization dependency is a misunderstanding. If you call kube-hetzner as a module, just depending on that module is enough. That's how I do it, no need for additional outputs. I do however use a 60s timeout between kube-hetzner and my follow-up kube stuff, to let the cluster "settle in".

[jr-dimedis]

@valkenburg-prevue-ch

Thanks for sharing this. Just waiting on the module is of course much simpler - I was not aware of that.

Finally, my solution now looks like this:

# File: init-wait-k8s.tf
resource "time_sleep" "kubernetes_ready" {
  depends_on = [module.kube-hetzner]
  create_duration = "60s"
}

# File: k8s-lets-encrypt.tf
data "kubectl_file_documents" "lets_encrypt_install" {
  content = file("yaml/lets-encrypt.yml")
}

resource "kubectl_manifest" "kubectl_lets_encrypt" {
  depends_on = [time_sleep.kubernetes_ready]
  for_each  = data.kubectl_file_documents.lets_encrypt_install.manifests
  yaml_body = each.value
}

I applied this to an existing cluster and yes: this took about 60s ;).

Because all my Kubernetes resources already existed, nothing else happened,
of course. But with a new cluster this should now work in one go.

Thanks a lot for your help!

[valkenburg-prevue-ch]

Yes, this looks perfect to me! It's exactly what I have too.

I have to add: one reason for the wait, is the kured goes through the nodes one by one and cordons them, in order to check for OS updates. The larger the cluster, the longer the wait, but I haven't spent any time figuring out how to optimize this. You could also go ahead and skip the wait, but then you might find that some nodes stay cordoned all the time.