How to separate installation of CRDs and using them in kustomization.yaml?

[fkuersch]

I'm trying to install & configure ArgoCD in customize.yaml.tpl

Unfortunately, when kustomize applies the ArgoCD application config, the CRDs are not available yet.

kustomization.yaml.tpl:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
namespace: argocd

resources:
- argocd-namespace.yaml
- https://raw.githubusercontent.com/argoproj/argo-cd/v2.5.2/manifests/ha/install.yaml
- argocd-applications.yaml

The first terraform apply fails with the following error:

error: resource mapping not found for name: "applications" namespace: "argocd" from "/var/user_kustomize/": no matches for kind "Application" in version "argoproj.io/v1alpha1"
ensure CRDs are installed first

When running terraform apply a second time, the argocd-applications.yaml will be applied correctly.

The current workaround appears to be installing the CRDs first, waiting for it to be established and then applying the config in a second step like this (for sealded-secrets):

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.13.1/controller.yaml
kubectl wait --for condition=established crd sealedsecrets.bitnami.com
kustomize build . | kubectl apply -f -

Source: kubernetes-sigs/kustomize#3502 (comment)

How do I do this in terraform-hcloud-kube-hetzner?

[captnCC]

The user_kustomize part has no logic for a staggered deployment or handling such problems.
I ran into a similar problem with the cert-manager taking a few minutes to get ready. To workaround this, I deployed my additional resources via the kubernetes and helm providers for terraform which allows for staggering in the deployment

[mysticaltech]

@captnCC If you could submit an example to the example folder, it would be awesome!

[captnCC]

On it!

[ctholho]

@captnCC Hi, is there any news on this? You'd help out enormously. I'm running into this problem when I want to install velero and apply a crd called schedule in extra-manifests/. Could you give me any pointers for a staggered deploy with kube-hetzner?

[fkuersch]

@ctholho I entirely gave up using kustomize in extra-manifests. Instead, I created a new terraform (sub)module and installed everything using terraform and helm.

I think that's the more terraformy and thus a cleaner approach that should probably be mentioned as the default way in the docs.

Example for ArgoCD

I could create a complete working example. But since I'm still learning, it would be nice if someone more experienced could have a look at my approach. I'm just copy-and-pasting the relevant parts for now, so the following is incomplete and untested. Hope this helps.

./kube.tf

# [...]

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.16.1"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "2.7.1"
    }
  }
}

provider "kubernetes" {
  host                   = module.kube-hetzner.kubeconfig_data.host
  client_certificate     = module.kube-hetzner.kubeconfig_data.client_certificate
  client_key             = module.kube-hetzner.kubeconfig_data.client_key
  cluster_ca_certificate = module.kube-hetzner.kubeconfig_data.cluster_ca_certificate
}

provider "helm" {
  kubernetes {
    host                   = module.kube-hetzner.kubeconfig_data.host
    client_certificate     = module.kube-hetzner.kubeconfig_data.client_certificate
    client_key             = module.kube-hetzner.kubeconfig_data.client_key
    cluster_ca_certificate = module.kube-hetzner.kubeconfig_data.cluster_ca_certificate
  }
}

module "cluster-bootstrap" {
  providers = {
    kubernetes = kubernetes
    helm       = helm
  }
  source = "./cluster-bootstrap"
  depends_on = [
    module.kube-hetzner
  ]

  # <variables here>
}

./cluster-bootstrap/main.tf

resource "kubernetes_namespace" "argocd" {
  metadata {
    name = "argocd"
  }
}

resource "helm_release" "argocd" {
  name       = "argo-cd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = "5.16.2"
  namespace  = kubernetes_namespace.argocd.metadata[0].name
  values = [
    "${file("cluster-bootstrap/helm-values/argo-cd.yaml")}"
  ]
}

resource "helm_release" "argocd-apps" {
  name       = "argocd-apps"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argocd-apps"
  version    = "0.0.4"
  namespace  = kubernetes_namespace.argocd.metadata[0].name
  values = [
    "${file("cluster-bootstrap/helm-values/argocd-apps.yaml")}"
  ]
  depends_on = [
    helm_release.argocd
  ]
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.16"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.7"
    }
  }
}

[mysticaltech]

@fkuersch Super interesting, please do not hesitate to submit a PR to include the above use case in our example folder and/or section in the readme!

[mashkovd]

i see
could not download chart: failed to download "https://github.com/argoproj/argo-helm/releases/download/argo-cd-5.16.2/argo-cd-5.16.2.tgz" at version "5.16.2"

when do command terrraform apply
what's problem may be with it

[fkuersch]

I deployed my additional resources via the kubernetes and helm providers for terraform which allows for staggering in the deployment

Thank you, I'll give it a try