Is the reconfiguration of applications supported?

[Mantany]

First of all: thank you for this great project!

I do however have a question: Is it possible to reconfigure the applications on a running cluster?
I build a working k3s-cluster with it, enabled cert-manager, but it seems when I try to disable it again, the changes do not get picked up. -> "No changes. Your infrastructure matches the configuration.".Also, when I try to add deployments like longhorn or want to switch from traefik to nginx, I get the same message.

[mysticaltech]

@Mantany Yes, it's one way for these applications via terraform. But those applications are configured via HelmChart definitions. You can use kubectl get helmchart -n kube-system, then you can do grab them with kubectl get helmchart <name> -o yaml same it to a file, and kubectl delete.

If you want a GUI for all of this, most probably Rancher is your best bet. I personally deploy Rancher as a single node (cx21) cluster with rancher enabled for this kind of things.

In the Readme, there's a lot of explanations on how HelmChart / HelmChart config work, please have a look.

[WolfspiritM]

Another way to modify and add stuff would be to use the created kustomization files and modify them.
There is a kustomize backup saved but it doesn't contain everything required.
However on the first control plane server you'll find a folder "/var/post_install/".
In there are all the files applied on the first install.

You can either change stuff in their and apply it via kustomize or copy it to your local machine and apply it from there.
What I did is moving this files to gitops (argocd) so they can be changed however needed.

Here is how it is applied on the first install so you can do the same:

terraform-hcloud-kube-hetzner/init.tf

Line 265 in 5becc6f

"kubectl apply -k /var/post_install",

If you want to add e.g. longhorn you can get the "longhorn.yaml" you need from the template here:
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/master/templates

In there you can find a "${values}" placeholder. This placeholder is filled by either a supplied yaml file on the first setup or by some default values. The default values for that can be found here:

terraform-hcloud-kube-hetzner/locals.tf

Lines 303 to 338 in e466f03

	default_cilium_values = <<EOT
	ipam:
	operator:
	clusterPoolIPv4PodCIDRList:
	- ${local.cluster_cidr_ipv4}
	devices: "eth1"
	EOT
	default_longhorn_values = <<EOT
	defaultSettings:
	defaultDataPath: /var/longhorn
	persistence:
	defaultFsType: ${var.longhorn_fstype}
	defaultClassReplicaCount: ${var.longhorn_replica_count}
	%{if var.disable_hetzner_csi~}defaultClass: true%{else~}defaultClass: false%{endif~}
	EOT
	default_nginx_ingress_values = <<EOT
	controller:
	watchIngressWithoutClass: "true"
	kind: "Deployment"
	replicaCount: ${(local.agent_count > 2) ? 3 : (local.agent_count == 2) ? 2 : 1}
	config:
	"use-forwarded-headers": "true"
	"compute-full-forwarded-for": "true"
	"use-proxy-protocol": "true"
	service:
	annotations:
	"load-balancer.hetzner.cloud/name": "${var.cluster_name}"
	"load-balancer.hetzner.cloud/use-private-ip": "true"
	"load-balancer.hetzner.cloud/disable-private-ingress": "true"
	"load-balancer.hetzner.cloud/ipv6-disabled": "${var.load_balancer_disable_ipv6}"
	"load-balancer.hetzner.cloud/location": "${var.load_balancer_location}"
	"load-balancer.hetzner.cloud/type": "${var.load_balancer_type}"
	"load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
	EOT

So for longhorn you would replace the "${values}" placeholder with the "default_longhorn_values" and use ext4 as fstype and 3 as replica count (or change it as you want).

Then you can either add that longhorn.yaml to the kustomization yaml and apply the kustomization or apply that file manually. That way you'd have longhorn deployed as it would have been on the first apply.

These "HelmChart" resources are just kubernetes resources that can be removed, modified and added while the k3s operator for helm charts makes sure they get installed as needed.

[Mantany]

Thank you for the great suggestions.
In the past, I used a git repo and fluxCD for automatic deployment. That way I have the complete state of the cluster in one repo. I guess, for now, I will copy the kustomization files manually over. Monorepo approach

Do you think think that it is useful to manage everything (longhorn, hetzner cloud manger, hetzner csi) via fluxCD? Do you have experience regarding bigger production clusters?

That way I would only use terraform for the infrastructure and fluxCD deployment and fluxCD to manage the complete state of the cluster.

[mysticaltech]

@WolfspiritM 100% I had forgotten to mention the /var/post_install folder with all the kustomization on the first control plane.

@Mantany Definitely, I think you could turn off every extra and start fresh, and use FluxCD for those.