Another "Error: remote-exec provisioner error"

[gefloh]

For some reason terraform apply -auto-approve doesn't work anymore with the latest version for me. Output:

(...)
  # module.kube-hetzner.null_resource.kustomization is tainted, so must be replaced
-/+ resource "null_resource" "kustomization" {
      ~ id       = "1682587839876891537" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 2 to add, 0 to change, 1 to destroy.
module.kube-hetzner.null_resource.kustomization: Destroying... [id=1682587839876891537]
module.kube-hetzner.null_resource.kustomization: Destruction complete after 0s
module.kube-hetzner.null_resource.kustomization: Creating...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Still creating... [10s elapsed]
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Still creating... [20s elapsed]
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Still creating... [30s elapsed]
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Still creating... [40s elapsed]
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Provisioning with 'file'...
module.kube-hetzner.null_resource.kustomization: Still creating... [50s elapsed]
module.kube-hetzner.null_resource.kustomization: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.null_resource.kustomization (remote-exec): (output suppressed due to sensitive value in config)
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.null_resource.kustomization,
│   on .terraform/modules/kube-hetzner/init.tf line 275, in resource "null_resource" "kustomization":
│  275:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_470358228.sh": Process exited with status 1

The thing is: It doesn't give me any hint what the problem might be. Any hints how I could debug this further to find out what the issue is?

I tried what is written in the docs regarding debugging, also sshd into the nodes. Probably something with the control plane nodes, because the logs for the other nodes were pretty empty for that timestamp.

[mysticaltech]

@gefloh First run terraform init -upgrade, if you do not have the ARM snapshot yet, to proceed with the image creation, first use hcloud image list and hcloud image delete ... to delete the old x86 snapshot, then run createkh in your project folder (no worries it's non-destructive).

Then apply again, and if it fails with the same error, login to the SSH node in question, and run that terraform sh file (located in /tmp/) to see the outputs and errors.

[gefloh]

Ok thanks @mysticaltech , I ran the script on the failing node manually and the issue was that it tried to create a file /var/post_install/hcloud-csi.yml but there was already an existing file /var/post_install containing a kured DaemonSet yaml. Hence it couldn't write the hcloud-csi.yml. Not sure why that happened. Anyway, removed that file, applied again and everything is green now. Thank you!