Terraforms stops with error "hcclient/WaitForActions ... unexpected EOF"

[jr-dimedis]

At first: thanks for this great peace of software! Up to now I used Terraform to create only the cluster resources at Hetzner but installed k3s more or less by hand. Getting all this with one Terraform module is awesome!

Recently I created a three node (control plane only) K8s testing cluster with kube-hetzner (v2.1.6) using a (near) default configuration and it worked flawlessly.

Today I tried to repeat this with a second cluster, which just has additional three agent nodes, and I am getting
the following error:

│ Error: hcclient/WaitForActions: Get "https://api.hetzner.cloud/v1/actions?id=1016980628&page=1": unexpected EOF
│
│ with module.kube-hetzner.module.control_planes["1-0-cp-nbg1"].hcloud_server.server,
│ on .terraform/modules/kube-hetzner/modules/host/main.tf line 22, in resource "hcloud_server" "server":
│ 22: resource "hcloud_server" "server" {
│

This error looks like a (temporary) problem at Hetzner API level for me, but Terraform stops processing and when applying it again, the server nodes are recreated due to their tainted state. This worked for some nodes and then randomly Terraform stops again with the above error message.

I even tried to remove the project at Hetzner completely and the Terraform state and started from the beginning. The same error happens again, up to the point that I reached my primary IP limit at Hetzner (I think because I recreated a bunch
of nodes in short time).

│ Error: Primary IP limit exceeded (resource_limit_exceeded)
│
│   with module.kube-hetzner.module.agents["0-1-wk-fsn1-cpx31"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 22, in resource "hcloud_server" "server":
│   22: resource "hcloud_server" "server" {
│

But this is not the issue here - I will contact Hetzner for that ;)

What I am asking is: is there anything I can do on my side to prevent this "unexpected EOF" error? Is this an issue of kube-hetzner or the hcloud Terraform provider? Or should I contact Hetzner?

If this is really a temporary Problem on the Hetzner API side, some retry logic would be fine instead of stopping Terraform and leaving the cluster in a tainted state.

Thanks for any help!

[mysticaltech]

@jr-dimedis Thanks for your comments! I think the EOF error was temporary error coming from Hetzner's API. Surely retry logic would be interesting, it should be possible in terraform, I will have a look, however since it's rare enough, it's not a huge priority. For cleanup (before retrying), follow the destroy section in the readme if you are not already doing so. Basically, we have a helper tool to ease the process!

[jr-dimedis]

@mysticaltech

Thanks for your answer. Next morning I did a "terraform apply" without changing anything and it was working fine. So I can confirm that this was a temporary problem as suspected.

I understand that this retry mechanism has no huge priority. I'll probably create more clusters in the near future and if that really happens more often, I might back to you ;)