Node Storage issues

[hypery2k]

Description

Getting this error when using longhorn from time to time:

Have to manually drain node, reboot and re-sync longhorn. Any ideas how to resolve that permanently?

Kube.tf file

locals {
  # Pass to control plane nodepools to reduce hard disk usage due to obsolete images, e.g. crictl rmi --prune
  control_plane_kubelet_args = ["image-gc-high-threshold=40", "image-gc-low-threshold=25"]
  worker_node_labels         = ["node.kubernetes.io/role=worker", "node.longhorn.io/create-default-disk=config"]
  gitlab_container_registry = {
    server   = "..."
    email    = "..."
    login    = "..."
    password =...
  }
  minio = {
    url       = "fsn1.your-objectstorage.com"
    accessKey = var.minio_access_key
    secretKey = var.minio_secret_key
    region    = "fsn1"
    bucket    = "..."
  }
  agent_nodepools = [
    {
      name        = "workload-cx42-fsn1",
      server_type = "cx42",
      location    = "fsn1",
      labels      = local.worker_node_labels,
      taints = [
      ],
      count = 1

      # In the case of using Longhorn, you can use Hetzner volumes instead of using the node's own storage by specifying a value from 10 to 10000 (in GB)
      # It will create one volume per node in the nodepool, and configure Longhorn to use them.
      # Something worth noting is that Volume storage is slower than node storage, which is achieved by not mentioning longhorn_volume_size or setting it to 0.
      # So for something like DBs, you definitely want node storage, for other things like backups, volume storage is fine, and cheaper.
      longhorn_volume_size = 0

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "workload-cx42-nbg1",
      server_type = "cx42",
      location    = "nbg1",
      labels      = local.worker_node_labels,
      taints = [
      ],
      count                = 1
      longhorn_volume_size = 0
    },
    {
      name        = "workload-cx42-hel1",
      server_type = "cx42",
      location    = "hel1",
      labels      = local.worker_node_labels,
      taints = [
      ],
      count                = 1
      longhorn_volume_size = 0
    },

  ]
}

data "dns_a_record_set" "gitlab" {
  host = "..."
}

data "dns_a_record_set" "home" {
  host = "..."
}
data "http" "current_ip" {
  url = "http://icanhazip.com"
}

module "k8s" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token

  # Then fill or edit the below values. Only the first values starting with a * are obligatory; the rest can remain with their default values, or you
  # could adapt them to your needs.

  # * For local dev, path to the git repo
  # source = "../../kube-hetzner/"
  # If you want to use the latest master branch
  # source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"
  # For normal use, this is the path to the terraform registry
  source = "kube-hetzner/kube-hetzner/hcloud"

  # renovate: datasource=githab-tags depname=kube-hetzner/terraform-hcloud-kube-hetzner
  version = "2.17.3"


  # * Your ssh public key
  ssh_public_key = ...
  # * Your private key must be "ssh_private_key = null" when you want to use ssh-agent for a Yubikey-like device authentification or an SSH key-pair with a passphrase.
  # For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md
  ssh_private_key = ...
  network_region = "eu-central" # change to `us-east` if location is ash


  control_plane_nodepools = [
    {
      name         = "control-plane-fsn1",
      server_type  = "cpx21",
      location     = "fsn1",
      labels       = [],
      taints       = [],
      count        = 1,
      kubelet_args = local.control_plane_kubelet_args

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name         = "control-plane-nbg1",
      server_type  = "cpx21",
      location     = "nbg1",
      labels       = [],
      taints       = [],
      count        = 1,
      kubelet_args = local.control_plane_kubelet_args

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name         = "control-plane-hel1",
      server_type  = "cpx21",
      location     = "hel1",
      labels       = [],
      taints       = [],
      count        = 1,
      kubelet_args = local.control_plane_kubelet_args

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    }
  ]

  agent_nodepools = local.agent_nodepools
  # Add custom control plane configuration options here.
  # E.g to enable monitoring for etcd, proxy etc:
  control_planes_custom_config = {
    etcd-expose-metrics         = true,
    kube-controller-manager-arg = "bind-address=0.0.0.0",
    kube-proxy-arg              = "metrics-bind-address=0.0.0.0",
    kube-scheduler-arg          = "bind-address=0.0.0.0",
  }

  # * LB location and type, the latter will depend on how much load you want it to handle, see https://www.hetzner.com/cloud/load-balancer
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  autoscaler_nodepools = [
    {
      name        = "autoscaler-cpx21"
      server_type = "cpx21" # must be same or better than the control_plane server type (regarding disk size)!
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 6
      labels = {
        "node.kubernetes.io/role" : "peak-workloads"
      }
    },
    {
      name        = "autoscaler-cpx31"
      server_type = "cpx31" # must be same or better than the control_plane server type (regarding disk size)!
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 2
      labels = {
        "node.kubernetes.io/role" : "peak-workloads"
      }
    }
  ]
  enable_delete_protection = {
    floating_ip   = true
    load_balancer = true
    volume        = true
  }
  cluster_autoscaler_extra_args = [
    "--ignore-daemonsets-utilization=true",
    "--enforce-node-group-min-size=true"
  ]

  # Enable etcd snapshot backups to S3 storage.
  # Just provide a map with the needed settings (according to your S3 storage provider) and backups to S3 will
  # be enabled (with the default settings for etcd snapshots).
  # Cloudflare's R2 offers 10GB, 10 million reads and 1 million writes per month for free.
  # For proper context, have a look at https://docs.k3s.io/datastore/backup-restore.
  # You also can use additional parameters from https://docs.k3s.io/cli/etcd-snapshot, such as `etc-s3-folder`
  etcd_s3_backup = {
    etcd-s3-endpoint            = local.minio.url
    etcd-s3-access-key          = local.minio.accessKey
    etcd-s3-secret-key          = local.minio.secretKey
    etcd-s3-bucket              = local.minio.bucket
    etcd-s3-folder              = "etcd"
    etcd-snapshot-schedule-cron = "0 3 * * *"
  }

  # To enable Hetzner Storage Box support, you can enable csi-driver-smb, default is "false".
  # enable_csi_driver_smb = true

  # To use local storage on the nodes, you can enable Longhorn, default is "false".
  # See a full recap on how to configure agent nodepools for longhorn here https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/373#discussioncomment-3983159
  # Also see Longhorn best practices here https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b
  enable_longhorn = true

  kured_options = {
    "reboot-days" : "we"
    "start-time" : "1am"
    "end-time" : "5am"
    # Reduce raise conditions
    "lock-ttl" : "30m"
    "drain-timeout" : "1h"
    "time-zone" : "Europe/Berlin"
    # Operations Bot
    "notify-url" : "slack://..."
  }
  # Allows you to specify either stable, latest, testing or supported minor versions.
  # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels
  # ⚠️ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to latest - 0.01,
  # ⚠️ Rancher currently only supports v1.25 and earlier versions: https://github.com/rancher/rancher/issues/41113
  # The default is "v1.26".
  initial_k3s_channel = "${split(".", var.initial_k3s_version)[0]}.${split(".", var.initial_k3s_version)[1]}"

  # The cluster name, by default "k3s"
  cluster_name = "production"
  k3s_registries = ...
  extra_firewall_rules = ...

  # Cert manager, all cert-manager helm values can be found at https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  cert_manager_values = <<EOT
installCRDs: true
prometheus:
  servicemonitor:
    enabled: true
resources:
  requests:
    cpu: "50m"
    memory: "50Mi"
  limits:
    cpu: "100m"
    memory: "150Mi"
EOT 

  # Longhorn, all Longhorn helm values can be found at https://github.com/longhorn/longhorn/blob/master/chart/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  longhorn_values = <<EOT
defaultSettings:
  backupTarget: s3://${local.minio.bucket}@${local.minio.region}/longhorn
  backupTargetCredentialSecret: minio-backup-longhorn
  defaultDataPath: /var/longhorn
  # ensure pod is moved to an healthy node if current node is down:
  node-down-pod-deletion-policy: delete-both-statefulset-and-deployment-pod
persistence:
  defaultFsType: ext4
  defaultClassReplicaCount: 3
  defaultClass: true
metrics:
  serviceMonitor:
    enabled: true
EOT
  # Traefik, all Traefik helm values can be found at https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml
  # The following is an example, please note that the current indentation inside the EOT is important.
  traefik_values = <<EOT
image:
  tag: 
deployment:
  replicas: 2
  podAnnotations:
    'cluster-autoscaler.kubernetes.io/safe-to-evict': 'true'
globalArguments: []
service:
  enabled: true
  type: LoadBalancer
  annotations:
    "load-balancer.hetzner.cloud/name": "production-traefik"
    "load-balancer.hetzner.cloud/use-private-ip": "true"
    "load-balancer.hetzner.cloud/disable-private-ingress": "true"
    "load-balancer.hetzner.cloud/disable-public-network": "false"
    "load-balancer.hetzner.cloud/ipv6-disabled": "false"
    "load-balancer.hetzner.cloud/location": "fsn1"
    "load-balancer.hetzner.cloud/type": "lb11"
    "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
    "load-balancer.hetzner.cloud/algorithm-type": "round_robin"
    "load-balancer.hetzner.cloud/health-check-interval": "15s"
    "load-balancer.hetzner.cloud/health-check-timeout": "10s"
    "load-balancer.hetzner.cloud/health-check-retries": "3"
deployment:
  podAnnotations:
    'cluster-autoscaler.kubernetes.io/safe-to-evict': 'true'
ports:
  web:
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
  websecure:
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
podDisruptionBudget:
  enabled: true
  maxUnavailable: 33%
additionalArguments:
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"
resources:
  requests:
    cpu: "50m"
    memory: "64Mi"
  limits:
    cpu: "100m"
    memory: "256Mi"
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
EOT

}

Screenshots

No response

Platform

Mac

[mysticaltech]

Hi @hypery2k,

Looking at your error screenshot, this appears to be a hardware-level disk I/O failure rather than a Longhorn or Terraform configuration issue. The errors show:

• EXT4-fs (sdf): I/O error while writing superblock
• Remounting filesystem read-only
• JBD2: I/O error when updating journal superblock for sdg-8

These indicate that the underlying disk is failing or experiencing hardware issues. This is not related to disk space or Longhorn configuration.

Immediate Actions:

1. Check the affected node's disk health:

   # SSH into the affected node
   dmesg | grep -i "i/o error"
   smartctl -a /dev/sdf  # or whichever disk is failing

2. Verify if this is a Hetzner Cloud Volume or local disk:

   lsblk
   mount | grep longhorn

Root Causes:

This can happen due to:

• Physical disk failure (if using local node storage)
• Corrupted filesystem requiring fsck

Recommended Solution:

Since you're using longhorn_volume_size = 0 (local node storage), this suggests the node's local disk is failing. You should:

1. Evacuate the node immediately:

   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

2. Contact Hetzner support if this is a hardware failure on their side

3. Replace the node, see the readme and kube.tf.example for how to do that properly, you can delete the node directly without touching the nodepool with hcloud and then apply terraform again to recreate it.

Prevention:

For better resilience against disk failures:

1. Use Hetzner Cloud Volumes instead of local storage:

   longhorn_volume_size = 100  # or appropriate size

2. Enable node monitoring to catch disk issues early:

   defaultSettings:
     nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod
     guaranteedEngineManagerCPU: 12
     guaranteedReplicaManagerCPU: 12

3. Set up proper backups which you already have with S3

This is not a bug in the Terraform module but rather a hardware failure that needs immediate attention to prevent data loss.