Storage Class for node local storage

[HendrikLevering]

First of all, thanks for the great work.
I have a running cluster. I created it by an almost plain kube.tf file (adjusted node pools to cx11 and deleted express node pool). I created Postgres via the crunchy operator. That worked but it uses Hetzner cloud volumes. By default crunchy uses the default storage class which hcloud-volumes. How can I create a storage class that uses hetzners node storage for volumes?

[transmogrifix]

You should use Longhorn, it will use NVME node storage if you use longhorn_volume_size = 0.
See https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b for more info.

[HendrikLevering]

I did some tests with the tag:
tag disk, no tag in yaml -> working
tag disk, tag in yaml -> working
no tag disk, no tag in yaml -> not working
no tag disk, tag in yaml -> not working

[HendrikLevering]

Maybe I did something wrong. It seems that no tag disk, no tag in yaml is working, now. I will repeat this test with a brand new cluster. Meanwhile, I found that is should be possible to automatically set node and disk tags (I did not test this, yet):
https://longhorn.io/docs/1.5.1/advanced-resources/default-disk-and-node-config/#customizing-default-disks-for-new-nodes
However, this relies on node annotations. Now the question is:
How can I set node annotations for a node pool in kube.tf? It seems, that only labels are supported.

[transmogrifix]

You are moving in the right direction. You can't set annotations as it is not supported (see #718). You need to iterate the nodes in that pool and set the annotation via kubectl after nodes are created.

[HendrikLevering]

I see. I think, I got as far as possible for the time being. Most important Postgres is working with the right volumes even if it required some manual work. Thanks for your help. Really appreciate this :).

[melalj]

On terraform I added the following modules to tag the nodes:

provider "kubernetes" {
  config_path = var.kubeconfig_path
}

data "kubernetes_nodes" "cluster" {}

resource "kubernetes_labels" "longhorn-labels" {
  for_each = { for node in data.kubernetes_nodes.cluster.nodes : node.metadata.0.name => node if length(regexall("node-", node.metadata.0.name)) > 0 }

  api_version = "v1"
  kind        = "Node"
  metadata {
    name = each.value.metadata.0.name
  }
  labels = {
    "node.longhorn.io/create-default-disk" = "config"
  }
}

resource "kubernetes_annotations" "longhorn-annotations" {
  for_each = { for node in data.kubernetes_nodes.cluster.nodes : node.metadata.0.name => node if length(regexall("node-", node.metadata.0.name)) > 0 }

  api_version = "v1"
  kind        = "Node"
  metadata {
    name = each.value.metadata.0.name
  }
  annotations = {
    "node.longhorn.io/default-disks-config" = "[ { \"path\":\"/var/lib/longhorn\",\"allowScheduling\":true, \"storageReserved\":21474836240, \"tags\":[ \"nvme\" ]}, { \"name\":\"hcloud-volume\", \"path\":\"/var/longhorn\",\"allowScheduling\":true, \"storageReserved\":10737418120,\"tags\":[ \"ssd\" ] }]"
  }
}

I create the following StorageClass:

resource "kubectl_manifest" "longhorn-sc" {
  yaml_body = <<-EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-fast
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fromBackup: ""
  fsType: "ext4"
  diskSelector: "nvme"
EOF
}

And I have added on the kube.tf the following option:

  longhorn_values = <<EOF
defaultSettings:
  createDefaultDiskLabeledNodes: true
  kubernetesClusterAutoscalerEnabled: false
  defaultDataPath: /var/lib/longhorn
  node-down-pod-deletion-policy: delete-both-statefulset-and-deployment-pod
persistence:
  defaultClass: true
  defaultFsType: ext4
  defaultClassReplicaCount: 3
EOF

For some reason I get the following error when I create a PVC:

failed to provision volume with StorageClass "longhorn-fast": rpc error: code = Internal desc = Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [message=specified disk tag nvme does not exist, code=Internal Server Error, detail=] from [http://longhorn-backend:9500/v1/volumes]