HA Database Setup

[0Tom2020]

Hello,

I would like to run a highly available database on my k3s cluster. I was thinking of https://cloudnative-pg.io/. I'm just wondering what the best way to run a database on the Hetzner stack is. Node storage would be best, but Hetzner tends to expand storage using Hetzner Volumes, which means the servers have little storage space. I currently have a Postgres setup that is a bit slow:

apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: db namespace: prod spec: instances: 3 storage: storageClass: hcloud-volumes size: 50Gi superuserSecret: name: db-superuser enableSuperuserAccess: true

How could I optimize it? Or should I take a completely different approach?

Thanks a lot in advance for your help.

[iannuzzelli]

I use CNPG on k3s at Hetzner, with Longhorn, and works well in terms of performance.
But using a specific class storage with

• numberOfReplicas: "1" let CNPG deal with replication
• diskSelector: "fast" which is my label for the local disks, do not use hcloud, it's slower, see https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b#file-01_longhorn_bestpractices-md
• dataLocality: "strict-local" force Longhorn to use the same node for storage and the db cluster instance pod

See https://cloudnative-pg.io/documentation/current/storage/#block-storage-considerations-cephlonghorn

You would need a server type for the worker nodes that has enough local storage for your database size... possibly cx42?

[iannuzzelli]

Another important reminder, use a dedicated volume (with similar storageClass) for the WAL, see https://cloudnative-pg.io/documentation/current/storage/#volume-for-wal

[0Tom2020]

Thanks for your reply. I've been trying to adjust my setup accordingly over the last few days, but I'm not having any luck. How did you configure your Longhorn so that you can use different disk labels? My current Longhorn nodes are defined as follows:
{
name = “base-nbg1”,
server_type = “cx22”,
location = “nbg1”,
labels = [
“node.longhorn.io/create-default-disk=true”
],
taints = [],
count = 1,
longhorn_volume_size = 50 # In GB
},

How could I now add a node with cx42 that has a disk label “fast”?
Of course, I would like to keep the longhorn hetzner volume config (which I currently have here) for “inexpensive storage” in the future.
Do I then create the storage class manually, or can I do this via kube-hetzner?

[AndrinGautschi]

You can label your node with node.longhorn.io/create-default-disk=config and afterwards annotate it with the disk config in which you specify custom disk labels (beware of the reservedStorage prop; at least 25% of the node storage should be for the system only):

kubectl get nodes -l your.identification.label.for.the.node/localstorage=160gb-for-all -o name | while read node; do
  kubectl annotate "$node" node.longhorn.io/default-disks-config='[ { "path":"/var/lib/longhorn","allowScheduling":true, "storageReserved":64424509440, "tags":[ "nvme-unshared" ]} ]' --overwrite
done

This works fine for a custom storage class like the following:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-local-unreplicated
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  numberOfReplicas: "1"
  dataLocality: strict-local
  staleReplicaTimeout: "2880" # 48 hours in minutes
  fsType: "ext4"
  diskSelector: "nvme-unshared"

Heads up: You WILL run into issues with kured; as soon as the node on which you run your db-instance (that uses the unshared longhorn storage class for volumes) gets scheduled for an update, kured will try to drain it. But draining will be prevented by the longhorn-manager with a complaint that it cannot be terminated since it manages the last replica of your volume. You will have to manually delete the volume for this to resolve. Furthermore, the new db instance will have trouble starting with a volume mount issue (since the volume is placed on another node).