Nooby questions - What's wrong with my setup ;)

[nupplaphil]

Hi all,

Unfortunately, I really struggle in creating even the most simple setups and I'm a little bit frustrated :( ..

I'm trying to move from my Docker root server (128GB RAM), which ran smoothly about ~4 years to kubernetes, because

• There are two heavy services (~each using ~50GB mysql DB RAM)
• I'm using the infrastructure for CI/CD - which creates a lot of spikes
• because of increasing usage of heavy services (since the Twitter-chaos ;-) ), the whole root server recently slowed down and I REALLY need to change something because of I/O issues

So I went to this project some month ago (maybe some know I added calico at first place) and stopped it because of lack of time.

Now I'm back and try to setup my cluser as a new home.

My services:

private usage (~6-7 users)

• Nextcloud with lot of space (=> longhorn support)
• Email & Groupware setup (=> egress service for smtp otgoing)
• Gitea instance
• Keycloak / SSO

small public services

• two wordpress instances
• personal Friendica instance

large public services

• Two Friendica setups (mysql, php-fpm, nginx, own asset service, cronjob service) with ~6-700 users
• CI/CD for open source projects (integration tests with mysql, redis, ... instances)

My current kube.tf:

locals {
  cluster_ipv4_cidr = "10.42.0.0/16"
}

module "kube-hetzner" {

  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token

  source = "./modules/kube-hetzner"

  ssh_public_key  = file(var.public_key)
  ssh_private_key = file(var.private_key)

  base_domain = "test.domain.com"
  ssh_port    = 2222

  cluster_ipv4_cidr = local.cluster_ipv4_cidr

  hetzner_ccm_version = "v1.17.1"
  hetzner_csi_version = "v2.3.2"
  kured_version       = "1.13.1"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 2
    },
    {
      name        = "control-plane-hel1",
      server_type = "cpx11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 2
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small-hel1",
      server_type = "cpx11",
      location    = "hel1",
      labels      = [
        "node.kubernetes.io/usage=app",
        "node.kubernetes.io/role=agent"
      ],
      taints = [],
      count  = 4
    },
    {
      name        = "agent-small-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [
        "node.kubernetes.io/usage=app",
        "node.kubernetes.io/role=agent"
      ],
      taints = [],
      count  = 4
    },
    {
      name        = "agent-large-nbg1",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [
        "node.kubernetes.io/usage=worker",
        "node.kubernetes.io/role=agent"
      ],
      taints = [],
      count  = 3
    },
    {
      name        = "agent-large-hel1",
      server_type = "cpx21",
      location    = "hel1",
      labels      = [
        "node.kubernetes.io/usage=worker",
        "node.kubernetes.io/role=agent"
      ],
      taints = [],
      count  = 3
    },
    {
      name        = "agent-storage-nbg1",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [
        "node.kubernetes.io/usage=storage",
        "node.kubernetes.io/role=agent",
        "node.longhorn.io/create-default-disk=config"
      ],
      taints = [],
      count  = 2
    },
    {
      name        = "agent-storage-hel1",
      server_type = "cpx21",
      location    = "hel1",
      labels      = [
        "node.kubernetes.io/usage=storage",
        "node.kubernetes.io/role=agent",
        "node.longhorn.io/create-default-disk=config"
      ],
      taints = [],
      count  = 2
    },
    {
      name        = "egress",
      server_type = "cpx11",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ],
      floating_ip = true
      count = 1
    }
  ]

  autoscaler_nodepools = [
    {
      name        = "autoscaled-small"
      server_type = "cpx21"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 5
    }
  ]

  etcd_s3_backup = {
    [...]
  }

  load_balancer_location     = "fsn1"
  load_balancer_type         = "lb11"

  enable_longhorn       = true
  enable_cert_manager   = true
  enable_csi_driver_smb = true
  enable_metrics_server = true

  cni_plugin    = "cilium"
  cilium_values = <<EOT
ipam:
  operator:
    clusterPoolIPv4PodCIDRList:
      - ${local.cluster_ipv4_cidr}
kubeProxyReplacement: strict
l7Proxy: "false"
bpf:
  masquerade: "true"
egressGateway:
  enabled: "true"
extraConfig:
  mtu: "1450"
EOT

  longhorn_values = <<EOT
defaultSettings:
  createDefaultDiskLabeledNodes: true
  kubernetesClusterAutoscalerEnabled: true
  defaultDataPath: /var/lib/longhorn
  node-down-pod-deletion-policy: delete-both-statefulset-and-deployment-pod
persistence:
  defaultClass: true
  defaultFsType: ext4
  defaultClassReplicaCount: 3
EOT
}

After after successful terraform apply, the whole setup starts crashing ..

log of one controller:

Jul 26 20:21:52 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:52.589Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 [logterm: 14, index: 26487, vote: 6862d5ef6fbce322] cast MsgPreVote for 6862d5ef6fbce322 [logterm: 14, index: 26487] at term 14"}
Jul 26 20:21:53 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:53.380Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0009c8540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jul 26 20:21:53 k3s-control-plane-hel1-rkf k3s[9174]: time="2023-07-26T20:21:53Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jul 26 20:21:53 k3s-control-plane-hel1-rkf k3s[9174]: time="2023-07-26T20:21:53Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.217Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 is starting a new election at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.218Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 became pre-candidate at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.218Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 received MsgPreVoteResp from b8721dfb5f0a2306 at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.218Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 [logterm: 14, index: 26487] sent MsgPreVote request to 37c27cde23b06db4 at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.218Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 [logterm: 14, index: 26487] sent MsgPreVote request to 6862d5ef6fbce322 at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.218Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 [logterm: 14, index: 26487] sent MsgPreVote request to 6c89d2f8e23fc045 at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.218Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 [logterm: 14, index: 26487] sent MsgPreVote request to 8a38914f73defeab at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.244Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 received MsgPreVoteResp from 6862d5ef6fbce322 at term 14"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:21:56.244Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 has received 2 MsgPreVoteResp votes and 0 vote rejections"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:56.296Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"8a38914f73defeab","rtt":"0s","error":"net/http: TLS handshake timeout"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:56.296Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"37c27cde23b06db4","rtt":"0s","error":"net/http: TLS handshake timeout"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:56.296Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"37c27cde23b06db4","rtt":"0s","error":"net/http: TLS handshake timeout"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:56.296Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"6c89d2f8e23fc045","rtt":"0s","error":"net/http: TLS handshake timeout"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:56.296Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"8a38914f73defeab","rtt":"0s","error":"net/http: TLS handshake timeout"}
Jul 26 20:21:56 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:21:56.296Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"6c89d2f8e23fc045","rtt":"0s","error":"net/http: TLS handshake timeout"}
Jul 26 20:21:58 k3s-control-plane-hel1-rkf k3s[9174]: time="2023-07-26T20:21:58Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
Jul 26 20:22:00 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:22:00.088Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b8721dfb5f0a2306 [logterm: 14, index: 26487, vote: 6862d5ef6fbce322] cast MsgPreVote for 6862d5ef6fbce322 [logterm: 14, index: 26487] at term 14"}
Jul 26 20:22:01 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"warn","ts":"2023-07-26T20:22:01.002Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0009c8540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""}
Jul 26 20:22:01 k3s-control-plane-hel1-rkf k3s[9174]: {"level":"info","ts":"2023-07-26T20:22:01.003Z","logger":"etcd-client","caller":"[email protected]/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}

Other control-plane nodes suddenly stops being available per SSH or per kubectl and showing OOM errors, like:
[ 1670.92471][ T6426] out of memory: Killed process 4046 (kured) total-vm:751132kb, anon-rss: 7040kB, file-rss:128kB, shmem-rss:0kB, UID:0 pgtables:176kB oom_score_adj:1000

Do I really need more resources just for the naked cluster without any application on it .. This seems like it's not properly working...

I would REALLY love to use this setup, the whole feature-catalog you show is exactly what I think, what would be necessary to move to the next level with my environment :)

HELP!!! :-)