Erhhung's Home Kubernetes Cluster

This Ansible-based project provisions Erhhung's high-availability Kubernetes cluster at home named homelab, and deploys services for monitoring various IoT appliances, as well as for deploying other personal projects, including self-hosted LLMs and AI pipelines that enable multi-source hybrid searches and agentic automations using local knowledge base containing vast amounts of personal and sensor data.

The approach taken on all service deployments is to treat the clusters as a production environment (to the extent possible with limited resources and scaling capacity across a few mini PCs). That means TLS everywhere and requiring authenticated user access, scraping metrics, and configuring dashboards and alerts.

Overview

The top-level Ansible playbook main.yml run by make will provision 7 VM hosts (rancher and k8s1..k8s6) in the existing XCP-ng Homelab pool, created using Terraform by the homelab-xcp project, all running Ubuntu Server 24.04 Minimal without customizations besides basic networking and authorized SSH key for user erhhung.

A single-node K3s Kubernetes cluster will be installed on host rancher along with Rancher Server on that cluster, and a 6-node RKE2 Kubernetes cluster with high-availability control plane using a virtual IP will be installed on hosts k8s1..k8s6. MetalLB will be installed and configured in BGP mode on the 6-node cluster to load-balance external traffic among cluster nodes using ECMP routing provided by pfSense and FRR.

Longhorn and NFS storage provisioners will be installed in each cluster to manage a pool of LVM logical volumes on each node, and to expand the overall storage capacity onto the QNAP NAS. MinIO will also be installed, serving as S3-compatible object storage backed by NFS volumes on QNAP.

All cluster services will be provisioned with TLS certificates from Erhhung's private CA server at pki.fourteeners.local (or its faster mirror at cosmos.fourteeners.local) with the help of cert-manager and Step CA.

Cluster Topology

Platform Services

AI Services

Service Endpoints

Service Endpoint	Description
https://rancher.fourteeners.local	Rancher Server console
https://harbor.fourteeners.local	Harbor OCI registry
https://velero.fourteeners.local	Velero console
https://minio.fourteeners.local	MinIO console
https://s3.fourteeners.local	MinIO S3 API
smtp://smtp.fourteeners.local:587	Mailpit SMTP
https://mailpit.fourteeners.local	Mailpit UI
https://opensearch.fourteeners.local:9200	OpenSearch
https://kibana.fourteeners.local	OpenSearch Dashboards
postgres.fourteeners.local:5432	PostgreSQL via Pgpool (mTLS only)
https://sso.fourteeners.local	Keycloak IAM console
valkey.fourteeners.local:6379 valkey{1..6}.fourteeners.local:6379	Valkey cluster (mTLS only)
https://grafana.fourteeners.local	Grafana dashboards
https://metrics.fourteeners.local	Prometheus UI (Keycloak SSO)
https://alerts.fourteeners.local	Alertmanager UI (Keycloak SSO)
https://thanos.fourteeners.local	Thanos Query UI
https://rule.thanos.fourteeners.local https://store.thanos.fourteeners.local https://bucket.thanos.fourteeners.local https://compact.thanos.fourteeners.local	Thanos components UI
https://slo.fourteeners.local	Pyrra dashboard
grpcs://otlp.fourteeners.local:4317 https://otlp.fourteeners.local:4318	OpenTelemetry collector
~~https://tracing.fourteeners.local~~	Jaeger UI (Tempo Query)
https://kiali.fourteeners.local	Kiali console (Keycloak SSO)
https://vault.fourteeners.local	Vault UI
https://policy.fourteeners.local	Policy Reporter UI
https://gitea.fourteeners.local	Gitea UI
ssh://git@gitea.fourteeners.local:2222	Gitea SSH Git
https://gitlab.fourteeners.local	GitLab UI
ssh://git@gitlab.fourteeners.local:2022	GitLab SSH Git
https://*.pages.gitlab.fourteeners.local	GitLab Pages
https://jenkins.fourteeners.local	Jenkins UI
https://buildkite.com/erhhung	Buildkite dashboard
grpcs://buildfarm.fourteeners.local	Buildfarm CAS+RBE (mTLS only)
grpcs://events.buildbuddy.fourteeners.local	BuildBuddy BES (mTLS only)
https://buildbuddy.fourteeners.local	BuildBuddy quickstart
https://vcluster.fourteeners.local	vCluster Platform UI
https://argocd.fourteeners.local	Argo CD UI
https://awx.fourteeners.local	Ansible AWX UI
https://qdrant.fourteeners.local	Qdrant dashboard
https://search.fourteeners.local	SearXNG search UI
wss://playwright.fourteeners.local	Playwright server
https://ollama.fourteeners.local	Ollama API server
https://litellm.fourteeners.local	LiteLLM dashboard
https://openwebui.fourteeners.local	Open WebUI portal
https://mcpo.fourteeners.local	MCP OpenAPI proxy
https://openclaw.fourteeners.local	OpenClaw control
wss://openclaw.fourteeners.local	OpenClaw gateway
https://flowise.fourteeners.local	Flowise designer
ssh://user@opencode.fourteeners.local:1222	VSCode remote SSH

Installation Sources

To-Do Tasks

Migrate manually provisioned certificates and secrets to ones issued by cert-manager with auto-rotation
Automate static DNS records creation in pfSense (dynamically assigned IPs still managed by ExternalDNS)
Install Traefik in RKE2 cluster with Gateway API support, and then gradually migrate Ingress to Gateway
Enable OIDC authentication for additional services: GitLab, Jenkins, AWX, ArgoCD, LiteLLM, Open WebUI
Switch the CNI on the RKE2 cluster from Canal to Cilium and install Hubble web UI to visualize L3/L4 traffic
Harden security posture by applying seccompProfile.type: RuntimeDefault to as many pods as possible
Identify & upload additional sources of personal documents into Open WebUI knowledge base collections

Requirements

# install Ansible and dependencies (pyproject.toml
# should be kept in-sync with requirements.txt)
pip3 install -U -r requirements.txt # or `uv sync -U`

# install required roles and collections
ansible-galaxy install -r requirements.yml

Ansible Vault

The Ansible Vault password is stored in macOS Keychain under item "Home-K8s" for account "ansible-vault"

export ANSIBLE_CONFIG="./ansible.cfg"
VAULTFILE="inventory/group_vars/all/vault.yml"

ansible-vault create $VAULTFILE
ansible-vault edit   $VAULTFILE
ansible-vault view   $VAULTFILE

Some variables stored in Ansible Vault (there are more)

Infrastructure Secrets	User Passwords
`sudo_pass.*`	`rancher_admin_pass`
`icloud_smtp.*`	`minio_root_pass`
`aws_profiles.*`	`minio_admin_pass`
`slack_webhook_urls.*`	`velero_admin_pass`
`docker_access_token`	`harbor_admin_pass`
`github_access_token`	`mailpit_ui_pass`
`age_secret_key`	`opensearch_admin_pass`
`sops_encryption_key`	`keycloak_admin_pass`
`yubikey_unlock_pin`	`thanos_admin_pass`
`pfsense_api_key`	`grafana_admin_pass`
`metallb_secret`	`vault_admin_pass`
`step_ca_provisioner_pass`	`gitlab_root_pass`
`minio_client_pass`	`gitlab_user_pass`
`velero_repo_pass`	`jenkins_admin_pass`
`velero_passphrase`	`vcluster_admin_pass`
`harbor_secret`	`argocd_admin_pass`
`mailpit_smtp_pass`	`awx_admin_pass`
`dashboards_os_pass`	`litellm_admin_pass`
`fluent_os_pass`	`openwebui_admin_pass`
`postgresql_pass`	`flowise_admin_pass`
`valkey_pass`
`oidc_client_secrets.*`
`oauth2_proxy_cookie_secret`
`monitoring_pass`
`policy_reporter_api_pass`
`gitea_secret_key`
`gitlab_secrets_data.*`
`gitlab_omniauth.*`
`jenkins_slack_token`
`buildkite_access_token`
`buildkite_agent_token`
`argocd_signing_key`
`awx_secret_key`
`hass_access_token`
`qdrant_api_key.*`
`searxng_secret_key`
`litellm_master_key`
`openwebui_secret_key`
`openwebui_pipelines_api_key`
`openwebui_mcpo_api_key`
`open_terminal_api_key`
`gogcli_keyring_pass`
`openclaw_gateway_token`
`openclaw_channel_secrets.*`
`flowise_encryption_key`
`anthropic_api_key`
`openai_api_key`
`groq_api_key`

Passwords & API Keys

Some useful commands to generate random passwords and API keys:

Passwords
- pwgen -cnys -r '"!&*\'"'" 12 1
API keys
- alpha-numeric:
  - head -c 4096 /dev/urandom | LC_CTYPE=C tr -cd '0-9a-zA-Z' | head -c 32
  - openssl rand -base64 32 | head -c 32
- hex-digits only:
  - head -c 4096 /dev/urandom | LC_CTYPE=C tr -cd '0-9a-f' | head -c 32
  - openssl rand -hex 16

Connections

All managed hosts are running Ubuntu 24.04 with SSH key from https://github.com/erhhung.keys already authorized.

Ansible will authenticate as user erhhung using private key "~/.ssh/erhhung.pem";
however, all privileged operations using sudo will require the password stored in Vault.

Playbooks

Install required packages

1.1. Tools — lsof, jq, yq, git, helm, etc.
1.2. Drivers — NFS and Intel client GPU drivers
1.3. Python — Ansible packages in virtual env
1.4. Helm — plugins like helm-diff, helm-git
1.5. Debugging — Tools like tcpdump, tshark
```
make packages
```

Configure system settings

2.1. Host — host name, time zone, and locale
2.2. Kernel — sysctl params and pam_limits
2.3. Network — DNS servers and search domains
2.4. Login — customize login MOTD messages
2.5. Certs — add CA certificates to trust store
```
make basics
```

Set up admin user's home directory

3.1. Dot files: .bash_aliases, etc.
3.2. Config files: htop, fastfetch
```
make files
```

Install Rancher Server on single-node K3s cluster
```
make rancher
```

Provision Kubernetes cluster with RKE on 6 nodes

Install RKE2 with a single control plane node and 5 worker nodes, all permitting workloads,
RKE2 in HA mode with 3 control plane nodes and 3 worker nodes, all permitting workloads.
Cluster will be accessible using a virtual IP address provisioned by kube-vip in HA mode.

5.1. Deploy another NGINX ingress controller for SSL passthrough
```
make cluster
```

Create cluster resources from static manifest files

IMPORTANT: Resource manifests must specify the namespaces they wish to be installed
into because the playbook simply applies each one without targeting specific namespaces.
```
make manifests
```

Install Kyverno policy engine and custom policies
Install Policy Reporter to view and notify findings
```
make kyverno policyreporter
```

Install MetalLB network load-balancer in BGP mode

8.1. Create BGPPeer, IPAddressPool, and BGPAdvertisement CRs
to complement FRR BGP configuration on pfSense, the local router
```
make metallb
```

Create static DNS records in pfSense DNS Resolver
```
make dns
```

Install ExternalDNS to manage DNS records in pfSense

10.1. Deploy webhook provider Unbound used by pfSense
```
make externaldns
```

Install cert-manager to automate certificate issuing

11.1. Connect to Step CA pki.fourteeners.local as a StepClusterIssuer
```
make certmanager
```

Install Node Feature Discovery to identify GPU nodes

12.1. Install Intel Device Plugins and GpuDevicePlugin
```
make nodefeatures
```

Install Wave to monitor ConfigMaps and Secrets
```
make wave
```

Install Longhorn dynamic PV provisioner
Install MinIO object storage in HA mode
Install Velero backup and restore tools

14.1. Create a pool of LVM logical volumes
14.2. Install Longhorn storage components
14.3. Install NFS dynamic PV provisioner
14.4. Install MinIO tenant using NFS PVs
14.5. Create MinIO buckets, users, groups
14.6. Install Velero using MinIO as target
14.7. Install Velero Dashboard
```
make storage minio velero
```

Install Harbor OCI & Helm registry

15.1. Mirror images like bitnamilegacy/* from registries
15.2. Mirror charts like bedag/raw from Helm repositories
```
make harbor
```

Install Trivy security scanner

16.1. Install Trivy Operator Polr Adapter to generate Open Reports CRs
```
make trivy
```

Install Mailpit e-mail service

17.1. Configure e-mail relay to iCloud only for specific trusted recipients
```
make mailpit
```

Install OpenSearch cluster in HA mode

18.1. Configure the OpenSearch security plugin (users and roles) for downstream applications
18.2. Install OpenSearch Dashboards UI
```
make opensearch
```

Install Fluent Bit to ingest logs into OpenSearch
```
make logging
```

Install PostgreSQL database in HA mode

20.1. Run initialization SQL script to create roles and databases for downstream applications
20.2. Create users in both PostgreSQL and Pgpool
```
make postgresql
```

Install Keycloak IAM & OIDC provider

21.1. Bootstrap PostgreSQL database with realm homelab, user erhhung, and OIDC clients
```
make keycloak
```

Install Valkey key-value store in HA mode

22.1. Deploy 6 nodes in total: 3 primaries and 3 replicas
```
make valkey
```

Install Prometheus, Alertmanager, and Thanos
Install Grafana for dashboards, traces, and logs
Install Pyrra to manage Service Level Objectives

23.1. Expose Prometheus & Alertmanager UIs via oauth2-proxy integration with Keycloak
23.2. Connect Thanos sidecars to MinIO to store scraped metrics in the telemetry bucket
23.3. Deploy and integrate additional Thanos components with Prometheus & Alertmanager
23.4. Import example SLOs to monitor K8s apiserver/kubelet/coredns and Prometheus
23.5. Add OpenSearch data source to Grafana to display application logs
23.6. Add Tempo data source to Grafana with traces-to-logs and -metrics
```
make monitoring thanos pyrra
```

Install OpenTelemetry collector and Tempo backend

24.1. Enable OTLP receiver and exporter as well as Prometheus exporter
24.2. Connect Tempo to MinIO to store traces in the telemetry bucket
```
make observability tempo
```

Install Istio service mesh in ambient mode
```
make istio
```

Install HashiCorp Vault in HA mode
Install External Secrets Operator

26.1. Initialize Vault cluster and unseal cluster pods
26.2. Create policies, Userpass accounts, k8s roles
26.3. Create KV mounts and populate secrets data
26.4. Create ESO's ClusterSecretStore for Vault
```
make vault externalsecrets
```

Install Gitea DevOps platform to deploy local projects

27.1. Create the Homelab organization, and import Erhhung's SSH and GPG keys
27.2. Configure and deploy Gitea Actions runner with two types of job containers
a. "host" mode for building images using buildah
b. DinD mode to use ubuntu-latest container
27.3. Use al2023-devops as the runner container and load common "step init" script
27.4. Migrate projects from GitHub and run workflows to build images for later installs
```
make gitea
```
Install GitLab EE CI/CD platform to deploy local projects

28.1. Import Erhhung's SSH and GPG public keys, and create the Homelab group
28.2. Configure Harbor and Slack integrations; connect to GitHub using OmniAuth
28.3. Configure and deploy Kubernetes runner for building images using buildah
28.4. Use al2023-devops as the build container and load common pre-build script
28.5. Import projects from GitHub and run pipelines to build images for later installs
28.6. Deploy CI Pipelines Exporter to export metrics and visualize them in Grafana
```
make gitlab
```

Install Jenkins CI/CD platform to deploy local projects

29.1. Configure and provision Jenkins agent for building images using buildah
29.2. Install and configure popular plugins for pipeline and job output visualization
29.3. Implicitly load shared library with Bash functions from Harbor in all pipelines
29.4. Create pipelines from GitHub repositories
```
make jenkins
```

Install Buildkite agent connected to buildkite.com

30.1. Configure agent pod spec with al2023-devops to build images using buildah
30.2. Mount Git, SSH, and Harbor credentials in checkout and command containers
30.3. Create YAML pipelines from GitHub repositories
```
make buildkite
```

Install Buildfarm build execution service
Install BuildBuddy FOSS events viewer
```
make buildfarm buildbuddy
```

Install vCluster Platform management plane
Create vCluster tenant cluster fleet in RKE

32.1. Install vCluster Platform with Free Tier license
32.2. Create tenant clusters with syncing of storage & ingress classes,
persistent volumes & ingresses, and cert-manager certificates to host
~~32.3. Expose Kyverno mutating webhooks from host in tenant clusters~~
```
make vclusters
```

Install Argo CD GitOps delivery in HA mode

33.1. Configure Argo CD to use Valkey for caching
33.2. Configure GitLab as an allowed SCM provider
```
make argocd
```

Install Ansible AWX automation platform

34.1. Create organization and custom execution environments based on al2023-devops
34.2. Create credentials for all homelab hosts and access tokens for GitHub and GitLab
34.3. Import this project and homelab-xcp, and inventories from their hosts.ini files
```
make awx
```

Install Metacontroller to create Operators
```
make metacontroller
```

Install Qdrant vector database in HA mode
```
make qdrant
```

Install SearXNG metasearch engine
Install Playwright WebSocket server
```
make searxng playwright
```

Install LiteLLM AI gateway with vendor models

38.1. Proxy gpt-5.x models through both ChatGPT subscription ("free") and API (metered)
38.2. Proxy Anthropic and Groq models through API (metered)
38.3. Proxy web search through local SearXNG instance
38.4. Send Slack alerts about LLM issues and spending reports
```
make litellm
```

Install Ollama LLM server with modest models
Install Open WebUI AI platform with Pipelines
Install MCP OpenAPI proxy with MCP servers

39.1. Add LiteLLM connection in Open WebUI to proxy OpenAI, Anthropic, and Groq models
39.2. Create Accounts knowledge base and Accounts custom model that embeds that KB
39.3. NOTE: Populate Accounts KB by running make openwebui -t knowledge separately
39.4. Deploy MCP tool servers, including time, browser, weather, lights and aws-api
39.5. Deploy Open Terminal on Mac & Linux hosts, then configure Open WebUI integrations
39.6. Define SLOs for HTTP success rates + latency for server, LiteLLM, Ollama, and MCPO
```
make ollama openwebui
```

Install OpenClaw AI agent gateway and skills

40.1. Proxy access to primary and fallback models, as well as web search, through LiteLLM
40.2. Install skills, such as gog and github, from ClawHub to enhance agent capabilities
40.3. Define user and agent identities through USER.md, IDENTITY.md and SOUL.md
40.4. Configure Slack messaging (requires manual creation of OpenClaw Slack app)
40.5. Automatically pair pending devices (clients)
```
make openclaw
```

Install Flowise AI platform with integrations

Current deployment uses local images in Harbor registry that were built by GitLab CI.
41.1. NOTE: Populate documents by running make flowise -t documents separately
```
make flowise
```

Deploy dev container with OpenCode agent

42.1. Install OpenSSH server for VSCode remote development
42.2. Customize remote terminal UX to match local desktop
42.3. Proxy model access from OpenCode through LiteLLM
```
make opencode
```

Install BentoPDF PDF tools
```
make bentopdf
```

Alternatively, run all playbooks automatically in order:

# specify options like -v or -t
make -- [ansible-playbook-opts]

# run all playbooks starting from "storage"
# ("storage" is a playbook tag in main.yml)
make -- storage-

# run all playbooks up to "dns" (inclusive)
make -- -dns

Output from playbook runs will be logged in "ansible.log".

VS Code Shortcuts

The default Bash shell for VS Code integrated terminal has been configured to load a custom .bash_profile containing aliases for common Ansible-related commands, as well as functions play and debug with completions for tags in playbooks main.yml and debug.yml, respectively.

Multipass Required

Due to the dependency chain of the Prometheus monitoring stack (Keycloak and Valkey), the monitoring.yml playbook must be run after most other playbooks. At the same time, those dependent services also want to create ServiceMonitor resources that require the Prometheus Operator CRDs. Therefore, a second pass through all playbooks, starting with certmanager.yml, is required to enable metrics collection on those services.

Optional Playbooks

Shut down all/specific VMs

make vmshutdown [{group|host}] [{group|host}]...

Create/revert/delete VM snapshots

2.1. Create new snaphots

make vmsnapshot create [targets={group|host},...] \
                           desc="text description"

2.2. Revert to snapshots

make vmsnapshot revert [targets={group|host},...]  \
                           desc="text description" \
                          [date="YYYY-mm-dd prefix"]

2.3. Delete old snaphots

make vmsnapshot delete [targets={group|host},...]  \
                           desc="text description" \
                           date="YYYY-mm-dd prefix"

Start all/specific VMs

make vmstart [{group|host}] [{group|host}]...

VM Storage

To expand the VM disk on a cluster node, the VM must be shut down (attempting to resize the disk from Xen Orchestra will fail with error: VDI in use).

Once the VM disk has been expanded, restart the VM and SSH into the node to resize the partition and LV.

$ sudo su

# verify new size
$ lsblk /dev/xvda

# resize partition
$ parted /dev/xvda
) print
Warning: Not all of the space available to /dev/xvda appears to be used...
Fix/Ignore? Fix

) resizepart 3 100%
# confirm new size
) print
) quit

# sync with kernel
$ partprobe

# confirm new size
$ lsblk /dev/xvda3

# resize VG volume
$ pvresize /dev/xvda3
Physical volume "/dev/xvda3" changed
1 physical volume(s) resized...

# confirm new size
$ pvdisplay

# show LV volumes
$ lvdisplay

# set exact LV size (G=GiB)
$ lvextend -vrL 50G /dev/ubuntu-vg/ubuntu-lv
# or grow LV by percentage
$ lvextend -vrl +90%FREE /dev/ubuntu-vg/ubuntu-lv
Extending logical volume ubuntu-vg/ubuntu-lv to up to...
fsadm: Executing resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now...

After expanding all desired disks, run ./diskfree.sh to confirm available disk space on all cluster nodes.

rancher
-------
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   44G   20G   22G  48% /

k8s1
----
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   37G   19G  67% /
/dev/mapper/ubuntu--vg-data--lv     60G   20G   41G  33% /data

k8s2
----
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   42G   14G  76% /
/dev/mapper/ubuntu--vg-data--lv     60G  4.4G   56G   8% /data

k8s3
----
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   39G   17G  71% /
/dev/mapper/ubuntu--vg-data--lv     60G   15G   46G  25% /data

k8s4
----
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   43G   13G  78% /
/dev/mapper/ubuntu--vg-data--lv     60G   31G   30G  51% /data

k8s5
----
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   45G   11G  81% /
/dev/mapper/ubuntu--vg-data--lv     60G   15G   46G  24% /data

k8s6
----
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   82G   63G   15G  81% /
/dev/mapper/ubuntu--vg-data--lv     60G   18G   43G  30% /data

NODE  DISK                       CAPACITY  ALLOCATABLE  AVAILABLE  SCHEDULABLE  SCHEDULED  RESERVED
---------------------------------------------------------------------------------------------------
k8s1  default-disk-fc0100000000       60G          14G        41G          14G        29G       18G
k8s2  default-disk-fc0100000000       60G          24G        56G          24G        19G       18G
k8s3  default-disk-fc0100000000       60G          19G        46G          19G        24G       18G
k8s4  default-disk-fc0100000000       60G        1010M        30G        1010M        41G       18G
k8s5  default-disk-fc0100000000       60G          18G        46G          18G        25G       18G
k8s6  default-disk-fc0100000000       60G          24G        43G          24G        19G       18G

Troubleshooting

Ansible's ad-hoc commands are useful in these scenarios.

Restart Kubernetes cluster services on all nodes

ansible rancher          -m ansible.builtin.service -b -a "name=k3s         state=restarted"
ansible control_plane_ha -m ansible.builtin.service -b -a "name=rke2-server state=restarted"
ansible workers_ha       -m ansible.builtin.service -b -a "name=rke2-agent  state=restarted"

NOTE: remove _ha suffix from target hosts if the RKE cluster was deployed in non-HA mode.

All kube-proxy static pods on continuous CrashLoopBackOff

This turns out to be a Linux kernel bug in linux-image-6.8.0-56-generic and above (discovered on upgrade to linux-image-6.8.0-57-generic), causing this error in the container logs:

ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"

Current workaround is to downgrade to an earlier kernel.

# list installed kernel images
ansible -v k8s_hosts -a 'bash -c "dpkg -l | grep linux-image"'

# install working kernel image
ansible -v k8s_hosts -b -a 'apt-get install -y linux-image-6.8.0-55-generic'

# GRUB use working kernel image
ansible -v k8s_hosts -m ansible.builtin.shell -b -a '
    kernel="6.8.0-55-generic"
    dvuuid=$(blkid -s UUID -o value /dev/mapper/ubuntu--vg-ubuntu--lv)
    menuid="gnulinux-advanced-$dvuuid>gnulinux-$kernel-advanced-$dvuuid"
    sed -Ei "s/^(GRUB_DEFAULT=).+$/\\1\"$menuid\"/" /etc/default/grub
    grep GRUB_DEFAULT /etc/default/grub
'
# update /boot/grub/grub.cfg
ansible -v k8s_hosts -b -a 'update-grub'

# reboot nodes, one at a time
ansible -v k8s_hosts -m ansible.builtin.reboot -b -a "post_reboot_delay=120" -f 1

# confirm working kernel image
ansible -v k8s_hosts -a 'uname -r'

# remove old backup kernels only
# (keep latest non-working kernel
# so upgrade won't install again)
ansible -v k8s_hosts -b -a 'apt-get autoremove -y --purge'

StatefulSet pod stuck on ContainerCreating due to MountDevice failed

Pod lifecycle events show an error like:

MountVolume.MountDevice failed for volume "pvc-4151d201-437b-4ceb-bbf6-c227ea49e285":
kubernetes.io/csi: attacher.MountDevice failed to create dir "/var/lib/kubelet/plugins/kubernetes.io/
csi/driver.longhorn.io/0bb8a8bc36ca16f14a425e5eaf35ed51af6096bf0302129a05394ce51393cecd/globalmount":
mkdir /var/lib/kubelet/plugins/kubernetes.io/.../globalmount: file exists

Problem is described by this GitHub issue, which may be caused by restarting the node while a Longhorn volume backup is in-progress.

An effective workaround is to unmount that volume.

$ ssh k8s1

$ mount | grep pvc-4151d201-437b-4ceb-bbf6-c227ea49e285

/dev/longhorn/pvc-4151d201-437b-4ceb-bbf6-c227ea49e285 on /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0bb8a8bc36ca16f14a425e5eaf35ed51af6096bf0302129a05394ce51393cecd/globalmount type xfs (rw,relatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/longhorn/pvc-4151d201-437b-4ceb-bbf6-c227ea49e285 on /var/lib/kubelet/pods/06fc67d7-833f-4ecd-810f-77787fd703e6/volumes/kubernetes.io~csi/pvc-4151d201-437b-4ceb-bbf6-c227ea49e285/mount type xfs (rw,relatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)

$ sudo umount /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0bb8a8bc36ca16f14a425e5eaf35ed51af6096bf0302129a05394ce51393cecd/globalmount

Or if pod events show an error like:

Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/
1508f1bfa1a751aaa24514b7576847e7f7ac042c6d8295a6d07417fb4e0068f1/globalmount:
mount system call failed: Structure needs cleaning.

Problem is likely caused by an abrupt node shutdown and file system was not unmounted cleanly.

An effective solution, albeit possibly with some data loss, is to repair that XFS volume.

$ ssh k8s4

# grep may or may not find a match, but /dev/longhorn/pvc-* should still exist
$ mount | grep 1508f1bfa1a751aaa24514b7576847e7f7ac042c6d8295a6d07417fb4e0068f1

/dev/longhorn/pvc-7bc42f2c-4bb6-42f4-ad31-a9fa27185103 on /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/
1508f1bfa1a751aaa24514b7576847e7f7ac042c6d8295a6d07417fb4e0068f1/globalmount type xfs (rw,relatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)

$ sudo xfs_repair -L /dev/longhorn/pvc-7bc42f2c-4bb6-42f4-ad31-a9fa27185103

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - scan filesystem freespace and inode maps...
clearing needsrepair flag and regenerating metadata
sb_fdblocks 1709737, counted 1762490
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
unknown block state, ag 1, blocks 555-1031
        - check for inodes claiming duplicate blocks...
        - agno = 1
        - agno = 2
        - agno = 0
entry "thanos.shipper.json" in shortform directory 131 references free inode 137
junking entry "thanos.shipper.json" in directory inode 131
        - agno = 3
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 134, moving to lost+found
Phase 7 - verify and correct link counts...
Maximum metadata LSN (6:55208) is ahead of log (1:8).
Format log to cycle 9.
done

💡 The above steps have been scripted into a Make target. Simply run: make volrepair pvc-<volume-uuid>

Then restart the pod, and it should run successfully.

Name		Name	Last commit message	Last commit date
.ansible/modules		.ansible/modules
.github/workflows		.github/workflows
.vscode		.vscode
docs/grafana		docs/grafana
files		files
images		images
inventory		inventory
manifests		manifests
pki		pki
playbooks		playbooks
plugins		plugins
scripts		scripts
tasks		tasks
templates		templates
tests/bazel		tests/bazel
vars		vars
.ansible-lint		.ansible-lint
.editorconfig		.editorconfig
.env.tpl		.env.tpl
.gitattributes		.gitattributes
.gitignore		.gitignore
.groovylintrc.json		.groovylintrc.json
AGENTS.md		AGENTS.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
ansible.cfg		ansible.cfg
main.yml		main.yml
mise.toml		mise.toml
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
requirements.yml		requirements.yml

Navigation Menu

Repository files navigation

Erhhung's Home Kubernetes Cluster

Overview

Cluster Topology

Platform Services

AI Services

Service Endpoints

Installation Sources

To-Do Tasks

Requirements

Ansible Vault

Passwords & API Keys

Connections

Playbooks

VS Code Shortcuts

Multipass Required

Optional Playbooks

VM Storage

Troubleshooting

About

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Erhhung's Home Kubernetes Cluster

Overview

Cluster Topology

Platform Services

AI Services

Service Endpoints

Installation Sources

To-Do Tasks

Requirements

Ansible Vault

Passwords & API Keys

Connections

Playbooks

VS Code Shortcuts

Multipass Required

Optional Playbooks

VM Storage

Troubleshooting

About

Topics

Resources

Stars

Watchers

Forks

Contributors

Languages