Outdated readme, don't believe it!

F9's Homelab

... managed with Flux, Doco-CD, Renovate and GitHub Actions

---

Overview

This is a mono repository for my entire homelab configuration, including my Kubernetes cluster and Docker instance. It uses Infrastructure as Code (IaC) and GitOps practices as much as possible using tools like Kubernetes, Flux, Renovate, Doco-CD, and GitHub Actions. All secrets are using 1Password Connect with the exception of 1 or 2 which are SOPS encrypted.

Kubernetes

My Kubernetes cluster is deployed with Talos. This is a semi-hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate TrueNAS server for NFS/SMB shares, AI, bulk file storage and backups.

Core Components

• cert-manager: Creates SSL certificates for services in my cluster.
• cilium: eBPF-based networking for my workloads.
• envoy: Modern Kubernetes Gateway provider
• rook: Distributed storage provider for peristent storage using CEPH
• volsync: Asynchronous data replication for Kubernetes volumes to NAS via NFS
• external-secrets: Managed Kubernetes secrets using 1Password Connect.
• sops: Managed secrets for Kubernetes and Terraform/OpenTofu which are commited to Git.

GitOps

Flux watches the clusters in my kubernetes folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository.

Flux will apply all in kubernetes/apps

Flux will recursively search sub folders until it finds the most top level kustomization.yaml per directory and then apply all the resources listed in it. That aforementioned kustomization.yaml will generally only have a namespace resource and one or many Flux kustomizations (ks.yaml). Under the control of those Flux kustomizations there will be a HelmRelease or other resources related to the application which will be applied.

Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When PRs are merged Flux applies the changes to my cluster.

---

Docker

My Docker instance is running on my TrueNAS server and managed by Doco-CD with GitOps practices, it runs AI workloads that require a dedicated NVidia GPU, Ser2Net for my Zigbee adapter and more, I've tried to limit as much as possible what the NAS does and where possible offloaded it to the Kubernetes cluster.

GitOps

Doco-CD watches my docker folder (see Directories below) and makes the changes based on the state of my Git repository.

Doco-CD is controlled mostly from a single file, the .doco-cd.yaml file. This is the file that tells Doco-CD what folders to search for compose files to deploy.

---

Let's Go!

Pre-requisites

Hardware

• 3x Computers (I'm using Dell Optiplex 3060's with 2.5Gbe Ethernet)

Software

• Linux or WSL on Windows is preferred
• Brew - /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Stage 1: Repository Preparation (Infrastructure)

1. Create a new repository by clicking the green Use this template button at the top of this page, then clone the new repo you just created and cd into it.

Warning

As this repository assumes it's for myself, there are many hardcoded domain name references currently set, I would recommend find and replacing all references of ${SECRET_EXTERNAL_DOMAIN} with your own tld and f9-casa with your own tld replacing dots with dashes, you will ofcourse also need to update any secret files with your own values.

Critical Files that will need updating are

• cert-manager/clusterissuer.yaml
• traefik/certificate.yaml
• flux-instance/values.yaml

Other files should be updated, but these ones will stop deployment working

Stage 2: Bootstrap Talos Infrastructure

Proxmox Talos VM's, Basic Kubernetes cluster with Cilium

1. Ensure the following enviroment variables are set to the correct values/paths PROXMOX_VE_USERNAME, PROXMOX_VE_PASSWORD

export PROXMOX_VE_USERNAME="root@pam"
export PROXMOX_VE_PASSWORD="password"

2. Edit the infrastructure/talos/prod.auto.tfvars file to reflect your desired configuration, refer to comments for explanations of eaach variable

3. Execute the Infrastructure Bootstrap script

rm infrastructure/talos/output/* # WARNING! This will delete your old kube-config and talos-confg
rm infrastructure/talos/terraform.tfstate # WARNING! This will delete state
rm infrastructure/talos/terraform.tfstate.backup # WARNING! This will delete state
chmod +x ./infrastructure//talos/bootstrap.sh
./infrastructure/talos/bootstrap.sh

Tip

Optionally add --dry-run/-d to test it, or if you are feeling brave --auto-approve/-y to skip confirmation prompts

Note

Approximate time to deploy: 5 Minutes (assuming NVME storage)

Stage 3: Bootstrap FluxCD Deployment

1. Ensure you have updated all your *sops* files in kubernetes/** to match your own values

Note

Not many of them have sample files currently but eventually i will ensure every *sops* file has a matching *.sample file [!TIP] If you are using VSCode you should be able to automatically encrypt your sops files.

2. Ensure the following enviroment variables are set to the correct values/paths SOPS_AGE_KEY_FILE, KUBECONFIG, GITHUB_OWNER, GITHUB_REPO

export SOPS_AGE_KEY_FILE=$PWD/.age.key
export KUBECONFIG=$PWD/infrastructure/talos/output/kube-config.yaml
export GITHUB_REPO="https://github.com/fma965/f9-homelab" # replace with your github repo url

3. Execute the Kubernetes Bootstrap script

chmod +x ./kubernetes/bootstrap.sh
./kubernetes/bootstrap.sh

Tip

Optionally if you are feeling brave --auto-approve/-y to skip confirmation prompts

Note

Approximate time to deploy: 10 Minutes (assuming NVME storage, excluding PV restore if applicable)

Stage 4: Bootstrap Docker Deployment (Semi-Manual)

Warning

If you do not intend to use Docker skip this stage

Note

Currently Komodo does not support adding of a Git Repo via the km CLI, once this is added we can automate this a bit more

1. Install Komodo from the AppStore (TrueNAS) or Docker Compose files (UnRaid / Other)
2. Access the Komodo WebUI
3. Navigate to "Settings" > "Profile" and create a "New Api Key", copy the Key and Secret in to the Komodo 1Password entry
4. Navigate to "Settings" > "Providers" and add a Github.com Account using your token (regenerate it in Github if needed)
5. Navigate to "Syncs" and Create a New Resource Sync called "f9-homelab"
6. Set the Mode to "Git Repo", Repo to "fma965/f9-homelab"
7. Set the Account to "fma965"
8. Add the following resource path docker/komodo.toml
9. Enable "Delete Unmatched Resources" and "Managed"
10. Make sure only "Sync Resources" is checked under the Include section
11. Click "Save", Click "Refresh" and the "Execute"

---

Git Repo Structure

Click here to the directories of this Git Repo with descriptions

.
├── 📂 docker
│
├── 📂 infrastructure
│   └── 📂 talos
│       ├── 📂 output              # Terraform output artifacts
│       └── 📂 talos               🤖 # Talos Linux Kubernetes configurations
│
└── 📂 kubernetes
    ├── 📂 apps
    │   ├── 📂 backup              💾 # Backup solutions
    │   ├── 📂 ceph-csi            💾 # Ceph CSI storage driver
    │   ├── 📂 cert-manager        📜 # SSL certificate management
    │   ├── 📂 default             🏠 # Dashboard and landing page
    │   ├── 📂 external-secrets    🤫 # External secret management
    │   ├── 📂 flux-system         ⚡ # GitOps management (FluxCD)
    │   ├── 📂 game                🦖 # Game servers
    │   ├── 📂 git                 ⎙ # Git services
    │   ├── 📂 kube-system         ⚙️ # Core Kubernetes system components
    │   ├── 📂 observability       👁️ # Monitoring and logging stack
    │   ├── 📂 openebs-system      💾 # Container Attached Storage (OpenEBS)
    │   ├── 📂 postgresql          🐘 # PostgreSQL databases
    │   ├── 📂 redis               🧠 # Redis key-value stores
    │   ├── 📂 security            🛡️ # Security tools
    │   ├── 📂 system-upgrade      ⬆️ # Kubernetes node upgrade controller
    │   ├── 📂 traefik             🚦 # Ingress controller and reverse proxy
    │   ├── 📂 volsync             🔄 # Volume snapshot and replication
    │   └── 📂 webdev              🌐 # Web development projects
    ├── 📂 components
    │   ├── 📂 common             ⚙️ # Shared Kubernetes components
    │   ├── 📂 gatus
    │   ├── 📂 volsync
    │   └── 📂 volsync-backuponly
    └── 📂 flux
        └── 📂 cluster             ⚡ # GitOps cluster definitions

---

Cloud Dependencies

Most of my infrastructure and workloads are self-hosted and do not rely upon cloud services, some however do, here is a list of them. Note that some of these may not be part of this repo but rather just things that i also use in relation to it.

Service	Use	Cost
1Password	Secrets with External Secrets	~$40/yr
Cloudflare	Domain	~£6/yr
GCP	Voice interactions with Home Assistant over Google Assistant	Free
GitHub	Hosting this repository and continuous integration/deployments	Free
Discord	Alerts and notifications	Free
Pushover	Kubernetes Alerts and application notifications	$5 OTP

Wiki

Check out my Wiki to see more about my hardware and much more

---

Gratitude and Thanks

Thanks to the Home Operations / OneDr0p Discord and community.