CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Overview

This is a GitOps repository managing a Kubernetes cluster using ArgoCD. All applications are declaratively defined and automatically synced from this repository to the cluster.

ArgoCD Application Pattern

All applications follow a consistent multi-source pattern defined in apps/*.yaml:

spec:
  sources:
  - repoURL: <upstream-helm-chart-repo>
    chart: <chart-name>
    targetRevision: <version>
    helm:
      valueFiles:
      - $values/<app-dir>/values.yaml
  - repoURL: https://github.com/Shion1305/k8s-GitOps.git
    targetRevision: HEAD
    ref: values  # Reference for Helm values
  - repoURL: https://github.com/Shion1305/k8s-GitOps.git
    targetRevision: HEAD
    path: <app-dir>  # Additional manifests (ingress, secrets, etc.)

Not all apps use this multi-source pattern. Some apps use a single-source pattern pointing directly at a repo directory (e.g., apps/ingress.yaml), where ArgoCD deploys raw manifests or Kustomize from the directory. Check the existing app manifest to see which pattern is used.

When modifying applications:

• Helm chart configurations go in <app-name>/values.yaml
• Additional Kubernetes manifests (Ingress, Secrets, Jobs) go in the app's directory, referenced by kustomization.yaml
• Version updates are done by changing targetRevision in apps/<app-name>-app.yaml
• To decommission an app, move its apps/*.yaml manifest to apps/archived/

Key Commands

ArgoCD Management

# View all applications
kubectl get applications -n argocd

# Force sync an application
kubectl patch application <app-name> -n argocd --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

# Access ArgoCD UI
# URL: https://argocd.shion1305.com

PostgreSQL Operations

Use the ./manage-postgres.sh script for PostgreSQL cluster management:

./manage-postgres.sh list                    # List clusters
./manage-postgres.sh status <cluster-name>   # Check cluster status
./manage-postgres.sh connect <cluster-name>  # Get connection info
./manage-postgres.sh users <cluster-name>    # List users and credentials
./manage-postgres.sh scale <cluster> <n>     # Scale to n instances
./manage-postgres.sh backup <cluster>        # Trigger manual backup

PostgreSQL clusters are managed by Zalando Postgres Operator in the postgres-clusters namespace.

Storage Setup

# Create storage directories on nodes
./setup-storage.sh

# Setup persistent volume directories for specific apps
./setup-pv-dirs.sh

Monitoring Access

# Grafana: http://<node-ip>:30080
# Admin credentials: see the `kube-prometheus-stack-grafana` Secret in the
# `grafana` namespace (or the in-cluster Vault path the chart values reference).

# Check Prometheus targets
kubectl port-forward -n grafana svc/kube-prometheus-stack-prometheus 9090:9090

Secret Management Architecture

This cluster uses a layered secret management approach:

HashiCorp Vault → External Secrets Operator (ESO) → Kubernetes Secrets

Vault Configuration

• Deployed in HA mode with 3 replicas (Raft storage)
• UI: https://vault.i.shion1305.com
• Default mount path: secret/ (KV v2); each app typically gets its own dedicated KV v2 mount

External Secrets Operator

• Reads from Vault using per-namespace SecretStore + ServiceAccount, each backed by a scoped Vault role/policy
• Also uses the Kubernetes provider to copy postgres-operator-generated DB credentials into app namespaces

Adding Secrets to Applications

1. Enable a per-service KV mount in Vault (vault secrets enable -path=<svc> kv-v2) and write the secret (vault kv put <svc>/<key> ...)
2. Create a Vault policy and a Kubernetes auth role bound to the namespace's ServiceAccount (add to vault/scripts/setup-eso-policies.sh)
3. Add a namespace-scoped SecretStore + ExternalSecret in the app directory referencing the Vault path
4. ESO syncs the Vault data to a Kubernetes Secret in that namespace

Identity & Authentication

Keycloak Setup

• Purpose: Provides OIDC/SAML authentication and Docker Registry v2 auth
• URL: https://keycloak.shion1305.com (legacy https://keycloak.k.shion1305.com 301-redirects to the new host)
• Admin access: master realm admin is shion1305 (passkey-only). The Keycloak Operator's bootstrap mechanism (KC_BOOTSTRAP_ADMIN_USERNAME/KC_BOOTSTRAP_ADMIN_PASSWORD and the keycloak-initial-admin Secret it generates) only fires on the very first server start when the DB has no admin; it is not re-evaluated on subsequent restarts. After a real admin exists, that Secret is intentionally absent. To recover from a lost admin: scale the StatefulSet to 0 and run kc.sh bootstrap-admin user --username <name> in a one-shot Pod that reuses the StatefulSet env/volumes.
• Realms:
  • master: Keycloak admin only (do not modify declaratively)
  • user: central human-user pool (passkey-only); brokered into child realms
  • zot: Docker registry auth (zot UI + GitHub Actions token-exchange)
  • ynufes-tech: GitHub-OAuth realm for the cloudflare-grafana audience
• GitHub Actions Integration: Uses OIDC token exchange via gha-exchanger client (in the zot realm)

Storage

Longhorn (Distributed Storage)

• Provides replicated block storage across cluster nodes
• Manages persistent volumes with replication and snapshots
• Backup target configured in longhorn/backup-target.yaml
• StorageClass: longhorn (defined in longhorn/storageclass.yaml)

Node-specific Storage

• Local storage provisioner for node-pinned workloads
• Setup scripts create directories in /var/local-storage on nodes
• Used by applications requiring specific node placement

Monitoring Stack

Deployed via kube-prometheus-stack (Grafana + Prometheus):

Components:

• Grafana: Visualization and dashboards (namespace: grafana)
• Prometheus: Metrics collection with 30-day retention, 200Gi storage
• AlertManager: Alert routing and management (5Gi storage)
• Node Exporter: Hardware/OS metrics from cluster nodes
• Kube State Metrics: Kubernetes object metrics

Pre-configured Dashboards:

• Kubernetes Cluster Monitoring (ID: 7249)
• Node Exporter Full (ID: 1860)
• Node Exporter Server Metrics (ID: 405)

Custom Exporters:

• Cloudflare Exporter: Monitors Cloudflare metrics with custom Grafana dashboard
• Airbyte Metrics Exporter: Monitors Airbyte data pipeline health

Service discovery via ServiceMonitors - Prometheus automatically discovers and scrapes targets across all namespaces.

Ingress Configuration

See docs/networking.md for the full architecture, conventions, and migration status.

Cluster traffic is fronted by Envoy Gateway (envoy-gateway-system namespace). Two Gateway resources expose two listener spaces:

1. external (141.147.189.36): publicly reachable. Listeners:

   • https on *.shion1305.com — canonical apex hostname for all externally-published apps (argocd, langfuse, keycloak, vault, openwebui, github-readme-stats, ynufes-cf grafana, …).
   • https-legacy-k on *.k.shion1305.com — legacy listener serving 301-redirect HTTPRoutes that move old *.k URLs to their apex equivalents. To be removed once all consumers have migrated.

2. internal (10.130.5.21, reachable via WireGuard): listener https on *.i.shion1305.com — used by atc/grafana, atc/prometheus, longhorn, mlflow, memgator, freqtrade, etc.

Apps publish themselves with HTTPRoute resources attached to one of these Gateways via parentRefs.sectionName (https or https-legacy-k). Cross-namespace parentRefs work because each Gateway's allowedRoutes.namespaces.from: All accepts routes from any namespace; per-app ReferenceGrant resources permit the Gateway namespace to dial backend Services.

Example HTTPRoute pattern:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <app>-external
  namespace: <app-ns>
spec:
  parentRefs:
    - name: external
      namespace: envoy-gateway-system
      sectionName: https
  hostnames:
    - <app>.shion1305.com
  rules:
    - backendRefs:
        - name: <service-name>
          port: <port>

Network Policy & Namespace Isolation

Cross-namespace traffic is default-denied on ingress. A Kyverno ClusterPolicy (network-policies/cross-ns-isolation-generator.yaml) generates a default-deny-ingress NetworkPolicy into every non-system namespace (current and future), allowing only same-namespace ingress. Egress is left open (the generated policy sets policyTypes: [Ingress] only). Cilium evaluates standard + Cilium policies as a union of allows.

The only cluster-wide ingress allows are in network-policies/allow-from-infra.yaml (additive CiliumClusterwideNetworkPolicy, all ports):

• node / kubelet / kube-apiserver / Cilium health
• the envoy-gateway-system namespace (the ingress proxy → any backend)
• the grafana namespace (Prometheus scrapes and Grafana datasource queries both originate here)

When adding an app, ask who needs to reach it on ingress:

• Reached only by the Gateway, by Prometheus/Grafana, or by same-namespace pods → no NetworkPolicy needed (covered above).
• Reached from any other namespace (e.g. another app's pod, or a controller that connects from its own namespace) → add an app-specific allow policy in the app dir; it unions with the generated default-deny.

Note that ServiceMonitor scrapes and cross-namespace GrafanaDatasource queries are both ingress from the grafana namespace, so both are already allowed. See network-policies/README.md for the full model.

Application Directory Structure

Each application directory typically contains:

• values.yaml: Helm chart overrides
• kustomization.yaml: Kustomize configuration for additional resources
• Additional manifests: ingress, secrets, jobs, custom resources
• README.md: Application-specific documentation (when present)

Documentation Responsibility

• Repository-level documents (README.md, CLAUDE.md, MONITORING.md): Must remain generic and describe cluster-wide infrastructure, patterns, and conventions. Do not add service-specific details here.
• Service-level documents (<service-dir>/README.md): Each service directory should contain its own README.md with service-specific details such as endpoints, credentials, architecture, and usage instructions.

Common Troubleshooting Patterns

Application not syncing

# Check application status
kubectl get application <app-name> -n argocd -o yaml

# View sync status
kubectl describe application <app-name> -n argocd

# Force refresh
kubectl delete application <app-name> -n argocd
kubectl apply -f apps/<app-name>-app.yaml

Secret synchronization issues

# Check ESO operator logs
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

# Verify ClusterSecretStore connection to Vault
kubectl describe clustersecretstore vault-cluster

# Force secret refresh
# On a ClusterExternalSecret the key MUST be namespaced — the controller
# propagates only `external-secrets.io/force-sync` down to the child
# ExternalSecrets, so a bare `force-sync` annotation silently no-ops.
kubectl annotate clusterexternalsecret <name> \
  external-secrets.io/force-sync="$(date +%s)" --overwrite

# On a plain ExternalSecret any annotation change triggers a resync, so the
# bare key is fine there.
kubectl annotate externalsecret -n <ns> <name> force-sync="$(date +%s)" --overwrite

Storage issues

# Check Longhorn status
kubectl get nodes -n longhorn-system
kubectl get volumes -n longhorn-system

# View PV/PVC status
kubectl get pv,pvc --all-namespaces

CI: Manifest render-validation

Every PR runs .github/workflows/render-validate.yaml, which executes scripts/render-validate.sh to render every ArgoCD Application in apps/ (Helm + Kustomize + raw paths) and pipe the output through kubeconform against Kubernetes API + CRD schemas.

• Helm charts are pulled per their targetRevision; external git repos (e.g. kubernetes-sigs/gateway-api) are shallow-cloned.
• CRD schemas come from the datreeio CRDs-catalog repo; unknown CRDs are skipped (-ignore-missing-schemas).
• Apps with known upstream chart bugs are tolerated via scripts/render-validate.allowlist. Each entry documents the upstream issue. When a chart bump fixes the bug the script will report "now passes — drop from allowlist".

To run locally: ./scripts/render-validate.sh. Requires helm, kustomize, kubeconform, yq, jq on PATH.

Dependency Management (Renovate)

Renovate runs in-cluster (deployed via Helm chart in renovate/) and automatically creates PRs for dependency updates:

• Configuration: renovate.json at repo root
• Manages: ArgoCD Helm chart versions in apps/*.yaml, Docker image tags across all YAML files, and Grafana dashboard revisions
• All dependency update PRs are set to automerge
• Commit prefix: chore(deps): with semantic commits and git sign-off
• PR limits: 5 concurrent, 2 per hour

Important Notes

• All applications use automated sync with self-healing enabled
• ArgoCD automatically creates namespaces via CreateNamespace=true sync option
• Most applications use ServerSideApply=true for better conflict resolution
• Domains: external apps publish at *.shion1305.com, internal apps at *.i.shion1305.com. Legacy *.k.shion1305.com URLs 301-redirect to their apex equivalents and will be retired once consumers migrate.
• Node labels and taints affect workload scheduling - check node status when troubleshooting pod placement

Repository Hygiene

This is a public repository on GitHub.

• Never commit real or example credentials, passwords, tokens, API keys, or admin secrets — including in docs, READMEs, comments, or example commands. Use <placeholder-name> style if a value must appear in an example.
• Real secrets live in Vault and are materialized into Kubernetes Secrets via External Secrets Operator. Documentation should reference the Vault path (e.g. vault kv get <svc>/<key>), not the value.
• For KeycloakRealmImport clients and IdPs, use the literal string PLACEHOLDER_REPLACE_AFTER_REALM_IMPORT. The Keycloak operator generates real values on first import; humans then write those values into Vault out-of-band.