Scope: live Hetzner release evidence for v3. Module source was the staging or release-candidate worktree unless noted. This file records release-gate evidence that ordinary fmt, validate, and static plan checks did not cover.
| Date | Scenario | Staging commit | Method | Result | Key observations |
|---|---|---|---|---|---|
| 2026-07-04/05 | k3s + Leap Micro fresh cluster: 1 control plane, 1 agent, Cilium, Traefik | e666a65 | Local Terraform apply from a live root with module source pointed at the staging worktree; Kubernetes/node readiness inspection after apply | PASS after Cilium fix | 45 resources; all nodes Ready on openSUSE Leap Micro 6.2; first attempt at db0e905^ failed on Cilium values YAML bug, fixed by 5639cf9. |
| 2026-07-04/05 | RKE2 + Leap Micro fresh cluster: 1 control plane, 1 agent | 8c6e411 | Local Terraform apply from a live root with staging module source; Kubernetes system pod and node OS inspection | PASS | 46 resources; both nodes Ready on Leap Micro 6.2; kube-scheduler Running; kube-reserved=512Mi live-confirmed on cx23; earlier attempts exposed reservation bugs fixed by cfe9ab6 and 8c6e411. |
| 2026-07-04/05 | v2.21.0 to v3 in-place upgrade: live 37-resource v2.21 cluster, MicroOS, Cilium, nginx | 8c6e411 | Switched live root module source to staging worktree, pinned k3s_channel = "v1.33", reviewed plan, applied, then inspected cluster health and node OS |
PASS | Plan had 0 destroy/replace of hcloud infrastructure; 4 in-place updates to firewall rules/server labels; 6 terraform_data re-runs; 12 new v3 resources; cluster healthy after apply; Cilium Running; nodes stayed MicroOS with no recreation; migration effort was source switch plus k3s_channel = "v1.33" pin. |
| 2026-07-04/05 | Autoscaler: 1 control plane, 1 agent, Cluster Autoscaler pool min 1 | 72e1c54^ (db0e905 era) |
Local Terraform apply from a live root with staging module source; verified autoscaler Deployment and autoscaler-created node readiness | PASS | 47 resources; autoscaler-created node joined Ready on Leap Micro 6.2; Cluster Autoscaler deployment healthy. |
| 2026-07-04/05 | NAT router: private-only nodes, control-plane load balancer, k3s + Leap Micro | 72e1c54^ (4311e98) |
Local Terraform apply from a live root with staging module source; router sshd -t; node readiness; private-node image-pull egress proof through NAT |
PASS after heredoc fix | 59-resource topology; router sshd -t valid; both private nodes Ready on Leap Micro; egress via NAT proven by image pulls; before 4311e98, every fresh NAT router bricked sshd about 2.5 minutes after boot. |
| 2026-07-05 | Redundant NAT router failover: enable_redundancy keepalived pair with private nodes |
3c576a3 | Live root with staging module source; powered off the primary NAT router and verified pod egress during failover | PASS | Redundant-NAT keepalived failover was proven by pod egress during primary poweroff, closing the previous live gap for enable_redundancy. |
| 2026-07-05 | Destroy-time ingress LoadBalancer cleanup after CCM adoption | 78e2c39 | Live destroy/re-destroy of a staging cluster with a module-managed ingress LoadBalancer; observed CCM cleanup and Terraform destroy behavior | PASS with retry caveat | Destroy-time LB cleanup is live-proven. A known already-detaching race can still require one terraform destroy retry; CI retries destroy once and plan 011 tracks the single-ownership design follow-up. |
| 2026-08-08 | v3.1.0 k3s + refreshed Leap Micro canary: 1 control plane, 1 agent, Flannel, Traefik, cert-manager, CSI | codex/fix-repeat-leapmicro-snapshot-builds |
Local matrix run 20260808-0109-v310-k3s-newimg pinned to signed snapshot 417523913; retained post-apply for Kubernetes, workload, storage, host, transactional-update, and reboot inspection |
PASS | 47 resources; both nodes Ready on k3s v1.36.3+k3s1; zero unhealthy nodes/pods/deployments/daemonsets/jobs; API ready/live, metrics, DNS/HTTP, certificate issuance, real HCloud CSI write/delete, and external Traefik passed. Both nodes completed a 69-package transactional update, booted snapshot #4 and kernel .36, preserved SELinux/SSH hardening, and returned Ready with zero failed units. Destroy removed all 47 resources; the superseded test snapshot was then deleted. |
| 2026-08-08 | Pre-final Leap Micro x86 k3s snapshot build | codex/fix-repeat-leapmicro-snapshot-builds |
Real Packer build using vendored openSUSE/Rancher trust anchors, reviewed openSUSE digest, real Rancher SELinux RPM, transactional updates, two reboots, and snapshot capture | PASS, superseded | Build completed in 4m38s and proved the initial hardened path, but later verifier, post-reboot, and runtime-fallback fixes changed the candidate bytes. Snapshot 417533886 is not final release evidence; immutable exact-tree build and canary results are recorded on the release PR because repository bytes freeze at functional commit A. |
| 2026-08-08 | Existing-state RKE2 upgrade plan (test13) |
codex/fix-repeat-leapmicro-snapshot-builds |
terraform init -upgrade, saved plan, and JSON action inspection against the non-empty local state |
PASS | Plan contains one expected internal terraform_data.configure_autoscaler delete/create to reapply the changed verified autoscaler payload. No HCloud server, network, subnet, load balancer, placement group, SSH key, firewall, or volume is destroyed or replaced; the existing cluster was not modified. |
| Date | Run | Staging commit | Method | Result | default Terraform/OpenTofu | nginx Terraform/OpenTofu | rke2 Terraform/OpenTofu | Notes |
|---|---|---|---|---|---|---|---|---|
| 2026-07-04/05 | 28718815894 | 136acfc^ | Test in Hetzner workflow, workflow_dispatch on staging |
PARTIAL | PASS/PASS | FAIL/FAIL | FAIL/FAIL | RKE2 failed from k3s-flavored Leap snapshot because per-distro secrets were missing; nginx failed from 67-character server names. |
| 2026-07-04/05 | 28719712912 + 28719694176 raced | 136acfc..2d0060d | Test in Hetzner workflow, overlapping workflow_dispatch runs on staging |
PARTIAL | PASS/PASS/PASS plus one quota loss | FAIL x4 | PASS x4 | RKE2 fixed by per-distro snapshots; nginx failures root-caused to load-balancer annotations spliced by strip-markers; quota losses came from 12 parallel clusters; max-parallel: 2 added in 2d0060d. |
| 2026-07-04/05 | 28723492149 (record run) | 5aafc86 | Test in Hetzner workflow, workflow_dispatch on staging |
PASS | PASS/PASS | PASS/PASS | PASS/PASS | FULL GREEN 6/6 -- serialized (max-parallel 2), per-distro Leap snapshots, honest OS assertion; locks the v3 CI record. |
| Date | Repository state | Method | Result | Evidence |
|---|---|---|---|---|
| 2026-08-08 | Canonical repository mysticaltech/terraform-hcloud-kube-hetzner |
Adversarial fixture contract followed by exact GitHub REST and GraphQL API readback with scripts/check-github-release-controls.sh |
PASS | master requires pull requests plus the strict GitHub Actions Validate Packer and supply-chain fixtures check bound to app ID 15368, with admin enforcement and conversation resolution; it rejects force-push/deletion, branch locking, signatures/restrictions, positive approval counts, code-owner review, last-push approval, and linear history so the required merge commit remains possible. Authoritative GraphQL readback separately proves the exact master protection rule requires no deployments, names no required deployment environments, and has zero pull-request bypass actors. Inherited ruleset discovery is fully paginated and flattened; the effective rules on master are exactly the administrator-only update/deletion/non-fast-forward controls, while additional active tag/push/repository rulesets are rejected. GitHub retains cheap lint/docs/publication only; HCloud credentials, smoke, cluster inspection, and cleanup remain local. The release workflow has no manual-dispatch trigger. Adversarial fixtures prove those predicates reject drift and missing authoritative fields. |
| 2026-08-08 | v3.1.0 release candidate | Iterative independent release review plus release-state, topology, authoritative-remote, workflow, generated-site, and live-control adversarial fixtures | PASS; A/B execution enforced | Earlier independent passes rejected stale pre-tag authority, incomplete merge-control coverage, first-page-only ruleset discovery, a branch-lock false pass, unchecked code-owner/last-push review switches, a nonexistent REST deployment field, unchecked pull-request bypass actors, and ambient-HEAD tag creation. The corrected gates run slow immutable-tree checks first, fully paginate inherited controls, use authoritative REST plus GraphQL protection fields, bind the tag to the exact reviewed commit, and use an atomic protected-master lease. The final focused review reran local contracts plus authenticated live readback and returned ALL CLEAR. PR/tag topology gates require functional commit A plus README-only pin commit B and reject intermediate modify/revert commits, mode changes, extra commits, unrelated merge parents, tree drift, or a wrong manifest. The generated site routes users to the release-tagged README, and snapshot guidance uses a fresh manifest-bound generated bundle plus the full OS/distribution/architecture matrix. |
These were invisible to ordinary fmt, validate, and static plan-only checks.
| Date | Staging commit | Method | Result | One-line description |
|---|---|---|---|---|
| 2026-07-04/05 | 5639cf9 | Live k3s + Cilium apply failure, rendered values inspection, yamldecode contract hardening |
FIXED | Fixed Cilium values indentation and added a yamldecode semantic contract so malformed rendered YAML fails before live apply. |
| 2026-07-04/05 | cfe9ab6 + 8c6e411 | Live RKE2 + Leap bootstrap failure on small cx23 control plane, kubelet reservation review, follow-up apply |
FIXED | Added size-aware kubelet reservations, avoided jsonencode optional-equality traps, and added safe fallbacks so RKE2 scheduler pods fit small control planes. |
| 2026-07-04/05 | 8c6e411 | CI/live OS assertions using Kubernetes node data | FIXED | Switched CI OS assertion to nodeInfo.osImage, verifying the node OS observed by Kubernetes rather than weaker host metadata. |
| 2026-07-04/05 | db0e905 | Live gate and changelog review | FIXED | Hardened Longhorn 1.5+ readiness lists, removed redundant nonsensitive() calls twice, added >50% reservation guard parsing, and added SSH revocation sidecar behavior. |
| 2026-07-04/05 | 136acfc | Hetzner CI failures and plan-name review | FIXED | Added per-distro Leap CI snapshots, shortened CI names, and added a 63-character server-name plan guard. |
| 2026-07-04/05 | 2d0060d | Hetzner CI quota/race review | FIXED | Added max-parallel: 2 to avoid quota loss from too many concurrent Hetzner clusters. |
| 2026-07-04/05 | 4311e98 | Fresh NAT router live apply, delayed SSH failure reproduction, repo heredoc sweep | FIXED | Fixed the NAT heredoc bug that wrote script text into SSH config and bricked fresh routers, then swept repo heredocs for the same class. |
| 2026-07-04/05 | 72e1c54 | Hetzner CI nginx failures, rendered Helm values inspection, plan-time semantic assertion | FIXED | Fixed ingress load-balancer annotation splicing and added a semantic adoption contract for rendered ingress values. |
| 2026-07-05 | e506cc4 + 78e2c39 | Live destroy after CCM-adopted ingress LoadBalancer survived cluster teardown; follow-up CI destroy behavior review | FIXED WITH RETRY CAVEAT | Restored fail-open destroy-time ingress Service cleanup so CCM releases the adopted LB before node/network teardown; removed the deterministic settle wait and retry destroys once for the known already-detaching race. |
| 2026-08-08 | codex/fix-repeat-leapmicro-snapshot-builds |
Fresh snapshot builds, two live k3s applies, and adversarial offline verifier fixtures | FIXED; exact-tree rerun pending | Replaced the stale Leap Micro image with collision-resistant snapshot builds; vendored and lifecycle-checked the full openSUSE/Rancher trust anchors; bound the requested appliance to version/architecture/flavor and a reviewed exact digest; verified SELinux RPM signer/package identity; and proved real transactional updates on both canary nodes without signature bypasses. The final exact-tree image evidence is recorded separately only after the post-review builds and canary complete. |
| 2026-08-08 | codex/fix-repeat-leapmicro-snapshot-builds |
First canary failed when cert-manager Helm reconciliation deleted cainjector between get and wait; hermetic replay plus second live apply |
FIXED | Made shared k3s/RKE2 deployment readiness retry transient replacement/NotFound races against one fixed deadline; the second canary exercised the interrupted cert-manager reinstall and completed successfully. |
| 2026-08-08 | codex/fix-repeat-leapmicro-snapshot-builds |
Maximum release-security review, iterative unbounded follow-up review, adversarial bootstrap/topology fixtures, and clean-source design review | HARDENED; A/B proof enforced | Replaced moving release/tag resolution and raw script execution with one commit-addressed Codeload archive verified before extraction, an independently pinned Packer manifest, and an explicit source-directory override. The release uses functional commit A plus README-only pin commit B to avoid a circular same-commit archive hash. Independent release-control follow-up is clean; CI and publication prove the final topology and execute the real archive after A is pushed and its exact digests are installed. |
| 2026-08-08 | v3.1.0 pre-freeze candidate | Exact-archive MicroOS x86 Packer build against the official Tumbleweed appliance sidecars | FIXED; re-freeze required | The verifier rejected a rotated x86 appliance because its validly signed checksum no longer matched the repository-owned reviewed digest. The current checksum selected exactly the canonical x86 pathname and was signed by the vendored openSUSE primary key AD485664E901B867051AB15F35A2F86E29B700A4; the reviewed x86 digest and bundle manifest were refreshed without weakening verification. The ARM digest was unchanged. |
| Date | Staging commit | Method | Result | Gap |
|---|---|---|---|---|
| 2026-07-04/05 | 72e1c54 | Not live-tested | NOT COVERED | Robot nodes remain untested. |
| 2026-07-04/05 | 72e1c54 | Not live-tested beyond preview constraints | NOT COVERED | Multinetwork remains an experimental preview, not release-grade live coverage. |
| 2026-07-04/05 | 72e1c54 | Workflow observation from canceled CI jobs | OPEN WORKFLOW GAP | Canceled CI jobs can orphan clusters; cleanup is manual today and should become a future workflow improvement. |
| 2026-07-04/05 | 72e1c54 / v2 master pending | Static pattern concern only; no explicit v2 live decision yet | PENDING DECISION | v2 master may carry the same ingress-annotation splice pattern; changing it on live v2 clusters needs an explicit behavior-risk decision. |
| 2026-07-05 | 78e2c39 | Live destroy/re-destroy and CI destroy behavior review | KNOWN RETRY CAVEAT | Destroy-time LB cleanup is live-proven, but a benign already-detaching race can still make one terraform destroy retry necessary until plan 011 resolves ingress LB single ownership. |