feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137)#150
feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137)#150wind-c merged 4 commits intowind-c:mainfrom
Conversation
Closes wind-c#137. This change adds three deliverables: 1. A maintained Helm chart at deploy/helm/comqtt supporting both single-node (Deployment) and clustered (StatefulSet + Raft + Gossip) modes. The chart includes: - values.schema.json with conditional rules (cluster mode requires odd replicaCount and persistence.enabled=true) - A runtime entrypoint shim that renders the broker config from a ConfigMap template, computing seed members from replicaCount + the headless Service FQDN, and enabling --raft-bootstrap only when both pod-0 AND the Raft data dir is empty (idempotent on restart) - PodDisruptionBudget pinned to Raft quorum (ceil((n+1)/2)) - Soft pod anti-affinity by default; hard via cluster.hardAntiAffinity - Per-replica PVCs via volumeClaimTemplates - liveness, readiness, and startup probes (all tunable) - Optional dashboard Ingress with documented MQTT-TCP caveats - Optional ServiceMonitor for kube-prometheus-stack - helm test pod that performs a mosquitto pub/sub round trip - chart README with full values reference, upgrade notes, and limitations (no operator, no automatic Raft member eviction) 2. .goreleaser.yaml building cmd/single + cmd/cluster across linux, darwin, windows × amd64, arm64. Builds multi-arch (amd64 + arm64) Docker images via goreleaser dockers + docker_manifests, publishing to ghcr.io/wind-c/comqtt with semver, minor, and latest tags. 3. .github/workflows/{release,chart-lint-test,chart-release}.yaml: - release.yaml triggers on tag push, runs GoReleaser, publishes binaries to GitHub Releases and images to GHCR - chart-lint-test runs ct lint + helm template + a kind boot test across single and cluster CI value files - chart-release runs helm/chart-releaser-action on push to main Implementation note: cmd/cluster/main.go replaces all CLI flag values with the loaded --conf file when one is supplied, so passing --raft-bootstrap as a CLI flag alongside --conf is silently ignored. The chart works around this by templating the config file itself at runtime via the entrypoint shim. Bitnami sub-charts (Redis/MySQL/Postgres) are intentionally NOT bundled because the public bitnami/* Docker images now require authentication. The chart documents bring-your-own and ships an example Valkey manifest at deploy/helm/comqtt/ci/valkey.yaml as the recommended OSS RESP store. Verification gauntlet (all passed locally on kind v0.31.0): - helm lint deploy/helm/comqtt - helm template (single + cluster) - kubectl --dry-run=client apply - helm install single + helm test (MQTT pub/sub round trip) - helm install cluster (3-node Raft, leader election + member join) - Cross-node MQTT: subscribe on cluster-comqtt-0, publish to cluster-comqtt-2, message delivered via shared Valkey state - Bootstrap idempotency: kubectl delete pod cluster-comqtt-0 -> "genesis pod but raft dir is non-empty; bootstrap suppressed"
…nd-c#150) # Conflicts: # .gitignore
The test-connection pod had hook-delete-policy "before-hook-creation, hook-succeeded", which deletes the pod immediately on success. The CI step `helm test single --logs` then errors out because the pod is gone before its logs can be fetched. Drop hook-succeeded; the pod still gets cleaned up before the next `helm test` run via before-hook-creation.
The workflow accepted a "tag" input but never used it. Without an existing v* tag, GoReleaser fell back to whatever the most recent tag was (here: chart-releaser's "comqtt-0.1.0") and failed to parse it as semver. On workflow_dispatch, create and push the requested tag before invoking GoReleaser. The push: tags: v* path is unaffected.
…owner GoReleaser failed at the archive step because cmd/single builds for 5 platforms (incl. windows) while cmd/cluster builds for 4 (no windows). The single shared archive then had different binary counts per platform and GoReleaser refused to package it. Set archives.allow_different_binary_count: true so the asymmetry is intentional (windows users get just comqtt; linux/darwin get both binaries). Also parameterize the GHCR image owner via $IMAGE_OWNER (lowercased github.repository_owner) so the same config publishes to ghcr.io/wind-c/comqtt on upstream and ghcr.io/debsahu/comqtt on the fork. Drop the hardcoded release.github.owner so GoReleaser uses the running repo automatically.
|
Great work, thanks! |
|
FYI ingress are being deprecated as a way of exposing routes from kubernetes. AFIK the nginx already is. Ref: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/ I would highly recomend using Gateway APIs. Like: https://gateway-api.sigs.k8s.io/guides/tcp/ You can use any of the gateway API providers viz: https://gateway.envoyproxy.io/ |
|
Heads-up follow-up on the chart-release piece of this PR — the
Three one-time setup steps on
After that, a re-run of the workflow (or the next push under |
|
Adds Gateway API resources to the chart and marks the Ingress block as deprecated, per @sansmoraxz's review feedback on #150 about ingress-nginx retirement (https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/). New resources: - `templates/httproute.yaml` (gateway.networking.k8s.io/v1) — routes the dashboard to a user-supplied Gateway via parentRefs. - `templates/tcproute.yaml` (gateway.networking.k8s.io/v1alpha2) — routes raw MQTT TCP. Requires a TCP-aware Gateway implementation (Envoy Gateway, Cilium, etc.). New `gateway:` values block with a master toggle, default parentRefs, and per-route overrides. Each route inherits the chart-level parentRefs when its own list is empty. Ingress deprecation: - values.yaml comment marks the block DEPRECATED with the upstream retirement reference. - values.schema.json marks the property `deprecated: true`. - NOTES.txt prints a migration warning when ingress.enabled=true. - README replaces the "Exposing MQTT externally" section with a Gateway-API-first guide; legacy Ingress kept as a fallback. The existing Ingress template is retained for back-compat. New deployments should set `gateway.enabled=true`. Render paths verified via helm template: - gateway.enabled=true with both routes + ingress.enabled=true (deprecation warning + both routes rendered) - gateway.dashboard.enabled=false, gateway.mqtt.enabled=true (TCPRoute only) - default values (no gateway resources rendered)
Closes #137.
This PR adds three deliverables that have been requested for productionising comqtt on Kubernetes and tightening up the release pipeline:
1. Helm chart at
deploy/helm/comqtt/Supports both single-node (Deployment) and clustered (StatefulSet + Raft + Gossip) modes from one chart, gated by
mode: single|cluster.Highlights:
values.schema.jsonwith conditional rules: cluster mode requires oddreplicaCount(Raft quorum) and forbidspersistence.enabled=false.replicaCountand the headless Service FQDN, so horizontal scaling does not require a chart upgrade,--raft-bootstrap=trueonly when both the pod is*-0and the Raft data directory is empty — making restarts idempotent so a populated cluster cannot be re-bootstrapped by accident.volumeClaimTemplatesfor per-replica PVCs.minAvailable: ⌈(replicas+1)/2⌉.cluster.hardAntiAffinity=trueto require distinct hostnames.helm testpod that runs amosquitto_pub/mosquitto_subround trip against the deployed broker.image.tag=latestis rejected by the chart helper (forced to trackChart.appVersionor an explicit pin).Implementation note on cluster mode
cmd/cluster/main.goreplaces all CLI flag values with the loaded--conffile when one is supplied. Passing--raft-bootstrapas a CLI flag alongside--confis therefore silently ignored. The chart works around this without any upstream Go changes by templating the config file itself at runtime via the entrypoint shim — we ship aconfig.tpl.ymlwith placeholders forNODE_NAME,BIND_ADDR,MEMBERS,RAFT_BOOTSTRAPthat the shim substitutes before exec'ingcomqtt-cluster --conf=....If maintainers prefer a different approach (e.g., flags-after-conf merging in
main.go), happy to follow up with a separate upstream patch and simplify the shim.Why no bundled Redis sub-chart
The chart historically would have used Bitnami Redis as a sub-chart. As of 2025 the public
bitnami/*images are gated behind authentication, which makes a bundled sub-chart unreliable for new users. Instead the chart documents bring your own RESP-compatible store and ships an example Valkey manifest atdeploy/helm/comqtt/ci/valkey.yamlas the recommended OSS option (Valkey is the actively-maintained Linux Foundation fork, RESP-protocol-compatible).2.
.goreleaser.yamlBuilds
cmd/singleandcmd/clusterbinaries forlinux | darwin | windows × amd64 | arm64, archives + checksums them, and builds multi-arch (amd64 + arm64) Docker images viadockers:+docker_manifests:, publishing toghcr.io/wind-c/comqtt:{version, vMAJOR.MINOR, latest}. The image build uses a slimDockerfile.goreleaserrather than the multi-stage Dockerfile at the repo root, since GoReleaser already produces the binaries.3. CI workflows
.github/workflows/release.yaml— triggers on tag push (v*), runs GoReleaser withpackages: writepermission against GHCR..github/workflows/chart-lint-test.yaml—ct lint+helm template+ a kind-based install test for both single and cluster modes (cluster job appliesci/valkey.yamlfirst)..github/workflows/chart-release.yaml—helm/chart-releaser-actionon push tomain, publishes to GitHub Pages so users canhelm repo add comqtt https://wind-c.github.io/comqtt.Verification gauntlet (run locally on kind v0.31.0)
Out of scope
CONTRIBUTING.mdlands or maintainers prefer it.tls.certManager.issuerRefpass-through.Files added/changed