Skip to content

feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137)#150

Merged
wind-c merged 4 commits intowind-c:mainfrom
debsahu:feat/helm-chart
May 6, 2026
Merged

feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137)#150
wind-c merged 4 commits intowind-c:mainfrom
debsahu:feat/helm-chart

Conversation

@debsahu
Copy link
Copy Markdown
Contributor

@debsahu debsahu commented May 4, 2026

Closes #137.

This PR adds three deliverables that have been requested for productionising comqtt on Kubernetes and tightening up the release pipeline:

1. Helm chart at deploy/helm/comqtt/

Supports both single-node (Deployment) and clustered (StatefulSet + Raft + Gossip) modes from one chart, gated by mode: single|cluster.

Highlights:

  • values.schema.json with conditional rules: cluster mode requires odd replicaCount (Raft quorum) and forbids persistence.enabled=false.
  • A runtime entrypoint shim (mounted from a ConfigMap) that:
    • computes seed members from replicaCount and the headless Service FQDN, so horizontal scaling does not require a chart upgrade,
    • sets --raft-bootstrap=true only when both the pod is *-0 and the Raft data directory is empty — making restarts idempotent so a populated cluster cannot be re-bootstrapped by accident.
  • StatefulSet uses volumeClaimTemplates for per-replica PVCs.
  • PodDisruptionBudget pinned to Raft quorum: minAvailable: ⌈(replicas+1)/2⌉.
  • Soft pod anti-affinity by default; flip cluster.hardAntiAffinity=true to require distinct hostnames.
  • Liveness, readiness, startup probes — all tunable.
  • Optional dashboard Ingress with documented MQTT-TCP caveats (HTTP Ingress can't proxy raw MQTT).
  • Optional ServiceMonitor for kube-prometheus-stack.
  • Bundled helm test pod that runs a mosquitto_pub/mosquitto_sub round trip against the deployed broker.
  • image.tag=latest is rejected by the chart helper (forced to track Chart.appVersion or an explicit pin).

Implementation note on cluster mode

cmd/cluster/main.go replaces all CLI flag values with the loaded --conf file when one is supplied. Passing --raft-bootstrap as a CLI flag alongside --conf is therefore silently ignored. The chart works around this without any upstream Go changes by templating the config file itself at runtime via the entrypoint shim — we ship a config.tpl.yml with placeholders for NODE_NAME, BIND_ADDR, MEMBERS, RAFT_BOOTSTRAP that the shim substitutes before exec'ing comqtt-cluster --conf=....

If maintainers prefer a different approach (e.g., flags-after-conf merging in main.go), happy to follow up with a separate upstream patch and simplify the shim.

Why no bundled Redis sub-chart

The chart historically would have used Bitnami Redis as a sub-chart. As of 2025 the public bitnami/* images are gated behind authentication, which makes a bundled sub-chart unreliable for new users. Instead the chart documents bring your own RESP-compatible store and ships an example Valkey manifest at deploy/helm/comqtt/ci/valkey.yaml as the recommended OSS option (Valkey is the actively-maintained Linux Foundation fork, RESP-protocol-compatible).

2. .goreleaser.yaml

Builds cmd/single and cmd/cluster binaries for linux | darwin | windows × amd64 | arm64, archives + checksums them, and builds multi-arch (amd64 + arm64) Docker images via dockers: + docker_manifests:, publishing to ghcr.io/wind-c/comqtt:{version, vMAJOR.MINOR, latest}. The image build uses a slim Dockerfile.goreleaser rather than the multi-stage Dockerfile at the repo root, since GoReleaser already produces the binaries.

3. CI workflows

  • .github/workflows/release.yaml — triggers on tag push (v*), runs GoReleaser with packages: write permission against GHCR.
  • .github/workflows/chart-lint-test.yamlct lint + helm template + a kind-based install test for both single and cluster modes (cluster job applies ci/valkey.yaml first).
  • .github/workflows/chart-release.yamlhelm/chart-releaser-action on push to main, publishes to GitHub Pages so users can helm repo add comqtt https://wind-c.github.io/comqtt.

Verification gauntlet (run locally on kind v0.31.0)

✅ helm lint deploy/helm/comqtt
✅ helm template ci deploy/helm/comqtt -f .../single-values.yaml  > /tmp/single.yaml
✅ helm template ci deploy/helm/comqtt -f .../cluster-values.yaml > /tmp/cluster.yaml
✅ kubectl --dry-run=client apply -f /tmp/single.yaml
✅ kubectl --dry-run=client apply -f /tmp/cluster.yaml
✅ kind create cluster + helm install single + helm test single
   → MQTT pub/sub round trip via mosquitto_pub/mosquitto_sub
✅ kubectl apply -f deploy/helm/comqtt/ci/valkey.yaml
✅ helm install cluster (3-replica StatefulSet)
   → pod-0 logs: "found raft leader: cluster-comqtt-0"
   → pod-1, pod-2 logs: "raft join cluster-comqtt-0"
✅ Cross-node MQTT: sub on cluster-comqtt-0, pub to cluster-comqtt-2
   → message delivered through shared Valkey state
✅ Bootstrap idempotency: kubectl delete pod cluster-comqtt-0
   → "genesis pod but raft dir is non-empty; bootstrap suppressed"
   → raft-bootstrap: false on restart, cluster reconverges

Out of scope

  • Conventional Commits / DCO sign-off — happy to amend if CONTRIBUTING.md lands or maintainers prefer it.
  • An operator that performs Raft member eviction on permanent pod loss — covered as a documented limitation in the chart README.
  • Cert-manager + ACME wiring beyond the tls.certManager.issuerRef pass-through.

Files added/changed

.github/workflows/release.yaml            new
.github/workflows/chart-lint-test.yaml    new
.github/workflows/chart-release.yaml      new
.goreleaser.yaml                          new
Dockerfile.goreleaser                     new
README.md                                 += "Using Kubernetes (Helm)" section
deploy/helm/comqtt/                       new (chart + CI value files + README)

Closes wind-c#137. This change adds three deliverables:

1. A maintained Helm chart at deploy/helm/comqtt supporting both
   single-node (Deployment) and clustered (StatefulSet + Raft + Gossip)
   modes. The chart includes:
   - values.schema.json with conditional rules (cluster mode requires odd
     replicaCount and persistence.enabled=true)
   - A runtime entrypoint shim that renders the broker config from a
     ConfigMap template, computing seed members from replicaCount + the
     headless Service FQDN, and enabling --raft-bootstrap only when both
     pod-0 AND the Raft data dir is empty (idempotent on restart)
   - PodDisruptionBudget pinned to Raft quorum (ceil((n+1)/2))
   - Soft pod anti-affinity by default; hard via cluster.hardAntiAffinity
   - Per-replica PVCs via volumeClaimTemplates
   - liveness, readiness, and startup probes (all tunable)
   - Optional dashboard Ingress with documented MQTT-TCP caveats
   - Optional ServiceMonitor for kube-prometheus-stack
   - helm test pod that performs a mosquitto pub/sub round trip
   - chart README with full values reference, upgrade notes, and
     limitations (no operator, no automatic Raft member eviction)

2. .goreleaser.yaml building cmd/single + cmd/cluster across linux,
   darwin, windows × amd64, arm64. Builds multi-arch (amd64 + arm64)
   Docker images via goreleaser dockers + docker_manifests, publishing
   to ghcr.io/wind-c/comqtt with semver, minor, and latest tags.

3. .github/workflows/{release,chart-lint-test,chart-release}.yaml:
   - release.yaml triggers on tag push, runs GoReleaser, publishes
     binaries to GitHub Releases and images to GHCR
   - chart-lint-test runs ct lint + helm template + a kind boot test
     across single and cluster CI value files
   - chart-release runs helm/chart-releaser-action on push to main

Implementation note: cmd/cluster/main.go replaces all CLI flag values
with the loaded --conf file when one is supplied, so passing
--raft-bootstrap as a CLI flag alongside --conf is silently ignored.
The chart works around this by templating the config file itself at
runtime via the entrypoint shim.

Bitnami sub-charts (Redis/MySQL/Postgres) are intentionally NOT bundled
because the public bitnami/* Docker images now require authentication.
The chart documents bring-your-own and ships an example Valkey manifest
at deploy/helm/comqtt/ci/valkey.yaml as the recommended OSS RESP store.

Verification gauntlet (all passed locally on kind v0.31.0):
- helm lint deploy/helm/comqtt
- helm template (single + cluster)
- kubectl --dry-run=client apply
- helm install single + helm test (MQTT pub/sub round trip)
- helm install cluster (3-node Raft, leader election + member join)
- Cross-node MQTT: subscribe on cluster-comqtt-0, publish to
  cluster-comqtt-2, message delivered via shared Valkey state
- Bootstrap idempotency: kubectl delete pod cluster-comqtt-0 ->
  "genesis pod but raft dir is non-empty; bootstrap suppressed"
debsahu added 3 commits May 4, 2026 19:53
The test-connection pod had hook-delete-policy "before-hook-creation,
hook-succeeded", which deletes the pod immediately on success. The CI
step `helm test single --logs` then errors out because the pod is gone
before its logs can be fetched.

Drop hook-succeeded; the pod still gets cleaned up before the next
`helm test` run via before-hook-creation.
The workflow accepted a "tag" input but never used it. Without an
existing v* tag, GoReleaser fell back to whatever the most recent tag
was (here: chart-releaser's "comqtt-0.1.0") and failed to parse it as
semver.

On workflow_dispatch, create and push the requested tag before
invoking GoReleaser. The push: tags: v* path is unaffected.
…owner

GoReleaser failed at the archive step because cmd/single builds for 5
platforms (incl. windows) while cmd/cluster builds for 4 (no windows).
The single shared archive then had different binary counts per platform
and GoReleaser refused to package it.

Set archives.allow_different_binary_count: true so the asymmetry is
intentional (windows users get just comqtt; linux/darwin get both
binaries).

Also parameterize the GHCR image owner via $IMAGE_OWNER (lowercased
github.repository_owner) so the same config publishes to
ghcr.io/wind-c/comqtt on upstream and ghcr.io/debsahu/comqtt on the
fork. Drop the hardcoded release.github.owner so GoReleaser uses the
running repo automatically.
@wind-c
Copy link
Copy Markdown
Owner

wind-c commented May 6, 2026

Great work, thanks!

@wind-c wind-c merged commit 56eb8b8 into wind-c:main May 6, 2026
@sansmoraxz
Copy link
Copy Markdown
Contributor

FYI ingress are being deprecated as a way of exposing routes from kubernetes. AFIK the nginx already is. Ref: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/

I would highly recomend using Gateway APIs. Like: https://gateway-api.sigs.k8s.io/guides/tcp/

You can use any of the gateway API providers viz: https://gateway.envoyproxy.io/

@debsahu
Copy link
Copy Markdown
Contributor Author

debsahu commented May 6, 2026

Heads-up follow-up on the chart-release piece of this PR — the chart-release.yaml workflow has been failing since it first ran. Log excerpt:

Packaging chart 'deploy/helm/comqtt'...
Successfully packaged chart in .cr-release-packages/comqtt-0.1.0.tgz
Releasing charts...
Updating charts repo index...
Loading index file from git repository .cr-index/index.yaml
fatal: invalid reference: origin/gh-pages
Error: exit status 128

helm/chart-releaser-action expects the gh-pages branch to already exist and GitHub Pages to be enabled — it doesn't bootstrap either. Confirmed the upstream repo has neither yet (git ls-remote ... gh-pages is empty; repos/wind-c/comqtt/pages returns 404).

Three one-time setup steps on wind-c/comqtt will unblock it:

  1. Create the gh-pages branch (orphan, empty):
    git checkout --orphan gh-pages
    git rm -rf .
    git commit --allow-empty -m "init pages"
    git push origin gh-pages
  2. Settings → Pages → Source = gh-pages branch, / (root).
  3. Settings → Actions → General → Workflow permissions → "Read and write permissions" (so GITHUB_TOKEN can push the updated index.yaml and create the chart release).

After that, a re-run of the workflow (or the next push under deploy/helm/**) will publish, and helm repo add comqtt https://wind-c.github.io/comqtt will work as documented in the chart README. Happy to send a small README/CONTRIBUTING note codifying these prereqs if useful.

@debsahu
Copy link
Copy Markdown
Contributor Author

debsahu commented May 6, 2026

FYI ingress are being deprecated as a way of exposing routes from kubernetes. AFIK the nginx already is. Ref: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/

I would highly recomend using Gateway APIs. Like: https://gateway-api.sigs.k8s.io/guides/tcp/

You can use any of the gateway API providers viz: https://gateway.envoyproxy.io/

hopefully #153 and #154 addresses this.

wind-c pushed a commit that referenced this pull request May 9, 2026
Adds Gateway API resources to the chart and marks the Ingress block as
deprecated, per @sansmoraxz's review feedback on #150 about
ingress-nginx retirement
(https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/).

New resources:

- `templates/httproute.yaml` (gateway.networking.k8s.io/v1) — routes
  the dashboard to a user-supplied Gateway via parentRefs.
- `templates/tcproute.yaml` (gateway.networking.k8s.io/v1alpha2) —
  routes raw MQTT TCP. Requires a TCP-aware Gateway implementation
  (Envoy Gateway, Cilium, etc.).

New `gateway:` values block with a master toggle, default parentRefs,
and per-route overrides. Each route inherits the chart-level parentRefs
when its own list is empty.

Ingress deprecation:
- values.yaml comment marks the block DEPRECATED with the upstream
  retirement reference.
- values.schema.json marks the property `deprecated: true`.
- NOTES.txt prints a migration warning when ingress.enabled=true.
- README replaces the "Exposing MQTT externally" section with a
  Gateway-API-first guide; legacy Ingress kept as a fallback.

The existing Ingress template is retained for back-compat. New
deployments should set `gateway.enabled=true`.

Render paths verified via helm template:
- gateway.enabled=true with both routes + ingress.enabled=true
  (deprecation warning + both routes rendered)
- gateway.dashboard.enabled=false, gateway.mqtt.enabled=true
  (TCPRoute only)
- default values (no gateway resources rendered)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Helm chart?

3 participants