feat: EMQX-flavor web dashboard with multi-user RBAC, SSE, and cluster-wide event aggregation

[debsahu]

EMQX-flavor web dashboard embedded in the comqtt broker. Closes the Roadmap "Dashboard" entry. 41 commits, ~8.9k LOC, all tests pass under -race.

What's in

Backend (Phase 1, 17 commits)

• Auth subsystem (mqtt/dashboard/auth/): HMAC-signed cookie sessions (~30 LOC, no JWT), file + Redis multi-user credential stores, in-memory lockout tracker, CSRF tokens, RBAC middleware (admin/viewer roles).
• 5 new REST endpoints with paginated responses: GET /clients (search by ClientID prefix), GET /subscriptions (filter by topic + clientid), GET /topics (tree), GET /retained (with optional payload preview, 4 KiB cap), GET /sessions. Plus DELETE /clients/{id}/subscriptions/{topic}, DELETE /retained/{topic}, DELETE /sessions/{id}.
• SSE pipeline (mqtt/dashboard/sse/): ring-buffered hub + mqtt-hook bridge + handler at /dashboard/events. The hub fan-outs are non-blocking with drop-on-full so the broker hot path can never stall.
• Cluster mirrors in cluster/rest/ for every new endpoint, fan-out via the existing fetchM helper.
• One small upstream-friendly change: added (*mqtt.Server).Hooks() accessor (5 lines) so REST handlers can inspect storage-backed sessions without monkey-patching the broker package.

Static UI (Phase 2, 15 commits)

• Stack: server-rendered Go html/template + htmx (vendored, ~50 KB) + Tailwind v3 (built by standalone CLI binary, no Node toolchain). All assets bundled via go:embed. go build produces a single binary.
• Pages: Overview (polling cards), Clients (paginated + search), Blacklist, Tools (publish), Settings (Chroma-rendered YAML), Users (admin), Account (read-only profile).
• Auth flows: login form, force-rotation on default password, password change, logout. Lockout fires after 5 fails / 5 min, holds for 10 min.
• Theme: light + dark with toggle, persisted in localStorage.
• Single binary: dashboard mounts on the existing :8080 listener alongside /api/v1/*. / redirects to /dashboard/.

Realtime + detail (Phase 3, 4 commits)

• SSE-driven Recent Events feed on Overview. Default response is HTML fragments suitable for htmx-sse direct DOM swap; ?as=json returns raw events for programmatic consumers.
• Inline-SVG sparkline helper (~50 LOC, no Chart.js). 60-bucket rolling history per metric.
• Client detail page with Info / Subscriptions (per-row Unsubscribe) / Events tabs.
• Subscriptions, Topics (collapsible tree), Retained, Sessions pages.

Cluster polish (Phase 4, 6 commits)

• Redis-backed cred store sharing user records across cluster nodes via a single hash, WATCH/MULTI/EXEC for atomic mutations.
• Redis-backed HMAC secret via EnsureSecret SETNX-with-race-loss-reread.
• Cluster page showing member list with leader badge.
• Redis pub/sub event bridge for cluster-wide event aggregation. Each node publishes its local events to channel comqtt:dashboard:events and subscribes to receive peer events. Self-echo filter + 4096-entry FIFO LRU dedup. The full pipeline is exercised by an integration test that spins two in-process brokers + miniredis and observes a connect on node-A from node-B's SSE stream.
• cmd/cluster/main.go plumbs the redis client through to the dashboard so cluster mode actually uses these components when redis is configured (storage-way=3 or auth-ds=1).

Hardening (Phase 6)

• go test -race ./mqtt/dashboard/... ./mqtt/rest/... is clean across all dashboard packages. 127 tests under race detector.
• Viewer role audit: every admin POST/DELETE route is wrapped in RequireRole(RoleAdmin) middleware (defense in depth at the server) AND every state-changing button renders with disabled attribute when User.Role != \"admin\" (UX layer). The viewer can browse every page and observe state, but can't mutate.

Tested

Layer	Status
go test -race ./mqtt/dashboard/... ./mqtt/rest/...	PASS (127 tests)
go test -tags=integration ./mqtt/dashboard/... -run TestCluster	PASS (cross-node event aggregation through miniredis)
go build ./cmd/single ./cmd/cluster	PASS (40 MB single binary including all dashboard assets)
Manual smoke: login form / static CSS / root redirect / API back-compat	PASS
Pre-existing failures unrelated to this change	QUIC flake (unrelated), postgres test (needs real DB)

Configuration

New keys under dashboard: in the broker config:

dashboard:
  enabled: true              # default
  session-secret: \"\"         # base64 or raw; auto-generated if empty
  password-expiry-days: 90   # 0 disables

Plus DASHBOARD_INITIAL_PASSWORD env var if you want to skip the random password printed on first boot.

In cluster mode: the dashboard reads the same cfg.Redis settings the broker already uses for storage-way=3. No extra redis config needed.

Deliberately deferred

These were all in scope per the design doc but pushed out for review tractability:

• TOTP/2FA, multi-tab session revocation, configuration editor - documented in the design as v2 features.
• Token-based embedded-link auth (EMQX 5.6+ feature) - same.
• Add/Remove peer actions on the Cluster page - the page is read-only in v1; admins use the existing POST /api/v1/cluster/peers endpoint.
• Helm chart updates - the chart lives in a separate PR (feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137) #150). Once both this PR and feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137) #150 are merged, a small follow-up will add dashboard.* values + Secret-mounted session secret to the chart.
• Per-client event filter on the Client detail page's Events tab - placeholder UI; events are visible on the global Overview feed.
• chromedp E2E smoke - skipped for v1; coverage from 127 unit tests + cluster integration test was the higher-value time investment.

File breakdown

Approximately:

• 21 files in mqtt/dashboard/ (auth, sse, handlers, dashboard.go, embed.go)
• 14 templates + 4 static assets (htmx, htmx-sse, tailwind.css, theme.js)
• 7 new endpoint files in mqtt/rest/
• 8 new mirror handlers in cluster/rest/rest.go
• Wiring touches in cmd/single/main.go, cmd/cluster/main.go, mqtt/server.go, cluster/agent.go, config/config.go
• 1 new dep: github.com/alecthomas/chroma/v2 (pure Go, ~1 MB binary bloat) for Settings page YAML highlighting.

Out of scope for this PR (already filed or to be filed)

• Helm chart dashboard.* values block (follow-up PR after feat(deploy): add Helm chart, GoReleaser, and GHCR release workflow (closes #137) #150 merges)
• Real ConnectedAt timestamp on the Client detail page (currently 0 placeholder)
• Per-client event filter on the Events tab
• chromedp E2E tests
• Hot-reloadable broker config

How to try locally

go build -o ./comqtt ./cmd/single
DASHBOARD_INITIAL_PASSWORD=mypass ./comqtt --storage-way=0
# then in a browser:
open http://localhost:8080/dashboard/
# username admin, password mypass

[wind-c]

Thanks for this impressive work — the dashboard is a major feature addition. I have one architectural suggestion before merge.

dashboard/ should be a top-level package, not nested under mqtt/

Current layout:

mqtt/dashboard/auth/
mqtt/dashboard/handlers/
mqtt/dashboard/sse/
mqtt/dashboard/templates/
mqtt/dashboard/static/
mqtt/dashboard/dashboard.go
mqtt/dashboard/embed.go

Proposed layout:

dashboard/auth/
dashboard/handlers/
dashboard/sse/
dashboard/templates/
dashboard/static/
dashboard/dashboard.go
dashboard/embed.go

Rationale

1. Separation of concerns — mqtt/ is the core broker package (protocol, packets, hooks, listeners). Dashboard is a separate subsystem with its own auth, routing, templates, and SSE pipeline. Nesting it under mqtt/ conflates two unrelated domains and bloats the core package.

2. Consistency with existing structure — cluster/ sits at the top level as an independent module that builds on top of the broker. Dashboard occupies the same tier: it is a consumer of the broker, not part of the broker itself.

3. Dashboard is not a hook — Unlike mqtt/hooks/* which implement the mqtt.Hook interface and are tightly coupled to the broker lifecycle, the dashboard is a self-contained web application that only needs a *mqtt.Server reference. It belongs alongside cluster/, not inside mqtt/hooks/ or mqtt/.

4. Independent evolution — UI templates, static assets, and auth flows change at a different cadence than the MQTT protocol engine. A top-level package makes it clear that dashboard changes do not affect core broker semantics and avoids unnecessary churn in mqtt/ during code reviews and releases.

5. Avoids leaking abstractions — This PR already adds a (*mqtt.Server).Hooks() accessor (5 lines) solely so the dashboard can inspect storage-backed sessions. Placing dashboard at the top level makes such cross-boundary dependencies explicit rather than hidden by package proximity.

Impact

This is a package rename — no logic changes required. The only adjustments are import paths and the wiring in cmd/single/main.go / cmd/cluster/main.go. Given that this PR has not yet merged, now is the best time to make this structural correction.

Would you be willing to restructure before merge?

[debsahu]

Yes will send update soon.

[debsahu]

Restructure pushed — mqtt/dashboard/ is now top-level dashboard/ (sibling of cluster/). Pure rename: imports, Makefile asset paths, the tailwind content glob, and one config.go doc-comment all updated. go test -race ./dashboard/... ./mqtt/rest/... ./cluster/rest/... passes (151 tests), the integration test still passes, and both cmd/single and cmd/cluster build clean.

Two quick questions while you're reviewing:

1. Dashboard colors / layout — the v1 theme is a fairly EMQX-style palette (slate/indigo accents on Tailwind defaults) with a light/dark toggle, plus a Vercel-ish cluster topology block on Overview. Before I lock it in, do you have a preference on the overall look — keep close to EMQX, lean toward something more neutral/admin-panel, or match any existing comqtt branding I missed? Same on layout density (current pages use comfortable spacing; some maintainers prefer a tighter, data-table-heavy view).

2. New dependency — this PR adds github.com/alecthomas/chroma/v2 (pure Go, ~1 MB binary bloat) only to syntax-highlight the YAML on the Settings page. If you'd rather keep the dep surface minimal I can drop it and render the YAML as plain <pre> — happy either way.

[wind-c]

Thanks for the quick restructure — top-level dashboard/ looks good.

1. Dashboard colors / layout

I'd recommend not following the EMQX palette directly — comqtt should have its own visual identity rather than mirroring a competitor's style.

Suggested palette (IoT-Blue):

Element	Color	Rationale
Primary (nav)	#0F172A (Slate 900)	Professional, grounded
Accent	#0EA5E9 (Sky 500)	IoT/connectivity feel — evokes message flow, distinct from EMQX's Indigo
Success / Online	#10B981 (Emerald 500)	Client connected, healthy
Warning	#F59E0B (Amber 500)	Needs attention, not urgent
Error / Offline	#EF4444 (Red 500)	Disconnected, anomaly
Background (light)	#F8FAFC (Slate 50)	Comfortable reading, not pure white
Background (dark)	#0F172A (Slate 900)	Unified with nav

Why Sky Blue over Indigo:

• EMQX uses Indigo/purple-blue — Sky Blue is immediately distinguishable
• Sky Blue communicates IoT, cloud, and connectivity, which aligns with comqtt's positioning as a lightweight, high-performance MQTT broker
• Light mode uses Slate 50 instead of pure white to reduce eye strain — consistent with the "lightweight" brand

Layout density: Prefer compact/tight spacing. The dashboard is an ops tool — information density over comfort. Data-table-heavy view is fine.

2. Chroma dependency

Keep it. 1 MB on a 40 MB binary is ~2.5% — negligible. YAML syntax highlighting on the Settings page is a high-frequency ops workflow where readability matters. Chroma is pure Go, no CGO, well-maintained — no concerns.

[debsahu]

For local testing of the latest push (IoT-Blue palette + themed select chrome):

# 1. Fetch the PR branch
gh pr checkout 151
# or, without gh:
# git fetch origin pull/151/head:pr-151 && git checkout pr-151

# 2. Build (static dashboard assets are committed, so plain `go build` works)
go build -o ./comqtt ./cmd/single

# 3. Run with a known admin password
DASHBOARD_INITIAL_PASSWORD=preview ./comqtt --storage-way=0

# 4. Open in a browser
open http://localhost:8080/dashboard/
# username: admin   password: preview

To populate the views (Topics, Subscriptions, Retained, and the Overview SSE feed), in a second terminal (needs mosquitto-clients):

# Persistent subscribers
mosquitto_sub -h localhost -i sub-sensors -t 'sensors/#' -q 1 &
mosquitto_sub -h localhost -i sub-status  -t 'devices/+/status' -q 0 &
mosquitto_sub -h localhost -i sub-events  -t 'events/#' &

# Retained seeds
for t in sensors/temp/room1 sensors/temp/room2 sensors/humidity/room1 \
         devices/d1/status devices/d2/status events/login; do
  mosquitto_pub -h localhost -t "$t" -m "{\"v\":$RANDOM}" -r -q 1
done

# Continuous traffic for the Recent Events feed and sparklines
while true; do
  mosquitto_pub -h localhost -t "events/burst/$((RANDOM % 10))" -m "tick-$RANDOM"
  sleep 2
done

Things to look at:

1. Sky 500 accent on primary buttons (Login, Publish, Save), focus rings on inputs, link colors.
2. Slate background (light: #F8FAFC, dark: #0F172A); toggle theme via the icon top-right.
3. Status colors: Emerald online indicators on Clients, Red destructive buttons, Amber wired but not yet used in templates (available as bg-warning / text-warning).
4. Native select dropdowns now render with a themed chevron and bg-card fill instead of OS chrome on a gray background. Visible on Tools (QoS), Sessions (filter), and Users (Role).
5. Cluster topology graphic on Overview (single-mode shows the standalone variant; cluster mode shows the multi-node view).

Layout density (your "compact, data-table-heavy" preference) is the next pass. Happy to send that as a separate commit if the palette looks right first.

[wind-c]

The dashboard you've implemented feels overly heavyweight.
You've added and modified 111 files in total, and the code volume has even surpassed that of the original comqtt codebase — I'm genuinely surprised by this!
Dashboards are generally designed to be lightweight.
After running and reviewing the actual dashboard pages and features, I don't believe such a massive overhaul is necessary to implement the functionality.
My apologies, but I need to carefully evaluate this before moving forward!

[debsahu]

Thanks for taking the time to look at this carefully and for the honest feedback. The scope concern is fair, and I'd rather get the direction right than push something into the codebase that you'll regret carrying.

Two paths I can offer, whichever fits comqtt's direction better.

Option 1: Slim down the PR significantly.

Trim aggressively to a minimal core: file-based single-user auth, six pages (Login, Overview, Clients, Subscriptions, Topics, Retained), polling instead of SSE, no cluster event aggregation, no Settings page (drops the chroma dep), no Tools page, no Users page, no sparklines. That brings the diff from roughly 8.9k LOC to around 3k LOC and removes most of what felt heavyweight. The cost is that Phase 3 and Phase 4 features (realtime, cluster polish, multi-user RBAC) do not ship; users who want them later need a follow-up PR.

Option 2: Keep comqtt lightweight; ship the dashboard as a separate add-on.

The dashboard touches comqtt at exactly one place: a 5-line (*mqtt.Server).Hooks() accessor so external hook authors can read storage-backed sessions. Everything else lives in its own package and can move to debsahu/comqtt-dashboard. Concretely:

1. A small standalone PR to comqtt that adds only the Hooks() accessor, framed for general hook authors rather than dashboard-specific. Five lines, zero feature surface in the broker.
2. This PR closes.
3. I maintain debsahu/comqtt-dashboard as a separate module that imports comqtt, registers the SSE hook, and exposes its own binaries (comqtt-dashboard, comqtt-cluster-dashboard) plus its own helm chart. Users who want the dashboard pull the add-on binary; users who want a lightweight broker stay on stock comqtt.
4. The two open chart PRs adjust accordingly: feat(helm): wire dashboard.* values + Secret-mounted session secret into chart #153 (dashboard helm values) closes and moves to the add-on repo; feat(helm): Gateway API support; deprecate Ingress #154 (Gateway API support) is independent and stays for upstream review.

Option 2 is what I would recommend if your read is that comqtt should stay focused on the broker itself. It directly addresses the "code volume surpasses comqtt" point: your repo gains five lines, not 8.9k. It also lets the dashboard iterate on its own cadence without burdening your review bandwidth.

Your call either way. If you want to explore Option 1 first, I can prepare a slimmed branch so you can compare; if Option 2 is the right framing, I'll close this PR and open the small Hooks() accessor PR with a clear standalone justification.

[wind-c]

I support Option 2!
Please go ahead and submit the PR to enable you to maintain the debsahu/comqtt-dashboard repository independently.
We will add a link to your dashboard project in the comqtt README.md, allowing interested developers and users to choose it freely.
Thank you for your understanding and suggestions!

[debsahu]

Thanks. Proceeding with Option 2.

Status:

• feat(server): add Server.Hooks() accessor for external hook authors #158 opened upstream with only the Server.Hooks() accessor (7 lines, framed as a general-purpose getter for hook authors and monitoring tools, no dashboard-specific framing in the rationale).
• feat(helm): wire dashboard.* values + Secret-mounted session secret into chart #153 (dashboard helm values follow-up) closed; that work moves into the add-on repo's own chart.
• feat(helm): Gateway API support; deprecate Ingress #154 (Gateway API support) stays open since it is independent of the dashboard.

Next: I will scaffold debsahu/comqtt-dashboard and migrate the dashboard package, the dashboard-specific REST endpoints, and the helm chart additions there. This PR will stay open as a draft until the add-on repo's first release is out, so the discussion thread is preserved; it will then close with a pointer to the new repo.

[debsahu]

Closing per the plan we agreed on: the dashboard moved to a separate add-on module so stock comqtt stays focused on the broker.

• Add-on repo: https://github.com/debsahu/comqtt-dashboard
• v0.1.0 (single-mode) released: https://github.com/debsahu/comqtt-dashboard/releases/tag/v0.1.0
• Cluster mode + helm chart land in v0.2.0.

The only upstream change the add-on needs is the small Server.Hooks() accessor in #158. Until that merges and a release is cut, the add-on uses a replace directive in go.mod to soft-fork comqtt at the same commit pushed in #158.

Thanks again for the careful review and for nudging this in a direction that keeps comqtt lightweight.

[wind-c]

Perfect, looking forward to it!