M8F-148: Add NATS UI for monitoring events and system stats; move NATS services to m8flow-nats-docker-compose.yml

[sonal-aot]

JIRA Ticket

M8F-148

Description

NATS UI & M8Flow NATS Implementation

NATS UI — NUI

NUI is an open-source browser-based dashboard for managing and monitoring a NATS server. In the M8Flow stack it runs as a Docker container (ghcr.io/nats-nui/nui:latest) accessible at port 8282.

Features

• Browse JetStream streams, subjects, and message counts
• Inspect durable consumers, pending counts, and lag
• View Key-Value buckets (e.g. m8flow-dedup)
• Manually publish test messages
• Monitor server health and connection status

---

M8Flow NATS Implementation

1. API Key Authentication

Tenant admins generate an API key via POST /m8flow/nats-tokens.

• The raw key (prefixed m8f_...) is shown once and never stored
• Stored as an HMAC-SHA256 hash (salted with M8FLOW_NATS_SALT) in m8flow_nats_tokens
• Consumer verifies using hmac.compare_digest to prevent timing attacks

2. NATS Consumer (consumer.py)

A standalone async Python service that bridges NATS JetStream to SpiffWorkflow:

Step	Description
Connect	Connects to NATS with username/password auth
Pull	Durable JetStream pull subscription on stream M8FLOW_EVENTS
Validate	Checks required fields, verifies api_key via NatsTokenService
Deduplicate	NATS KV bucket (m8flow-dedup) blocks duplicate tenant_id + event_id pairs (configurable TTL)
Instantiate	Calls ProcessInstanceService directly inside Flask app context — no HTTP hop
Retry	msg.nak(delay=5) on failure; msg.ack() on success

3. Docker Split Architecture

Stack	Compose File	Services
m8flow-nats-stack	m8flow-nats-docker-compose.yml	NATS server, NUI dashboard
m8flow-default	m8flow-docker-compose.yml	Backend, frontend, NATS consumer

The consumer connects via the shared m8flow_default Docker network, so the NATS stack can be restarted independently. The consumer auto-reconnects on disconnect.

---

Event Message Schema

{
  "id": "unique-event-uuid",
  "tenant_id": "your-tenant-uuid",
  "api_key": "m8f_...",
  "process_identifier": "group-name/process-model-name",
  "username": "username@tenant_slug",
  "payload": {}
}

Type

• Feature
• Bug fix
• Documentation
• Other

Changes

• Backend
• Frontend
• Documentation

Testing

Prerequisites

• All stacks running:

  docker compose -f docker/m8flow-nats-docker-compose.yml up -d
  docker compose -f docker/m8flow-docker-compose.yml up -d

• NUI dashboard accessible at http://localhost:8282

---

1. Connect NUI to NATS Server

1. Open http://localhost:8282
2. Add a new connection:
   • Host: nats://nats:4222
   • Username: admin (or value of M8FLOW_NATS_USER)
   • Password: (value of M8FLOW_NATS_PASSWORD)
3. Click Connect — status should show Connected

---

2. Verify Stream Exists

1. Navigate to Streams in NUI
2. Confirm stream M8FLOW_EVENTS is listed with subject m8flow.events.>

---

3. Publish a Test Event via NUI

1. Go to Streams → M8FLOW_EVENTS → Publish
2. Set subject: m8flow.events.<your-tenant-uuid>.trigger
3. Paste the message body:

   {
     "id": "test-event-001",
     "tenant_id": "your-tenant-uuid",
     "api_key": "m8f_...",
     "process_identifier": "group/process-model",
     "username": "user@tenant",
     "payload": {}
   }

4. Click Publish

---

4. Verify Consumer Processed the Event

1. In NUI → Streams → M8FLOW_EVENTS → Consumers
2. Confirm durable consumer m8flow-engine-consumer shows 0 pending messages
3. Cross-check consumer logs:

   docker logs m8flow-nats-consumer --follow

   Expected:

   [INFO] Process instance created | tenant=... identifier=... instance_id=...
   

---

5. Verify Deduplication via NUI

1. Go to Key-Value in NUI
2. Open bucket m8flow-dedup
3. Confirm an entry exists for key <tenant-id>-test-event-001
4. Re-publish the same message from NUI (same id)
5. Consumer log should show:

   [WARNING] Duplicate event id='test-event-001' — already processed. Discarding.
   

Related Issues

Closes #M8F-148

[andrepestana-aot]

Shouldn't we move "m8flow-nats-consumer" to m8flow-nats-docker-compose.yml ?

[sonal-aot]

to optimize the core deployment, the NATS server and NATS-UI have been moved to a standalone Docker Compose configuration. While the m8flow-nats-consumer remains part of the main application build, its execution is conditionally controlled by the M8FLOW_NATS_ENABLED environment variable. This modular approach reduces the resource overhead of the main stack and gives customers the flexibility to use their own NATS infrastructure or alternative UI tools if preferred.

[andrepestana-aot]

Please update docs, README.md etc. describing how to use this docker compose.

[auslin-aot]

m8flow-backend still depends on a nats service that no longer exists in that file.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud