Jetstream and Gateway could not be enabled at the sametime

[veilnet-admin]

What version were you using?

2.12.3-alpine

What environment was the server running in?

k3s

Is this defect reproducible?

If Jetstream is enabled, and Gateway is enabled at the same time, the gateway will refuse any connection from other clusters.

Given the capability you are leveraging, describe your expectation?

A supercluster should be formed by the gateway.

Given the expectation, what is the defect you are observing?

This is not an issue of NATS itself, but a misconfiguration of the readiness probe of the Helm chart. The defect is caused by:

1. Jetstream and Gateway are both enabled. NATS node expects to connect with all explicit gateways before a meta leader for Jetstream can be elected.

2. Due to the readiness probe configuration, the NATS node is considered not ready, and the service does not forward traffic to the NATS node.

3. So connections to explicit gateways all got connection refused because the remote cluster is in the same state.

4. So the meta leader can not be elected, and NATS nodes never become ready and stall forever.

The fix is very simple:

service:
enabled: true
ports:
nats:
enabled: true
gateway:
enabled: true
merge:
spec:
type: LoadBalancer
publishNotReadyAddresses: true # force loadbalancer or any type of service to publish regardless

I hope this helps anyone stuck on this. It took me 3 days to figure it out...

[avacaru]

I'm seeing the same issue, so helm upgrades and especially node upgrades put the NATS cluster in a bad state that is impossible to recover from. The service.spec.publishNotReadyAddresses might be a work-around, but definitely not a solution.