JetStream on cluster should be shared with leafnode, all using TLS. But streams aren't visible on both ends

[tbalbers]

Hi,
I'm obviously missing something here. I've set up a 3-node cluster with JetStream using TLS certs for internal communication in the cluster and I've also managed to connect a leaf node also using TLS certs.
The certificates all contain an email address(which is different from user to user) in their SAN-field which I map to a user account.
All certs are from the same CA.

So what I have is this:

3 clusternodes using certA for internal stuff and client connections.
1 leafnode using certB for connecting to the cluster.
I've also set the jetstream domain to the same value on all 4 hosts

All clusternodes connects fine to one another and the leafnode also connects to the cluster.

However, while a JetStream created on one clusternode is visible on all nodes, it doesn't show up on the leafnode.
I've set the 'jetstream domain' to the same value, and I can see in the logs that the leafnode is connected.

I've also tried to get the system account connected between the cluster and the leafnode -but I'm not sure that part works since it seems impossible to be able to specify a password for a user on the system_account when using TLS.

And these are my issues:

a) If I create a jetstream stream on the cluster, it doesn't show up on the leafnode.
b) If I create a jetstream stream on the leafnode, it doesn't show up on the cluster.
c) If I do a 'server report jetstream' on the leafnode, it only lists the leafnode.
d) If I do a 'server report jetstream' on the cluster, it only lists the cluster nodes.

BUT:

e) If I change the jetstream domain to be e.g. "nlx000013" on the cluster nodes and "notnlx000013" on the leafnode(using --js-domain), and I then create a jetstream stream on either the cluster or the leafnode, it shows up in both places. That is really weird.

Any advice is most welcome, the documentation is unfortunately very limited when it comes to using TLS.
Thanks in advance,

/tony

Configuration:

This is nats.conf on a cluster node:

http: 0.0.0.0:8222      # HTTP monitoring port
#syslog: true

jetstream: enabled
server_name: natsc01

# Client connects here
listen: 0.0.0.0:4222
client_advertise: 192.168.122.131:4222

# Logging
debug: false
trace: false
logtime: true
logfile_size_limit: 1GB
logfile_max_num: 4
log_file: "/var/log/nats/nats-server.log"
jetstream {
    # JetStream storage location, limits and encryption
        store_dir: /opt/nats/jetstream
        max_mem: 1G
        max_file: 2G
        domain: nlx000013
}

tls: { 
    # Configuration map for tls parameters used for client connections, 
    # routes and https monitoring connections.
    
    cert_file: "/usr/local/etc/tls/natsc01-cnode.pem"
    key_file: "/usr/local/etc/tls/natsc01-cnode-key.pem"
    ca_file: "/usr/local/etc/tls/ca.pem"
    min_version: "1.2"
    timeout: 2
    verify_and_map: true
}

cluster: {
    name: nats01
    host: 192.168.122.131
    port: 4248

    routes: [
        tls://nats-cnode@nlx000013.dk@192.168.122.132:4248
        tls://nats-cnode@nlx000013.dk@192.168.122.136:4248
    ]
    authorization: {
        user: nats-cnode@nlx000013.dk
#        password: routing
        timeout: 0.5
    }
    tls: { 
        cert_file: "/usr/local/etc/tls/natsc01-cnode.pem"
        key_file: "/usr/local/etc/tls/natsc01-cnode-key.pem"
        ca_file: "/usr/local/etc/tls/ca.pem"
        verify_and_map: true
        min_version: "1.2"
    }

}


include ./nats.d/nats-auth.conf
include ./nats.d/nats-leafnodes.conf

And this is nats-leafnodes.conf on a cluster node:

leafnodes: {
    # Configuration map for leafnodes. LeafNodes are lightweight clusters.
  port: 7422
  tls: {
    cert_file: "/usr/local/etc/tls/natsc03-cnode.pem"
    key_file: "/usr/local/etc/tls/natsc03-cnode-key.pem"
    ca_file: "/usr/local/etc/tls/ca.pem"
    verify_and_map: true
  }
  authorization: {
    timeout: 0.5
    users = [
      { user: nats-admin@nlx000013.dk, account: SYSTEM }
      { user: nats-lnode@nlx000013.dk, account: SENTINEL }
      { user: tonalb, password: tonalb123, account: SENTINEL }
    ]
  }
}

And this is nats-auth.conf on a cluster node:

system_account: SYSTEM
accounts: {
    # List of accounts and user within accounts
    # User may have an authorization and authentication section
  SYSTEM: {
    users: [
      { user: nats-admin, password: $2a$11$5r1xNXkaastrrb/jTgKF7eix4zf3vZ2gUHGAjCqRPBxHd7P6BgKRu }
      { user: nats-admin@nlx000013.dk }
    ]
  }

  SENTINEL: {
    jetstream: enabled
    jetstream: {
      max_mem: 512M
      max_file: 512K  
      max_streams: 10
      max_consumers: 100
    },
    default_permissions: {
      publish: ">"
      subscribe: ">"
    },
    users: [
      { user: nats-sentinel@nlx000013.dk }
      { user: nats-lnode@nlx000013.dk }
    ],
    mappings: {
    }
  }
}

And nats.conf on the leafnode:

server_name: natsl01.nlx000013.dk
listen: 192.168.122.133:4222    # host/port to listen for client connections
#listen: localhost:4222 # host/port to listen for client connections
http: localhost:8222    # HTTP monitoring port
#syslog: true
# Logging
debug: true
trace: false
logtime: true
logfile_size_limit: 1GB
logfile_max_num: 4
log_file: "/var/log/nats/nats-server.log"

jetstream: enabled

jetstream {
    # JetStream storage location, limits and encryption
        store_dir: /opt/nats/jetstream
        max_mem: 1G                                  
        max_file: 2G
        domain: nlx000013
}

tls: { 
  # Configuration map for tls parameters used for client connections, 
  # routes and https monitoring connections.
  cert_file: "/usr/local/etc/tls/natsl01-lnode.pem"
  key_file: "/usr/local/etc/tls/natsl01-lnode-key.pem"
  ca_file: "/usr/local/etc/tls/ca.pem"
  min_version: "1.2"
  timeout: 2
  verify_and_map: true
}

include ./nats.d/nats-auth.conf
include ./nats.d/nats-leafnodes.conf

nats-leafnodes.conf on the leafnode:

leafnodes: {
  remotes: [
    {
      urls: [
        "tls://natsc01.nlx000013.dk:7422"
        "tls://natsc02.nlx000013.dk:7422"
        "tls://natsc03.nlx000013.dk:7422"
      ]
      tls: {
        cert_file: "/usr/local/etc/tls/natsl01-lnode.pem"
        key_file: "/usr/local/etc/tls/natsl01-lnode-key.pem"
        ca_file: "/usr/local/etc/tls/ca.pem"
        verify_and_map: true
      }
      account: SENTINEL
   }
  ]
}

nats-auth.conf on the leafnode:

system_account: SYSTEM
accounts: {
    # List of accounts and user within accounts
    # User may have an authorization and authentication section
  SYSTEM: {
    users: [
      { user: nats-admin, password: $2a$11$5r1xNXkaastrrb/jTgKF7eix4zf3vZ2gUHGAjCqRPBxHd7P6BgKRu }
      { user: nats-admin@nlx000013.dk }
    ]
  }

  SENTINEL: {
    jetstream: enabled
    jetstream: {
      max_mem: 512M
      max_file: 512K
      max_streams: 10
      max_consumers: 100
    },
    default_permissions: {
      publish: ">"
      subscribe: ">"
    },
    users: [
      { user: nats-sentinel@nlx000013.dk }
      { user: nats-lnode@nlx000013.dk }
    ],
    mappings: {
    }
  }
}

[tbalbers]

oh dear, github sux at handling code blocks it seems.. I'll see if I can get it to work..

done.

[tbalbers]

And of course I just found out why (at least I think so):
Just leave out the jetstream domain name on the leaf node.

When I do that, and create a stream on the leafnode, it shows up in the cluster. But does it then have disk persistency on the leafnode if the connection to the cluster is lost?

[tbalbers]

na.. I'll start over

[tbalbers]

closing