Subscribe to LEAFNODE.CONNECT and LEAFNODE.DISCONNECT events

[ZarkoRunjevac]

Hi, I am trying to subscribe to LEAFNODE.CONNECT and LEAFNODE.DISCONNECT events. My hub is configured as

jetstream {
    store_dir: "/data"
    domain=hub
    max_memory_store: 8GB
    max_file_store: 1TB
}
accounts: {
    SYS: {
        users: [
            { 
                user: "admin", 
                password: "changeit"
            }
        ]
    }
}
system_account: SYS
leafnodes {
    port: 7422
}
port: 4222
http_port: 8222

and leaf node is configured as

jetstream {
    store_dir: "/data1"
    domain=leaf1
}
port: 4333
leafnodes {
    remotes [
        {
            urls: ["nats://nats-main:7422"]
        }
    ]
}

When I subscribe to events on hubusing command:

 nats -s localhost:4222 sub '$SYS.>' --user admin --password changeit

I never receive LEAFNODE.CONNECT and LEAFNODE.DISCONNECT events.
Do I need to setup anything else in order to receive these events?

[derekcollison]

From the system account you can subscribe to the following.

$SYS.ACCOUNT.*.CONNECT

This will show all connections of all types for all accounts. The third token is the account.

e.g.

[#1] Received on "$SYS.ACCOUNT.ADEREKYKHPGHKP3UOOWSTAVFTYSFSXZELVLTRXOWQ74RG7CI2BFUKOQ3.CONNECT"
{"type":"io.nats.server.advisory.v1.client_connect","id":"jf9HGwRRs9OWS65PwUxOVc","timestamp":"2024-09-18T21:28:49.735218109Z","server":{"name":"aws-uswest2-natscj1-3","host":"52.26.44.194","id":"NCYS3GAN7R5FEPE5AZSRMGLLMXV7MTIFI4UDCEACTQHHP4NBRHYI5A3I","cluster":"ngsprod-aws-uswest2","domain":"ngs","ver":"2.10.21-RC.1","tags":["cloud:aws","cluster:ngsprod-aws-uswest2","country:us","geo:us","geo:us-west","index:3","instance:aws-uswest2-natscj1-3","pool:natscj1","region:uswest2","stack:ngsprod","zone:uswest2c"],"jetstream":true,"flags":3,"seq":546495,"time":"2024-09-18T21:28:49.735390986Z"},"client":{"start":"2024-09-18T21:28:49.566571361Z","host":"76.91.2.75","id":82322,"acc":"ADEREKYKHPGHKP3UOOWSTAVFTYSFSXZELVLTRXOWQ74RG7CI2BFUKOQ3","user":"UDFQRH4SMFSFFR5CQUYSQQXZQZ3RUPUGCRQZ5KHACCPTSTVQTXQKFEDW","name":"ngs-leaf-derek","ver":"2.11.0-dev","jwt":"ey...","issuer_key":"ADEREKYKHPGHKP3UOOWSTAVFTYSFSXZELVLTRXOWQ74RG7CI2BFUKOQ3","name_tag":"derek","kind":"Leafnode"}}

If you want only the leafnodes, you can subscribe to the following, again from the system account.

$SYS.ACCOUNT.*.LEAFNODE.>

e.g.

[#1] Received on "$SYS.ACCOUNT.ADEREKYKHPGHKP3UOOWSTAVFTYSFSXZELVLTRXOWQ74RG7CI2BFUKOQ3.LEAFNODE.CONNECT"
{"server":{"name":"aws-uswest2-natscj1-4","host":"44.235.52.92","id":"NBDVJN5SI4SN62JKEHFG3PLDOJ6YD4NJ5QI7477ESGXEGNA6NIYF5OEV","cluster":"ngsprod-aws-uswest2","domain":"ngs","ver":"2.10.21-RC.1","tags":["cloud:aws","cluster:ngsprod-aws-uswest2","country:us","geo:us","geo:us-west","index:4","instance:aws-uswest2-natscj1-4","pool:natscj1","region:uswest2","stack:ngsprod","zone:uswest2a"],"jetstream":true,"flags":3,"seq":534314,"time":"2024-09-18T21:26:26.622509189Z"},"acc":"ADEREKYKHPGHKP3UOOWSTAVFTYSFSXZELVLTRXOWQ74RG7CI2BFUKOQ3"}

Verified with Synadia Cloud.

[ZarkoRunjevac]

Hi, I managed to receive connections when I subscribe to $SYS.ACCOUNT..CONNECT, but still I can't receive anything when I subscribe to $SYS.ACCOUNT..LEAFNODE.>.

My docker-compose is

version: '3'

services:
  nats-main:
    image: nats:2.10.20
    container_name: nats-main
    ports:
      - "4222:4222"
      - "8222:8222"
    command: ["--config", "/etc/nats/nats-main.conf"]
    volumes:
      - ./nats-main.conf:/etc/nats/nats-main.conf:ro
    networks:
      - nats-network

  nats-leaf:
    image: nats:2.10.20
    container_name: nats-leaf
    depends_on:
      - nats-main
    ports:
      - "4111:4111"  
      - "8223:8222"  
    command: ["--config", "/etc/nats/nats-leaf.conf"]
    volumes:
      - ./nats-leaf.conf:/etc/nats/nats-leaf.conf:ro
    networks:
      - nats-network

networks:
  nats-network:
    driver: bridge

nats-main.conf

port: 4222
http_port: 8222

server_name: hub

server_name: hub-server
jetstream {
    store_dir="./store_server"
    domain=hub
}
leafnodes {
    port: 7422
}
accounts {
    SYS: {
        users: [{user: admin, password: admin}]
    },
    ACC22: {
        users: [{user: acc, password: acc}],
        jetstream: enabled
    }
}
system_account: SYS

and nats-leaf.conf is:

port: 4111
server_name: leaf-server
jetstream {
    store_dir="./store_leaf"
    domain=leaf
}
leafnodes {
    remotes = [
        {
            urls: ["nats-leaf://admin:admin@0.0.0.0:7422"]
            account: "SYS"
        },
        {
            urls: ["nats-leaf://acc:acc@nats-main:7422"]
            account: "ACC1"
        }
    ]
}
accounts {
    SYS: {
        users: [{user: admin, password: admin}]
    },
    ACC1: {
        users: [{user: acc, password: acc}],
        jetstream: enabled
    }
}
system_account: SYS

[derekcollison]

You should receive those events under system account on the hub side like I did with Synadia Cloud.

Unclear why you are not.

What is topology of the hub system?
What server version?

[ZarkoRunjevac]

My server info:
nats --user admin --password admin -s "nats://127.0.0.1:4222" server info

Server information for hub-server (NCPPJYAPI2JDRLFHSU7UUMOXG2SJ5MF2MRGDEPR7SOC2DKPJRJMTCY76)

Process Details:

                     Version: 2.10.20
                  Git Commit: 7140387
                  Go Version: go1.22.6
                  Start Time: 2024-09-24 16:22:03
            Config Load Time: 2024-09-24 16:22:03
                      Uptime: 15h46m25s

Since I am just evaluating, I keep things simple, so my topology is one hub and one leaf node(I use docker compose file).

One interesting thing is when I run command
nats --user admin --password admin -s "nats://127.0.0.1:4222" events to listen to all events on SYS account in hub, I can only listen to events listed:

Listening for Client Connection events on $SYS.ACCOUNT.*.CONNECT
Listening for Client Disconnection events on $SYS.ACCOUNT.*.DISCONNECT
Listening for Authentication Errors events on $SYS.SERVER.*.CLIENT.AUTH.ERR

and other system events are missing.

And when I subscribe to
nats --user admin --password admin -s "nats://127.0.0.1:4222" sub ">"

I only receive events
$SYS.SERVER.ACCOUNT..CONNS
$SYS.SERVER..STATSZ

[derekcollison]

Would need to have someone jump on a call to triage your setup and try to understand why you are not seeing events like we do in Synadia Cloud or any of our managed customer systems.

[ZarkoRunjevac]

That would be really nice. I found this question with same issue nats-io/nats.net#604

[derekcollison]

I am wondering if this is only presenting as an issue when the hub is a single server vs a cluster or more advanced system?