Use thousands of leaf nodes at the edge

[pgentile2]

Hi,

I work for a retail company, and I'm currently studying NATS for edge use cases. We're planning to install a NATS leaf node in each of our 1,700 stores, all connected to a central cluster hosted at our cloud provider. The data exchanges will be between the stores and the central cluster, with no communication occurring between stores directly.
We intend to use JetStream source and mirror to synchronize data between the stores and the central cluster. This will help our in-store applications become more resilient to Internet connectivity issues.

However, managing 1,700 leaf nodes is a significant concern, particularly due to the potential increase in NATS gossip between all nodes.

• Is our approach of using leaf nodes suitable for this use case?
• How can we reduce the amount of NATS gossip between all nodes?
• Is there a way to configure a leaf node deployed in one store to be unaware of other store leaf nodes?

Regards,

[ripienaar]

Leafnodes do not gossip between other leaf nodes unless there is legitimate interest, is that the Gossip you are concerned about?

[pgentile2]

I've started a bunch of NATS leaf nodes in debug mode and I monitor the exchanges between the nodes.

When the Nth leaf node starts, it seems that it receives N -1 leaf subscriptions. Here is an example of a log a leaf node receives during startup:

[63038] 2024/10/08 14:41:41.505207 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $LDS.DNhM0XvErkdhDHR3gYcy2m]

When I configure the domain on each leaf node Jetstream, I also receive subscriptions for all the other leaf node Jetstream. Example here of messages received by a node:

[63038] 2024/10/08 14:41:45.513578 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.INFO]
[63038] 2024/10/08 14:41:45.513654 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.$OBJ.>]
[63038] 2024/10/08 14:41:45.513691 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.$KV.>]
[63038] 2024/10/08 14:41:45.513751 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.META.>]
[63038] 2024/10/08 14:41:45.513764 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.STREAM.>]
[63038] 2024/10/08 14:41:45.513777 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.DIRECT.>]
[63038] 2024/10/08 14:41:45.513791 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.SERVER.>]
[63038] 2024/10/08 14:41:45.513803 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.CONSUMER.>]
[63038] 2024/10/08 14:41:45.513815 [TRC] 100.64.1.4:443 - lid_ws:5 - <<- [LS+ $JS.store-15.API.ACCOUNT.>]

I'm concerned that when a new leaf node is added, it receives subscriptions from the other N-1 leaf nodes. With thousands of leaf nodes, this could significantly impact our network bandwidth between the stores and the central hosting.

[ripienaar]

Right, yeah that's normal and I agree can be an issue - you could maybe put each store in its own account which is probably a good idea in general if you do care for isolation?

[pgentile2]

Unfortunately, even after using an account to isolate each store, I continued to receive LS+ protocol messages from other leaf nodes.

But I tried Derek trick and it suppressed all LS+ messages between leaf nodes.

[derekcollison]

The trick to avoid lots of interest graph propagation between leafnodes when you do not need or desire east/west communications is to have them use the same cluster name. This prevents the hub/cloud side from having to deal with interest graph propagation between each leafnode.

We will formalize this in 2.11, but for now use that trick in setups like these.

[pgentile2]

Using the same cluster name for all leaf nodes made it, resulting in communication exclusively between the store and the leaf nodes.

Thanks for your help 🙏

[akhil-keus]

Hi @pgentile2, can you please share your example leaf node config with same cluster name? I think I am facing exact same issue which is causing spikes in network traffic.

[derekcollison]

Use same cluster name for leafs if you do not need east - west traffic..