Source stream form multiple leaf nodes

[ZarkoRunjevac]

Hi,
I am evaluating using Nats Jetstream with leaf nodes in one Iot project. I would like to implement Nats leaf nodes on my Iot devices. To sync data after disconnects I plan to use source streams to aggregate data from all devices to one stream on hub. I'm interested in synchronizing data from multiple NATS leaf nodes into a single stream on the hub using source streams.

I'm having difficulty understanding how to implement this when I don't know the final number of leaf nodes, as I'm constantly adding new ones. ( in stream add command with source option we always need to specify source streams and stream created in this manner can't be updated).Can I use multiple sourced streams on a single hub for each leaf node? Is there a use case or best practice for implementing this scenario?
Specifically:

1.How can I dynamically add new leaf node sources to my hub stream?
2. Is there a limit to the number of sources a single stream can have?
3. What's the recommended approach for scaling this kind of setup as the number of leaf nodes grows?
4. Are there any performance considerations I should be aware of when using many sourced streams?

[jnmoyne]

1. You can update a stream's configuration to add or remove sources at any time.
   Using the nats CLI tool one easy way to do that is through using JSON config files. For example if you have a stream 'c' that sources a stream 'a' and wanted to add a sourcing for stream 'b' to 'c'.

First get the current config for 'c'
nats stream info c --json | jq .config > c-config.json

{
  "name": "c",
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "old",
  "num_replicas": 1,
  "duplicate_window": 120000000000,
  "sources": [
    {
      "name": "a"
    }
  ],
  "sealed": false,
  "deny_delete": false,
  "deny_purge": false,
  "allow_rollup_hdrs": false,
  "allow_direct": true,
  "mirror_direct": false,
  "metadata": {
    "_nats.created.server.api_level": "1",
    "_nats.created.server.version": "2.11.0-dev",
    "_nats.server.api_level": "1",
    "_nats.server.require.api_level": "0",
    "_nats.server.version": "2.11.0-dev"
  },
  "consumer_limits": {}
}

Just edit the JSON to add {"name":"b"} in the sources array (you can also optionally specify a subject filter and subject transform destination for the source).
Then update the stream configuration nats stream edit --config c-config.json

You can also do this using nats s edit -i (though the configuration in that mode is in YAML rather than JSON), and of course you can also use the client library in your language of choice to update the stream's configuration.

2. No hard coded limit
3. You can use more than one 'intermediary' stream to source from the leaf-node's streams (e.g. 2 sourcing stream, one sourcing from half the LNs and the other one sourcing from the other half) and then you can have a final 'central' stream that sources from those 'intermediary' streams.
4. Nothing specific

[ZarkoRunjevac]

Hi,
is there a protection against concurrent modifications when updating sources array(e.g. if want add two streams from Leafs at a same time)?

[derekcollison]

That is fine, a stream can have multiple sources.

[ZarkoRunjevac]

Also,
in leaf node i have a stream data that is sourced to hub stream data_hub. Also on hub I have stream stream_partition_x that listens on same topics. After the disconnect messages are only sync to data_hub stream and not to stream_partition_x. Is it possible to sync those retroactive messages to stream_partition_x stream without changing topics?
If not what is best approach to handle this situation?

[ZarkoRunjevac]

Hi, just one more question: is there a limit of number of streams that one hub can have(in context of performances)? Is it acceptable that for every leaf node, we create sourced stream on hub in order to sync data after disconnect?
My use case is that we will dynamically add new leaf nodes for edge devices, and official documentation is kind not clear, what is the preferred way when syncing data:
a) source all streams from leaf nodes to one aggregate stream on hub, and manage updating aggregate streams like you described
b) create for every stream on leaf nodes one stream on hub, to avoid updating streams

Thank you vary much for support.

[derekcollison]

We have worked with multiple customers on solutions for synching across edge nodes. Each is a little different but in general the hub/cloud side has muxed streams that hold up to N fleet nodes. Each architecture here is usually referred to as a "cell", where a cell has a capacity and can be deployed anywhere as the fleet increases and for latency concerns.

We do not recommend cloud side has a 1:1 stream for every fleet member.