What are strategies for moving a stream to a different set of nodes in the cluster?

[bruth]

The driving motivation is to deal with situations when and existing NATS cluster need to be migrated to new VMs/servers/hardware. The goal would be to not require clients to change the address they are connecting to. Obviously if client connections are temporarily interrupted and auto-reconnect, then that is fine (no different then a network hiccup).

To setup a scenario, let's assume I have a 3-node cluster with Jetstream enabled on all nodes. I have one or more streams some of which are configured to have 3 replicas. The problem is that I am running out of storage and need to add more storage for existing streams but also new streams.

I am aware that additional nodes can be added to a cluster with JS enabled and NATS will distribute stream replicas across the set of available nodes. My understanding is that replication/quorum happens on a per stream basis.

Would it be possible to increase the size of the cluster, with the new nodes having the new storage requirements, and the stream be moved transparently to these new nodes? I know a mirror could be setup, but this would require clients to know to switch over to publishing and consuming to this new stream.

Is there some other transparent way of handling this?

[ripienaar]

Generally you'd want to keep your server names static, so in your 3 node cluster case you'd bring up replacement nodes on the same name - possibly CNAME of course - and with appropriate TLS certs etc.

When you cleanly shut down a old one and bring a new one it would start syncing data soon as it comes online of any streams assigned to it. You want to wait for this process to fully complete, all streams in all accounts to be up to date - then move to the next machine. Always use Lame Duck Mode to prepare a server for replacement.

Otoh if you increase size and you want to redistribute streams, you'll have to use the cli to remove one peer from a stream and it will select a new one at random

Both processes quite involved and manual and requires careful testing and planning.

[bruth]

Thanks. To confirm the strategy:

1. Add the new desired nodes to the cluster
2. Take one of the existing nodes offline
3. Explicitly remove the node as a peer nats cluster peer-remove <stream> <peer>
   • This will cause JS to pick one of the new nodes at random and replicate the data to it
4. Repeat 2-3 until all existing nodes are gone

I understand this is tricky since there could be multiple streams and needing to wait/ensure replication is complete.

[ripienaar]

Yes, if adding new nodes isn’t a problem (names changing etc)

replacing them one by one is the other option

[derekcollison]

Depending on your needs you can also backup and restore into a new cluster with different storage properties.

[bruth]

Thanks. This is indeed the simplest approach if a bit of downtime is allowed. I am not ruling this out for my purpose, but I was curious about the no downtime approach.

[derekcollison]

For no downtime, I would expand the cluster (JetStream assets live within a cluster scope, so any option of no downtime needs to work inside that structure) with additional nodes running JS. I would then move the non-leader peers for each asset that needs to be moved onto the new servers. This may be tricky since right now if you remove a peer it will pick a new one but could pick one of the old ones. Will think more on that.

Then move the leader by asking the asset group to elect a new leader and then remove that peer after that operation has been successful.

[bruth]

New ~6m with the incorporated changes: https://www.youtube.com/watch?v=3HhvgV9WEYI

[ripienaar]

Nice, this seemed a bit rougher going than the first attempt (saw some n2! near the end there), when cycling machines you have to be careful that all the streams thats being removed from a machine you're retiring has been fully synced onto new machines.

Imagine you have a R3 on nodes 1,2,3. You take out 1 and lets say all goes to 4 instead. If 2 and 3 have not finished syncing up 4 completely 100% over all streams in all accounts and you move to take out 2, you can end up with data loss. So you want to be super careful around how you decide to move to the next server in the loop.

[bruth]

Yea, the rougher going was due to cycling the nodes rather than the streams. Intuitively (and as you said in the previous comment), moving a single stream at a time should be simpler than a full node where it can disrupt multiple streams.

For the purpose of the demo, I shortened the cycle time, but in practice it could use some signal of replication status before doing the next node cycle. Ideally not dependent on time, but health of all stream replicas on the node being cycled.

[bruth]

My next effort is going to be extracting some stats (replica lag, active replicas, etc) across all streams and consumers as it applies to a node being removed before moving onto the next node to migrate.

[bruth]

I wrote a little health summary to visualize the replication lag of the node that is in LDM. I am not sure what other metrics I would add in to model the health of a stream or consumer.

One thing I did observe is that even if a new node is available to be replicated onto, the server doesn't proactively start replicating to a new node when another node goes into LDM. It seems like it's only when the LDM node is removed. Is this by design?

I suppose there is the use case for putting a server in LDM to restart it and then its available again as a peer in the cluster? If so, I wonder if there could be a LDM option indicating the eventual removal of the node, so replicas should start moving right away to other nodes if available.

[ripienaar]

LDM gracefully shuts a node down moving its leadships elsewhere etc. you should always shut down a JS node that way if possible.

but it might still come, maybe just maintenance or something. It’s not till you remove it from the RAFT group that a new member is picked in it’s place

[bruth]

Thanks, that I understand. I guess I was just suggesting some kind of joint "LDM + intention of RAFT group removal" signal. So the server knows that the node will be removed or it can even remove it automatically once all clients disconnect and replicas have been moved to other nodes.

[bruth]

Created a repo with the progress thus far: https://github.com/bruth/nats-zero-down/tree/main/replace-cluster-nodes. Further work will be updated there.

[jnmoyne]

This question is pretty old but in the interest of cleanup: you can now control the placement of streams within a cluster or even between clusters using the placement tags.

You can even move existing streams using those placements tags, without any interruption of service, and in any case the client applications do not need to know where a particular stream is located it's all completely transparent to the client applications.