❓ Unable to remove an already shut down cluster member

[pwhsoft]

Hello,

I’m running into an issue when trying to remove a node from a cluster after it has already been shut down. I’m not sure if I’m doing something wrong or if this is a bug or missing feature.

🧩 Scenario

I have a cluster consisting of 3 nodes.

When a node shuts down (ASP.NET Core exit), it attempts to:
1. Send a signal to the cluster leader to gracefully unregister itself.
2. If the shutting-down node is the leader, it unregisters itself as leader und then sends a leave signal.

If this does not work, the leader runs a cleanup routine that should automatically remove the unreachable node after 15 seconds.

However — no matter what I try, the node simply does not get removed.

⚙️ Technical Setup
• 3-node cluster
• Graceful shutdown implemented
• Leader fallback cleanup routine for unreachable nodes
• Persistence engine enabled
• Everything else in the cluster works fine — only removing a member fails

❗ The Problem

The node cannot be removed during graceful shutdown nor by the leader’s cleanup logic.
It stays in the cluster state indefinitely, even though it is completely offline and unreachable.

❓ Question

What is the correct way to remove a node that is already unreachable?
And how should a node perform a proper graceful shutdown and ensure it is fully removed from the cluster?

🧭 Why this is needed
• High-availability scaling
• Handling random node failures safely

Thanks in advance for any guidance!

[pwhsoft]

Hello,
Do you have any suggestions?

[sakno]

Sorry, I was focused on another things in .NEXT. I'll try to take a look soon.

Could you specify which implementation are you using? DotNext.Cluster.AspNetCore via HTTPS or default implementation from DotNext.Net.Cluster with TCP transport?