2.4 Remove Search Node

[8inaryMata1eao]

The current deployment I am using is distributed with 1 manager, 2 search nodes and 3 sensors. After a few weeks of usage the search node in a different network is causing the entire grid to be sluggish. I removed the search node in accordance with 'https://docs.securityonion.net/en/2.4/removing-a-node.html' but now my manager shows constant error state and the elasticsearch log is throwing errors. After searching more I found "#11556", which is for 2.3. As of now i rebooted the old search node and added it back to the cluster. What is the proper procedure to remove a search node for 2.4+?

[cm-ops]

You would probably want to exclude the ip of the search node first to move data off the node before shutting it down.

Example:

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}

If you shut the node down withoit moving data, you probably have shards unassigned. Check GET /_cat/shards.

[8inaryMata1eao]

I was able to get the search node back online and into the cluster with no errors so I am starting from the beginning and want to do it right.

To make sure I am following correctly, you are saying run that command in the kibana dev tools console screen, with the appropriate IP of the search node. Then follow the instructions on "https://docs.securityonion.net/en/2.4/removing-a-node.html". Do I have to wait or do anything extra? It isn't clear in the documentation the order of operations.

We made the mistake of running two search nodes in different geo-locations so naturally it KILLS the IO wait and slows everything down. So we had to learn that the hard way and are now trying to kill the non local search node gracefully.

[cm-ops]

Yes, after that command, you want to wait until all the data is off the node. If you have a lot of data and you think Elasticsearch may restart, you could use persistent versus transient in your settings command. Then follow the remove a node from the docs.

[8inaryMata1eao]

Thanks for the support. The shard reallocation failed after multiple attempts and the removal of search node was unsuccessful. We had to nuke and rebuild.