PSA: What does "Pending" mean on the grid screen?

[TOoSmOotH]

In 2.4.40 we introduced two new features that monitor the health of the grid and make the user aware of certain conditions. In previous versions of Security Onion, the auto-update process would update the kernel on the grid members and only notify the user by modifying the motd of devices that needed a reboot. In 2.3 we felt this was sufficient because users connected to the manager via SSH to tune rules etc. Since 2.4 removes the need to ssh to the manager to accomplish normal tasks we needed a way to notify the user of pending reboots. If a node goes into a pending state, the first thing to do is expand it and see what has caused the pending state. If a reboot is required, the grid screen will highlight the reboot button for you. Please keep in mind that although it is important to reboot, this should not impact the operation of the grid while in this state. These reboots can be done during non-peak times to avoid losing critical visibility during peak usage times.

The second condition that causes pending is the health of the elastic cluster. In previous versions, we had no way to notify the user when the cluster was in any other state than green. If the cluster moves from green to yellow or red you will see the manager go into a pending state. When expanding the node in grid you will see the "Elasticsearch Status" turn to pending.

We are aware of an issue where some indices are not getting the default template and are creating replicas when they shouldn't. This is typically not an issue for users with multiple search nodes as there is a node for the replica to land on. This is not the case for standalone nodes. The endpoint.diagnostic.collection index is such an index that isn't using the SO index template. Even if you have fixed it in the past it will continue to send the manager into a pending status when the index rolls over since it will yet again create a replica. To fix the cluster run the following commands:

sudo so-elasticsearch-query _cluster/health
sudo so-elasticsearch-query _cat/shards | grep UN
sudo so-elasticsearch-query .ds-.logs-endpoint.diagnostic.collection-default-2024.01.23-000010/_settings -d '{"number_of_replicas":0}' -XPUT

Make sure the name of the index matches that of the one that has an unassigned replica. You can then run the _cluster/health again and should see the status turn to green. The status in grid should change back to "OK" after a few minutes.

It is important to note that when a node is pending due to elasticsearch, rebooting will NOT fix it.

This issue should be resolved in the upcoming 2.4.50 release. Please start a new discussion if you are having issues getting your cluster to green.