Search failure in the elasticsearch cluster error help please .. SO version 2.4.40 (logs attached)

[plaotec]

PLEASE STOP AND READ THIS INFORMATION!

If you are creating an issue just to ask a question, you will likely get faster and better responses by posting to our discussions forum instead:
https://securityonion.net/discuss

If you think you have found a possible bug or are observing a behavior that you weren't expecting, use the discussion forum to start a conversation about it instead of creating an issue.

If you are very familiar with the latest version of the product and are confident you have found a bug in Security Onion, you can continue with creating an issue here, but please make sure you have done the following:

• duplicated the issue on a fresh installation of the latest version
• provide information about your system and how you installed Security Onion
• include relevant log files
• include reproduction steps

[plaotec]

Reference : #11500

We updated to SO version 2.4.40 but still getting the error:

V2.4.20, we are experiencing a search failure in the elastic cluster. After further review of the logs, it appears that 4 shards failed to initialize kicking a 502 error again

[plaotec]

salt-relay.log
salt-relay-20240116.log.gz
salt-relay-20240117.log.gz
salt-relay-20240118.log.gz
salt-relay-20240119.log.gz
salt-relay-20240120.log.gz
salt-relay-20240121.log.gz
salt-relay-20240122.log.gz
salt-relay-20240123.log.gz
salt-relay-20240124.log.gz
salt-relay-20240125.log.gz
salt-relay-20240126.log.gz
salt-relay-20240127.log.gz
salt-relay-20240128.log.gz
salt-relay-20240129.log.gz

[plaotec]

Uploading sensoroni-server.zip…

[plaotec]

sensoroni-server-20240116.log.gz
sensoroni-server-20240117.log.gz
sensoroni-server-20240118.log.gz
sensoroni-server-20240119.log.gz
sensoroni-server-20240120.log.gz
sensoroni-server-20240121.log.gz
sensoroni-server-20240122.log.gz
sensoroni-server-20240123.log.gz
sensoroni-server-20240124.log.gz
sensoroni-server-20240125.log.gz
sensoroni-server-20240126.log.gz
sensoroni-server-20240127.log.gz
sensoroni-server-20240128.log.gz
sensoroni-server-20240129.log.gz

[plaotec]

sync.log
sync-20240116.log.gz
sync-20240117.log.gz
sync-20240118.log.gz
sync-20240119.log.gz
sync-20240120.log.gz
sync-20240121.log.gz
sync-20240122.log.gz
sync-20240123.log.gz
sync-20240124.log.gz
sync-20240125.log.gz
sync-20240126.log.gz
sync-20240127.log.gz
sync-20240128.log.gz
sync-20240129.log.gz

[plaotec]

[plaotec]

for now, we deleted the indexes with the red health status, and the elasticsearch error went away .. will monitor to see if error comes back after awhile

[reyesj2]

run sudo so-elasticsearch-query _cat/shards any unassigned shards you can look and try to figure out why with the command below (replace .ds-logs-suricata-so-2024.01.30-000001 with whatever shard is showing unassigned)

sudo so-elasticsearch-query _cluster/allocation/explain?pretty -d' {  "index": ".ds-logs-suricata-so-2024.01.30-000001", "shard": 0, "primary": true }'

What does your grid consist of ? Managersearch + sensor? Manager + sensor(s) + searchnode(s)?

[plaotec]

thanks