Elasticsearch Status: Pending even when all index green after upgrade

[geistchevalier]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

64GB

Storage for /

500GB

Storage for /nsm

2TB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I got the Elasticsearch Status: Pending warning after upgrading my SO from 2.4.30 to 2.4.40 and proceeded to apply the fix prescribed here
https://docs.securityonion.net/en/2.4/release-notes.html#known-issues

but after I cleared all the affected index it is still showing Elasticsearch Status: Pending

Running sudo so-elasticsearch-indices-list | grep -vE "green|health" shows no results, meaning I've cleared all the affected indices as far as I know

Are there any other index that I need to fix? or any specific thing to look for in the logs?

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[KimkimCyber]

Maybe a stupid question but did you try rebooting the server completely?
We had a similar issue with Zeek and were trying to find the cause of the problem during a whole day to no avail.
Until we decided to reboot our server and then the problem was fixed

[geistchevalier]

I did do a sudo reboot after an hour of it not changing the status, I'll try a sudo poweroff and power it on and see if that fixes it when I have the chance

[Chrischevy80]

Same issue here. The command sudo so-elasticsearch-indices-list | grep -vE "green|health" shows no result and the grid status still shows Pending.

[TOoSmOotH]

Try so-elasticsearch-query _cat/shards | grep UNASSIGNED and see if you have unassigned shards.

[geistchevalier]

I found 1 index from that query .ds-.logs-endpoint.diagnostic.collection-default-2024.01.20-000008 0 r UNASSIGNED

tried to delete it with sudo so-elasticsearch-query $index -XDELETE but got an error instead

{"error":
{"root_cause":[
{"type":"illegal_argument_exception","reason":"index [.ds-.logs-endpoint.diagnostic.collection-default-2024.01.20-000008] is the write index for data stream [.logs-endpoint.diagnostic.collection-default] and cannot be deleted"}],
"type":"illegal_argument_exception","reason":"index [.ds-.logs-endpoint.diagnostic.collection-default-2024.01.20-000008] is the write index for data stream

I'm unsure on how to proceed

edit: grep-ing the result for the index name nets me this result

[user@host ~]$ sudo so-elasticsearch-query _cat/shards | grep .ds-.logs-endpoint.diagnostic.collection-default-2024.01.20
.ds-.logs-endpoint.diagnostic.collection-default-2024.01.20-000008 0 p STARTED           16  73.1kb [IP Addr] security-onion-sa
.ds-.logs-endpoint.diagnostic.collection-default-2024.01.20-000008 0 r UNASSIGNED

[bushcraftbuddy]

I'm in the same situation. I had over 50 indices that were yellow and now that I cleared those I have the same unassigned index as you so still digging into it as well

Edit: I got it working for me by running:
sudo so-elasticsearch-query .ds-.logs-endpoint.diagnostic.collection-default-2024.01.22-000006/_settings -d '{"number_of_replicas":0}' -XPUT

it went ok after about a minute and then I rebooted as a sanity check and right now it's up and showing ok so you might try it

[geistchevalier]

Ok this worked for me. But after running it on the UNASSIGNED index, a few yellow indices popped up for me so I had to set the replicas to 0 for those too. I got OK instead of Pending a few seconds after the last index was fixed.

Thanks to all of the participants in this discussion 😄

[Chrischevy80]

This also worked for me.
I had to run sudo so-elasticsearch-query .ds-.logs-endpoint.diagnostic.collection-default-2024.01.23-000010/_settings -d '{"number_of_replicas":0}' -XPUT
All green now !

[rlberry]

Can confirm the solution by running sudo so-elasticsearch-indices-list | grep -vE "green|health" returned nothing, sudo so-elasticsearch-query _cat/shards | grep UNASSIGNED showed one item, and sudo so-elasticsearch-query .ds-.logs-endpoint.diagnostic.collection-default-202X.0X.XX-0000XX/_settings -d '{"number_of_replicas":0}' -XPUT cleared the issue.

[gsprojectacc]

For my 2.4.50 instance so-elasticsearch-query .apm-source-map/_settings -d '{"number_of_replicas":0}' -XPUT didn't work, but so-elasticsearch-query .apm-source-map/_settings -d '{"auto_expand_replicas":"0-1"}' -XPUT was a solution.