Elastalert index won't come on and 0b storage size

[byronsims]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

16

Storage for /

163G

Storage for /nsm

327G

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I ran into a storage issue. Now the elastalert index will not come online. Is there a way to rebuild that? The command "sudo so-elasticsearch-query _cluster/allocation/explain pretty" comes back with this:

{"note":"No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.","index":"elastalert","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2025-01-06T14:07:48.125Z","last_allocation_status":"no_valid_shard_copy"},"can_allocate":"no_valid_shard_copy","allocate_explanation":"Elasticsearch can't allocate this shard because all the copies of its data in the cluster are stale or corrupt. Elasticsearch will allocate this shard when a node containing a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.","node_allocation_decisions":[{"node_id":"D-IPZt8BSPOPuWCrkiG-KQ","node_name":"store1","transport_address":"192.168.2.148:9300","node_attributes":{"xpack.installed":"true","transform.config_version":"10.0.0","ml.config_version":"12.0.0"},"roles":["data","data_hot","ingest","transform"],"node_decision":"no","store":{"found":false}},{"node_id":"lWml0Gk1QUuMBy4wYRDoxw","node_name":"store2","transport_address":"192.168.2.149:9300","node_attributes":{"xpack.installed":"true","transform.config_version":"10.0.0","ml.config_version":"12.0.0"},"roles":["data","data_hot","ingest","transform"],"node_decision":"no","store":{"found":false}},{"node_id":"rrPVzpcXR8S1eFxvL6e0GQ","node_name":"store3","transport_address":"192.168.2.179:9300","node_attributes":{"transform.config_version":"10.0.0","xpack.installed":"true","ml.config_version":"12.0.0"},"roles":["data","data_hot","ingest","transform"],"node_decision":"no","store":{"in_sync":true,"allocation_id":"wV7Nfa08SvqKXEyUX7rJAA","store_exception":{"type":"corrupt_index_exception","reason":"failed engine (reason: [recovery]) (resource=preexisting_corruption)","caused_by":{"type":"i_o_exception","reason":"failed engine (reason: [recovery])","caused_by":{"type":"corrupt_index_exception","reason":"verification failed (hardware problem?) : expected=d3q6jj actual=14nmo3i footer=<not checked> writtenLength=2241948 expectedLength=2241948 (resource=name [_bk.cfs], length [2241948], checksum [d3q6jj], writtenBy [9.7.0]) (resource=VerifyingIndexOutput(_bk.cfs))"}}}}},{"node_id":"sPUPVcClTRq_WQK3nYHvsw","node_name":"master","transport_address":"192.168.2.147:9300","node_attributes":{"transform.config_version":"10.0.0","xpack.installed":"true","ml.config_version":"12.0.0"},"roles":["data","master","remote_cluster_client","transform"],"node_decision":"no","store":{"in_sync":false,"allocation_id":"d68HvYxbS4Wt1rTlR0PItA","store_exception":{"type":"index_not_found_exception","reason":"no segments* file found in NIOFSDirectory@/usr/share/elasticsearch/data/indices/rq0a1XRAQVSxANVoKO2bwQ/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@c8080e6: files: [write.lock]"}}}]}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

You should be able to remove the Elastalert indices and restart the service to recreate them,

sudo so-elasticsearch-query _cat/shards | grep elastalert 
sudo so-elasticsearch-query $INDEX_NAME -XDELETE
sudo so-elastalert-restart

[byronsims]

That worked perfectly. Thanks!