Unassigned indicies after purging /nsm/elasticsearch/indices folder

[netc444t]

Hello, community and SO developers.

Thank you for this product; I've been using it for years, and SO has never disappointed me in any of my security needs.

I encountered a big problem: Elasticsearch on my production Manager node crashed and didn't respond to any localhost:9200 requests. In an attempt to fix it, I decided to purge all data in the /nsm/elasticsearch/indices directory on Manager, as the indices were not critical, and I was prepared to lose them. However, I didn't realize that this directory contained all Kibana and security indices and shards. As a result, the Elasticsearch service on my Manager node got stuck.

If anyone has any advice, I will attach the response of the so-elasticsearch-shards-list command:

.ds-ilm-history-5-2023.07.27-000004 0 p UNASSIGNED
.tasks                              0 p UNASSIGNED
.kibana_8.6.2_001                   0 p UNASSIGNED
.apm-agent-configuration            0 p UNASSIGNED
.security-7                         0 p UNASSIGNED
.kibana_security_session_1          0 p UNASSIGNED
.kibana_task_manager_8.6.2_001      0 p UNASSIGNED
.security-profile-8                 0 p UNASSIGNED
.apm-custom-link                    0 p UNASSIGNED
.async-search                       0 p UNASSIGNED

The so-index-list command gives an empty response, as expected.

The output of curl -K /opt/so/conf/elasticsearch/curl.config -k -X GET "https://localhost:9200/_cluster/allocation/explain" | jq returns the following:

{
  "note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
  "index": ".ds-ilm-history-5-2023.07.27-000004",
  "shard": 0,
  "primary": true,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "CLUSTER_RECOVERED",
    "at": "2023-07-28T10:31:41.563Z",
    "last_allocation_status": "no_valid_shard_copy"
  },
  "can_allocate": "no_valid_shard_copy",
  "allocate_explanation": "Elasticsearch can't allocate this shard because there are no copies of its data in the cluster. Elasticsearch will allocate this shard when a node holding a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.",
  "node_allocation_decisions": [
    {
      "node_id": "4nzbe_VSQHalCjgf8Vzhww",
      "node_name": "mymanagernoder",
      "transport_address": "manageriphere",
      "node_attributes": {
        "xpack.installed": "true",
        "box_type": "hot"
      },
      "node_decision": "no",
      "store": {
        "found": false
      }
    }
  ]
}

I've also tried to delete shards via the API endpoint or reroute them, but neither approach yielded any result. I've spent a lot of time trying to fix this, but now I believe that the only way to resolve my problem is to reinstall Manager.

[netc444t]

Update:
I managed to solve my problem by disabling xpack.security in /opt/so/saltstack/local/salt/elasticsearch/defaults.yaml.
Then, the unsecured API allowed me to purge all indices and data streams needed. I reenabled xpack.security, reapplied the Elasticsearch Salt state, and the system recreated the indices I needed.
Now, my system works as expected. I hope my topic will help someone else who may encounter such a disaster.

My issue was:
Due to purging all indices in /nsm/elasticsearch/indices, my security and other indices were also purged.
Deletion of shards and indices was also not available due to broken xpack.security roles.

Steps to fix:

1.Disable shards allocation on the Manager Node

curl -k -X PUT "https://localhost:9200/_cluster/settings" -H "Content-Type: application/json" -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}'

2. Copy /opt/so/saltstack/default/salt/elasticsearch/defaults.yaml to /opt/so/saltstack/local/salt/elasticsearch/defaults.yaml on Manager Node and disable security module:

      security:
        enabled: false

3. Reapply elasticsearch Salt state on Manager node:

salt-call state.apply elasticsearch

4. Delete broken indicies via HTTP API:

curl -K /opt/so/conf/elasticsearch/curl.config -XDELETE -k -L http://"172.16.10.174:9200/INDEX_NAME"

5. Reenable shards allocation so Elasticsearch can recreate indicies on restart and reassign new shards to them:

curl -K /opt/so/conf/elasticsearch/curl.config -k -X PUT "https://localhost:9200/_cluster/settings" -H "Content-Type: application/json" -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}'

6. Reenable security module in /opt/so/saltstack/local/salt/elasticsearch/defaults.yaml or delete this file if no other customisations for Manager node are required:

      security:
        enabled: true

7. Reapply Elasticsearch Salt state on Manager node:

salt-call state.apply elasticsearch