Can't delete corrupted kibana_task_manager shard

[6161726f6e]

Version

2.4.120

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16

Storage for /

65GB

Storage for /nsm

125G

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

I have this corrupted shard that I cannot delete:

[root@securityonion elasticsearch]# so-elasticsearch-query _cluster/allocation/explain?pretty                                                                                                                 
{                                                                                                                                                                                                             
  "note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for
 different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must speci
fy the target shard in the request to this API.",                                                                                                                                                             
  "index" : ".kibana_task_manager_8.14.3_001",                                                                                                                                                                
  "shard" : 0,                                                                                                                                                                                                
  "primary" : true,                                                                                                                                                                                           
  "current_state" : "unassigned",                                                                                                                                                                             
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2025-03-04T21:32:09.182Z",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "Elasticsearch can't allocate this shard because all the copies of its data in the cluster are stale or corrupt. Elasticsearch will allocate this shard when a node containing a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.",
  "node_allocation_decisions" : [
    {
      "node_id" : "FJSVcscFTmW_frz3igQoLg",
      "node_name" : "securityonion",
      "transport_address" : "192.168.29.252:9300",
      "node_attributes" : {
        "transform.config_version" : "10.0.0",
        "xpack.installed" : "true",
        "ml.config_version" : "12.0.0"
      },
      "roles" : [
        "data",
        "data_hot",
        "ingest",
        "master",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "eTFicQBaTOS8afFGPwh8vA", 
        "store_exception" : {
          "type" : "corrupt_index_exception",
          "reason" : "failed engine (reason: [refresh failed source[api]]) (resource=preexisting_corruption)",
          "caused_by" : {
            "type" : "i_o_exception",
            "reason" : "failed engine (reason: [refresh failed source[api]])",
            "caused_by" : {
              "type" : "corrupt_index_exception",
              "reason" : "checksum failed (hardware problem?) : expected=fe4e347c actual=a888453d (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path=\"/usr/share/elasticsearch/data/indices/vAbkZy6KSi
qD_X_m5b2_3g/0/index/_3da_1.fnm\")))"
            }
          }
        }
      }
    }
  ]
}

When I try to remove it, I am getting this error:

[root@securityonion elasticsearch]# so-elasticsearch-query .kibana_task_manager_8.14.3_001 -XDELETE
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana_task_manager_8.14.3_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana_task_manager_8.14.3_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

#13630 includes a command that might help you delete that index. They are protected indices for a reason, please note that the OP from the linked discussion later came back to report they ended up doing a rebuild anyways.

Do you have any clue what caused the initial corruption? Do you have any backups available like a snapshot? https://docs.securityonion.net/en/2.4/backup.html#elasticsearch

[6161726f6e]

Hi Jorge, the server got into this state after an ungraceful shutdown.

This command did resolve the issue:

kpass=$(salt-call pillar.get elasticsearch:auth:users:so_kibana_user:pass --out=newline_values_only)
curl --user "so_kibana:$kpass" -s -k -L -H "Content-Type:application/json" "https://localhost:9200/.kibana_task_manager_8.14.3_001" -XDELETE

I followed it with so-kibana-restart

And I am able to access kibana now.

Thank You!!