Seeing a Fault status for Elastic Search though everything working not a normal unassigned shard

[DCraigRich]

Version

2.4.211

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

1 socket 16 cores

RAM

48GB

Storage for /

/dev/mapper/system-root 958G 65G 893G 7% /

Storage for /nsm

/dev/mapper/nsm-nsm 10T 711G 9.4T 7% /nsm

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Normally I find an unassigned shard log to fix but never seen one with a transform like this:

[root@seconion10 secadmin]# curl -k -u @'https://localhost:9200/_cat/shards?v=true&h=index,shard,prirep,state,unassigned.reason&s=state'
Enter host password for user '************':
index shard prirep state unassigned.reason
.transform-internal-007 0 p UNASSIGNED CLUSTER_RECOVERED
logs-ti_eset_latest.dest_botnet-3 0 p STARTED

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Is that the only unassigned shard? What does so-elasticsearch-query _cluster/allocation/explain?pretty show?

I see you are on 2.4.211, what version were you on when you upgraded?

[DCraigRich]

That is the only unassigned shard. I normally keep everything up to date. Run soup often to keep up to date. I think the previous version was 2.4.210

[root@seconion10 secadmin]# so-elasticsearch-query _cluster/allocation/explain?pretty
{
"note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API. See https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-cluster-allocation-explain?version=9.0 for more information.",
"index" : ".transform-internal-007",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2026-03-27T20:20:34.308Z",
"last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "no_valid_shard_copy",
"allocate_explanation" : "Elasticsearch can't allocate this shard because all the copies of its data in the cluster are stale or corrupt. Elasticsearch will allocate this shard when a node containing a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.",
"node_allocation_decisions" : [
{
"node_id" : "7OXBdK6vQFy7b779uXSYgw",
"node_name" : "seconion10",
"transport_address" : "192.168.40.10:9300",
"node_attributes" : {
"xpack.installed" : "true",
"ml.config_version" : "12.0.0",
"transform.config_version" : "10.0.0"
},
"roles" : [
"data",
"data_hot",
"ingest",
"master",
"remote_cluster_client",
"transform"
],
"node_decision" : "no",
"store" : {
"in_sync" : true,
"allocation_id" : "BNo4aTwVR_qVjAiRAMeVjg",
"store_exception" : {
"type" : "corrupt_index_exception",
"reason" : "failed engine (reason: [refresh failed source[refresh_flag_index]]) (resource=preexisting_corruption)",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "failed engine (reason: [refresh failed source[refresh_flag_index]])",
"caused_by" : {
"type" : "corrupt_index_exception",
"reason" : "checksum failed (hardware problem?) : expected=219c2c61 actual=57e0bb18 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/share/elasticsearch/data/indices/-utBQnBLTHC2Gh72Gnx45g/0/index/_1nisw_1.fnm")))"
}
}
}
}
}
]
}
[root@seconion10 secadmin]#

Actually seeing two unassigned now:

[root@seconion10 secadmin]# ^C
[root@seconion10 secadmin]# curl -k -u @ 'https://localhost:9200/_cat/shards?v=true&h=index,shard,prirep,state,unassigned.reason&s=state'
Enter host password for user '********':
index shard prirep state unassigned.reason
.transform-internal-007 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-logs-fortinet_fortigate.log-default-2026.03.20-000017 0 p UNASSIGNED ALLOCATION_FAILED
logs-ti_cybersixgill_latest.dest_threat-3 0 p STARTED
.ds-logs-elastic_agent.fleet_server-default-2025.12.10-000003 0 p STARTED

There was only one before the transform which I had never seen. Normally ones like that firewall log shard occur and I fix.

[DCraigRich]

I cleaned up the fortigate firewall log:
Enter host password for user 'craig@richhabitat.com':
index shard prirep state unassigned.reason
.transform-internal-007 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-metrics-fleet_server.agent_versions-default-2026.03.10-000006 0 p STARTED
.kibana_alerting_cases_8.18.6_001 0 p STARTED
logs-ti_eclecticiq_latest.threat-4 0 p STARTED

[cm-ops]

Was there an issue with the disk at some point? "reason" : "checksum failed (hardware problem?) : expected=219c2c61 actual=57e0bb18 The index is corrupted.

You could try and remove it, but you would need to create a system indices role to do it. It will get recreated when you restart the services.

To do this, create the role needed to delete a system index:

PUT /_security/role/system_index_admin
{
  "indices": [
    {
      "names": [".*"],
      "privileges": ["delete_index", "manage", "all"],
      "allow_restricted_indices": true
    }
  ],
  "cluster": ["all"],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["all"],
      "resources": ["*"]
    }
  ]
}

I usually create a user and assign the role:

POST /_security/user/sysadmin
{
  "password" : "password",
  "roles" : [ "system_index_admin" ],
  "full_name" : "System Index Admin",
  "email" : "admin@example.com"
}

Log out of Kibana and log in with the user sysadmin and password you set above.

In Stack Management > Index Management find the index .transform-internal-007 and delete.

Restart Elasticsearch.

To remove the role and user, log out and log back in as your regular user.

DELETE /_security/user/sysadmin
DELETE /_security/role/system_index_admin

You should see the index recreated.

[DCraigRich]

Thanks I was able to delete it restart the elastic search services and no shards now.