search query failure within elasticsearch cluster

[witekenea]

Hello, I have a security onion distributed version with one sensor node, one search node and a manager node. when i try the test data reply from sensor node i got the error search query failure within the elasticsearch cluster, even though elastic search is running and healthy. i try restarting elasticsearch service and runnig so-checkin but it doesn't fix this issue. Any idea how i can fix this issue? below is the output of so-elasticsearch-query_cat/shards on the search node

[root@search elasticsearch]# so-elasticsearch-query _cat/shards
metrics-endpoint.metadata_current_default 0 p STARTED 0 249b 10.123.15.112 manager
metrics-endpoint.metadata_current_default 0 r STARTED 0 249b 10.123.15.116 search
.ds-logs-elasticsearch.server-default-2024.03.29-000001 0 p UNASSIGNED
.internal.alerts-observability.logs.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-observability.logs.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
.ds-.kibana-event-log-8.10.4-2024.03.28-000001 0 p STARTED 45 95.2kb 10.123.15.112 manager
.ds-.kibana-event-log-8.10.4-2024.03.28-000001 0 r STARTED 45 95.2kb 10.123.15.116 search
.apm-source-map 0 p STARTED 0 249b 10.123.15.112 manager
.apm-source-map 0 r STARTED 0 249b 10.123.15.116 search
.ds-logs-syslog-so-2024.03.30-000001 0 p UNASSIGNED
.fleet-policies-leader-7 0 p STARTED 4 18.9kb 10.123.15.112 manager
.fleet-policies-leader-7 0 r STARTED 4 24.2kb 10.123.15.116 search
.internal.alerts-observability.uptime.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-observability.uptime.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
elastalert_error 0 p UNASSIGNED
.apm-agent-configuration 0 p STARTED 0 249b 10.123.15.112 manager
.apm-agent-configuration 0 r STARTED 0 249b 10.123.15.116 search
elastalert_past 0 p STARTED 0 249b 10.123.15.112 manager
.logs-osquery_manager.action.responses-default 0 p UNASSIGNED
.kibana_task_manager_8.10.4_001 0 p STARTED 25 138.4kb 10.123.15.112 manager
.kibana_task_manager_8.10.4_001 0 r STARTED 25 111.4kb 10.123.15.116 search
.ds-logs-elastic_agent.osquerybeat-default-2024.03.29-000001 0 p STARTED 10492 3mb 10.123.15.112 manager
.transform-internal-007 0 p STARTED 120 253.1kb 10.123.15.112 manager
.transform-internal-007 0 r STARTED 120 275.3kb 10.123.15.116 search
.internal.alerts-observability.slo.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-observability.slo.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
elastalert 0 p UNASSIGNED
.fleet-servers-7 0 p STARTED 3 44.2kb 10.123.15.112 manager
.fleet-servers-7 0 r STARTED 3 62kb 10.123.15.116 search
.ds-logs-kratos-so-2024.03.29-000001 0 p UNASSIGNED
.ds-logs-system.auth-default-2024.03.29-000001 0 p UNASSIGNED
.internal.alerts-stack.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-stack.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
.ds-logs-elastic_agent.fleet_server-default-2024.03.29-000001 0 p UNASSIGNED
.security-profile-8 0 p STARTED 1 9.2kb 10.123.15.112 manager
.security-profile-8 0 r STARTED 1 9.2kb 10.123.15.116 search
.ds-ilm-history-5-2024.03.28-000001 0 p STARTED 82 106.8kb 10.123.15.112 manager
.ds-ilm-history-5-2024.03.28-000001 0 r STARTED 82 105.7kb 10.123.15.116 search
.ds-logs-elastic_agent.metricbeat-default-2024.03.29-000001 0 p STARTED 28 35.4kb 10.123.15.112 manager
.ds-logs-soc-so-2024.03.29-000001 0 p UNASSIGNED
.fleet-artifacts-7 0 p STARTED 15 14.8kb 10.123.15.112 manager
.fleet-artifacts-7 0 r STARTED 15 14.8kb 10.123.15.116 search
.internal.alerts-security.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-security.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
.kibana_security_session_1 0 p STARTED 1 6.7kb 10.123.15.112 manager
.kibana_security_session_1 0 r STARTED 1 6.7kb 10.123.15.116 search
elastalert_silence 0 p STARTED 0 249b 10.123.15.112 manager
.metrics-endpoint.metadata_united_default 0 p STARTED 6 265kb 10.123.15.112 manager
.metrics-endpoint.metadata_united_default 0 r STARTED 6 188.2kb 10.123.15.116 search
.fleet-policies-7 0 p STARTED 38 443.5kb 10.123.15.112 manager
.fleet-policies-7 0 r STARTED 38 443.5kb 10.123.15.116 search
.apm-custom-link 0 p STARTED 0 249b 10.123.15.112 manager
.apm-custom-link 0 r STARTED 0 249b 10.123.15.116 search
.slo-observability.summary-v2 0 p STARTED 0 249b 10.123.15.112 manager
.slo-observability.summary-v2 0 r STARTED 0 249b 10.123.15.116 search
.transform-notifications-000002 0 p STARTED 1256 357.8kb 10.123.15.112 manager
.transform-notifications-000002 0 r STARTED 1256 357.8kb 10.123.15.116 search
.internal.alerts-observability.apm.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-observability.apm.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
.ds-logs-system.syslog-default-2024.03.29-000001 0 p STARTED 603274 85.6mb 10.123.15.112 manager
.ds-logs-suricata-so-2024.03.30-000001 0 p STARTED 1255 6.4mb 10.123.15.112 manager
.security-7 0 p STARTED 159 476.9kb 10.123.15.112 manager
.security-7 0 r STARTED 159 462.2kb 10.123.15.116 search
.fleet-enrollment-api-keys-7 0 p STARTED 4 26.2kb 10.123.15.112 manager
.fleet-enrollment-api-keys-7 0 r STARTED 4 26.2kb 10.123.15.116 search
.kibana-observability-ai-assistant-conversations-000001 0 p STARTED 0 249b 10.123.15.112 manager
.kibana-observability-ai-assistant-conversations-000001 0 r STARTED 0 249b 10.123.15.116 search
.ds-logs-zeek-so-2024.03.29-000001 0 p STARTED 11545 13.5mb 10.123.15.112 manager
.ds-logs-zeek-so-2024.03.29-000001 1 p UNASSIGNED
.fleet-agents-7 0 p STARTED 6 219.9kb 10.123.15.112 manager
.fleet-agents-7 0 r STARTED 6 217.8kb 10.123.15.116 search
.async-search 0 p STARTED 1 9.5kb 10.123.15.112 manager
.async-search 0 r STARTED 1 9.3kb 10.123.15.116 search
.slo-observability.sli-v2 0 p STARTED 0 249b 10.123.15.112 manager
.slo-observability.sli-v2 0 r STARTED 0 249b 10.123.15.116 search
.geoip_databases 0 p STARTED 53 56.1mb 10.123.15.112 manager
.geoip_databases 0 r STARTED 53 56.1mb 10.123.15.116 search
.internal.alerts-observability.metrics.alerts-default-000001 0 p STARTED 0 249b 10.123.15.112 manager
.internal.alerts-observability.metrics.alerts-default-000001 0 r STARTED 0 249b 10.123.15.116 search
.kibana_security_solution_8.10.4_001 0 p STARTED 2709 3.7mb 10.123.15.112 manager
.kibana_security_solution_8.10.4_001 0 r STARTED 2709 3.7mb 10.123.15.116 search
.kibana_ingest_8.10.4_001 0 p STARTED 7568 76.3mb 10.123.15.112 manager
.kibana_ingest_8.10.4_001 0 r STARTED 7568 76.3mb 10.123.15.116 search
elastalert_status 0 p STARTED 0 249b 10.123.15.112 manager
.kibana_8.10.4_001 0 p STARTED 124 206.5kb 10.123.15.112 manager
.kibana_8.10.4_001 0 r STARTED 124 199.5kb 10.123.15.116 search
.ds-logs-elastic_agent-default-2024.03.29-000001 0 p STARTED 6600 2.2mb 10.123.15.112 manager
.slo-observability.summary-v2.temp 0 p STARTED 0 249b 10.123.15.112 manager
.slo-observability.summary-v2.temp 0 r STARTED 0 249b 10.123.15.116 search
.ds-logs-elastic_agent.filebeat-default-2024.03.29-000001 0 p UNASSIGNED
logs-ti_recordedfuture_latest.threat-1 0 p STARTED 0 249b 10.123.15.112 manager
logs-ti_recordedfuture_latest.threat-1 0 r STARTED 0 249b 10.123.15.116 search
.kibana_analytics_8.10.4_001 0 p STARTED 7999 5.1mb 10.123.15.112 manager
.kibana_analytics_8.10.4_001 0 r STARTED 7999 5.1mb 10.123.15.116 search
.logs-osquery_manager.actions-default 0 p STARTED 0 249b 10.123.15.112 manager
.kibana_alerting_cases_8.10.4_001 0 p STARTED 1 6.8kb 10.123.15.112 manager
.kibana_alerting_cases_8.10.4_001 0 r STARTED 1 6.8kb 10.123.15.116 search
.ds-logs-strelka-so-2024.03.30-000001 0 p STARTED 28 796.7kb 10.123.15.112 manager
logs-ti_otx_latest.dest_pulses_subscribed-1 0 p STARTED 0 249b 10.123.15.112 manager
logs-ti_otx_latest.dest_pulses_subscribed-1 0 r STARTED 0 249b 10.123.15.116 search

[witekenea]

update:
GET /_cat/health?v (on devtools in kibana) returns
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1712579886 12:38:06 securityonion red 2 2 89 50 0 0 11 0 - 89.0%

[cm-ops]

GET /_cluster/health you have quite a few shards UNASSIGNED, check them with https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html and see why they are unassigned.

[witekenea]

thank you for your response, I checked on how to assign the unassigned shards and use this command
POST /_cluster/reroute
{
"commands": [
{
"allocate_stale_primary": {
"index": ".ds-logs-syslog-so-2024.03.30-000001",
"shard": 0,
"node": "searchnode2",
"accept_data_loss": true
}
}
]
}

by exchanging the node to manager and searchnode2 but get this error:

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "No data for shard [0] of index [.ds-logs-syslog-so-2024.03.30-000001] found on any node"
}
],
"type": "illegal_argument_exception",
"reason": "No data for shard [0] of index [.ds-logs-syslog-so-2024.03.30-000001] found on any node"
},
"status": 400
}
for both manager and searchnode2 and i also tried the method to restore from backup repo but couldn't find any

this is what i currently have

.fleet-enrollment-api-keys-7 0 r STARTED searchnode2
.fleet-enrollment-api-keys-7 0 p STARTED manager
.internal.alerts-observability.slo.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-observability.slo.alerts-default-000001 0 p STARTED manager
.internal.alerts-observability.metrics.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-observability.metrics.alerts-default-000001 0 p STARTED manager
.kibana_task_manager_8.10.4_001 0 r STARTED searchnode2
.kibana_task_manager_8.10.4_001 0 p STARTED manager
.fleet-servers-7 0 r STARTED searchnode2
.fleet-servers-7 0 p STARTED manager
.ds-logs-syslog-so-2024.03.30-000001 0 p UNASSIGNED
.security-profile-8 0 r STARTED searchnode2
.security-profile-8 0 p STARTED manager
.logs-osquery_manager.action.responses-default 0 p UNASSIGNED
metrics-endpoint.metadata_current_default 0 r STARTED searchnode2
metrics-endpoint.metadata_current_default 0 p STARTED manager
.kibana_analytics_8.10.4_001 0 r STARTED searchnode2
.kibana_analytics_8.10.4_001 0 p STARTED manager
.ds-logs-elastic_agent-default-2024.03.29-000001 0 p STARTED manager
.kibana-observability-ai-assistant-conversations-000001 0 r STARTED searchnode2
.kibana-observability-ai-assistant-conversations-000001 0 p STARTED manager
elastalert 0 p UNASSIGNED
.ds-logs-strelka-so-2024.03.30-000001 0 p STARTED manager
.ds-ilm-history-5-2024.03.28-000001 0 r STARTED searchnode2
.ds-ilm-history-5-2024.03.28-000001 0 p STARTED manager
.slo-observability.summary-v2 0 r STARTED searchnode2
.slo-observability.summary-v2 0 p STARTED manager
.fleet-policies-leader-7 0 r STARTED searchnode2
.fleet-policies-leader-7 0 p STARTED manager
.kibana_8.10.4_001 0 r STARTED searchnode2
.kibana_8.10.4_001 0 p STARTED manager
elastalert_status 0 p STARTED manager
.transform-internal-007 0 r STARTED searchnode2
.transform-internal-007 0 p STARTED manager
.metrics-endpoint.metadata_united_default 0 r STARTED searchnode2
.metrics-endpoint.metadata_united_default 0 p STARTED manager
.ds-logs-suricata-so-2024.03.30-000001 0 p STARTED manager
.internal.alerts-observability.uptime.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-observability.uptime.alerts-default-000001 0 p STARTED manager
.internal.alerts-observability.logs.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-observability.logs.alerts-default-000001 0 p STARTED manager
elastalert_silence 0 p STARTED manager
.apm-source-map 0 r STARTED searchnode2
.apm-source-map 0 p STARTED manager
.ds-logs-elastic_agent.filebeat-default-2024.03.29-000001 0 p UNASSIGNED
.apm-agent-configuration 0 r STARTED searchnode2
.apm-agent-configuration 0 p STARTED manager
.kibana_security_solution_8.10.4_001 0 r STARTED searchnode2
.kibana_security_solution_8.10.4_001 0 p STARTED manager
.internal.alerts-stack.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-stack.alerts-default-000001 0 p STARTED manager
.slo-observability.summary-v2.temp 0 r STARTED searchnode2
.slo-observability.summary-v2.temp 0 p STARTED manager
.transform-notifications-000002 0 r STARTED searchnode2
.transform-notifications-000002 0 p STARTED manager
.security-7 0 r STARTED searchnode2
.security-7 0 p STARTED manager
.ds-logs-system.auth-default-2024.03.29-000001 0 p UNASSIGNED
.ds-logs-soc-so-2024.03.29-000001 0 p UNASSIGNED
.fleet-agents-7 0 r STARTED searchnode2
.fleet-agents-7 0 p STARTED manager
.fleet-artifacts-7 0 r STARTED searchnode2
.fleet-artifacts-7 0 p STARTED manager
logs-ti_recordedfuture_latest.threat-1 0 r STARTED searchnode2
logs-ti_recordedfuture_latest.threat-1 0 p STARTED manager
.ds-logs-kratos-so-2024.03.29-000001 0 p UNASSIGNED
.ds-logs-elastic_agent.metricbeat-default-2024.03.29-000001 0 p STARTED manager
.kibana_alerting_cases_8.10.4_001 0 r STARTED searchnode2
.kibana_alerting_cases_8.10.4_001 0 p STARTED manager
elastalert_past 0 p STARTED manager
.geoip_databases 0 r STARTED searchnode2
.geoip_databases 0 p STARTED manager
.kibana_ingest_8.10.4_001 0 r STARTED searchnode2
.kibana_ingest_8.10.4_001 0 p STARTED manager
.apm-custom-link 0 r STARTED searchnode2
.apm-custom-link 0 p STARTED manager
.ds-logs-system.syslog-default-2024.03.29-000001 0 p STARTED manager
.ds-logs-elastic_agent.osquerybeat-default-2024.03.29-000001 0 p STARTED manager
.ds-logs-elasticsearch.server-default-2024.03.29-000001 0 p UNASSIGNED
.ds-.kibana-event-log-8.10.4-2024.03.28-000001 0 r STARTED searchnode2
.ds-.kibana-event-log-8.10.4-2024.03.28-000001 0 p STARTED manager
.ds-logs-elastic_agent.fleet_server-default-2024.03.29-000001 0 p UNASSIGNED
logs-ti_otx_latest.dest_pulses_subscribed-1 0 r STARTED searchnode2
logs-ti_otx_latest.dest_pulses_subscribed-1 0 p STARTED manager
.async-search 0 r STARTED searchnode2
.async-search 0 p STARTED manager
.kibana_security_session_1 0 r STARTED searchnode2
.kibana_security_session_1 0 p STARTED manager
.slo-observability.sli-v2 0 r STARTED searchnode2
.slo-observability.sli-v2 0 p STARTED manager
.fleet-policies-7 0 r STARTED searchnode2
.fleet-policies-7 0 p STARTED manager
.internal.alerts-observability.apm.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-observability.apm.alerts-default-000001 0 p STARTED manager
.logs-osquery_manager.actions-default 0 p STARTED manager
elastalert_error 0 p UNASSIGNED
.ds-logs-zeek-so-2024.03.29-000001 0 p STARTED manager
.ds-logs-zeek-so-2024.03.29-000001 1 p UNASSIGNED
.internal.alerts-security.alerts-default-000001 0 r STARTED searchnode2
.internal.alerts-security.alerts-default-000001 0 p STARTED manager

is there any other way i can assign the unassigned shards? am new for both security onion and ELK stack

[cm-ops]

Try the below and see what returns for that shard:

GET _cluster/allocation/explain
{
  "index": ".ds-logs-syslog-so-2024.03.30-000001",
  "shard": 0,
  "primary": true
}

[witekenea]

This is the output

{
"index": ".ds-logs-syslog-so-2024.03.30-000001",
"shard": 0,
"primary": true,
"current_state": "unassigned",
"unassigned_info": {
"reason": "CLUSTER_RECOVERED",
"at": "2024-04-11T06:18:02.519Z",
"last_allocation_status": "no_valid_shard_copy"
},
"can_allocate": "no_valid_shard_copy",
"allocate_explanation": "Elasticsearch can't allocate this shard because there are no copies of its data in the cluster. Elasticsearch will allocate this shard when a node holding a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.",
"node_allocation_decisions": [
{
"node_id": "4Bi7arSNT2uEZKJmA4UzaQ",
"node_name": "manager",
"transport_address": "10.123.15.112:9300",
"node_attributes": {
"xpack.installed": "true",
"transform.config_version": "10.0.0"
},
"node_decision": "no",
"store": {
"found": false
}
},
{
"node_id": "ZwzcVI56QjGuCru6yHkZbw",
"node_name": "searchnode2",
"transport_address": "10.123.15.116:9300",
"node_attributes": {
"xpack.installed": "true",
"transform.config_version": "10.0.0"
},
"node_decision": "no",
"store": {
"found": false
}
}
]
}