problem with the distribution of logs between search nodes

[tuamortemo]

After upgrading to version 2.3.240 from version 2.3.220, there was a problem with the distribution of logs between search nodes.
I'm not sure if this is related to the upgrade since the problem appeared a week later.
Most of the events are handled by a single node.
Upgrading to 2.3.250 did not bring results

The node configuration files are the same, no changes have been made
in which direction should I move to find the cause and solution it?

[dougburks]

Are all of the search nodes in the same data center?

Are all of the search nodes connected to the network at the same speed?

Have you checked the redis logs on the manager for any possible clues?
https://docs.securityonion.net/en/2.3/redis.html

Have you checked the diagnostic logs on the search nodes at /opt/so/log/?
https://docs.securityonion.net/en/2.3/logstash.html
https://docs.securityonion.net/en/2.3/elasticsearch.html

[tuamortemo]

The search nodes are working on different hosts of the same vcenter and have the same speed

Have you checked the redis logs on the manager for any possible clues? https://docs.securityonion.net/en/2.3/redis.html

In redis logs on the manager I didn't find any warnings

Have you checked the diagnostic logs on the search nodes at /opt/so/log/? https://docs.securityonion.net/en/2.3/logstash.html https://docs.securityonion.net/en/2.3/elasticsearch.html

In logstash logs I don't see any differences between the first and second node and it looks like this, only in their count:

On the first:
[2023-05-30T13:19:21,031][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"so-kibana-2023.05.30", :routing=>nil, :pipeline=>"filebeat-8.7.1-kibana -log-pipeline"}, {"message"=>"{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-30T13:19:10.768+00:00","message":"Fleet Usage: {\"agents_enabled\":t rue,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_sta tuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":8},"trace ":{"id":"1fe470bde0548148e25a464d1b866aec"},"transaction":{"id":"97d72eceeffe9192"}}", "fileset"=>{"name"=>"log"}, "input"=>{"type"=>"log"}, "tags"=>["beats_input_codec_plain_applied"], "event"=>{"dataset"=>"kibana.log ", "module"=>"kibana"}, "@timestamp"=>2023-05-30T13:19:18.924Z, "ecs"=>{"version"=>"1.12.0"}, "service"=>{"type"=>"kibana"}, "agent"=>{"id"=>"62634bec-85e0-40e3-a935-b84877bc9239", "name"=>"seconion-manager", "version"=>"8.7.1", "ephemeral_id"=>"0dd78ef2-5db6-419e-9cfe-e4c954d1011a", "type"=>"filebeat"}, "json"=>{"@timestamp"=>"2023-05-30T13:19:10.768+00:00", "message"=>"Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0 ,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0," num_host_urls":0}}", "ecs"=>{"version"=>"8.6.0"}, "trace"=>{"id"=>"1fe470bde0548148e25a464d1b866aec"}, "service"=>{"node"=>{"roles"=>["background_tasks", "ui"]}}, "transaction"=>{"id"=>"97d72eceeffe9192"}, "log"=>{"level"=>"INFO ", "logger"=>"plugins.fleet"}, "process"=>{"pid"=>8}}, "metadata"=>{"beat"=>"filebeat", "ip_address"=>"172.17.0.1", "version"=>"8.7.1", "pipeline"=>"filebeat-8.7.1-kibana-log-pipeline", "type"=>"_doc"}, "@Version"=>"1", "host"=>{ "name"=>"seconion-manager"}, "log"=>{"file"=>{"path"=>"/logs/kibana/kibana.log"}, "offset"=>43178}, "type"=>"redis-input"}], response: {"index"=>{"_index"=>"so-kibana-2023.05.30", "_id"=>nil, "status"=>400, "error"=>{"type"=>"ill egal_argument_exception", "reason"=>"pipeline with id [filebeat-8.7.1-kibana-log-pipeline] does not exist"}}}

On the second:

[2023-05-30T01:33:36,249][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"so-kibana-2023.05.30", :routing=>nil, :pipeline=>"filebeat-8.7.1-kibana-log-pipeline"}, {"message"=>"{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-30T01:33:31.195+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":8},"trace":{"id":"1fe470bde0548148e25a464d1b866aec"},"transaction":{"id":"7a2439309e40dd81"}}", "tags"=>["beats_input_codec_plain_applied"], "json"=>{"transaction"=>{"id"=>"7a2439309e40dd81"}, "message"=>"Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}}", "ecs"=>{"version"=>"8.6.0"}, "log"=>{"logger"=>"plugins.fleet", "level"=>"INFO"}, "trace"=>{"id"=>"1fe470bde0548148e25a464d1b866aec"}, "service"=>{"node"=>{"roles"=>["background_tasks", "ui"]}}, "process"=>{"pid"=>8}, "@timestamp"=>"2023-05-30T01:33:31.195+00:00"}, "type"=>"redis-input", "service"=>{"type"=>"kibana"}, "@timestamp"=>2023-05-30T01:33:32.191Z, "metadata"=>{"pipeline"=>"filebeat-8.7.1-kibana-log-pipeline", "beat"=>"filebeat", "version"=>"8.7.1", "ip_address"=>"172.17.0.1", "type"=>"_doc"}, "event"=>{"dataset"=>"kibana.log", "module"=>"kibana"}, "ecs"=>{"version"=>"1.12.0"}, "input"=>{"type"=>"log"}, "fileset"=>{"name"=>"log"}, "log"=>{"file"=>{"path"=>"/logs/kibana/kibana.log"}, "offset"=>4848}, "host"=>{"name"=>"seconion-manager"}, "@Version"=>"1", "agent"=>{"ephemeral_id"=>"0dd78ef2-5db6-419e-9cfe-e4c954d1011a", "id"=>"62634bec-85e0-40e3-a935-b84877bc9239", "name"=>"seconion-manager", "version"=>"8.7.1", "type"=>"filebeat"}}], response: {"index"=>{"_index"=>"so-kibana-2023.05.30", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [filebeat-8.7.1-kibana-log-pipeline] does not exist"}}}

In elasticsearch I have only the following warning on all nodes
[2023-05-30T04:38:45,667][WARN ][org.elasticsearch.cluster.coordination.ClusterBootstrapService] this node is locked into cluster UUID [WMev8P7PT0S0Ft8eiS7FxA] but [cluster.initial_master_nodes] is set to [search-0*]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.7/important-settings.html#initial_master_nodes

[dougburks]

Do the search nodes all have the same amount of CPU, RAM, storage, and other specs?

[tuamortemo]

It seems that the reason was in the datacenter. After an unplanned reboot, everything returned to normal state
Ty for the quick reaction