Logstash failing to send events

[witekenea]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

64

Storage for /

500Gb

Storage for /nsm

250Gb

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

i have been using seconion for more than a week now and everything was working fine, after the complete installation of soc every service is running ok and i have data of three days. after that on the 4th day i set the redis max memory to 2147483648 using the redis>config>maxmemory on the gui, after that i was unable to get 7 days of data and on the dashboard i only have data of the first three days. below is the log from logstash and so-redis-count has 2038252

/opt/so/log/logstash/logstash.log has
[WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@securityonionmanager:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1610:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1981:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
how can i fix this?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Are there any errors in the Logstash log on your search node(s)? Increasing Redis maxmemory will allow for a larger queue, but if there is an error in the pipeline somewhere all you will get is a larger Redis count.

[witekenea]

yes there is an error log.
[2024-10-29T13:26:04,206][INFO ][logstash.javapipeline ] Pipeline Java execution initialization time {"seconds"=>1.7}
[2024-10-29T13:26:04,244][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@securityonionmanager:9696/0 list:logstash:unparsed"}
[2024-10-29T13:26:04,244][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@securityonionmanager:9696/0 list:logstash:unparsed"}
[2024-10-29T13:26:04,245][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@securityonionmanager:9696/0 list:logstash:unparsed"}
[2024-10-29T13:26:04,245][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@securityonionmanager:9696/0 list:logstash:unparsed"}
[2024-10-29T13:26:04,246][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@securityonionmanager:9696/0 list:logstash:unparsed"}
[2024-10-29T13:26:04,246][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@securityonionmanager:9696/0 list:logstash:unparsed"}
[2024-10-29T13:26:04,256][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"search"}
[2024-10-29T13:26:04,273][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:search], :non_running_pipelines=>[]}
[2024-10-29T13:26:14,277][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:14,277][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:14,278][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:14,279][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:14,280][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:14,280][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:17,465][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@securityonionmanager:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://securityonionmanager:9200/][Manticore::ConnectTimeout] Connect to securityonionmanager:9200 [securityonionmanager/x.x.x.x] failed: Connect timed out"}
[2024-10-29T13:26:25,291][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:25,291][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:25,295][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:25,295][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:25,296][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:25,298][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on securityonionmanager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-10-29T13:26:27,426][ERROR][logstash.outputs.elasticsearch] Unable to get license information {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '503' contacting Elasticsearch at URL 'https://securityonionsearch:9200/_license'"}
[2024-10-29T13:26:27,428][ERROR][logstash.outputs.elasticsearch] Could not connect to a compatible version of Elasticsearch {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/"}
[2024-10-29T13:26:28,348][ERROR][logstash.outputs.elasticsearch] Unable to get license information {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '503' contacting Elasticsearch at URL 'https://securityonionsearch:9200/_license'"}
[2024-10-29T13:26:28,348][ERROR][logstash.outputs.elasticsearch] Could not connect to a compatible version of Elasticsearch {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/"}
[2024-10-29T13:26:28,447][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/"}
[2024-10-29T13:26:28,488][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@securityonionmanager:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://securityonionmanager:9200/][Manticore::ConnectTimeout] Connect to securityonionmanager:9200 [securityonionmanager/x.x.x.x] failed: Connect timed out"}
[2024-10-29T13:26:29,242][ERROR][logstash.outputs.elasticsearch] Unable to get license information {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '503' contacting Elasticsearch at URL 'https://securityonionsearch:9200/_license'"}
[2024-10-29T13:26:29,243][ERROR][logstash.outputs.elasticsearch] Could not connect to a compatible version of Elasticsearch {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/"}
[2024-10-29T13:26:29,361][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/"}
[2024-10-29T13:26:30,263][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@securityonionsearch:9200/"}
i tried restarting logstash on the search node but didn't change anything

[cm-ops]

A couple of things to check. On the manager node, look in /opt/so/log/elasticsearch/securityonion.log for any errors. Also, would you run sudo so-elasticsearch-query _cluster/health?pretty to make sure the cluster is okay and showing all the data nodes.

From your search node, run nc -vz securityonionmanager 9200 if the name cannot be resolved, substitute the manager's IP in the command for the hostname.

[witekenea]

Thank you for your response, here is what i got

• the log On the manager node, in /opt/so/log/elasticsearch/securityonion.log

tail -f /opt/so/log/elasticsearch/securityonion.log
at org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:333) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$2.doRun(TransportBulkAction.java:305) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
[2024-10-30T13:38:06,929][WARN ][org.elasticsearch.cluster.coordination.Coordinator] This node is a fully-formed single-node cluster with cluster UUID [4wkCptBnQP6azniqPGzJQw], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[securityonionmanager, securityonionsearch]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.
[2024-10-30T13:38:36,930][WARN ][org.elasticsearch.cluster.coordination.Coordinator] This node is a fully-formed single-node cluster with cluster UUID [4wkCptBnQP6azniqPGzJQw], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[securityonionmanager, securityonionsearch]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.
[2024-10-30T13:38:51,932][WARN ][rest.suppressed ] path: /metrics-fleet_server.agent_status-default/_doc, params: {refresh=true, index=metrics-fleet_server.agent_status-default, op_type=create}, status: 500
java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.
at org.elasticsearch.action.ingest.IngestActionForwarder.randomIngestNode(IngestActionForwarder.java:54) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.ingest.IngestActionForwarder.forwardIngestRequest(IngestActionForwarder.java:44) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$doInternalExecute$2(TransportBulkAction.java:345) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:333) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$2.doRun(TransportBulkAction.java:305) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
[2024-10-30T13:39:06,930][WARN ][org.elasticsearch.cluster.coordination.Coordinator] This node is a fully-formed single-node cluster with cluster UUID [4wkCptBnQP6azniqPGzJQw], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[securityonionmanager, securityonionsearch]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.

• so-elasticsearch-query _cluster/health?pretty

{
"cluster_name" : "securityonion",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 82,
"active_shards" : 82,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

• nc -vz securityonionmanager 9200 on the search node

Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to x.x.x.x:9200. (x.x.x.x is local ip)
Ncat: 0 bytes sent, 0 bytes received in 0.11 seconds.

• tail -f /opt/so/log/elasticsearch/securityonion.log on the search node

[2024-10-30T13:47:46,862][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:47:48,474][WARN ][org.elasticsearch.discovery.PeerFinder] address [publicip:9300], node [unknown discovery result: [][publicip:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:47:56,863][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:06,864][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:16,865][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:18,491][WARN ][org.elasticsearch.discovery.PeerFinder] address [publicip:9300], node [unknown discovery result: [][publicip:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:26,866][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:36,866][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:46,867][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:48,509][WARN ][org.elasticsearch.discovery.PeerFinder] address [publicip:9300], node [unknown discovery result: [][publicip:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:48:56,867][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:49:06,868][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:49:16,869][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:49:18,524][WARN ][org.elasticsearch.discovery.PeerFinder] address [publicip:9300], node [unknown discovery result: [][publicip:9300] connect_timeout[30s]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html
[2024-10-30T13:49:26,870][WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{securityonionsearch}{kuXUetoaS5SzI16tgFRFtg}{TCw6dyRXTtaWYofuHi-aMQ}{securityonionsearch}{securityonionsearch}{localipofsearchnode:9300}{dhit}{8.14.3}{7000099-8505000}]; discovery will continue using [publicip:9300] from hosts providers and [] from last-known cluster state; node term 14, last-accepted version 3136 in term 14; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.14/discovery-troubleshooting.html

this is what i got. and the manager node has 2 nics one for public ip and the other for private ip. am accessing the gui using the public ip. (in the time of deployment the manager has only one nic which is the local ip and after connecting with the search and sensor node i add the second nic).

[cm-ops]

Can the search node reach the manager on port 9300? It does not look like it joined the cluster. You only have one node in the cluster it looks like.