Fleet agents not able to send data

[ws6543]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

16

Storage for /

300

Storage for /nsm

1000

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

This is a setup for a home lab before we can deploy to a production environment.

I did a fresh install of 2.4.30 and then updated with salt after. No errors on that part.

In Admin, Config, FW, hostgroups, elastic_agent_endpoint i added the IP;s of my VM;s (single ip per line, no , after)
in elastic fleet, i added 6 hosts to monitor. all show healthy.

if i click on an agent, then more agent metrics it is blank (ie no data). clicking on dashboard, agent info i see this error over and over for every host:
failed to publish events: write tcp --agent-source-ip--:60718->--SO-destintation-IP--:5055: write: connection reset by peer
Failed to publish events caused by: write tcp --agent-source-ip--:60718->--SO-destintation-IP--:5055: write: connection reset by peer

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[cm-ops]

Does your Fleet server show healthy as well?

[ws6543]

so-status looks clean:

looking at logs, elasticflreet-agent-startup is 0 bits. elasticfleet-agent-date doesn't have anything in it except for info stuff (there are 3 errors but i caused them playing with 1 agent).

i plan on opening another case later on this one but i am also having an issue digesting logs from my pfsense. i am not sure if it is a firewall issue or something else. if i set the host and port profile on docker-user i keep getting hits on IPTables (rejecting it) but if i set it to input i get nothing. it was setup to use the fleet pfsense agent on tcp 9001.

[ws6543]

i might not have been correct in my last reply. looking at the log more i am seeing this:
{"log.level":"error","@timestamp":"2024-01-11T01:34:56.165Z","message":"Failed to publish events caused by: EOF","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"logstash","log.origin":{"file.line":280,"file.name":"logstash/async.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-11T01:34:56.165Z","message":"Failed to publish events caused by: client is not connected","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"logstash","log.origin":{"file.line":280,"file.name":"logstash/async.go"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-11T01:34:57.684Z","message":"failed to publish events: client is not connected","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go"},"ecs.version":"1.6.0"}

[ws6543]

in logstash.log i see this also:
[2024-01-10T02:48:03,023][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl_certificate_verification" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_verification_mode' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"ssl_certificate_verification", :plugin=><LogStash::Outputs::ElasticSearch ssl_certificate_verification=>false, password=>, hosts=>[//sec-onion], data_stream=>"true", id=>"a5a52ea7526e536cad7c88c510b834027f1d364c0c1cdc9e41d04c9125388992", user=>"so_elastic", ssl=>true, ecs_compatibility=>:v8, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_685e6960-34c1-4657-b9a1-cf41b2c1ed08", enable_metric=>true, charset=>"UTF-8">, workers=>1, ssl_verification_mode=>"full", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, retry_initial_interval=>2, retry_max_interval=>64, dlq_on_failed_indexname_interpolation=>true, data_stream_type=>"logs", data_stream_dataset=>"generic", data_stream_namespace=>"default", data_stream_sync_fields=>true, data_stream_auto_routing=>true, manage_template=>true, template_overwrite=>false, template_api=>"auto", doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy">}

[2024-01-10T02:48:03,064][INFO ][logstash.javapipeline ] Pipeline search is configured with pipeline.ecs_compatibility: disabled setting. All plugins in this pipeline will default to ecs_compatibility => disabled unless explicitly configured otherwise.
[2024-01-10T02:48:03,068][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"manager", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["/usr/share/logstash/pipelines/manager/0011_input_endgame.conf", "/usr/share/logstash/pipelines/manager/0012_input_elastic_agent.conf", "/usr/share/logstash/pipelines/manager/0013_input_lumberjack_fleet.conf", "/usr/share/logstash/pipelines/manager/9999_output_redis.conf"], :thread=>"#<Thread:0x288dc37d /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-01-10T02:48:03,070][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//sec-onion"]}
[2024-01-10T02:48:03,072][WARN ][logstash.outputs.elasticsearch] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set ssl_verification_mode => full
[2024-01-10T02:48:03,143][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://so_elastic:xxxxxx@sec-onion:9200/]}}
[2024-01-10T02:48:03,299][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@sec-onion:9200/"}
[2024-01-10T02:48:03,304][INFO ][logstash.outputs.elasticsearch] Elasticsearch version determined (8.10.4) {:es_version=>8}
[2024-01-10T02:48:03,304][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-01-10T02:48:03,310][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//sec-onion"]}
[2024-01-10T02:48:03,311][WARN ][logstash.outputs.elasticsearch] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set ssl_verification_mode => full
[2024-01-10T02:48:03,314][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://so_elastic:xxxxxx@sec-onion:9200/]}}
[2024-01-10T02:48:03,349][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@sec-onion:9200/"}
[2024-01-10T02:48:03,353][INFO ][logstash.outputs.elasticsearch] Elasticsearch version determined (8.10.4) {:es_version=>8}
[2024-01-10T02:48:03,354][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-01-10T02:48:03,365][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//sec-onion"]}
[2024-01-10T02:48:03,365][WARN ][logstash.outputs.elasticsearch] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set ssl_verification_mode => full
[2024-01-10T02:48:03,368][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://so_elastic:xxxxxx@sec-onion:9200/]}}
[2024-01-10T02:48:03,396][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@sec-onion:9200/"}
[2024-01-10T02:48:03,399][INFO ][logstash.outputs.elasticsearch] Elasticsearch version determined (8.10.4) {:es_version=>8}
[2024-01-10T02:48:03,400][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-01-10T02:48:03,404][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//sec-onion"]}
[2024-01-10T02:48:03,404][WARN ][logstash.outputs.elasticsearch] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set ssl_verification_mode => full
[2024-01-10T02:48:03,410][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://so_elastic:xxxxxx@sec-onion:9200/]}}
[2024-01-10T02:48:03,441][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://so_elastic:xxxxxx@sec-onion:9200/"}
[2024-01-10T02:48:03,445][INFO ][logstash.outputs.elasticsearch] Elasticsearch version determined (8.10.4) {:es_version=>8}
[2024-01-10T02:48:03,446][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-01-10T02:48:03,449][INFO ][logstash.outputs.elasticsearch] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While ecs_compatibility can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an ecs_compatibility mode, and the pipeline.ecs_compatibility setting can be used to opt-in for all plugins in a pipeline.
[2024-01-10T02:48:03,449][INFO ][logstash.outputs.elasticsearch] Data streams auto configuration (data_stream => auto or unset) resolved to false
[2024-01-10T02:48:03,467][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2024-01-10T02:48:03,469][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"search", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["/usr/share/logstash/pipelines/search/0900_input_redis.conf", "/usr/share/logstash/pipelines/search/9805_output_elastic_agent.conf", "/usr/share/logstash/pipelines/search/9900_output_endgame.conf"], :thread=>"#<Thread:0x5a385fe5 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-01-10T02:48:03,630][INFO ][logstash.javapipeline ] Pipeline Java execution initialization time {"seconds"=>0.56}
[2024-01-10T02:48:03,715][INFO ][logstash.javapipeline ] Pipeline Java execution initialization time {"seconds"=>0.25}
[2024-01-10T02:48:03,726][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@sec-onion:9696/0 list:logstash:unparsed"}
[2024-01-10T02:48:03,726][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@sec-onion:9696/0 list:logstash:unparsed"}
[2024-01-10T02:48:03,726][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@sec-onion:9696/0 list:logstash:unparsed"}
[2024-01-10T02:48:03,726][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@sec-onion:9696/0 list:logstash:unparsed"}
[2024-01-10T02:48:03,727][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@sec-onion:9696/0 list:logstash:unparsed"}
[2024-01-10T02:48:03,727][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@sec-onion:9696/0 list:logstash:unparsed"}
[2024-01-10T02:48:03,728][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"search"}
[2024-01-10T02:48:04,015][INFO ][logstash.inputs.beats ] Starting input listener {:address=>"0.0.0.0:5055"}
[2024-01-10T02:48:04,165][INFO ][logstash.inputs.beats ] Starting input listener {:address=>"0.0.0.0:5056"}
[2024-01-10T02:48:04,276][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"manager"}
[2024-01-10T02:48:04,277][INFO ][logstash.inputs.http ] Starting http input listener {:address=>"0.0.0.0:3765", :ssl=>"true"}
[2024-01-10T02:48:04,281][INFO ][org.logstash.beats.Server] Starting server on port: 5055
[2024-01-10T02:48:04,282][INFO ][org.logstash.beats.Server] Starting server on port: 5056
[2024-01-10T02:48:04,283][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:manager, :search], :non_running_pipelines=>[]}
[2024-01-10T05:55:17,632][INFO ][org.logstash.beats.BeatsHandler] [local: 172.17.255.29:5055, remote: 192.168.168.10:49684] Handling exception: java.io.IOException: Connection timed out (caused by: java.io.IOException: Connection timed out)
[2024-01-10T05:55:17,639][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection timed out
at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47) ~[?:?]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:330) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:284) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:259) ~[?:?]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417) ~[?:?]
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
[noonewh0@sec-onion logstash]$

also FWIW, i run 172.17.1.0/24 as a local subnet to i had to change the docker subnet to 172.17.255.0/24

[ws6543]

I completely reinstalled the security onion server today:
-selected stand alone server and to use the ip address for connection
-changed the docker pool from 172.17.1.0 to 172.17.1.255.0 due to internal IP conflict
-install finished without errors

I logged into the GUI, went to admin/config/firewall/hostgroups/elastic_agent_endpoint. selected my node and then added my single test client address to it.

went to fleet, and added node. node shows healthy in fleet but still getting the same errors and no host data in kibana.

Jan 14, 2024 @ 15:50:52.300
--vmhostname--
failed to publish events: write tcp ----57336->----:5055: write: connection reset by peer

Jan 14, 2024 @ 15:50:50.963
--vmhostname--
Failed to publish events caused by: write tcp ----:57336->----:5055: write: connection reset by peer

[cm-ops]

Did you use the installer from SOC > Downloads?

[ws6543]

from here which redirects to git hub - https://securityonionsolutions.com/software
verified hash also after downloading

[cm-ops]

This test client has the Elastic Agent installed on it or are you talking about the standalone SO server?

[ws6543]

the test client has the agent installed in it from the SO server. used the link and install instructions on the fleet to do the install. client is ubuntu server 22.04 w/ all current patches

[cm-ops]

^ get your package from here in SOC and install it on your server.

[ws6543]

thanks for the reply chris.

sorry, i did not download the clients from there. i removed all the clients i had and re-installed using the download that you showed. all installed without issue on the client side but i noticed this in the logs after when it started to pull policy (showed for each client):

Jan 20, 2024 @ 23:51:07.375
--ubuntu_vm---
Entry.cpp:452 Failed to apply initial policy from on disk configuration (Initial Policy Application Failed)

Jan 20, 2024 @ 23:51:07.259
--ubuntu_vm---
Artifacts.cpp:770 Failed to initialize artifact, identifier: diagnostic-configuration-v1, reason: Invalid url

Jan 20, 2024 @ 23:51:07.259
--ubuntu_vm---
Artifacts.cpp:3128 Failed to download artifact diagnostic-configuration-v1 - Invalid url

after a few minutes, i started to see the same errors again though:
Jan 21, 2024 @ 00:00:43.619
--ubuntu_vm---
failed to publish events: write tcp 172.18.18.20:39764->172.18.18.200:5055: write: connection reset by peer

Jan 21, 2024 @ 00:07:53.748
--ubuntu_vm---
Failed to publish events caused by: write tcp 172.18.18.220:52238->172.18.18.200:5055: write: connection reset by peer

again, this is a stand alone VM.

[cm-ops]

Did you allow the agent in the firwall on your SO Standalone? https://docs.securityonion.net/en/2.4/elastic-agent.html#deployment

[ws6543]

sorry for the delayed response, i was traveling last week

yes i did, i added the single IP's of each endpoint i was going to log from:

[cm-ops]

In your Fleet Server, do the agents show healthy or not? I would also tcpdump on your standalone for port 5055 and see if that shows anything of note.

[ws6543]

sorry for the delay, more work travel.

so i see the hosts as healthy in elastic fleet:

for the tcpdump, i don't see anything adnormal:
from the host:
[---@unifi:]$ sudo tcpdump -nn dst 172.18.18.200 and port 5055 -vv
tcpdump: listening on ens192, link-type EN10MB (Ethernet), snapshot length 262144 bytes
04:02:42.525053 IP (tos 0x0, ttl 64, id 27903, offset 0, flags [DF], proto TCP (6), length 1183)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [P.], cksum 0x8e1e (correct), seq 2521230959:2521232090, ack 3691631911, win 501, options [nop,nop,TS val 740917915 ecr 4164380837], length 1131
04:02:42.525844 IP (tos 0x0, ttl 64, id 27904, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [.], cksum 0xb95e (correct), seq 1131, ack 45, win 501, options [nop,nop,TS val 740917916 ecr 4164385838], length 0
04:02:46.791226 IP (tos 0x0, ttl 64, id 20006, offset 0, flags [DF], proto TCP (6), length 914)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0x7814 (correct), seq 3697418374:3697419236, ack 234592577, win 501, options [nop,nop,TS val 740922181 ecr 4164384103], length 862
04:02:46.791882 IP (tos 0x0, ttl 64, id 20007, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0x2566 (correct), seq 862, ack 45, win 501, options [nop,nop,TS val 740922182 ecr 4164390104], length 0
04:02:47.526361 IP (tos 0x0, ttl 64, id 27905, offset 0, flags [DF], proto TCP (6), length 825)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [P.], cksum 0x4478 (correct), seq 1131:1904, ack 45, win 501, options [nop,nop,TS val 740922916 ecr 4164385838], length 773
04:02:47.526943 IP (tos 0x0, ttl 64, id 27906, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [.], cksum 0x8f1a (correct), seq 1904, ack 89, win 501, options [nop,nop,TS val 740922917 ecr 4164390840], length 0
04:02:49.734165 IP (tos 0x0, ttl 64, id 27018, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0xf5a1 (correct), seq 2364474085:2364475533, ack 1945973880, win 501, options [nop,nop,TS val 740925124 ecr 4164363105], length 1448
04:02:49.734174 IP (tos 0x0, ttl 64, id 27019, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0xd590 (correct), seq 1448:2896, ack 1, win 501, options [nop,nop,TS val 740925124 ecr 4164363105], length 1448
04:02:49.734175 IP (tos 0x0, ttl 64, id 27020, offset 0, flags [DF], proto TCP (6), length 384)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [P.], cksum 0x1935 (correct), seq 2896:3228, ack 1, win 501, options [nop,nop,TS val 740925124 ecr 4164363105], length 332
04:02:49.736578 IP (tos 0x0, ttl 64, id 27021, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x5605 (correct), seq 3228, ack 45, win 501, options [nop,nop,TS val 740925127 ecr 4164393049], length 0
04:02:49.836757 IP (tos 0x0, ttl 64, id 27022, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0xd7b7 (correct), seq 3228:4676, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 1448
04:02:49.836767 IP (tos 0x0, ttl 64, id 27023, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x2592 (correct), seq 4676:6124, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 1448
04:02:49.836768 IP (tos 0x0, ttl 64, id 27024, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x68c6 (correct), seq 6124:7572, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 1448
04:02:49.836769 IP (tos 0x0, ttl 64, id 27025, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x3928 (correct), seq 7572:9020, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 1448
04:02:49.836769 IP (tos 0x0, ttl 64, id 27026, offset 0, flags [DF], proto TCP (6), length 1500)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [P.], cksum 0x7a31 (correct), seq 9020:10468, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 1448
04:02:49.836774 IP (tos 0x0, ttl 64, id 27027, offset 0, flags [DF], proto TCP (6), length 116)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [P.], cksum 0xc753 (correct), seq 10468:10532, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 64
04:02:49.844202 IP (tos 0x0, ttl 64, id 27028, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x387a (correct), seq 10532, ack 89, win 501, options [nop,nop,TS val 740925234 ecr 4164393157], length 0
04:02:50.521420 IP (tos 0x0, ttl 64, id 27907, offset 0, flags [DF], proto TCP (6), length 795)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [P.], cksum 0x82fe (correct), seq 1904:2647, ack 89, win 501, options [nop,nop,TS val 740925911 ecr 4164390840], length 743
04:02:50.522010 IP (tos 0x0, ttl 64, id 27908, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [.], cksum 0x74a1 (correct), seq 2647, ack 133, win 501, options [nop,nop,TS val 740925912 ecr 4164393835], length 0
04:02:52.792245 IP (tos 0x0, ttl 64, id 20008, offset 0, flags [DF], proto TCP (6), length 1090)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0xbb91 (correct), seq 862:1900, ack 45, win 501, options [nop,nop,TS val 740928182 ecr 4164390104], length 1038
04:02:52.793086 IP (tos 0x0, ttl 64, id 20009, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0xf248 (correct), seq 1900, ack 89, win 501, options [nop,nop,TS val 740928183 ecr 4164396106], length 0
04:02:54.792752 IP (tos 0x0, ttl 64, id 20010, offset 0, flags [DF], proto TCP (6), length 922)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0xa341 (correct), seq 1900:2770, ack 89, win 501, options [nop,nop,TS val 740930183 ecr 4164396106], length 870
04:02:54.793502 IP (tos 0x0, ttl 64, id 20011, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0xdf15 (correct), seq 2770, ack 133, win 501, options [nop,nop,TS val 740930184 ecr 4164398106], length 0
04:02:56.792494 IP (tos 0x0, ttl 64, id 20012, offset 0, flags [DF], proto TCP (6), length 914)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0x40dd (correct), seq 2770:3632, ack 133, win 501, options [nop,nop,TS val 740932183 ecr 4164398106], length 862
04:02:56.793178 IP (tos 0x0, ttl 64, id 20013, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0xcbec (correct), seq 3632, ack 177, win 501, options [nop,nop,TS val 740932183 ecr 4164400106], length 0
^C
25 packets captured
25 packets received by filter
0 packets dropped by kernel
----@unifi:~$

from the SO server:
[root@sec-onion ----]# tcpdump -i ens192 -nn src host 172.18.18.20 and dst port 5055 -vv
dropped privs to tcpdump
tcpdump: listening on ens192, link-type EN10MB (Ethernet), snapshot length 262144 bytes
04:02:42.511878 IP (tos 0x0, ttl 64, id 27903, offset 0, flags [DF], proto TCP (6), length 1183)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [P.], cksum 0x8e1e (correct), seq 2521230959:2521232090, ack 3691631911, win 501, options [nop,nop,TS val 740917915 ecr 4164380837], length 1131
04:02:42.512652 IP (tos 0x0, ttl 64, id 27904, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [.], cksum 0xb95e (correct), seq 1131, ack 45, win 501, options [nop,nop,TS val 740917916 ecr 4164385838], length 0
04:02:46.778025 IP (tos 0x0, ttl 64, id 20006, offset 0, flags [DF], proto TCP (6), length 914)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0x7814 (correct), seq 3697418374:3697419236, ack 234592577, win 501, options [nop,nop,TS val 740922181 ecr 4164384103], length 862
04:02:46.778666 IP (tos 0x0, ttl 64, id 20007, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0x2566 (correct), seq 862, ack 45, win 501, options [nop,nop,TS val 740922182 ecr 4164390104], length 0
04:02:47.513163 IP (tos 0x0, ttl 64, id 27905, offset 0, flags [DF], proto TCP (6), length 825)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [P.], cksum 0x4478 (correct), seq 1131:1904, ack 45, win 501, options [nop,nop,TS val 740922916 ecr 4164385838], length 773
04:02:47.513729 IP (tos 0x0, ttl 64, id 27906, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [.], cksum 0x8f1a (correct), seq 1904, ack 89, win 501, options [nop,nop,TS val 740922917 ecr 4164390840], length 0
04:02:49.720971 IP (tos 0x0, ttl 64, id 27018, offset 0, flags [DF], proto TCP (6), length 3280)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [P.], cksum 0x89c3 (incorrect -> 0x45ce), seq 2364474085:2364477313, ack 1945973880, win 501, options [nop,nop,TS val 740925124 ecr 4164363105], length 3228
04:02:49.723351 IP (tos 0x0, ttl 64, id 27021, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x5605 (correct), seq 3228, ack 45, win 501, options [nop,nop,TS val 740925127 ecr 4164393049], length 0
04:02:49.823578 IP (tos 0x0, ttl 64, id 27022, offset 0, flags [DF], proto TCP (6), length 7292)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [P.], cksum 0x996f (incorrect -> 0xfb74), seq 3228:10468, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 7240
04:02:49.823579 IP (tos 0x0, ttl 64, id 27027, offset 0, flags [DF], proto TCP (6), length 116)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [P.], cksum 0xc753 (correct), seq 10468:10532, ack 45, win 501, options [nop,nop,TS val 740925227 ecr 4164393049], length 64
04:02:49.830978 IP (tos 0x0, ttl 64, id 27028, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.60782 > 172.18.18.200.5055: Flags [.], cksum 0x387a (correct), seq 10532, ack 89, win 501, options [nop,nop,TS val 740925234 ecr 4164393157], length 0
04:02:50.508216 IP (tos 0x0, ttl 64, id 27907, offset 0, flags [DF], proto TCP (6), length 795)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [P.], cksum 0x82fe (correct), seq 1904:2647, ack 89, win 501, options [nop,nop,TS val 740925911 ecr 4164390840], length 743
04:02:50.508786 IP (tos 0x0, ttl 64, id 27908, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.52478 > 172.18.18.200.5055: Flags [.], cksum 0x74a1 (correct), seq 2647, ack 133, win 501, options [nop,nop,TS val 740925912 ecr 4164393835], length 0
04:02:52.779019 IP (tos 0x0, ttl 64, id 20008, offset 0, flags [DF], proto TCP (6), length 1090)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0xbb91 (correct), seq 862:1900, ack 45, win 501, options [nop,nop,TS val 740928182 ecr 4164390104], length 1038
04:02:52.779853 IP (tos 0x0, ttl 64, id 20009, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0xf248 (correct), seq 1900, ack 89, win 501, options [nop,nop,TS val 740928183 ecr 4164396106], length 0
04:02:54.779548 IP (tos 0x0, ttl 64, id 20010, offset 0, flags [DF], proto TCP (6), length 922)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0xa341 (correct), seq 1900:2770, ack 89, win 501, options [nop,nop,TS val 740930183 ecr 4164396106], length 870
04:02:54.780270 IP (tos 0x0, ttl 64, id 20011, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0xdf15 (correct), seq 2770, ack 133, win 501, options [nop,nop,TS val 740930184 ecr 4164398106], length 0
04:02:56.779279 IP (tos 0x0, ttl 64, id 20012, offset 0, flags [DF], proto TCP (6), length 914)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [P.], cksum 0x40dd (correct), seq 2770:3632, ack 133, win 501, options [nop,nop,TS val 740932183 ecr 4164398106], length 862
04:02:56.779950 IP (tos 0x0, ttl 64, id 20013, offset 0, flags [DF], proto TCP (6), length 52)
172.18.18.20.53874 > 172.18.18.200.5055: Flags [.], cksum 0xcbec (correct), seq 3632, ack 177, win 501, options [nop,nop,TS val 740932183 ecr 4164400106], length 0
^C
19 packets captured
19 packets received by filter
0 packets dropped by kernel
[root@sec-onion ----]#

[cm-ops]

Check the Logstash log as well /opt/so/log/logstash/logstash.log for any errors.

If you check that endpoints-initial policy, what integrations are showing?

[ws6543]

logstash log has these errors over and over again:
[2024-02-06T04:43:11,456][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"sec-onion:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::OrgApacheHttp::NoHttpResponseException: sec-onion:9200 failed to respond>}

and this:
[2024-02-06T04:43:11,481][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://sec-onion:9200/_bulk][Manticore::ClientProtocolException] sec-onion:9200 failed to respond {:url=>https://so_elastic:xxxxxx@sec-onion:9200/, :error_message=>"Elasticsearch Unreachable: [https://sec-onion:9200/_bulk][Manticore::ClientProtocolException] sec-onion:9200 failed to respond", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2024-02-06T04:43:11,489][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://sec-onion:9200/_bulk][Manticore::ClientProtocolException] sec-onion:9200 failed to respond", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}

and this:
[2024-02-06T06:36:20,480][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>4, :exception=>"Redis::CannotConnectError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:398:in establish_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:115:in block in connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:344:in with_reconnect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:114:in connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:409:in ensure_connected'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:269:in block in process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:375:in logging'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:268:in process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:161:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:11:in llen'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:138:in congestion_check'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:151:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:112:in block in buffer_initialize'", "org/jruby/RubyKernel.java:1586:in loop'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:110:in block in buffer_initialize'"]} [2024-02-06T06:36:20,480][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://<password>@sec-onion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CannotConnectError: Error connecting to Redis on sec-onion:6379 (Errno::ECONNREFUSED)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:398:in establish_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:115:in block in connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:344:in with_reconnect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:114:in connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:409:in ensure_connected'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:269:in block in process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:375:in logging'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:268:in process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:161:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:11:in llen'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:138:in congestion_check'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:151:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:112:in block in buffer_initialize'", "org/jruby/RubyKernel.java:1586:in loop'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:110:in block in buffer_initialize'"]}

sec-onion resolves to 127.0.0.1

[ws6543]

i am going to download the 2.4.40 cd and reinstall from scratch. i will post once i am done if it worked or not

[ws6543]

rebuilt from scratch. all base settings for stand alone except changed the docker IP subnet (due to local conflict). added ip for test vm to endpoint group in the firewall settings. reinstalled client on test vm.

tcpdump looks clean and i see traffic from both the host side and client side

looking at the logstash i see this:
[2024-02-08T19:04:06,784][INFO ][org.logstash.beats.BeatsHandler] [local: 172.32.190.29:5055, remote: 172.18.18.20:36564] Handling exception: java.net.SocketException: Connection reset (caused by: java.net.SocketException: Connection reset)
[2024-02-08T19:04:06,795][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.net.SocketException: Connection reset
at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]

this log does show that info is sent to the SO server but i don't see where it is going

going to elasticagent dashboard and filtering for just the SO server, i see this:

[ws6543]

SO server resolves sec-onion to 127.0.0.1
client resolves sec-onion to correct machine IP

[samuel809]

download from elastic and extract the specific agent for your distribution on your target host, and do an install from the directory with your token and share results

[ws6543]

i removed the existing client, purged directories, and downloaded/installed client from elastic. still getting the same error.

[ws6543]

any update on this issue?

[cm-ops]

What does this output show? nc -vz 172.18.18.200 8220

[ws6543]

sorry for the delay.

rebuilt the SO server with .60 today. same settings (set user/passes, ip's, mgmt network, changed docker network due to IP conflict, and added IP's of test host to elastic_agent fw setting).

from the test host:
admin2@unifi:/var/log$ nc -vz 172.18.18.200 8220
Connection to 172.18.18.200 8220 port [tcp/] succeeded!
admin2@unifi:/var/log$ nc -vz 172.18.18.200 5055
Connection to 172.18.18.200 5055 port [tcp/] succeeded!

[cm-ops]

Are you seeing data now?

[ws6543]

NoSent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Chris Morgret ***@***.***> Date: 4/3/24 07:58 (GMT-06:00) To: Security-Onion-Solutions/securityonion ***@***.***> Cc: ws6543 ***@***.***>, Author ***@***.***> Subject: Re: [Security-Onion-Solutions/securityonion] Fleet agents not able to send data (Discussion #12140) Are you seeing data now? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[cm-ops]

Okay, do you see any errors or issues in the manager pipeline - sudo so-logstash-pipeline-stats manager. Since you are running a standalone, your server also is running the search pipeline, if there are no issues and you see data flowing through the manager pipeline, check the search pipeline with the same command, replacing manager with search. If one or both return empty/nothing, check your logstash.log.

[ws6543]

so if i am reading this right events are hitting the manager and being passed to redis:

[root@sec-onion logstash]# sudo so-logstash-pipeline-stats manager
{
"events": {
"out": 22693,
"in": 22693,
"queue_push_duration_in_millis": 136,
"duration_in_millis": 6638,
"filtered": 22693
},
"flow": {
"output_throughput": {
"current": 3.283,
"last_1_minute": 7.476,
"last_5_minutes": 44.82,
"lifetime": 25.8
},
"queue_backpressure": {
"current": 0.0,
"last_1_minute": 0.0000666,
"last_5_minutes": 0.0002032,
"lifetime": 0.0001546
},
"worker_concurrency": {
"current": 0.0009947,
"last_1_minute": 0.002564,
"last_5_minutes": 0.01099,
"lifetime": 0.007547
},
"input_throughput": {
"current": 3.283,
"last_1_minute": 7.476,
"last_5_minutes": 44.82,
"lifetime": 25.8
},
"filter_throughput": {
"current": 3.283,
"last_1_minute": 7.476,
"last_5_minutes": 44.82,
"lifetime": 25.8
}
},
"plugins": {
"inputs": [
{
"id": "4fd1d833001b263cbf74a4479d004c3aebb48397f6c1811a8a03565076942491",
"current_connections": 3,
"flow": {
"throughput": {
"current": 3.283,
"last_1_minute": 7.476,
"last_5_minutes": 44.82,
"lifetime": 25.8
}
},
"events": {
"out": 22693,
"queue_push_duration_in_millis": 133
},
"name": "beats",
"peak_connections": 5
},
{
"id": "fleet-lumberjack-in",
"flow": {
"throughput": {
"current": 0.0,
"last_1_minute": 0.0,
"last_5_minutes": 0.0,
"lifetime": 0.0
}
},
"events": {
"out": 0,
"queue_push_duration_in_millis": 0
},
"name": "beats"
},
{
"id": "endgame_data",
"flow": {
"throughput": {
"current": 0.0,
"last_1_minute": 0.0,
"last_5_minutes": 0.0,
"lifetime": 0.0
}
},
"events": {
"out": 0,
"queue_push_duration_in_millis": 0
},
"name": "http"
}
],
"codecs": [
{
"id": "es_bulk_c7d53f9a-d34f-4409-bf8d-b59b6e56e517",
"name": "es_bulk",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
},
{
"id": "json_554abae5-912d-418a-b596-4a9ce51111bb",
"name": "json",
"encode": {
"writes_in": 22693,
"duration_in_millis": 1280
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
},
{
"id": "json_0af46fe5-bb2b-4c54-a9d9-bfc869428ae2",
"name": "json",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
},
{
"id": "plain_0a549d4a-e7ad-4cbf-9819-0b34b88263b7",
"name": "plain",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
}
],
"filters": [
{
"id": "5314dd7b439cb20b193cdd401f56dc3dffe5e77894f6a2914c14d9f306dd2426",
"flow": {
"worker_utilization": {
"current": 0.0,
"last_1_minute": 0.0,
"last_5_minutes": 0.00008194,
"lifetime": 0.0001563
},
"worker_millis_per_event": {
"last_5_minutes": "Infinity",
"lifetime": "Infinity"
}
},
"events": {
"out": 0,
"in": 0,
"duration_in_millis": 11
},
"name": "mutate"
},
{
"id": "eddfc7e55e29bf9ab23f4ea9ba79c86e6745186ee3b36fe5b9e21f2dcae876da",
"flow": {
"worker_utilization": {
"current": 0.002487,
"last_1_minute": 0.002706,
"last_5_minutes": 0.0111,
"lifetime": 0.009024
},
"worker_millis_per_event": {
"current": 0.06061,
"last_1_minute": 0.02895,
"last_5_minutes": 0.01982,
"lifetime": 0.02798
}
},
"events": {
"out": 22693,
"in": 22693,
"duration_in_millis": 635
},
"name": "mutate"
}
],
"outputs": [
{
"id": "7cff96d95560161bb05259b9bfc0edfafacdcd342cd599fcc7f8d6a3c6b82bed",
"flow": {
"worker_utilization": {
"current": 0.008704,
"last_1_minute": 0.0281,
"last_5_minutes": 0.1236,
"lifetime": 0.08295
},
"worker_millis_per_event": {
"current": 0.2121,
"last_1_minute": 0.3007,
"last_5_minutes": 0.2207,
"lifetime": 0.2572
}
},
"events": {
"out": 22693,
"in": 22693,
"duration_in_millis": 5837
},
"name": "redis"
}
]
},
"reloads": {
"failures": 0,
"successes": 0,
"last_success_timestamp": null,
"last_error": null,
"last_failure_timestamp": null
},
"queue": {
"type": "memory",
"events_count": 0,
"queue_size_in_bytes": 0,
"max_queue_size_in_bytes": 0
},
"hash": "eb14154e62c947be227ef50105b31d32d4b2a06f5674a5bae4d389fe411f0cff",
"ephemeral_id": "701e56bd-ed59-44bb-8649-9fc792b72430"

but events from redis are not getting to elastic search?
[root@sec-onion logstash]# sudo so-logstash-pipeline-stats search
{
"events": {
"out": 22889,
"in": 22889,
"queue_push_duration_in_millis": 747,
"duration_in_millis": 21599,
"filtered": 22889
},
"flow": {
"output_throughput": {
"current": 3.291,
"last_1_minute": 3.141,
"last_5_minutes": 42.66,
"last_15_minutes": 25.0,
"lifetime": 24.26
},
"queue_backpressure": {
"current": 0.0,
"last_1_minute": 0.00003126,
"last_5_minutes": 0.002223,
"last_15_minutes": 0.000816,
"lifetime": 0.0007917
},
"worker_concurrency": {
"current": 0.01023,
"last_1_minute": 0.009018,
"last_5_minutes": 0.02718,
"last_15_minutes": 0.02294,
"lifetime": 0.02289
},
"input_throughput": {
"current": 3.291,
"last_1_minute": 3.141,
"last_5_minutes": 42.66,
"last_15_minutes": 25.0,
"lifetime": 24.26
},
"filter_throughput": {
"current": 3.291,
"last_1_minute": 3.141,
"last_5_minutes": 42.66,
"last_15_minutes": 25.0,
"lifetime": 24.26
}
},
"plugins": {
"inputs": [
{
"id": "ca8e8e839d6a8e9aca3e045354c04e84e5205138fa1e1c7cd3832d500b8f1ff1",
"flow": {
"throughput": {
"current": 3.291,
"last_1_minute": 3.141,
"last_5_minutes": 42.66,
"last_15_minutes": 25.0,
"lifetime": 24.26
}
},
"events": {
"out": 22889,
"queue_push_duration_in_millis": 743
},
"name": "redis"
}
],
"codecs": [
{
"id": "json_2bf7d71f-53cb-4a99-b4a8-3bbb2f167cd1",
"name": "json",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 22889,
"writes_in": 22889,
"duration_in_millis": 2208
}
},
{
"id": "plain_ef2f1ede-342e-486f-93b6-1768a1c5a518",
"name": "plain",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
},
{
"id": "plain_fd0eb9b9-cef9-4a93-b853-9899197f2ae4",
"name": "plain",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
},
{
"id": "plain_15c86694-df6c-48c5-a38e-45548daa103b",
"name": "plain",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
},
{
"id": "plain_5490bd8b-ab98-420c-8b86-732c6e0541c0",
"name": "plain",
"encode": {
"writes_in": 0,
"duration_in_millis": 0
},
"decode": {
"out": 0,
"writes_in": 0,
"duration_in_millis": 0
}
}
],
"filters": [
{
"id": "2e1c3fc09afe281818a8a579c18829a50a00c83ce31c05c66019898c2f02f602",
"flow": {
"worker_utilization": {
"current": 0.0008943,
"last_1_minute": 0.0001954,
"last_5_minutes": 0.0001618,
"last_15_minutes": 0.0002871,
"lifetime": 0.0002915
},
"worker_millis_per_event": {
"current": "Infinity",
"last_1_minute": "Infinity",
"last_5_minutes": "Infinity",
"last_15_minutes": "Infinity",
"lifetime": "Infinity"
}
},
"events": {
"out": 0,
"in": 0,
"duration_in_millis": 22
},
"name": "mutate"
}
],
"outputs": [
{
"id": "11ed62ede549c69d28c7b6e44ac2b553d09aced81f2b8dc27c01851a9aed081d",
"flow": {
"worker_utilization": {
"current": 0.0,
"last_1_minute": 0.0001954,
"last_5_minutes": 0.0002427,
"last_15_minutes": 0.0004375,
"lifetime": 0.0004637
},
"worker_millis_per_event": {
"last_1_minute": "Infinity",
"last_5_minutes": "Infinity",
"last_15_minutes": "Infinity",
"lifetime": "Infinity"
}
},
"events": {
"out": 0,
"in": 0,
"duration_in_millis": 35
},
"name": "elasticsearch"
},
{
"id": "c4f92446780eeed8b9a4738b09c1329e75f17172239bb15b7e28d32c595bc403",
"bulk_requests": {
"responses": {
"200": 1518
},
"successes": 1518
},
"flow": {
"worker_utilization": {
"current": 0.07781,
"last_1_minute": 0.07814,
"last_5_minutes": 0.0995,
"last_15_minutes": 0.1493,
"lifetime": 0.15
},
"worker_millis_per_event": {
"current": 2.806,
"last_1_minute": 2.778,
"last_5_minutes": 1.545,
"last_15_minutes": 1.604,
"lifetime": 1.662
}
},
"events": {
"out": 6815,
"in": 6815,
"duration_in_millis": 11326
},
"name": "elasticsearch",
"documents": {
"successes": 6815
}
},
{
"id": "c7321925a66b78cf8b24896095b210161489812b4bb1de7c90aa02d8dea23e53",
"bulk_requests": {
"responses": {
"200": 948
},
"successes": 948
},
"flow": {
"worker_utilization": {
"current": 0.04919,
"last_1_minute": 0.0338,
"last_5_minutes": 0.2373,
"last_15_minutes": 0.1347,
"lifetime": 0.1332
},
"worker_millis_per_event": {
"current": 3.667,
"last_1_minute": 3.035,
"last_5_minutes": 0.5061,
"last_15_minutes": 0.6139,
"lifetime": 0.6254
}
},
"events": {
"out": 16074,
"in": 16074,
"duration_in_millis": 10053
},
"name": "elasticsearch",
"documents": {
"successes": 16074
}
},
{
"id": "endgame_es_output",
"flow": {
"worker_utilization": {
"current": 0.0,
"last_1_minute": 0.0001954,
"last_5_minutes": 0.0001213,
"last_15_minutes": 0.0002051,
"lifetime": 0.0001987
},
"worker_millis_per_event": {
"last_1_minute": "Infinity",
"last_5_minutes": "Infinity",
"last_15_minutes": "Infinity",
"lifetime": "Infinity"
}
},
"events": {
"out": 0,
"in": 0,
"duration_in_millis": 15
},
"name": "elasticsearch"
}
]
},
"reloads": {
"failures": 0,
"successes": 0,
"last_success_timestamp": null,
"last_error": null,
"last_failure_timestamp": null
},
"queue": {
"type": "memory",
"events_count": 0,
"queue_size_in_bytes": 0,
"max_queue_size_in_bytes": 0
},
"hash": "8b13f8decd9aa7ca13f6b09d1aaa0ee6c471c7a0d044d4b8fb87ed4fe7b2792b",
"ephemeral_id": "b092bd1f-fb26-48c8-86f3-6cec81242c32"
}
[root@sec-onion logstash]#

i attached the logstash, redis, and elasticsearch logs. i did restart the redis service at 04:27 so ignore those errors.
logstash log shows warning deprecated ssl config settings
redis log is in debug mode and i don't see anything adnormal in it.
elasticsearch: per the documentation the log should be named the hostname of the system (sec-onion) but it is not, it is named the default system name. also there are 2 errors about a transforms that is not starting:

[org.elasticsearch.xpack.transform.transforms.TransformTask] [endpoint.metadata_united-default-8.10.2] is already failed but encountered new failure; reason [Failed to load transform configuration for transform [endpoint.metadata_united-default-8.10.2

[2024-04-06T04:19:34,991][ERROR][org.elasticsearch.xpack.transform.transforms.TransformPersistentTasksExecutor] Failed to load transform configuration for transform [endpoint.metadata_united-default-8.10.2]
org.elasticsearch.ResourceNotFoundException: Transform with id [endpoint.metadata_united-default-8.10.2] could not be found
at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.lambda$getTransformConfiguration$10(IndexBasedTransformConfigManager.java:478)

logstash.log
redis-server.log
securityonion.log

[ws6543]

after looking at the transform issue, i was able to do the following:
-found the bad transform was from elastic defend integration which was not shown as installed
-removed the bad transform shown
-installed elastic defend integration
-transform endpoint.metadata_united-default-8.10.2 now shown as installed and healthy. also tranform endpoint.metadata_current-default-8.10.2 was installed and shows as healthy.
-all slo- tranforms still show as degraded

also i am still getting the failed to publish events log message

[cm-ops]

In Elastic Fleet, can you show me what your agent polices show?

[ws6543]

do you have a system that you can try to replicate this issue with? the install is on vmware 8.0.2 (most current patch). 300 gb primary and 1tb nsm drive

[cm-ops]

Would you put Logstash in debug logging and share the log? In SOC > Configuration > Administration > logstash > advanced

logstash:
  config:
    log.level: debug

Restart Logstash.

[ws6543]

sorry for the delay, i thought i uploaded this already but must have forgot to. i regenerated the log tonight
logstash.log