An existing connection was forcibly closed by the remote host. (No alert is being generated for the agents)

[hanifmoha]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

16GB

Storage for /

220GB

Storage for /nsm

158

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have installed Security Onion Version 2.4.111 in standalone mode on Hyper-V. The grid status shows "OK," and the SO-status indicates that all containers are running. The agent and Security Onion are on the same network, and both ping and tracert work without any issues. Running nslookup securityonion on both the agent and Security Onion resolves the correct DNS record. It appears that the agents can communicate with the server, as running tcpdump captures traffic on Security Onion, and the required ports are open. All subnets are allowed under Administration -> Configuration -> Firewall.
beats_endpoint / elastic_agent_endpoint / fleet / standalone

Alerts are being created for failed login attempts to the portal and are displayed as: Security Onion - SOC Login Failure. However, no alert is generated for the agents. I tested this in different ways, including browsing the URL: https://testmynids.org/uid/index.html.

Windows agents were added by running the script, with the agent policy set to endpoints-initial, and enrolled in Fleet. They installed successfully and appear healthy.

Checking the Kibana Elastic Agent info shows below logs for the SecurityOnion

Failed to connect to failover(backoff(async(tcp://172.16.16.92:5055)),backoff(async(tcp://securityonion:5055))): dial tcp [::1]:5055: connect: network is unreachable

elastic-agent-diagnostics-2025-02-17T06-16-36Z-00.zip

Failed to connect to failover(backoff(async(tcp://172.16.16.92:5055)),backoff(async(tcp://securityonion:5055))): dial tcp 172.16.16.92:5055: connect: no route to host

ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:so-grid-nodes_general:22:1, type: POLICY_CHANGE]' for elastic-agent '0a35ca2d-a8d7-4c3a-b563-6f0866ec8df8' failed: fail to ack to fleet: all hosts failed: 2 errors occurred:
* requester 0/2 to host https://172.16.16.92:8220/ errored: Post "https://172.16.16.92:8220/api/fleet/agents/0a35ca2d-a8d7-4c3a-b563-6f0866ec8df8/acks?": dial tcp 172.16.16.92:8220: connect: no route to host
* requester 1/2 to host https://securityonion:8220/ errored: Post "https://securityonion:8220/api/fleet/agents/0a35ca2d-a8d7-4c3a-b563-6f0866ec8df8/acks?": dial tcp 127.0.0.1:8220: connect: connection refused

Filtering the logs for the AMESYDW-JPB-P01 agent shows
Failed to publish events: write tcp 172.16.16.80:61836->172.16.16.92:5055: wsasend: An existing connection was forcibly closed by the remote host.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

Can try the command so-logstash-restart
Also be very sure that fleet is configured to look at the right endpoints and using the right IPs. And of course check to make sure there's no firewall dropping the connection.

[hanifmoha]

so-logstash-restart executed, however server was rebooted few times before that.
There is no firewall between agents and SO
I'm sure Fleet hosts/URLs set correctly

[hanifmoha]

so-logstash-restart executed, however server was rebooted few times before that.
There is no firewall between agents and SO
I'm sure Fleet hosts/URLs set correctly

[dougburks]

I tested this in different ways, including browsing the URL: https://testmynids.org/uid/index.html.

Are you sniffing network traffic from a tap or span port?

Windows agents were added by running the script, with the agent policy set to endpoints-initial, and enrolled in Fleet. They installed successfully and appear healthy.

Go to SOC Dashboards, click the dropdown menu, and select the Elastic Agent Overview dashboard. Do you see logs from your Elastic Agents there?

[hanifmoha]

Sorry, I'm late. The tap port on Hyper-V was not configured properly. Once it was configured, some alerts started coming through. However, some agents are still generating the same error in Elastic:

Failed to publish events: write tcp 172.16.16.15:49985->172.16.16.92:5055: wsasend: An existing connection was forcibly closed by the remote host.

Running tcpdump on the Security Onion, eth1 mirror port, shows all the traffic.

In the SOC Dashboards, click the dropdown menu and select the Elastic Agent Overview dashboard. => Yes

[hanifmoha]

For troubleshooting, I uninstalled the fleet agent, reboot the VM, rebooted the SO but it made no difference.

[reyesj2]

Can you review logs at /opt/so/log/logstash/logstash.log & /opt/so/log/elasticsearch/securityonion.log

Also check influxdb for resource utilization (specifically RAM) overtime. Is it possible logstash is crashing due to running out of memory?

If possible try increasing memory and see if that smooths out your data ingest

[hanifmoha]

I did not find anything helpful in logs at /opt/so/log/logstash/logstash.log & /opt/so/log/elasticsearch/securityonion.log
What I supposed to search for?
RAM was more than 85% usage and I added extra to bring to 25GB and increased the CPU cores to 10 , however it's the same.

Low Traffic Volume on Monitor Interface, Is that relevant?

[hanifmoha]

so-test