Elastic agent fails to connect to Fleet

[securityfai]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16

Storage for /

200

Storage for /nsm

200

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,
I have a distributed architecture with a fleet node. When I enroll elastic agents with the manager node, the registration is successfully done, but when I try to enroll them with The fleet node, I got the following error in the endpoint: fail to checkin to fleet-server, also I checked in the fleet node using tcpdump to capture traffic from one of the hosts I try to install the agent on , I got the following error: "admin prohibited"
Please note that all the required ports between hosts & fleet are correctly permitted & there is no drop. what could be the problem? any help?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

All three of these ports are open on the fleet server TCP/8220, TCP/8443, TCP/5055 for the agent?

[securityfai]

I allow the agent to connect by adding the subnet of the host in [Administration–> Configuration –> firewall –> hostgroups –>elastic agent endpoints] as described in this video: https://www.youtube.com/watch?v=cGmQMsFuAvw

Should I add the three ports explicitly in the firewall of SOC?

Regards,

[cm-ops]

Did you also add your Elastic Fleet Node to connect to Manager? Administration > Configuration > firewall > hostgroups > fleet

[securityfai]

When I installed the Fleet Node, I ran this command in the manager CLI : "so-firewall-minion --role=fleet --ip= Fleet IP". After 15 min the IP address of the fleet was automatically added to the path you mentioned.
Also note that I added different fleet nodes & I have always the same problem, only the manager is able to enroll agents

[cm-ops]

Were you still using the installers from SOC from before adding the Fleet server?

[securityfai]

After adding the Fleet server, I downloaded the new elastic agent installer from the SOC. when I launch the installer I got the error shown in the attached screenshot.
I don't have any communication drop and the traffic can reach the Fleet

[cm-ops]

Is the node able to resolve so-fleet-1?

[securityfai]

Yes the node is able to resolve so-fleet-1

[cm-ops]

What do you see if you run nc -vz 192.168.33.150 8220 nc -vz 192.168.33.150 8220 nc -vz so-manager 8220 nc -vz so-fleet-1 8220?

[securityfai]

please find below the output of the command

[cm-ops]

Looks like the agent can't get to that IP on port 8220 Are the other nc commands showing the same No route to host on that port?

[securityfai]

Only the agents we want to enroll with the fleet are showing this problem.
Agents directly connected to the manager are working

[cm-ops]

No route to host is a network problem. This was added to the documentation for grannular control over agent connections recently https://docs.securityonion.net/en/2.4/elastic-fleet.html#settings

[securityfai]

We followed the steps as recommended, but we have always the same problem.

Please note that agents connected to the manager are successfully enrolled, agents are in different subnet than manager and fleet, all network permissions are set correctly and we don't have any drop. Also to be sure that we don't have any network problem, we installed another fleet instance in the same subnet as the agent but we got the same problem.

We can't figure this out , any further checks please?
Thank You for your help, I really I appreciate it.

[cm-ops]

Does the Standalone Fleet node in the different subnet than the manager communicate with the manager without issue?