Dedicated Fleet Node not creating a new Fleet server in fleet

[jaypfe]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

16

Storage for /

128

Storage for /nsm

128

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I spun up a fleet node to put in my DMZ. It joined the grid okay, and I can see the machine got added as an agent under the grid_nodes_general policy. But a separate fleet server was never added. I can see a policy was created, just no fleet server.

I have tried twice, powering off machine, removing from the grid nodes, but same result.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[defensivedepth]

Hello there!

I have tried twice, powering off machine, removing from the grid nodes, but same result.

What do you mean when you say removing from the grid nodes?

Do you see the Fleet Server in the Grid page in the Security Onion Console? Does it show any errors?

[jaypfe]

No errors with that command. But afterwards the container went missing for a bit, but it did come back
I have also observed a steady increase in ram usage up to about 92% slowly day over day of 16gb before I restarted.

So interestingly when I tried the first time to add a fleet node, it added another fleet server tied to the manager. The status of the original manager fleet server went inactive so I un-enrolled it. Then I tried again to add a new fleet node but I did not see that behavior replicated. Just the addition of the new node under the grid nodes policy.

Does this work for you guys in house? I am not against trying again with a new node. I am not absolutely against redeploying again either, but would like to be reasonably certain that would fix it lol.

It is hosted in my DMZ, I have these ports open to it

Security onion 04 is 8443

[defensivedepth]

So the Fleet Servers run in containers and store their state in Elasticsearch. So if a container is removed & replaced (normal thing that happens when so-elastic-fleet-restart is run), the new container enrolls as a new fleet server - this is by design, directly from Elastic. So you will see older fleet servers in the Fleet listing, but they will go inactive and be hidden from the default view.

It's possible that manually unenrolling it caused the enrollment key to expire.

Try running the following on the Fleet Node:

sudo docker logs so-elastic-fleet > logs
cat logs

Any errors from those logs? It should give you the enrollment status.

And yes, we have a long-running internal distributed grid with a dedicated fleet node working as expected. As well as nightly automated testing for various architectures, including this one.

[jaypfe]

There are several of these errors:
{"log.level":"error","@timestamp":"2023-10-20T14:48:56.897Z","message":"fail elasticsearch info","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"cluster.addr":["tul-so-manager01:9200"],"cluster.maxConnsPersHost":128,"error.message":"dial tcp xx.x.xx.xxx:9200: i/o timeout","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}

Does 9200 need to be open between this node and the manager? I did not see that port required in the docs.

[defensivedepth]

Ah yes, I missed that in your port list screencap. Yes, the Fleet server needs to be able to access ES (TCP/9200) on the Manager.

I will get the docs updated.

[jaypfe]

Sweeeeeet. I will make that change. I will let update if it works just with that, or if it needs a new node spun in place.
Thanks for your help on this one Josh,

[defensivedepth]

Let me know how it goes and if there are other issues that you run into. I think the Fleet node + Elastic Agents are a powerful addition to SO and we want to make it as usable as possible.

[jaypfe]

After restarting, I still see that it cannot connect. I will try to spin up a new node and see what it does.

[jaypfe]

There she be!